Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-13-g0278677bbc8c) initialized Starting run at 09:06:56 UTC on 2022-01-10. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-13-g0278677bb Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.041708562 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024765135 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.147345889 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.070335343 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072435355 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.085704761 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.085842844 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040127428 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.072460355 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.070810762 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.088223858 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.08186124 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.090822331 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.042101148 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025159305 seconds Ending run at 09:06:57 UTC on 2022-01-10. Run time = 1.140789409 Run time without initialization = 0.973862412 Average number of zones advanced per microsecond: 2.692 Average number of zones advanced per microsecond per rank: 2.692 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 1.141 ... 1.141 ... 1.141 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2458 0.2458 0.2458 21.55% VisMF::Write(FabArray) 11 0.1655 0.1655 0.1655 14.51% MLCellLinOp::applyBC() 4433 0.1184 0.1184 0.1184 10.38% MLPoisson::Fsmooth() 3280 0.07473 0.07473 0.07473 6.55% FillBoundary_nowait() 4023 0.04014 0.04014 0.04014 3.52% MLCGSolver::bicgstab 82 0.03815 0.03815 0.03815 3.34% FabArray::setVal() 1140 0.03126 0.03126 0.03126 2.74% MultiFab::Dot() 1114 0.02802 0.02802 0.02802 2.46% StateData::FillBoundary(geom) 328 0.02627 0.02627 0.02627 2.30% MLCellLinOp::smooth() 1640 0.02378 0.02378 0.02378 2.08% FabArray::FillBoundary() 4023 0.0231 0.0231 0.0231 2.03% FabArray::ParallelCopy_nowait() 861 0.02226 0.02226 0.02226 1.95% MultiFab::LinComb() 1586 0.01998 0.01998 0.01998 1.75% StateDataPhysBCFunct::() 41 0.01971 0.01971 0.01971 1.73% MLPoisson::Fapply() 1142 0.01589 0.01589 0.01589 1.39% MLCellLinOp::defineAuxData() 11 0.01399 0.01399 0.01399 1.23% Castro::computeTemp() 63 0.01321 0.01321 0.01321 1.16% Gravity::fill_multipole_BCs() 11 0.0128 0.0128 0.0128 1.12% FabArrayBase::getFB() 4023 0.01192 0.01192 0.01192 1.04% MLCellLinOp::apply() 1142 0.009999 0.009999 0.009999 0.88% MLMG::addInterpCorrection() 410 0.009934 0.009934 0.009934 0.87% FabArrayBase::getCPC() 1323 0.00939 0.00939 0.00939 0.82% amrex::average_down 410 0.009224 0.009224 0.009224 0.81% CGSolver::sxay() 1586 0.0092 0.0092 0.0092 0.81% MultiFab::Xpay() 585 0.008897 0.008897 0.008897 0.78% Castro::enforce_min_density() 62 0.008572 0.008572 0.008572 0.75% Castro::reset_internal_energy() 63 0.006324 0.006324 0.006324 0.55% MLCGSolver::ParallelAllReduce 1514 0.006213 0.006213 0.006213 0.54% FabArray::setDomainBndry() 41 0.005554 0.005554 0.005554 0.49% Castro::estTimeStep() 21 0.005497 0.005497 0.005497 0.48% FabArray::ParallelCopy() 861 0.005082 0.005082 0.005082 0.45% Amr::checkPoint() 3 0.004885 0.004885 0.004885 0.43% Castro::expand_state() 10 0.004577 0.004577 0.004577 0.40% BndryData::define() 11 0.004532 0.004532 0.004532 0.40% MLCellLinOp::correctionResidual() 492 0.004245 0.004245 0.004245 0.37% Castro::do_advance_ctu() 10 0.003852 0.003852 0.003852 0.34% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003685 0.003685 0.003685 0.32% Amr::writePlotFile() 2 0.003364 0.003364 0.003364 0.29% MLMG::mgVcycle() 82 0.003088 0.003088 0.003088 0.27% Castro::construct_new_gravity_source() 10 0.002977 0.002977 0.002977 0.26% Castro::normalize_species() 62 0.00293 0.00293 0.00293 0.26% Castro::enforce_speed_limit() 62 0.002926 0.002926 0.002926 0.26% Gravity::get_new_grav_vector() 11 0.002909 0.002909 0.002909 0.25% MLMG:computeResOfCorrection() 410 0.002403 0.002403 0.002403 0.21% MLMG::ResNormInf() 93 0.00235 0.00235 0.00235 0.21% MLMG::oneIter() 82 0.002322 0.002322 0.002322 0.20% Castro::construct_old_gravity_source() 10 0.002218 0.002218 0.002218 0.19% Amr::defBaseLevel() 1 0.002217 0.002217 0.002217 0.19% MLCellLinOp::compGrad() 11 0.002128 0.002128 0.002128 0.19% FabArrayBase::CPC::define() 454 0.001973 0.001973 0.001973 0.17% Gravity::get_old_grav_vector() 10 0.001945 0.001945 0.001945 0.17% MultiFab::Saxpy() 20 0.001938 0.001938 0.001938 0.17% MLCellLinOp::setLevelBC() 11 0.001682 0.001682 0.001682 0.15% MLMG::prepareForSolve() 11 0.001659 0.001659 0.001659 0.15% Gravity::actual_solve_with_mlmg() 11 0.001619 0.001619 0.001619 0.14% FabArray::mult() 43 0.001514 0.001514 0.001514 0.13% MLMG::mgVcycle_down::1 82 0.00146 0.00146 0.00146 0.13% MLMG::mgVcycle_down::0 82 0.00144 0.00144 0.00144 0.13% MLMG::mgVcycle_down::2 82 0.001437 0.001437 0.001437 0.13% MLMG::mgVcycle_down::3 82 0.001434 0.001434 0.001434 0.13% MLMG::mgVcycle_down::4 82 0.001423 0.001423 0.001423 0.12% MLCellLinOp::prepareForSolve() 11 0.001414 0.001414 0.001414 0.12% Castro::initData() 1 0.001386 0.001386 0.001386 0.12% MLMG::actualBottomSolve() 82 0.001229 0.001229 0.001229 0.11% MLMG::solve() 11 0.001008 0.001008 0.001008 0.09% MLMG::mgVcycle_up::1 82 0.001005 0.001005 0.001005 0.09% MLMG::mgVcycle_up::4 82 0.0009806 0.0009806 0.0009806 0.09% MLMG::mgVcycle_up::2 82 0.0009488 0.0009488 0.0009488 0.08% MLMG::mgVcycle_up::3 82 0.0009471 0.0009471 0.0009471 0.08% MLMG::mgVcycle_up::0 82 0.0009383 0.0009383 0.0009383 0.08% Castro::clean_state() 62 0.000924 0.000924 0.000924 0.08% MLCellLinOp::solutionResidual() 93 0.0008221 0.0008221 0.0008221 0.07% Gravity::update_max_rhs() 11 0.0007399 0.0007399 0.0007399 0.06% FillPatchIterator::Initialize 41 0.0006281 0.0006281 0.0006281 0.06% MultiFab::Copy() 11 0.000608 0.000608 0.000608 0.05% Gravity::solve_for_phi() 10 0.000584 0.000584 0.000584 0.05% Amr::InitAmr() 1 0.0005116 0.0005116 0.0005116 0.04% MLMG::computeResidual() 82 0.000494 0.000494 0.000494 0.04% MLMG::mgVcycle_bottom 82 0.0004781 0.0004781 0.0004781 0.04% Castro::subcycle_advance_ctu() 10 0.0004509 0.0004509 0.0004509 0.04% Castro::initialize_advance() 10 0.0003973 0.0003973 0.0003973 0.03% Amr::coarseTimeStep() 10 0.0003963 0.0003963 0.0003963 0.03% FillPatchSingleLevel 41 0.0003794 0.0003794 0.0003794 0.03% MultiFab::min() 10 0.0003716 0.0003716 0.0003716 0.03% main() 1 0.0003603 0.0003603 0.0003603 0.03% Castro::construct_new_gravity() 10 0.000346 0.000346 0.000346 0.03% makeSFC 55 0.0003408 0.0003408 0.0003408 0.03% Castro::do_new_sources() 10 0.0003293 0.0003293 0.0003293 0.03% MultiFab::max() 11 0.0003278 0.0003278 0.0003278 0.03% Amr::timeStep() 10 0.0003126 0.0003126 0.0003126 0.03% Castro::do_old_sources() 10 0.0002921 0.0002921 0.0002921 0.03% MLLinOp::defineGrids() 11 0.000291 0.000291 0.000291 0.03% Castro::construct_new_source() 50 0.0002842 0.0002842 0.0002842 0.02% Castro::advance() 10 0.0002815 0.0002815 0.0002815 0.02% MLCellLinOp::defineBC() 11 0.0002747 0.0002747 0.0002747 0.02% FabArrayBase::FB::FB() 56 0.0002573 0.0002573 0.0002573 0.02% Gravity::actual_multilevel_solve() 1 0.0002472 0.0002472 0.0002472 0.02% MLMG::MLRhsNormInf() 11 0.0002394 0.0002394 0.0002394 0.02% Castro::construct_old_source() 50 0.0002328 0.0002328 0.0002328 0.02% MLLinOp::makeAgglomeratedDMap 11 0.0001962 0.0001962 0.0001962 0.02% StateData::checkPoint() 12 0.000192 0.000192 0.000192 0.02% DistributionMapping::Distribute() 56 0.0001696 0.0001696 0.0001696 0.01% MLPoisson::define() 11 0.0001637 0.0001637 0.0001637 0.01% Gravity::solve_phi_with_mlmg() 11 0.0001577 0.0001577 0.0001577 0.01% Castro::initialize_do_advance() 10 0.000156 0.000156 0.000156 0.01% Castro::finalize_advance() 10 0.0001558 0.0001558 0.0001558 0.01% Castro::apply_source_to_state() 20 0.0001228 0.0001228 0.0001228 0.01% Gravity::swapTimeLevels() 10 0.00012 0.00012 0.00012 0.01% AmrLevel::checkPoint() 3 0.0001147 0.0001147 0.0001147 0.01% Castro::post_timestep() 10 9.217e-05 9.217e-05 9.217e-05 0.01% MLMG::getGradSolution() 11 8.108e-05 8.108e-05 8.108e-05 0.01% Castro::swap_state_time_levels() 10 7.65e-05 7.65e-05 7.65e-05 0.01% MLLinOp::define() 11 7.366e-05 7.366e-05 7.366e-05 0.01% MLPoisson::prepareForSolve() 11 6.834e-05 6.834e-05 6.834e-05 0.01% Castro::construct_old_gravity() 10 6.787e-05 6.787e-05 6.787e-05 0.01% Castro::finalize_do_advance() 10 6.675e-05 6.675e-05 6.675e-05 0.01% MLMG::computeMLResidual() 11 6.617e-05 6.617e-05 6.617e-05 0.01% MLMG::MLResNormInf() 11 6.433e-05 6.433e-05 6.433e-05 0.01% Amr::FinalizeInit() 1 5.788e-05 5.788e-05 5.788e-05 0.01% Castro::computeNewDt() 9 5.413e-05 5.413e-05 5.413e-05 0.00% StateData::define() 4 5.217e-05 5.217e-05 5.217e-05 0.00% Castro::enforce_consistent_e() 1 3.958e-05 3.958e-05 3.958e-05 0.00% MLMG::buildFineMask() 11 3.604e-05 3.604e-05 3.604e-05 0.00% Gravity::set_mass_offset() 11 3.475e-05 3.475e-05 3.475e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.267e-05 3.267e-05 3.267e-05 0.00% Castro::create_source_corrector() 10 3.237e-05 3.237e-05 3.237e-05 0.00% Castro::retry_advance_ctu() 10 3.104e-05 3.104e-05 3.104e-05 0.00% Castro::FluxRegFineAdd() 10 3.017e-05 3.017e-05 3.017e-05 0.00% AmrLevel::AmrLevel(dm) 1 2.596e-05 2.596e-05 2.596e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.117e-05 2.117e-05 2.117e-05 0.00% Castro::post_init() 1 1.825e-05 1.825e-05 1.825e-05 0.00% AmrLevel::checkPointPost() 3 1.429e-05 1.429e-05 1.429e-05 0.00% Amr::writeSmallPlotFile() 1 1.275e-05 1.275e-05 1.275e-05 0.00% Amr::init() 1 1.275e-05 1.275e-05 1.275e-05 0.00% Castro::computeInitialDt() 2 1.271e-05 1.271e-05 1.271e-05 0.00% Amr::InitializeInit() 1 1.196e-05 1.196e-05 1.196e-05 0.00% AmrLevel::checkPointPre() 3 1.033e-05 1.033e-05 1.033e-05 0.00% Amr::initialInit() 1 9.541e-06 9.541e-06 9.541e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 8.791e-06 8.791e-06 8.791e-06 0.00% Amr::initSubcycle() 1 8.542e-06 8.542e-06 8.542e-06 0.00% Castro::post_regrid() 1 6.916e-06 6.916e-06 6.916e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 1.141 1.141 1.141 100.00% Amr::coarseTimeStep() 10 0.9484 0.9484 0.9484 83.14% Amr::timeStep() 10 0.8629 0.8629 0.8629 75.63% Castro::advance() 10 0.856 0.856 0.856 75.03% Castro::subcycle_advance_ctu() 10 0.8408 0.8408 0.8408 73.70% Castro::do_advance_ctu() 10 0.8403 0.8403 0.8403 73.66% Gravity::solve_phi_with_mlmg() 11 0.573 0.573 0.573 50.22% Gravity::actual_solve_with_mlmg() 11 0.5598 0.5598 0.5598 49.07% MLMG::solve() 11 0.5259 0.5259 0.5259 46.10% Castro::construct_new_gravity() 10 0.5092 0.5092 0.5092 44.63% MLMG::oneIter() 82 0.5011 0.5011 0.5011 43.93% MLMG::mgVcycle() 82 0.4988 0.4988 0.4988 43.72% Gravity::solve_for_phi() 10 0.4884 0.4884 0.4884 42.81% Castro::construct_ctu_hydro_source() 10 0.2458 0.2458 0.2458 21.55% MLCellLinOp::smooth() 1640 0.2406 0.2406 0.2406 21.09% MLCellLinOp::applyBC() 4433 0.1938 0.1938 0.1938 16.99% Amr::init() 1 0.1663 0.1663 0.1663 14.57% VisMF::Write(FabArray) 11 0.1655 0.1655 0.1655 14.51% MLMG::mgVcycle_bottom 82 0.148 0.148 0.148 12.98% MLMG::actualBottomSolve() 82 0.1476 0.1476 0.1476 12.93% MLCGSolver::bicgstab 82 0.1453 0.1453 0.1453 12.73% Amr::checkPoint() 3 0.1241 0.1241 0.1241 10.88% AmrLevel::checkPoint() 3 0.1192 0.1192 0.1192 10.45% StateData::checkPoint() 12 0.1191 0.1191 0.1191 10.44% Amr::initialInit() 1 0.09968 0.09968 0.09968 8.74% Amr::FinalizeInit() 1 0.0891 0.0891 0.0891 7.81% Castro::post_init() 1 0.0883 0.0883 0.0883 7.74% Gravity::multilevel_solve_for_new_phi() 1 0.08545 0.08545 0.08545 7.49% Gravity::actual_multilevel_solve() 1 0.08543 0.08543 0.08543 7.49% MLCellLinOp::apply() 1142 0.07681 0.07681 0.07681 6.73% FabArray::FillBoundary() 4023 0.07542 0.07542 0.07542 6.61% MLPoisson::Fsmooth() 3280 0.07473 0.07473 0.07473 6.55% FillPatchIterator::Initialize 41 0.05738 0.05738 0.05738 5.03% MLMG::mgVcycle_down::0 82 0.05506 0.05506 0.05506 4.83% FillBoundary_nowait() 4023 0.05231 0.05231 0.05231 4.59% FillPatchSingleLevel 41 0.0512 0.0512 0.0512 4.49% Amr::writePlotFile() 2 0.05005 0.05005 0.05005 4.39% StateDataPhysBCFunct::() 41 0.04597 0.04597 0.04597 4.03% MLMG::mgVcycle_up::0 82 0.0459 0.0459 0.0459 4.02% MLCellLinOp::correctionResidual() 492 0.04471 0.04471 0.04471 3.92% MLMG:computeResOfCorrection() 410 0.04022 0.04022 0.04022 3.53% MLMG::mgVcycle_down::1 82 0.03888 0.03888 0.03888 3.41% MLMG::mgVcycle_down::2 82 0.03492 0.03492 0.03492 3.06% FabArray::ParallelCopy() 861 0.03484 0.03484 0.03484 3.05% Castro::clean_state() 62 0.03398 0.03398 0.03398 2.98% MLMG::mgVcycle_down::3 82 0.03392 0.03392 0.03392 2.97% MLMG::mgVcycle_down::4 82 0.03302 0.03302 0.03302 2.89% FabArray::setVal() 1140 0.03126 0.03126 0.03126 2.74% FabArray::ParallelCopy_nowait() 861 0.02975 0.02975 0.02975 2.61% CGSolver::sxay() 1586 0.02917 0.02917 0.02917 2.56% MultiFab::Dot() 1114 0.02802 0.02802 0.02802 2.46% MLPoisson::define() 11 0.02762 0.02762 0.02762 2.42% MLMG::mgVcycle_up::2 82 0.02714 0.02714 0.02714 2.38% MLMG::mgVcycle_up::1 82 0.02677 0.02677 0.02677 2.35% StateData::FillBoundary(geom) 328 0.02627 0.02627 0.02627 2.30% amrex::average_down 410 0.02623 0.02623 0.02623 2.30% MLMG::mgVcycle_up::3 82 0.02606 0.02606 0.02606 2.28% MLMG::mgVcycle_up::4 82 0.02602 0.02602 0.02602 2.28% Castro::initialize_do_advance() 10 0.02566 0.02566 0.02566 2.25% Gravity::get_new_grav_vector() 11 0.02306 0.02306 0.02306 2.02% MLMG::addInterpCorrection() 410 0.02291 0.02291 0.02291 2.01% MLCGSolver::ParallelAllReduce 1514 0.02258 0.02258 0.02258 1.98% MultiFab::LinComb() 1586 0.01998 0.01998 0.01998 1.75% MLCellLinOp::defineAuxData() 11 0.01978 0.01978 0.01978 1.73% Castro::expand_state() 10 0.01965 0.01965 0.01965 1.72% Castro::computeTemp() 63 0.01954 0.01954 0.01954 1.71% Castro::construct_old_gravity() 10 0.01919 0.01919 0.01919 1.68% Gravity::get_old_grav_vector() 10 0.01913 0.01913 0.01913 1.68% MLPoisson::Fapply() 1142 0.01589 0.01589 0.01589 1.39% Castro::initialize_advance() 10 0.01472 0.01472 0.01472 1.29% Gravity::fill_multipole_BCs() 11 0.0128 0.0128 0.0128 1.12% FabArrayBase::getFB() 4023 0.01218 0.01218 0.01218 1.07% MLCellLinOp::solutionResidual() 93 0.01214 0.01214 0.01214 1.06% Castro::do_new_sources() 10 0.01206 0.01206 0.01206 1.06% FabArrayBase::getCPC() 1323 0.01136 0.01136 0.01136 1.00% MLMG::computeResidual() 82 0.01102 0.01102 0.01102 0.97% Amr::InitializeInit() 1 0.01057 0.01057 0.01057 0.93% Amr::defBaseLevel() 1 0.01056 0.01056 0.01056 0.93% MultiFab::Xpay() 585 0.008897 0.008897 0.008897 0.78% Castro::enforce_min_density() 62 0.008572 0.008572 0.008572 0.75% Castro::do_old_sources() 10 0.008519 0.008519 0.008519 0.75% MLMG::prepareForSolve() 11 0.008372 0.008372 0.008372 0.73% MLCellLinOp::defineBC() 11 0.006574 0.006574 0.006574 0.58% Castro::post_timestep() 10 0.006557 0.006557 0.006557 0.57% Castro::reset_internal_energy() 63 0.006324 0.006324 0.006324 0.55% BndryData::define() 11 0.006299 0.006299 0.006299 0.55% FabArray::setDomainBndry() 41 0.005554 0.005554 0.005554 0.49% Castro::estTimeStep() 21 0.005497 0.005497 0.005497 0.48% Castro::initData() 1 0.003731 0.003731 0.003731 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003685 0.003685 0.003685 0.32% Castro::construct_new_source() 50 0.003261 0.003261 0.003261 0.29% Castro::construct_new_gravity_source() 10 0.002977 0.002977 0.002977 0.26% MLMG::getGradSolution() 11 0.002966 0.002966 0.002966 0.26% Castro::normalize_species() 62 0.00293 0.00293 0.00293 0.26% Castro::enforce_speed_limit() 62 0.002926 0.002926 0.002926 0.26% MLCellLinOp::compGrad() 11 0.002885 0.002885 0.002885 0.25% Castro::construct_old_source() 50 0.002451 0.002451 0.002451 0.21% MLMG::ResNormInf() 93 0.00235 0.00235 0.00235 0.21% Castro::computeNewDt() 9 0.002345 0.002345 0.002345 0.21% Castro::construct_old_gravity_source() 10 0.002218 0.002218 0.002218 0.19% Castro::apply_source_to_state() 20 0.002061 0.002061 0.002061 0.18% FabArrayBase::CPC::define() 454 0.001973 0.001973 0.001973 0.17% MultiFab::Saxpy() 20 0.001938 0.001938 0.001938 0.17% MLCellLinOp::setLevelBC() 11 0.001682 0.001682 0.001682 0.15% MLMG::computeMLResidual() 11 0.00168 0.00168 0.00168 0.15% FabArray::mult() 43 0.001514 0.001514 0.001514 0.13% MLPoisson::prepareForSolve() 11 0.001483 0.001483 0.001483 0.13% MLCellLinOp::prepareForSolve() 11 0.001414 0.001414 0.001414 0.12% Gravity::swapTimeLevels() 10 0.001352 0.001352 0.001352 0.12% Gravity::update_max_rhs() 11 0.001252 0.001252 0.001252 0.11% MLLinOp::define() 11 0.0011 0.0011 0.0011 0.10% MLLinOp::defineGrids() 11 0.001026 0.001026 0.001026 0.09% Castro::computeInitialDt() 2 0.0007115 0.0007115 0.0007115 0.06% MLLinOp::makeAgglomeratedDMap 11 0.0007027 0.0007027 0.0007027 0.06% MultiFab::Copy() 11 0.000608 0.000608 0.000608 0.05% Amr::InitAmr() 1 0.0005202 0.0005202 0.0005202 0.05% Castro::post_regrid() 1 0.0005166 0.0005166 0.0005166 0.05% makeSFC 55 0.0005065 0.0005065 0.0005065 0.04% MultiFab::min() 10 0.0003716 0.0003716 0.0003716 0.03% MLMG::MLResNormInf() 11 0.0003621 0.0003621 0.0003621 0.03% MultiFab::max() 11 0.0003278 0.0003278 0.0003278 0.03% FabArrayBase::FB::FB() 56 0.0002573 0.0002573 0.0002573 0.02% MLMG::MLRhsNormInf() 11 0.0002394 0.0002394 0.0002394 0.02% Castro::finalize_advance() 10 0.000186 0.000186 0.000186 0.02% DistributionMapping::Distribute() 56 0.0001696 0.0001696 0.0001696 0.01% AmrLevel::AmrLevel(dm) 1 7.813e-05 7.813e-05 7.813e-05 0.01% Castro::swap_state_time_levels() 10 7.65e-05 7.65e-05 7.65e-05 0.01% Castro::finalize_do_advance() 10 6.675e-05 6.675e-05 6.675e-05 0.01% StateData::define() 4 5.217e-05 5.217e-05 5.217e-05 0.00% Castro::enforce_consistent_e() 1 3.958e-05 3.958e-05 3.958e-05 0.00% MLMG::buildFineMask() 11 3.604e-05 3.604e-05 3.604e-05 0.00% Gravity::set_mass_offset() 11 3.475e-05 3.475e-05 3.475e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.267e-05 3.267e-05 3.267e-05 0.00% Castro::create_source_corrector() 10 3.237e-05 3.237e-05 3.237e-05 0.00% Castro::retry_advance_ctu() 10 3.104e-05 3.104e-05 3.104e-05 0.00% Castro::FluxRegFineAdd() 10 3.017e-05 3.017e-05 3.017e-05 0.00% AmrLevel::checkPointPost() 3 1.429e-05 1.429e-05 1.429e-05 0.00% Amr::writeSmallPlotFile() 1 1.275e-05 1.275e-05 1.275e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.271e-05 1.271e-05 1.271e-05 0.00% AmrLevel::checkPointPre() 3 1.033e-05 1.033e-05 1.033e-05 0.00% Amr::initSubcycle() 1 8.542e-06 8.542e-06 8.542e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-13-g0278677bbc8c) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-13-g0278677bbc8c) initialized Starting run at 09:06:57 UTC on 2022-01-10. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-13-g0278677bb Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.668684284 Restart time = 0.059967662 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.145396088 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.075010785 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.074858827 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.085140049 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.080585983 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027537109 seconds Ending run at 09:06:58 UTC on 2022-01-10. Run time = 0.553105444 Run time without initialization = 0.492490029 Average number of zones advanced per microsecond: 2.661 Average number of zones advanced per microsecond per rank: 2.661 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 0.5531 ... 0.5531 ... 0.5531 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1309 0.1309 0.1309 23.66% MLCellLinOp::applyBC() 1946 0.05506 0.05506 0.05506 9.95% VisMF::Read() 3 0.04545 0.04545 0.04545 8.22% MLPoisson::Fsmooth() 1440 0.03281 0.03281 0.03281 5.93% VisMF::Write(FabArray) 1 0.02604 0.02604 0.02604 4.71% FabArray::setVal() 533 0.01898 0.01898 0.01898 3.43% FillBoundary_nowait() 1766 0.01803 0.01803 0.01803 3.26% MLCGSolver::bicgstab 36 0.01677 0.01677 0.01677 3.03% StateData::FillBoundary(geom) 160 0.0124 0.0124 0.0124 2.24% MultiFab::Dot() 484 0.01227 0.01227 0.01227 2.22% MLCellLinOp::smooth() 720 0.01041 0.01041 0.01041 1.88% FabArray::ParallelCopy_nowait() 380 0.01035 0.01035 0.01035 1.87% FabArray::FillBoundary() 1766 0.01016 0.01016 0.01016 1.84% StateDataPhysBCFunct::() 20 0.009713 0.009713 0.009713 1.76% MultiFab::LinComb() 690 0.008812 0.008812 0.008812 1.59% MLCellLinOp::defineAuxData() 6 0.007632 0.007632 0.007632 1.38% MLPoisson::Fapply() 500 0.00702 0.00702 0.00702 1.27% Gravity::fill_multipole_BCs() 6 0.006928 0.006928 0.006928 1.25% Castro::computeTemp() 30 0.00607 0.00607 0.00607 1.10% FabArrayBase::getFB() 1766 0.005304 0.005304 0.005304 0.96% Amr::writePlotFile() 1 0.005128 0.005128 0.005128 0.93% Amr::restart() 1 0.004936 0.004936 0.004936 0.89% MLMG::addInterpCorrection() 180 0.004476 0.004476 0.004476 0.81% MLCellLinOp::apply() 500 0.00436 0.00436 0.00436 0.79% Castro::expand_state() 5 0.004287 0.004287 0.004287 0.78% amrex::average_down 180 0.004155 0.004155 0.004155 0.75% FabArray::setDomainBndry() 20 0.004115 0.004115 0.004115 0.74% CGSolver::sxay() 690 0.004011 0.004011 0.004011 0.73% MultiFab::Xpay() 258 0.003951 0.003951 0.003951 0.71% Castro::do_advance_ctu() 5 0.002909 0.002909 0.002909 0.53% FabArrayBase::getCPC() 632 0.002852 0.002852 0.002852 0.52% MLCGSolver::ParallelAllReduce 659 0.002707 0.002707 0.002707 0.49% Castro::reset_internal_energy() 30 0.002632 0.002632 0.002632 0.48% Castro::estTimeStep() 10 0.002511 0.002511 0.002511 0.45% Castro::enforce_min_density() 30 0.002509 0.002509 0.002509 0.45% BndryData::define() 6 0.002486 0.002486 0.002486 0.45% FabArray::ParallelCopy() 380 0.002256 0.002256 0.002256 0.41% Gravity::get_new_grav_vector() 5 0.001999 0.001999 0.001999 0.36% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001997 0.001997 0.001997 0.36% MLCellLinOp::correctionResidual() 216 0.001916 0.001916 0.001916 0.35% Castro::construct_new_gravity_source() 5 0.001652 0.001652 0.001652 0.30% MLCellLinOp::compGrad() 6 0.001477 0.001477 0.001477 0.27% Castro::normalize_species() 30 0.001472 0.001472 0.001472 0.27% Castro::construct_old_gravity_source() 5 0.00138 0.00138 0.00138 0.25% MLMG::mgVcycle() 36 0.001352 0.001352 0.001352 0.24% Castro::enforce_speed_limit() 30 0.001099 0.001099 0.001099 0.20% FabArrayBase::CPC::define() 244 0.001092 0.001092 0.001092 0.20% MLMG::ResNormInf() 42 0.00107 0.00107 0.00107 0.19% MLMG:computeResOfCorrection() 180 0.001053 0.001053 0.001053 0.19% MLMG::oneIter() 36 0.001037 0.001037 0.001037 0.19% Gravity::get_old_grav_vector() 5 0.001034 0.001034 0.001034 0.19% MultiFab::Saxpy() 10 0.000959 0.000959 0.000959 0.17% MLCellLinOp::setLevelBC() 6 0.000958 0.000958 0.000958 0.17% Castro::construct_new_source() 25 0.0009386 0.0009386 0.0009386 0.17% MLMG::prepareForSolve() 6 0.0009096 0.0009096 0.0009096 0.16% Gravity::actual_solve_with_mlmg() 6 0.0008208 0.0008208 0.0008208 0.15% MLCellLinOp::prepareForSolve() 6 0.0007697 0.0007697 0.0007697 0.14% FabArray::mult() 22 0.0007585 0.0007585 0.0007585 0.14% Castro::post_timestep() 5 0.0006992 0.0006992 0.0006992 0.13% Gravity::update_max_rhs() 6 0.000661 0.000661 0.000661 0.12% Castro::subcycle_advance_ctu() 5 0.000651 0.000651 0.000651 0.12% MLMG::mgVcycle_down::3 36 0.0006503 0.0006503 0.0006503 0.12% MLMG::mgVcycle_down::1 36 0.0006419 0.0006419 0.0006419 0.12% MLMG::mgVcycle_down::0 36 0.0006415 0.0006415 0.0006415 0.12% MLMG::mgVcycle_down::4 36 0.0006354 0.0006354 0.0006354 0.11% MLMG::mgVcycle_down::2 36 0.0006228 0.0006228 0.0006228 0.11% MLMG::actualBottomSolve() 36 0.0005375 0.0005375 0.0005375 0.10% MLMG::solve() 6 0.0004698 0.0004698 0.0004698 0.08% Castro::clean_state() 30 0.0004522 0.0004522 0.0004522 0.08% MLMG::mgVcycle_up::0 36 0.0004448 0.0004448 0.0004448 0.08% Amr::InitAmr() 1 0.0004388 0.0004388 0.0004388 0.08% MLMG::mgVcycle_up::4 36 0.0004347 0.0004347 0.0004347 0.08% MLMG::mgVcycle_up::2 36 0.0004287 0.0004287 0.0004287 0.08% MultiFab::Copy() 6 0.0004249 0.0004249 0.0004249 0.08% MLMG::mgVcycle_up::3 36 0.0004141 0.0004141 0.0004141 0.07% MLMG::mgVcycle_up::1 36 0.0004127 0.0004127 0.0004127 0.07% MLCellLinOp::solutionResidual() 42 0.0003664 0.0003664 0.0003664 0.07% main() 1 0.0003458 0.0003458 0.0003458 0.06% Gravity::solve_for_phi() 5 0.000298 0.000298 0.000298 0.05% FillPatchIterator::Initialize 20 0.0002935 0.0002935 0.0002935 0.05% Castro::construct_new_gravity() 5 0.0002796 0.0002796 0.0002796 0.05% Gravity::actual_multilevel_solve() 1 0.0002306 0.0002306 0.0002306 0.04% Castro::initialize_advance() 5 0.0002225 0.0002225 0.0002225 0.04% Castro::do_new_sources() 5 0.0002115 0.0002115 0.0002115 0.04% MLMG::computeResidual() 36 0.0002103 0.0002103 0.0002103 0.04% MLMG::mgVcycle_bottom 36 0.000209 0.000209 0.000209 0.04% Castro::construct_old_source() 25 0.0002012 0.0002012 0.0002012 0.04% Amr::coarseTimeStep() 5 0.000193 0.000193 0.000193 0.03% MultiFab::min() 5 0.0001848 0.0001848 0.0001848 0.03% FillPatchSingleLevel 20 0.0001812 0.0001812 0.0001812 0.03% makeSFC 30 0.0001798 0.0001798 0.0001798 0.03% MultiFab::max() 6 0.0001635 0.0001635 0.0001635 0.03% Castro::do_old_sources() 5 0.000159 0.000159 0.000159 0.03% MLLinOp::defineGrids() 6 0.0001446 0.0001446 0.0001446 0.03% MLMG::MLRhsNormInf() 6 0.0001439 0.0001439 0.0001439 0.03% Castro::advance() 5 0.0001428 0.0001428 0.0001428 0.03% MLCellLinOp::defineBC() 6 0.0001415 0.0001415 0.0001415 0.03% Amr::timeStep() 5 0.0001385 0.0001385 0.0001385 0.03% FabArrayBase::FB::FB() 26 0.0001384 0.0001384 0.0001384 0.03% Castro::create_source_corrector() 5 0.0001265 0.0001265 0.0001265 0.02% MLLinOp::makeAgglomeratedDMap 6 0.0001212 0.0001212 0.0001212 0.02% StateData::restartDoit() 4 0.0001158 0.0001158 0.0001158 0.02% AmrLevel::restart() 1 9.883e-05 9.883e-05 9.883e-05 0.02% Castro::initialize_do_advance() 5 9.658e-05 9.658e-05 9.658e-05 0.02% DistributionMapping::Distribute() 31 9.463e-05 9.463e-05 9.463e-05 0.02% MLPoisson::define() 6 8.883e-05 8.883e-05 8.883e-05 0.02% Gravity::solve_phi_with_mlmg() 6 8.279e-05 8.279e-05 8.279e-05 0.01% Castro::construct_old_gravity() 5 7.058e-05 7.058e-05 7.058e-05 0.01% Castro::finalize_advance() 5 6.95e-05 6.95e-05 6.95e-05 0.01% Castro::apply_source_to_state() 10 6.116e-05 6.116e-05 6.116e-05 0.01% Gravity::swapTimeLevels() 5 5.892e-05 5.892e-05 5.892e-05 0.01% MLLinOp::define() 6 4.346e-05 4.346e-05 4.346e-05 0.01% Castro::swap_state_time_levels() 5 4.112e-05 4.112e-05 4.112e-05 0.01% MLPoisson::prepareForSolve() 6 3.762e-05 3.762e-05 3.762e-05 0.01% Castro::post_restart() 1 3.742e-05 3.742e-05 3.742e-05 0.01% MLMG::computeMLResidual() 6 3.621e-05 3.621e-05 3.621e-05 0.01% MLMG::getGradSolution() 6 3.604e-05 3.604e-05 3.604e-05 0.01% MLMG::MLResNormInf() 6 3.546e-05 3.546e-05 3.546e-05 0.01% Castro::finalize_do_advance() 5 3.425e-05 3.425e-05 3.425e-05 0.01% Amr::writeSmallPlotFile() 1 3.354e-05 3.354e-05 3.354e-05 0.01% Castro::computeNewDt() 5 3.054e-05 3.054e-05 3.054e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.246e-05 2.246e-05 2.246e-05 0.00% MLMG::buildFineMask() 6 1.996e-05 1.996e-05 1.996e-05 0.00% Gravity::set_mass_offset() 6 1.938e-05 1.938e-05 1.938e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.771e-05 1.771e-05 1.771e-05 0.00% Castro::retry_advance_ctu() 5 1.6e-05 1.6e-05 1.6e-05 0.00% Castro::FluxRegFineAdd() 5 1.512e-05 1.512e-05 1.512e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 8.709e-06 8.709e-06 8.709e-06 0.00% Amr::initSubcycle() 1 8.583e-06 8.583e-06 8.583e-06 0.00% Amr::init() 1 6.834e-06 6.834e-06 6.834e-06 0.00% AmrLevel::AmrLevel() 1 3.666e-06 3.666e-06 3.666e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.5531 0.5531 0.5531 100.00% Amr::coarseTimeStep() 5 0.4611 0.4611 0.4611 83.36% Amr::timeStep() 5 0.4595 0.4595 0.4595 83.07% Castro::advance() 5 0.4568 0.4568 0.4568 82.59% Castro::subcycle_advance_ctu() 5 0.4444 0.4444 0.4444 80.34% Castro::do_advance_ctu() 5 0.4437 0.4437 0.4437 80.22% Castro::construct_new_gravity() 5 0.2649 0.2649 0.2649 47.90% Gravity::solve_phi_with_mlmg() 6 0.2609 0.2609 0.2609 47.16% Gravity::actual_solve_with_mlmg() 6 0.2537 0.2537 0.2537 45.87% Gravity::solve_for_phi() 5 0.2534 0.2534 0.2534 45.81% MLMG::solve() 6 0.2349 0.2349 0.2349 42.46% MLMG::oneIter() 36 0.2223 0.2223 0.2223 40.20% MLMG::mgVcycle() 36 0.2213 0.2213 0.2213 40.01% Castro::construct_ctu_hydro_source() 5 0.1309 0.1309 0.1309 23.66% MLCellLinOp::smooth() 720 0.108 0.108 0.108 19.52% MLCellLinOp::applyBC() 1946 0.08869 0.08869 0.08869 16.03% MLMG::mgVcycle_bottom 36 0.06556 0.06556 0.06556 11.85% MLMG::actualBottomSolve() 36 0.06536 0.06536 0.06536 11.82% MLCGSolver::bicgstab 36 0.06436 0.06436 0.06436 11.63% Amr::init() 1 0.06001 0.06001 0.06001 10.85% Amr::restart() 1 0.06001 0.06001 0.06001 10.85% AmrLevel::restart() 1 0.04659 0.04659 0.04659 8.42% StateData::restartDoit() 4 0.04648 0.04648 0.04648 8.40% VisMF::Read() 3 0.04545 0.04545 0.04545 8.22% MLCellLinOp::apply() 500 0.03485 0.03485 0.03485 6.30% FabArray::FillBoundary() 1766 0.03363 0.03363 0.03363 6.08% MLPoisson::Fsmooth() 1440 0.03281 0.03281 0.03281 5.93% Amr::writePlotFile() 1 0.03116 0.03116 0.03116 5.63% FillPatchIterator::Initialize 20 0.0292 0.0292 0.0292 5.28% VisMF::Write(FabArray) 1 0.02604 0.02604 0.02604 4.71% FillPatchSingleLevel 20 0.02479 0.02479 0.02479 4.48% MLMG::mgVcycle_down::0 36 0.02478 0.02478 0.02478 4.48% FillBoundary_nowait() 1766 0.02347 0.02347 0.02347 4.24% StateDataPhysBCFunct::() 20 0.02211 0.02211 0.02211 4.00% MLMG::mgVcycle_up::0 36 0.02075 0.02075 0.02075 3.75% MLCellLinOp::correctionResidual() 216 0.02021 0.02021 0.02021 3.65% FabArray::setVal() 533 0.01898 0.01898 0.01898 3.43% MLMG:computeResOfCorrection() 180 0.01811 0.01811 0.01811 3.27% MLMG::mgVcycle_down::1 36 0.01567 0.01567 0.01567 2.83% MLMG::mgVcycle_down::2 36 0.01562 0.01562 0.01562 2.82% Castro::initialize_do_advance() 5 0.01559 0.01559 0.01559 2.82% MLPoisson::define() 6 0.01512 0.01512 0.01512 2.73% MLMG::mgVcycle_down::3 36 0.01502 0.01502 0.01502 2.72% MLMG::mgVcycle_down::4 36 0.01497 0.01497 0.01497 2.71% FabArray::ParallelCopy() 380 0.01438 0.01438 0.01438 2.60% Castro::clean_state() 30 0.01424 0.01424 0.01424 2.57% Castro::expand_state() 5 0.01303 0.01303 0.01303 2.36% CGSolver::sxay() 690 0.01282 0.01282 0.01282 2.32% StateData::FillBoundary(geom) 160 0.0124 0.0124 0.0124 2.24% MultiFab::Dot() 484 0.01227 0.01227 0.01227 2.22% Castro::initialize_advance() 5 0.0122 0.0122 0.0122 2.21% FabArray::ParallelCopy_nowait() 380 0.01212 0.01212 0.01212 2.19% MLMG::mgVcycle_up::2 36 0.01212 0.01212 0.01212 2.19% MLMG::mgVcycle_up::1 36 0.01197 0.01197 0.01197 2.16% MLMG::mgVcycle_up::4 36 0.01183 0.01183 0.01183 2.14% MLMG::mgVcycle_up::3 36 0.01167 0.01167 0.01167 2.11% Gravity::get_new_grav_vector() 5 0.01123 0.01123 0.01123 2.03% MLCellLinOp::defineAuxData() 6 0.01081 0.01081 0.01081 1.95% MLMG::addInterpCorrection() 180 0.01039 0.01039 0.01039 1.88% amrex::average_down 180 0.01013 0.01013 0.01013 1.83% MLCGSolver::ParallelAllReduce 659 0.009879 0.009879 0.009879 1.79% Castro::construct_old_gravity() 5 0.009727 0.009727 0.009727 1.76% Gravity::get_old_grav_vector() 5 0.009657 0.009657 0.009657 1.75% MultiFab::LinComb() 690 0.008812 0.008812 0.008812 1.59% Castro::computeTemp() 30 0.008702 0.008702 0.008702 1.57% Castro::post_restart() 1 0.008479 0.008479 0.008479 1.53% Gravity::multilevel_solve_for_new_phi() 1 0.007995 0.007995 0.007995 1.45% Gravity::actual_multilevel_solve() 1 0.007972 0.007972 0.007972 1.44% Castro::do_new_sources() 5 0.007343 0.007343 0.007343 1.33% MLPoisson::Fapply() 500 0.00702 0.00702 0.00702 1.27% Gravity::fill_multipole_BCs() 6 0.006928 0.006928 0.006928 1.25% MLCellLinOp::solutionResidual() 42 0.005526 0.005526 0.005526 1.00% FabArrayBase::getFB() 1766 0.005442 0.005442 0.005442 0.98% MLMG::prepareForSolve() 6 0.005024 0.005024 0.005024 0.91% MLMG::computeResidual() 36 0.004826 0.004826 0.004826 0.87% Castro::do_old_sources() 5 0.004513 0.004513 0.004513 0.82% FabArray::setDomainBndry() 20 0.004115 0.004115 0.004115 0.74% MultiFab::Xpay() 258 0.003951 0.003951 0.003951 0.71% FabArrayBase::getCPC() 632 0.003945 0.003945 0.003945 0.71% MLCellLinOp::defineBC() 6 0.003618 0.003618 0.003618 0.65% BndryData::define() 6 0.003477 0.003477 0.003477 0.63% Castro::reset_internal_energy() 30 0.002632 0.002632 0.002632 0.48% Castro::construct_new_source() 25 0.00259 0.00259 0.00259 0.47% Castro::post_timestep() 5 0.002534 0.002534 0.002534 0.46% Castro::estTimeStep() 10 0.002511 0.002511 0.002511 0.45% Castro::enforce_min_density() 30 0.002509 0.002509 0.002509 0.45% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001997 0.001997 0.001997 0.36% MLMG::getGradSolution() 6 0.001954 0.001954 0.001954 0.35% MLCellLinOp::compGrad() 6 0.001918 0.001918 0.001918 0.35% Castro::construct_new_gravity_source() 5 0.001652 0.001652 0.001652 0.30% Castro::construct_old_source() 25 0.001582 0.001582 0.001582 0.29% Castro::normalize_species() 30 0.001472 0.001472 0.001472 0.27% Castro::computeNewDt() 5 0.001409 0.001409 0.001409 0.25% Castro::construct_old_gravity_source() 5 0.00138 0.00138 0.00138 0.25% Castro::enforce_speed_limit() 30 0.001099 0.001099 0.001099 0.20% FabArrayBase::CPC::define() 244 0.001092 0.001092 0.001092 0.20% MLMG::ResNormInf() 42 0.00107 0.00107 0.00107 0.19% Castro::apply_source_to_state() 10 0.00102 0.00102 0.00102 0.18% Gravity::swapTimeLevels() 5 0.000987 0.000987 0.000987 0.18% MultiFab::Saxpy() 10 0.000959 0.000959 0.000959 0.17% MLCellLinOp::setLevelBC() 6 0.000958 0.000958 0.000958 0.17% MLMG::computeMLResidual() 6 0.0009465 0.0009465 0.0009465 0.17% Gravity::update_max_rhs() 6 0.000927 0.000927 0.000927 0.17% MLPoisson::prepareForSolve() 6 0.0008073 0.0008073 0.0008073 0.15% MLCellLinOp::prepareForSolve() 6 0.0007697 0.0007697 0.0007697 0.14% FabArray::mult() 22 0.0007585 0.0007585 0.0007585 0.14% MLLinOp::define() 6 0.0005973 0.0005973 0.0005973 0.11% MLLinOp::defineGrids() 6 0.0005538 0.0005538 0.0005538 0.10% Amr::InitAmr() 1 0.0004473 0.0004473 0.0004473 0.08% MultiFab::Copy() 6 0.0004249 0.0004249 0.0004249 0.08% MLLinOp::makeAgglomeratedDMap 6 0.0003915 0.0003915 0.0003915 0.07% makeSFC 30 0.0002703 0.0002703 0.0002703 0.05% MLMG::MLResNormInf() 6 0.0001972 0.0001972 0.0001972 0.04% MultiFab::min() 5 0.0001848 0.0001848 0.0001848 0.03% MultiFab::max() 6 0.0001635 0.0001635 0.0001635 0.03% MLMG::MLRhsNormInf() 6 0.0001439 0.0001439 0.0001439 0.03% FabArrayBase::FB::FB() 26 0.0001384 0.0001384 0.0001384 0.03% Castro::create_source_corrector() 5 0.0001265 0.0001265 0.0001265 0.02% DistributionMapping::Distribute() 31 9.463e-05 9.463e-05 9.463e-05 0.02% Castro::finalize_advance() 5 8.463e-05 8.463e-05 8.463e-05 0.02% Castro::swap_state_time_levels() 5 4.112e-05 4.112e-05 4.112e-05 0.01% Castro::finalize_do_advance() 5 3.425e-05 3.425e-05 3.425e-05 0.01% Amr::writeSmallPlotFile() 1 3.354e-05 3.354e-05 3.354e-05 0.01% MLMG::buildFineMask() 6 1.996e-05 1.996e-05 1.996e-05 0.00% Gravity::set_mass_offset() 6 1.938e-05 1.938e-05 1.938e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.771e-05 1.771e-05 1.771e-05 0.00% Castro::retry_advance_ctu() 5 1.6e-05 1.6e-05 1.6e-05 0.00% Castro::FluxRegFineAdd() 5 1.512e-05 1.512e-05 1.512e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.283e-05 1.283e-05 1.283e-05 0.00% Amr::initSubcycle() 1 8.583e-06 8.583e-06 8.583e-06 0.00% AmrLevel::AmrLevel() 1 3.666e-06 3.666e-06 3.666e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-13-g0278677bbc8c) finalized