Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-26-g0d136ea53aed) initialized Starting run at 08:25:03 UTC on 2022-04-20. Successfully read inputs file ... Castro git describe: 22.04-12-g78961ce2c AMReX git describe: 22.04-26-g0d136ea53 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038147968 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022386396 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.107504296 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051379572 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048500686 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059212961 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.060038504 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035991878 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.05151595 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049328287 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058684351 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060660036 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063462469 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.034831207 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.021886424 seconds Ending run at 08:25:03 UTC on 2022-04-20. Run time = 0.824726685 Run time without initialization = 0.703483002 Average number of zones advanced per microsecond: 3.726 Average number of zones advanced per microsecond per rank: 3.726 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8248 ... 0.8248 ... 0.8248 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2114 0.2114 0.2114 25.63% VisMF::Write(FabArray) 11 0.1462 0.1462 0.1462 17.73% MLCellLinOp::applyBC() 4433 0.09265 0.09265 0.09265 11.23% MLPoisson::Fsmooth() 3280 0.05964 0.05964 0.05964 7.23% FabArray::setVal() 1144 0.02491 0.02491 0.02491 3.02% StateData::FillBoundary(geom) 328 0.02358 0.02358 0.02358 2.86% MLCGSolver::bicgstab 82 0.02165 0.02165 0.02165 2.62% MultiFab::Dot() 1114 0.02118 0.02118 0.02118 2.57% FillBoundary_nowait() 4023 0.01695 0.01695 0.01695 2.06% StateDataPhysBCFunct::() 41 0.01665 0.01665 0.01665 2.02% FabArray::ParallelCopy_nowait() 861 0.01608 0.01608 0.01608 1.95% Castro::computeTemp() 63 0.01469 0.01469 0.01469 1.78% MultiFab::LinComb() 1586 0.01209 0.01209 0.01209 1.47% Gravity::fill_multipole_BCs() 11 0.01176 0.01176 0.01176 1.43% Castro::enforce_min_density() 62 0.01164 0.01164 0.01164 1.41% MLPoisson::Fapply() 1142 0.01053 0.01053 0.01053 1.28% MLCellLinOp::defineAuxData() 11 0.0103 0.0103 0.0103 1.25% MLMG::addInterpCorrection() 410 0.007051 0.007051 0.007051 0.85% amrex::average_down 410 0.006577 0.006577 0.006577 0.80% MultiFab::Xpay() 585 0.006009 0.006009 0.006009 0.73% FabArray::setDomainBndry() 41 0.005538 0.005538 0.005538 0.67% Castro::reset_internal_energy() 63 0.005346 0.005346 0.005346 0.65% Castro::estTimeStep() 21 0.005021 0.005021 0.005021 0.61% Castro::expand_state() 10 0.004764 0.004764 0.004764 0.58% Castro::normalize_species() 62 0.004395 0.004395 0.004395 0.53% Amr::checkPoint() 3 0.004122 0.004122 0.004122 0.50% BndryData::define() 11 0.003648 0.003648 0.003648 0.44% Castro::do_advance_ctu() 10 0.003116 0.003116 0.003116 0.38% Amr::writePlotFile() 2 0.002935 0.002935 0.002935 0.36% Castro::enforce_speed_limit() 62 0.002925 0.002925 0.002925 0.35% Gravity::get_new_grav_vector() 11 0.002628 0.002628 0.002628 0.32% Castro::construct_new_gravity_source() 10 0.002527 0.002527 0.002527 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002273 0.002273 0.002273 0.28% Castro::construct_old_gravity_source() 10 0.002104 0.002104 0.002104 0.26% MLCellLinOp::compGrad() 11 0.001981 0.001981 0.001981 0.24% MLMG::ResNormInf() 93 0.001944 0.001944 0.001944 0.24% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% Gravity::get_old_grav_vector() 10 0.001758 0.001758 0.001758 0.21% MLMG::oneIter() 82 0.0017 0.0017 0.0017 0.21% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% Gravity::actual_solve_with_mlmg() 11 0.001358 0.001358 0.001358 0.16% MLCellLinOp::prepareForSolve() 11 0.001341 0.001341 0.001341 0.16% FabArray::mult() 43 0.001322 0.001322 0.001322 0.16% Castro::initData() 1 0.001221 0.001221 0.001221 0.15% MLCellLinOp::smooth() 1640 0.001184 0.001184 0.001184 0.14% MultiFab::contains_nan() 20 0.001164 0.001164 0.001164 0.14% FabArrayBase::getCPC() 1323 0.0008716 0.0008716 0.0008716 0.11% FabArray::FillBoundary() 4023 0.0008629 0.0008629 0.0008629 0.10% MLMG::prepareForSolve() 11 0.0007802 0.0007802 0.0007802 0.09% FabArrayBase::CPC::define() 454 0.0007539 0.0007539 0.0007539 0.09% FabArrayBase::getFB() 4023 0.0006801 0.0006801 0.0006801 0.08% MLCellLinOp::apply() 1142 0.000552 0.000552 0.000552 0.07% Gravity::update_max_rhs() 11 0.0005419 0.0005419 0.0005419 0.07% MultiFab::Copy() 11 0.000508 0.000508 0.000508 0.06% CGSolver::sxay() 1586 0.0004548 0.0004548 0.0004548 0.06% Gravity::solve_for_phi() 10 0.000442 0.000442 0.000442 0.05% Amr::InitAmr() 1 0.0004262 0.0004262 0.0004262 0.05% MLMG::mgVcycle() 82 0.0004131 0.0004131 0.0004131 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003346 0.0003346 0.0003346 0.04% MultiFab::min() 10 0.0003269 0.0003269 0.0003269 0.04% FabArray::ParallelCopy() 861 0.0003026 0.0003026 0.0003026 0.04% main() 1 0.0002796 0.0002796 0.0002796 0.03% FillPatchIterator::Initialize 41 0.0002634 0.0002634 0.0002634 0.03% MultiFab::max() 11 0.0002523 0.0002523 0.0002523 0.03% MLCellLinOp::correctionResidual() 492 0.0002426 0.0002426 0.0002426 0.03% Gravity::actual_multilevel_solve() 1 0.0002217 0.0002217 0.0002217 0.03% Castro::construct_new_gravity() 10 0.0002117 0.0002117 0.0002117 0.03% Amr::coarseTimeStep() 10 0.0002073 0.0002073 0.0002073 0.03% Amr::timeStep() 10 0.0001971 0.0001971 0.0001971 0.02% MLMG::MLRhsNormInf() 11 0.0001967 0.0001967 0.0001967 0.02% MLCellLinOp::defineBC() 11 0.0001865 0.0001865 0.0001865 0.02% MLLinOp::defineGrids() 11 0.0001773 0.0001773 0.0001773 0.02% Amr::defBaseLevel() 1 0.0001472 0.0001472 0.0001472 0.02% MLMG:computeResOfCorrection() 410 0.0001431 0.0001431 0.0001431 0.02% StateData::checkPoint() 12 0.0001357 0.0001357 0.0001357 0.02% Castro::subcycle_advance_ctu() 10 0.0001332 0.0001332 0.0001332 0.02% MLMG::actualBottomSolve() 82 0.0001127 0.0001127 0.0001127 0.01% MLMG::mgVcycle_down::0 82 9.208e-05 9.208e-05 9.208e-05 0.01% FabArrayBase::FB::FB() 56 9.158e-05 9.158e-05 9.158e-05 0.01% Castro::post_timestep() 10 9.115e-05 9.115e-05 9.115e-05 0.01% AmrLevel::checkPoint() 3 8.166e-05 8.166e-05 8.166e-05 0.01% MLMG::mgVcycle_down::1 82 8.081e-05 8.081e-05 8.081e-05 0.01% Castro::initialize_advance() 10 7.894e-05 7.894e-05 7.894e-05 0.01% MLMG::solve() 11 7.868e-05 7.868e-05 7.868e-05 0.01% MLMG::mgVcycle_down::2 82 7.861e-05 7.861e-05 7.861e-05 0.01% MLMG::mgVcycle_down::4 82 7.56e-05 7.56e-05 7.56e-05 0.01% MLMG::mgVcycle_down::3 82 7.495e-05 7.495e-05 7.495e-05 0.01% Castro::clean_state() 62 7.297e-05 7.297e-05 7.297e-05 0.01% Castro::advance() 10 7.272e-05 7.272e-05 7.272e-05 0.01% Castro::construct_new_source() 50 6.87e-05 6.87e-05 6.87e-05 0.01% MLMG::mgVcycle_up::4 82 6.64e-05 6.64e-05 6.64e-05 0.01% Castro::finalize_advance() 10 5.766e-05 5.766e-05 5.766e-05 0.01% Castro::initialize_do_advance() 10 5.577e-05 5.577e-05 5.577e-05 0.01% MLMG::mgVcycle_up::0 82 5.553e-05 5.553e-05 5.553e-05 0.01% MLMG::mgVcycle_up::3 82 5.272e-05 5.272e-05 5.272e-05 0.01% MLMG::mgVcycle_up::2 82 5.072e-05 5.072e-05 5.072e-05 0.01% MLMG::mgVcycle_up::1 82 5.059e-05 5.059e-05 5.059e-05 0.01% MLCellLinOp::solutionResidual() 93 4.815e-05 4.815e-05 4.815e-05 0.01% Castro::finalize_do_advance() 10 4.466e-05 4.466e-05 4.466e-05 0.01% StateData::define() 4 4.303e-05 4.303e-05 4.303e-05 0.01% Castro::swap_state_time_levels() 10 3.935e-05 3.935e-05 3.935e-05 0.00% MLMG::computeResidual() 82 3.527e-05 3.527e-05 3.527e-05 0.00% Castro::enforce_consistent_e() 1 3.443e-05 3.443e-05 3.443e-05 0.00% MLMG::mgVcycle_bottom 82 3.422e-05 3.422e-05 3.422e-05 0.00% MLLinOp::define() 11 3.144e-05 3.144e-05 3.144e-05 0.00% FillPatchSingleLevel 41 3.022e-05 3.022e-05 3.022e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.919e-05 2.919e-05 2.919e-05 0.00% makeSFC 55 2.616e-05 2.616e-05 2.616e-05 0.00% MLPoisson::define() 11 2.411e-05 2.411e-05 2.411e-05 0.00% Amr::writeSmallPlotFile() 1 2.408e-05 2.408e-05 2.408e-05 0.00% Castro::construct_old_source() 50 2.251e-05 2.251e-05 2.251e-05 0.00% Amr::FinalizeInit() 1 2.103e-05 2.103e-05 2.103e-05 0.00% Castro::do_new_sources() 10 1.788e-05 1.788e-05 1.788e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.722e-05 1.722e-05 1.722e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.709e-05 1.709e-05 1.709e-05 0.00% Castro::do_old_sources() 10 1.628e-05 1.628e-05 1.628e-05 0.00% DistributionMapping::Distribute() 56 1.615e-05 1.615e-05 1.615e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.437e-05 1.437e-05 1.437e-05 0.00% Castro::apply_source_to_state() 20 1.266e-05 1.266e-05 1.266e-05 0.00% Castro::check_for_nan() 20 1.251e-05 1.251e-05 1.251e-05 0.00% Castro::construct_old_gravity() 10 1.099e-05 1.099e-05 1.099e-05 0.00% Gravity::swapTimeLevels() 10 1.066e-05 1.066e-05 1.066e-05 0.00% Amr::initSubcycle() 1 9.48e-06 9.48e-06 9.48e-06 0.00% MLPoisson::prepareForSolve() 11 8.793e-06 8.793e-06 8.793e-06 0.00% MLMG::computeMLResidual() 11 6.613e-06 6.613e-06 6.613e-06 0.00% Castro::computeNewDt() 9 5.935e-06 5.935e-06 5.935e-06 0.00% MLMG::getGradSolution() 11 5.833e-06 5.833e-06 5.833e-06 0.00% MLMG::buildFineMask() 11 5.753e-06 5.753e-06 5.753e-06 0.00% AmrLevel::checkPointPost() 3 5.742e-06 5.742e-06 5.742e-06 0.00% Amr::InitializeInit() 1 5.605e-06 5.605e-06 5.605e-06 0.00% Castro::create_source_corrector() 10 5.249e-06 5.249e-06 5.249e-06 0.00% MLMG::MLResNormInf() 11 4.832e-06 4.832e-06 4.832e-06 0.00% Gravity::set_mass_offset() 11 4.648e-06 4.648e-06 4.648e-06 0.00% Castro::retry_advance_ctu() 10 4.266e-06 4.266e-06 4.266e-06 0.00% Castro::post_init() 1 3.95e-06 3.95e-06 3.95e-06 0.00% Castro::FluxRegFineAdd() 10 3.187e-06 3.187e-06 3.187e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.603e-06 2.603e-06 2.603e-06 0.00% Amr::init() 1 2.497e-06 2.497e-06 2.497e-06 0.00% Castro::computeInitialDt() 2 2.311e-06 2.311e-06 2.311e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.253e-06 2.253e-06 2.253e-06 0.00% AmrLevel::checkPointPre() 3 1.899e-06 1.899e-06 1.899e-06 0.00% Castro::post_regrid() 1 1.152e-06 1.152e-06 1.152e-06 0.00% Amr::initialInit() 1 1.004e-06 1.004e-06 1.004e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8247 0.8247 0.8247 100.00% Amr::coarseTimeStep() 10 0.6814 0.6814 0.6814 82.62% Amr::timeStep() 10 0.6076 0.6076 0.6076 73.67% Castro::advance() 10 0.6002 0.6002 0.6002 72.78% Castro::subcycle_advance_ctu() 10 0.5857 0.5857 0.5857 71.02% Castro::do_advance_ctu() 10 0.5856 0.5856 0.5856 71.00% Gravity::solve_phi_with_mlmg() 11 0.3232 0.3232 0.3232 39.19% Gravity::actual_solve_with_mlmg() 11 0.3112 0.3112 0.3112 37.74% Castro::construct_new_gravity() 10 0.2942 0.2942 0.2942 35.67% MLMG::solve() 11 0.2884 0.2884 0.2884 34.97% Gravity::solve_for_phi() 10 0.2755 0.2755 0.2755 33.40% MLMG::oneIter() 82 0.2727 0.2727 0.2727 33.07% MLMG::mgVcycle() 82 0.271 0.271 0.271 32.86% Castro::construct_ctu_hydro_source() 10 0.2114 0.2114 0.2114 25.63% VisMF::Write(FabArray) 11 0.1462 0.1462 0.1462 17.73% MLCellLinOp::smooth() 1640 0.1433 0.1433 0.1433 17.38% Amr::init() 1 0.1207 0.1207 0.1207 14.63% MLCellLinOp::applyBC() 4433 0.1112 0.1112 0.1112 13.49% Amr::checkPoint() 3 0.1091 0.1091 0.1091 13.23% AmrLevel::checkPoint() 3 0.105 0.105 0.105 12.73% StateData::checkPoint() 12 0.1049 0.1049 0.1049 12.72% MLMG::mgVcycle_bottom 82 0.07708 0.07708 0.07708 9.35% MLMG::actualBottomSolve() 82 0.07705 0.07705 0.07705 9.34% MLCGSolver::bicgstab 82 0.07631 0.07631 0.07631 9.25% Amr::initialInit() 1 0.06001 0.06001 0.06001 7.28% MLPoisson::Fsmooth() 3280 0.05964 0.05964 0.05964 7.23% Amr::FinalizeInit() 1 0.05172 0.05172 0.05172 6.27% Castro::post_init() 1 0.05099 0.05099 0.05099 6.18% FillPatchIterator::Initialize 41 0.05029 0.05029 0.05029 6.10% Gravity::multilevel_solve_for_new_phi() 1 0.04842 0.04842 0.04842 5.87% Gravity::actual_multilevel_solve() 1 0.04841 0.04841 0.04841 5.87% FillPatchSingleLevel 41 0.04449 0.04449 0.04449 5.39% Amr::writePlotFile() 2 0.04439 0.04439 0.04439 5.38% StateDataPhysBCFunct::() 41 0.04022 0.04022 0.04022 4.88% MLCellLinOp::apply() 1142 0.03927 0.03927 0.03927 4.76% Castro::clean_state() 62 0.03842 0.03842 0.03842 4.66% MLMG::mgVcycle_down::0 82 0.03771 0.03771 0.03771 4.57% MLMG::mgVcycle_up::0 82 0.03227 0.03227 0.03227 3.91% FabArray::setVal() 1144 0.02491 0.02491 0.02491 3.02% Castro::initialize_do_advance() 10 0.02404 0.02404 0.02404 2.91% StateData::FillBoundary(geom) 328 0.02358 0.02358 0.02358 2.86% MLCellLinOp::correctionResidual() 492 0.02213 0.02213 0.02213 2.68% MultiFab::Dot() 1114 0.02118 0.02118 0.02118 2.57% Gravity::get_new_grav_vector() 11 0.02089 0.02089 0.02089 2.53% Castro::computeTemp() 63 0.02003 0.02003 0.02003 2.43% MLMG:computeResOfCorrection() 410 0.01915 0.01915 0.01915 2.32% FabArray::FillBoundary() 4023 0.01859 0.01859 0.01859 2.25% MLMG::mgVcycle_down::1 82 0.01851 0.01851 0.01851 2.24% MLMG::mgVcycle_down::2 82 0.01787 0.01787 0.01787 2.17% Castro::expand_state() 10 0.01778 0.01778 0.01778 2.16% FillBoundary_nowait() 4023 0.01772 0.01772 0.01772 2.15% MLPoisson::define() 11 0.01744 0.01744 0.01744 2.11% FabArray::ParallelCopy() 861 0.01728 0.01728 0.01728 2.09% MLMG::mgVcycle_down::3 82 0.01702 0.01702 0.01702 2.06% FabArray::ParallelCopy_nowait() 861 0.01697 0.01697 0.01697 2.06% Castro::construct_old_gravity() 10 0.01648 0.01648 0.01648 2.00% Gravity::get_old_grav_vector() 10 0.01646 0.01646 0.01646 2.00% MLMG::mgVcycle_down::4 82 0.01625 0.01625 0.01625 1.97% Castro::initialize_advance() 10 0.01435 0.01435 0.01435 1.74% MLMG::mgVcycle_up::2 82 0.0139 0.0139 0.0139 1.68% MLMG::mgVcycle_up::1 82 0.01371 0.01371 0.01371 1.66% MLMG::addInterpCorrection() 410 0.01359 0.01359 0.01359 1.65% MLMG::mgVcycle_up::3 82 0.01321 0.01321 0.01321 1.60% amrex::average_down 410 0.01308 0.01308 0.01308 1.59% MLMG::mgVcycle_up::4 82 0.01308 0.01308 0.01308 1.59% MLCGSolver::ParallelAllReduce 1514 0.01277 0.01277 0.01277 1.55% CGSolver::sxay() 1586 0.01255 0.01255 0.01255 1.52% MultiFab::LinComb() 1586 0.01209 0.01209 0.01209 1.47% MLCellLinOp::defineAuxData() 11 0.01204 0.01204 0.01204 1.46% Castro::do_new_sources() 10 0.01178 0.01178 0.01178 1.43% Gravity::fill_multipole_BCs() 11 0.01176 0.01176 0.01176 1.43% Castro::enforce_min_density() 62 0.01164 0.01164 0.01164 1.41% MLPoisson::Fapply() 1142 0.01053 0.01053 0.01053 1.28% Castro::do_old_sources() 10 0.008374 0.008374 0.008374 1.02% Amr::InitializeInit() 1 0.008292 0.008292 0.008292 1.01% Amr::defBaseLevel() 1 0.008286 0.008286 0.008286 1.00% MLCellLinOp::solutionResidual() 93 0.007656 0.007656 0.007656 0.93% Castro::post_timestep() 10 0.007189 0.007189 0.007189 0.87% MLMG::computeResidual() 82 0.006627 0.006627 0.006627 0.80% MultiFab::Xpay() 585 0.006009 0.006009 0.006009 0.73% MLMG::prepareForSolve() 11 0.005756 0.005756 0.005756 0.70% FabArray::setDomainBndry() 41 0.005538 0.005538 0.005538 0.67% Castro::reset_internal_energy() 63 0.005346 0.005346 0.005346 0.65% MLCellLinOp::defineBC() 11 0.005103 0.005103 0.005103 0.62% Castro::estTimeStep() 21 0.005021 0.005021 0.005021 0.61% BndryData::define() 11 0.004916 0.004916 0.004916 0.60% Castro::normalize_species() 62 0.004395 0.004395 0.004395 0.53% Castro::initData() 1 0.003305 0.003305 0.003305 0.40% Castro::enforce_speed_limit() 62 0.002925 0.002925 0.002925 0.35% Castro::construct_new_source() 50 0.002596 0.002596 0.002596 0.31% Castro::construct_new_gravity_source() 10 0.002527 0.002527 0.002527 0.31% MLMG::getGradSolution() 11 0.002522 0.002522 0.002522 0.31% MLCellLinOp::compGrad() 11 0.002516 0.002516 0.002516 0.31% Castro::computeNewDt() 9 0.002342 0.002342 0.002342 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002273 0.002273 0.002273 0.28% Castro::construct_old_source() 50 0.002127 0.002127 0.002127 0.26% Castro::construct_old_gravity_source() 10 0.002104 0.002104 0.002104 0.26% MLMG::ResNormInf() 93 0.001944 0.001944 0.001944 0.24% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.22% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% FabArrayBase::getCPC() 1323 0.001625 0.001625 0.001625 0.20% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% MLPoisson::prepareForSolve() 11 0.001349 0.001349 0.001349 0.16% MLCellLinOp::prepareForSolve() 11 0.001341 0.001341 0.001341 0.16% FabArray::mult() 43 0.001322 0.001322 0.001322 0.16% Castro::check_for_nan() 20 0.001176 0.001176 0.001176 0.14% MultiFab::contains_nan() 20 0.001164 0.001164 0.001164 0.14% MLMG::computeMLResidual() 11 0.001071 0.001071 0.001071 0.13% Gravity::update_max_rhs() 11 0.000936 0.000936 0.000936 0.11% Gravity::swapTimeLevels() 10 0.0008818 0.0008818 0.0008818 0.11% FabArrayBase::getFB() 4023 0.0007716 0.0007716 0.0007716 0.09% FabArrayBase::CPC::define() 454 0.0007539 0.0007539 0.0007539 0.09% Castro::computeInitialDt() 2 0.0005202 0.0005202 0.0005202 0.06% MultiFab::Copy() 11 0.000508 0.000508 0.000508 0.06% Castro::post_regrid() 1 0.0004966 0.0004966 0.0004966 0.06% Amr::InitAmr() 1 0.0004357 0.0004357 0.0004357 0.05% MultiFab::min() 10 0.0003269 0.0003269 0.0003269 0.04% MLLinOp::define() 11 0.0002692 0.0002692 0.0002692 0.03% MultiFab::max() 11 0.0002523 0.0002523 0.0002523 0.03% MLMG::MLResNormInf() 11 0.0002519 0.0002519 0.0002519 0.03% MLLinOp::defineGrids() 11 0.0002378 0.0002378 0.0002378 0.03% MLMG::MLRhsNormInf() 11 0.0001967 0.0001967 0.0001967 0.02% FabArrayBase::FB::FB() 56 9.158e-05 9.158e-05 9.158e-05 0.01% Castro::finalize_advance() 10 6.085e-05 6.085e-05 6.085e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.819e-05 5.819e-05 5.819e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.74e-05 5.74e-05 5.74e-05 0.01% Castro::finalize_do_advance() 10 4.466e-05 4.466e-05 4.466e-05 0.01% StateData::define() 4 4.303e-05 4.303e-05 4.303e-05 0.01% makeSFC 55 4.109e-05 4.109e-05 4.109e-05 0.00% Castro::swap_state_time_levels() 10 3.935e-05 3.935e-05 3.935e-05 0.00% Castro::enforce_consistent_e() 1 3.443e-05 3.443e-05 3.443e-05 0.00% Amr::writeSmallPlotFile() 1 2.408e-05 2.408e-05 2.408e-05 0.00% DistributionMapping::Distribute() 56 1.615e-05 1.615e-05 1.615e-05 0.00% Amr::initSubcycle() 1 9.48e-06 9.48e-06 9.48e-06 0.00% MLMG::buildFineMask() 11 5.753e-06 5.753e-06 5.753e-06 0.00% AmrLevel::checkPointPost() 3 5.742e-06 5.742e-06 5.742e-06 0.00% Castro::create_source_corrector() 10 5.249e-06 5.249e-06 5.249e-06 0.00% Gravity::set_mass_offset() 11 4.648e-06 4.648e-06 4.648e-06 0.00% Castro::retry_advance_ctu() 10 4.266e-06 4.266e-06 4.266e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.823e-06 3.823e-06 3.823e-06 0.00% Castro::FluxRegFineAdd() 10 3.187e-06 3.187e-06 3.187e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.253e-06 2.253e-06 2.253e-06 0.00% AmrLevel::checkPointPre() 3 1.899e-06 1.899e-06 1.899e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-26-g0d136ea53aed) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-26-g0d136ea53aed) initialized Starting run at 08:25:04 UTC on 2022-04-20. Successfully read inputs file ... Castro git describe: 22.04-12-g78961ce2c AMReX git describe: 22.04-26-g0d136ea53 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.483810178 Restart time = 0.052353676 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.110027063 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049078074 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051781253 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.053718485 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.056416292 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03935841 seconds Ending run at 08:25:04 UTC on 2022-04-20. Run time = 0.413627794 Run time without initialization = 0.360721312 Average number of zones advanced per microsecond: 3.634 Average number of zones advanced per microsecond per rank: 3.634 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4137 ... 0.4137 ... 0.4137 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1221 0.1221 0.1221 29.52% VisMF::Read() 3 0.04137 0.04137 0.04137 10.00% MLCellLinOp::applyBC() 1946 0.03738 0.03738 0.03738 9.04% MLPoisson::Fsmooth() 1440 0.02539 0.02539 0.02539 6.14% VisMF::Write(FabArray) 1 0.02168 0.02168 0.02168 5.24% Amr::writePlotFile() 1 0.01776 0.01776 0.01776 4.29% FabArray::setVal() 537 0.01515 0.01515 0.01515 3.66% StateData::FillBoundary(geom) 160 0.01143 0.01143 0.01143 2.76% MLCGSolver::bicgstab 36 0.009213 0.009213 0.009213 2.23% MultiFab::Dot() 484 0.009054 0.009054 0.009054 2.19% FabArray::ParallelCopy_nowait() 380 0.007074 0.007074 0.007074 1.71% FillBoundary_nowait() 1766 0.007049 0.007049 0.007049 1.70% StateDataPhysBCFunct::() 20 0.006477 0.006477 0.006477 1.57% Gravity::fill_multipole_BCs() 6 0.005937 0.005937 0.005937 1.44% MLCellLinOp::defineAuxData() 6 0.0055 0.0055 0.0055 1.33% MultiFab::LinComb() 690 0.005158 0.005158 0.005158 1.25% MLPoisson::Fapply() 500 0.004564 0.004564 0.004564 1.10% Castro::expand_state() 5 0.004168 0.004168 0.004168 1.01% FabArray::setDomainBndry() 20 0.003889 0.003889 0.003889 0.94% Castro::computeTemp() 30 0.00374 0.00374 0.00374 0.90% Castro::do_advance_ctu() 5 0.003295 0.003295 0.003295 0.80% MLMG::addInterpCorrection() 180 0.002957 0.002957 0.002957 0.71% Amr::restart() 1 0.002946 0.002946 0.002946 0.71% amrex::average_down 180 0.00276 0.00276 0.00276 0.67% MultiFab::Xpay() 258 0.00261 0.00261 0.00261 0.63% Castro::estTimeStep() 10 0.002557 0.002557 0.002557 0.62% Castro::reset_internal_energy() 30 0.002425 0.002425 0.002425 0.59% Castro::enforce_min_density() 30 0.002286 0.002286 0.002286 0.55% Castro::normalize_species() 30 0.002238 0.002238 0.002238 0.54% BndryData::define() 6 0.001943 0.001943 0.001943 0.47% Castro::enforce_speed_limit() 30 0.001426 0.001426 0.001426 0.34% MLCellLinOp::compGrad() 6 0.001399 0.001399 0.001399 0.34% Castro::construct_new_gravity_source() 5 0.001277 0.001277 0.001277 0.31% Gravity::get_new_grav_vector() 5 0.001237 0.001237 0.001237 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001203 0.001203 0.001203 0.29% Castro::construct_old_gravity_source() 5 0.001014 0.001014 0.001014 0.25% MultiFab::Saxpy() 10 0.0009182 0.0009182 0.0009182 0.22% Gravity::get_old_grav_vector() 5 0.0009095 0.0009095 0.0009095 0.22% MLMG::ResNormInf() 42 0.0008524 0.0008524 0.0008524 0.21% Castro::post_timestep() 5 0.000841 0.000841 0.000841 0.20% MLCellLinOp::setLevelBC() 6 0.0008224 0.0008224 0.0008224 0.20% MLMG::oneIter() 36 0.0007468 0.0007468 0.0007468 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007178 0.0007178 0.0007178 0.17% MLCellLinOp::prepareForSolve() 6 0.0006868 0.0006868 0.0006868 0.17% FabArray::mult() 22 0.000655 0.000655 0.000655 0.16% MultiFab::contains_nan() 10 0.0005869 0.0005869 0.0005869 0.14% MLCellLinOp::smooth() 720 0.0005428 0.0005428 0.0005428 0.13% Gravity::update_max_rhs() 6 0.0005279 0.0005279 0.0005279 0.13% FabArrayBase::getCPC() 632 0.0004546 0.0004546 0.0004546 0.11% FabArrayBase::CPC::define() 244 0.0004454 0.0004454 0.0004454 0.11% MLMG::prepareForSolve() 6 0.0004313 0.0004313 0.0004313 0.10% FabArray::FillBoundary() 1766 0.0004166 0.0004166 0.0004166 0.10% Amr::InitAmr() 1 0.0003623 0.0003623 0.0003623 0.09% MultiFab::Copy() 6 0.0003112 0.0003112 0.0003112 0.08% FabArrayBase::getFB() 1766 0.0002984 0.0002984 0.0002984 0.07% main() 1 0.0002883 0.0002883 0.0002883 0.07% Gravity::actual_multilevel_solve() 1 0.0002374 0.0002374 0.0002374 0.06% MLCellLinOp::apply() 500 0.0002233 0.0002233 0.0002233 0.05% Gravity::solve_for_phi() 5 0.0002152 0.0002152 0.0002152 0.05% CGSolver::sxay() 690 0.000197 0.000197 0.000197 0.05% Castro::construct_new_gravity() 5 0.0001968 0.0001968 0.0001968 0.05% MLMG::mgVcycle() 36 0.0001814 0.0001814 0.0001814 0.04% MultiFab::min() 5 0.0001621 0.0001621 0.0001621 0.04% Castro::construct_new_source() 25 0.0001541 0.0001541 0.0001541 0.04% FabArray::ParallelCopy() 380 0.0001451 0.0001451 0.0001451 0.04% MLCGSolver::ParallelAllReduce 659 0.0001437 0.0001437 0.0001437 0.03% MultiFab::max() 6 0.0001403 0.0001403 0.0001403 0.03% FillPatchIterator::Initialize 20 0.0001272 0.0001272 0.0001272 0.03% Amr::coarseTimeStep() 5 0.0001109 0.0001109 0.0001109 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% Amr::timeStep() 5 0.0001035 0.0001035 0.0001035 0.03% MLCellLinOp::correctionResidual() 216 0.0001028 0.0001028 0.0001028 0.02% MLCellLinOp::defineBC() 6 0.0001006 0.0001006 0.0001006 0.02% MLLinOp::defineGrids() 6 9.531e-05 9.531e-05 9.531e-05 0.02% AmrLevel::restart() 1 7.58e-05 7.58e-05 7.58e-05 0.02% StateData::restartDoit() 4 7.491e-05 7.491e-05 7.491e-05 0.02% FabArrayBase::FB::FB() 26 6.473e-05 6.473e-05 6.473e-05 0.02% Castro::create_source_corrector() 5 6.39e-05 6.39e-05 6.39e-05 0.02% Castro::initialize_do_advance() 5 6.369e-05 6.369e-05 6.369e-05 0.02% Castro::subcycle_advance_ctu() 5 6.105e-05 6.105e-05 6.105e-05 0.01% MLMG:computeResOfCorrection() 180 5.799e-05 5.799e-05 5.799e-05 0.01% MLMG::actualBottomSolve() 36 4.787e-05 4.787e-05 4.787e-05 0.01% Castro::construct_old_source() 25 4.675e-05 4.675e-05 4.675e-05 0.01% Castro::advance() 5 4.459e-05 4.459e-05 4.459e-05 0.01% Castro::clean_state() 30 3.978e-05 3.978e-05 3.978e-05 0.01% Castro::initialize_advance() 5 3.749e-05 3.749e-05 3.749e-05 0.01% MLMG::mgVcycle_down::0 36 3.621e-05 3.621e-05 3.621e-05 0.01% MLMG::solve() 6 3.551e-05 3.551e-05 3.551e-05 0.01% MLMG::mgVcycle_down::1 36 3.438e-05 3.438e-05 3.438e-05 0.01% Castro::do_new_sources() 5 3.323e-05 3.323e-05 3.323e-05 0.01% MLMG::mgVcycle_down::2 36 3.107e-05 3.107e-05 3.107e-05 0.01% MLLinOp::define() 6 2.921e-05 2.921e-05 2.921e-05 0.01% MLMG::mgVcycle_down::4 36 2.911e-05 2.911e-05 2.911e-05 0.01% MLMG::mgVcycle_down::3 36 2.908e-05 2.908e-05 2.908e-05 0.01% Castro::swap_state_time_levels() 5 2.846e-05 2.846e-05 2.846e-05 0.01% Castro::finalize_advance() 5 2.78e-05 2.78e-05 2.78e-05 0.01% MLMG::mgVcycle_up::4 36 2.619e-05 2.619e-05 2.619e-05 0.01% Amr::writeSmallPlotFile() 1 2.532e-05 2.532e-05 2.532e-05 0.01% Castro::post_restart() 1 2.354e-05 2.354e-05 2.354e-05 0.01% MLCellLinOp::solutionResidual() 42 2.316e-05 2.316e-05 2.316e-05 0.01% Castro::construct_old_gravity() 5 2.313e-05 2.313e-05 2.313e-05 0.01% MLMG::mgVcycle_up::0 36 2.275e-05 2.275e-05 2.275e-05 0.01% Castro::finalize_do_advance() 5 2.133e-05 2.133e-05 2.133e-05 0.01% MLMG::mgVcycle_up::3 36 2.024e-05 2.024e-05 2.024e-05 0.00% MLMG::mgVcycle_up::2 36 1.994e-05 1.994e-05 1.994e-05 0.00% MLMG::mgVcycle_up::1 36 1.916e-05 1.916e-05 1.916e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.675e-05 1.675e-05 1.675e-05 0.00% MLPoisson::define() 6 1.611e-05 1.611e-05 1.611e-05 0.00% MLMG::mgVcycle_bottom 36 1.552e-05 1.552e-05 1.552e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.536e-05 1.536e-05 1.536e-05 0.00% MLMG::computeResidual() 36 1.517e-05 1.517e-05 1.517e-05 0.00% makeSFC 30 1.517e-05 1.517e-05 1.517e-05 0.00% FillPatchSingleLevel 20 1.51e-05 1.51e-05 1.51e-05 0.00% DistributionMapping::Distribute() 31 9.603e-06 9.603e-06 9.603e-06 0.00% Amr::initSubcycle() 1 9.013e-06 9.013e-06 9.013e-06 0.00% Castro::do_old_sources() 5 8.857e-06 8.857e-06 8.857e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.186e-06 8.186e-06 8.186e-06 0.00% Castro::check_for_nan() 10 6.555e-06 6.555e-06 6.555e-06 0.00% Castro::apply_source_to_state() 10 6.177e-06 6.177e-06 6.177e-06 0.00% Gravity::swapTimeLevels() 5 4.939e-06 4.939e-06 4.939e-06 0.00% MLPoisson::prepareForSolve() 6 4.708e-06 4.708e-06 4.708e-06 0.00% Castro::computeNewDt() 5 3.473e-06 3.473e-06 3.473e-06 0.00% MLMG::buildFineMask() 6 3.454e-06 3.454e-06 3.454e-06 0.00% MLMG::computeMLResidual() 6 3.412e-06 3.412e-06 3.412e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.408e-06 3.408e-06 3.408e-06 0.00% MLMG::getGradSolution() 6 3.343e-06 3.343e-06 3.343e-06 0.00% Gravity::set_mass_offset() 6 2.661e-06 2.661e-06 2.661e-06 0.00% MLMG::MLResNormInf() 6 2.421e-06 2.421e-06 2.421e-06 0.00% Castro::retry_advance_ctu() 5 1.84e-06 1.84e-06 1.84e-06 0.00% Castro::FluxRegFineAdd() 5 1.782e-06 1.782e-06 1.782e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.225e-06 1.225e-06 1.225e-06 0.00% AmrLevel::AmrLevel() 1 1.06e-06 1.06e-06 1.06e-06 0.00% Amr::init() 1 1.051e-06 1.051e-06 1.051e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4136 0.4136 0.4136 100.00% Amr::coarseTimeStep() 5 0.3211 0.3211 0.3211 77.63% Amr::timeStep() 5 0.3198 0.3198 0.3198 77.32% Castro::advance() 5 0.3169 0.3169 0.3169 76.60% Castro::subcycle_advance_ctu() 5 0.3071 0.3071 0.3071 74.25% Castro::do_advance_ctu() 5 0.3071 0.3071 0.3071 74.23% Castro::construct_new_gravity() 5 0.1439 0.1439 0.1439 34.78% Gravity::solve_phi_with_mlmg() 6 0.1402 0.1402 0.1402 33.90% Gravity::solve_for_phi() 5 0.1348 0.1348 0.1348 32.60% Gravity::actual_solve_with_mlmg() 6 0.1341 0.1341 0.1341 32.43% Castro::construct_ctu_hydro_source() 5 0.1221 0.1221 0.1221 29.52% MLMG::solve() 6 0.1215 0.1215 0.1215 29.38% MLMG::oneIter() 36 0.114 0.114 0.114 27.56% MLMG::mgVcycle() 36 0.1132 0.1132 0.1132 27.38% MLCellLinOp::smooth() 720 0.05935 0.05935 0.05935 14.35% Amr::init() 1 0.05239 0.05239 0.05239 12.66% Amr::restart() 1 0.05239 0.05239 0.05239 12.66% MLCellLinOp::applyBC() 1946 0.04521 0.04521 0.04521 10.93% AmrLevel::restart() 1 0.04219 0.04219 0.04219 10.20% StateData::restartDoit() 4 0.04211 0.04211 0.04211 10.18% VisMF::Read() 3 0.04137 0.04137 0.04137 10.00% Amr::writePlotFile() 1 0.03944 0.03944 0.03944 9.54% MLMG::mgVcycle_bottom 36 0.03254 0.03254 0.03254 7.87% MLMG::actualBottomSolve() 36 0.03253 0.03253 0.03253 7.86% MLCGSolver::bicgstab 36 0.03221 0.03221 0.03221 7.79% MLPoisson::Fsmooth() 1440 0.02539 0.02539 0.02539 6.14% FillPatchIterator::Initialize 20 0.02402 0.02402 0.02402 5.81% VisMF::Write(FabArray) 1 0.02168 0.02168 0.02168 5.24% FillPatchSingleLevel 20 0.02001 0.02001 0.02001 4.84% StateDataPhysBCFunct::() 20 0.01791 0.01791 0.01791 4.33% MLCellLinOp::apply() 500 0.0163 0.0163 0.0163 3.94% MLMG::mgVcycle_down::0 36 0.01586 0.01586 0.01586 3.83% FabArray::setVal() 537 0.01515 0.01515 0.01515 3.66% Castro::initialize_do_advance() 5 0.01389 0.01389 0.01389 3.36% MLMG::mgVcycle_up::0 36 0.0135 0.0135 0.0135 3.26% Castro::clean_state() 30 0.01216 0.01216 0.01216 2.94% Castro::expand_state() 5 0.01175 0.01175 0.01175 2.84% StateData::FillBoundary(geom) 160 0.01143 0.01143 0.01143 2.76% Castro::initialize_advance() 5 0.009656 0.009656 0.009656 2.33% MLPoisson::define() 6 0.009376 0.009376 0.009376 2.27% MLCellLinOp::correctionResidual() 216 0.009249 0.009249 0.009249 2.24% MultiFab::Dot() 484 0.009054 0.009054 0.009054 2.19% Gravity::get_new_grav_vector() 5 0.008831 0.008831 0.008831 2.13% MLMG:computeResOfCorrection() 180 0.00801 0.00801 0.00801 1.94% FabArray::FillBoundary() 1766 0.007829 0.007829 0.007829 1.89% MLMG::mgVcycle_down::1 36 0.007751 0.007751 0.007751 1.87% Castro::construct_old_gravity() 5 0.007681 0.007681 0.007681 1.86% FabArray::ParallelCopy() 380 0.007663 0.007663 0.007663 1.85% Gravity::get_old_grav_vector() 5 0.007658 0.007658 0.007658 1.85% FabArray::ParallelCopy_nowait() 380 0.007518 0.007518 0.007518 1.82% FillBoundary_nowait() 1766 0.007412 0.007412 0.007412 1.79% MLMG::mgVcycle_down::2 36 0.007379 0.007379 0.007379 1.78% MLMG::mgVcycle_down::3 36 0.007036 0.007036 0.007036 1.70% MLMG::mgVcycle_down::4 36 0.006711 0.006711 0.006711 1.62% MLCellLinOp::defineAuxData() 6 0.006451 0.006451 0.006451 1.56% Castro::post_restart() 1 0.006224 0.006224 0.006224 1.50% Castro::computeTemp() 30 0.006165 0.006165 0.006165 1.49% Gravity::fill_multipole_BCs() 6 0.005937 0.005937 0.005937 1.44% Gravity::multilevel_solve_for_new_phi() 1 0.005845 0.005845 0.005845 1.41% Gravity::actual_multilevel_solve() 1 0.005828 0.005828 0.005828 1.41% MLMG::addInterpCorrection() 180 0.005756 0.005756 0.005756 1.39% MLMG::mgVcycle_up::2 36 0.005752 0.005752 0.005752 1.39% MLMG::mgVcycle_up::1 36 0.005679 0.005679 0.005679 1.37% amrex::average_down 180 0.005539 0.005539 0.005539 1.34% MLCGSolver::ParallelAllReduce 659 0.005492 0.005492 0.005492 1.33% MLMG::mgVcycle_up::3 36 0.005446 0.005446 0.005446 1.32% MLMG::mgVcycle_up::4 36 0.005408 0.005408 0.005408 1.31% CGSolver::sxay() 690 0.005355 0.005355 0.005355 1.29% MultiFab::LinComb() 690 0.005158 0.005158 0.005158 1.25% Castro::do_new_sources() 5 0.004971 0.004971 0.004971 1.20% MLPoisson::Fapply() 500 0.004564 0.004564 0.004564 1.10% FabArray::setDomainBndry() 20 0.003889 0.003889 0.003889 0.94% MLCellLinOp::solutionResidual() 42 0.003366 0.003366 0.003366 0.81% Castro::do_old_sources() 5 0.003301 0.003301 0.003301 0.80% MLMG::prepareForSolve() 6 0.003173 0.003173 0.003173 0.77% Castro::post_timestep() 5 0.002875 0.002875 0.002875 0.70% MLMG::computeResidual() 36 0.002792 0.002792 0.002792 0.67% MLCellLinOp::defineBC() 6 0.002752 0.002752 0.002752 0.67% BndryData::define() 6 0.002651 0.002651 0.002651 0.64% MultiFab::Xpay() 258 0.00261 0.00261 0.00261 0.63% Castro::estTimeStep() 10 0.002557 0.002557 0.002557 0.62% Castro::reset_internal_energy() 30 0.002425 0.002425 0.002425 0.59% Castro::enforce_min_density() 30 0.002286 0.002286 0.002286 0.55% Castro::normalize_species() 30 0.002238 0.002238 0.002238 0.54% MLMG::getGradSolution() 6 0.001687 0.001687 0.001687 0.41% MLCellLinOp::compGrad() 6 0.001684 0.001684 0.001684 0.41% Castro::construct_new_source() 25 0.001431 0.001431 0.001431 0.35% Castro::enforce_speed_limit() 30 0.001426 0.001426 0.001426 0.34% Castro::construct_new_gravity_source() 5 0.001277 0.001277 0.001277 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001203 0.001203 0.001203 0.29% Castro::computeNewDt() 5 0.001166 0.001166 0.001166 0.28% Castro::construct_old_source() 25 0.001061 0.001061 0.001061 0.26% Castro::construct_old_gravity_source() 5 0.001014 0.001014 0.001014 0.25% Castro::apply_source_to_state() 10 0.0009244 0.0009244 0.0009244 0.22% MultiFab::Saxpy() 10 0.0009182 0.0009182 0.0009182 0.22% FabArrayBase::getCPC() 632 0.0009 0.0009 0.0009 0.22% Gravity::swapTimeLevels() 5 0.0008612 0.0008612 0.0008612 0.21% MLMG::ResNormInf() 42 0.0008524 0.0008524 0.0008524 0.21% MLCellLinOp::setLevelBC() 6 0.0008224 0.0008224 0.0008224 0.20% Gravity::update_max_rhs() 6 0.0007454 0.0007454 0.0007454 0.18% MLPoisson::prepareForSolve() 6 0.0006915 0.0006915 0.0006915 0.17% MLCellLinOp::prepareForSolve() 6 0.0006868 0.0006868 0.0006868 0.17% FabArray::mult() 22 0.000655 0.000655 0.000655 0.16% MLMG::computeMLResidual() 6 0.0005934 0.0005934 0.0005934 0.14% Castro::check_for_nan() 10 0.0005934 0.0005934 0.0005934 0.14% MultiFab::contains_nan() 10 0.0005869 0.0005869 0.0005869 0.14% FabArrayBase::CPC::define() 244 0.0004454 0.0004454 0.0004454 0.11% Amr::InitAmr() 1 0.0003713 0.0003713 0.0003713 0.09% FabArrayBase::getFB() 1766 0.0003631 0.0003631 0.0003631 0.09% MultiFab::Copy() 6 0.0003112 0.0003112 0.0003112 0.08% MultiFab::min() 5 0.0001621 0.0001621 0.0001621 0.04% MLLinOp::define() 6 0.0001571 0.0001571 0.0001571 0.04% MultiFab::max() 6 0.0001403 0.0001403 0.0001403 0.03% MLMG::MLResNormInf() 6 0.000133 0.000133 0.000133 0.03% MLLinOp::defineGrids() 6 0.0001279 0.0001279 0.0001279 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% FabArrayBase::FB::FB() 26 6.473e-05 6.473e-05 6.473e-05 0.02% Castro::create_source_corrector() 5 6.39e-05 6.39e-05 6.39e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.132e-05 3.132e-05 3.132e-05 0.01% Castro::finalize_advance() 5 2.958e-05 2.958e-05 2.958e-05 0.01% Castro::swap_state_time_levels() 5 2.846e-05 2.846e-05 2.846e-05 0.01% Amr::writeSmallPlotFile() 1 2.532e-05 2.532e-05 2.532e-05 0.01% makeSFC 30 2.314e-05 2.314e-05 2.314e-05 0.01% Castro::finalize_do_advance() 5 2.133e-05 2.133e-05 2.133e-05 0.01% DistributionMapping::Distribute() 31 9.603e-06 9.603e-06 9.603e-06 0.00% Amr::initSubcycle() 1 9.013e-06 9.013e-06 9.013e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.038e-06 5.038e-06 5.038e-06 0.00% MLMG::buildFineMask() 6 3.454e-06 3.454e-06 3.454e-06 0.00% Gravity::set_mass_offset() 6 2.661e-06 2.661e-06 2.661e-06 0.00% Castro::retry_advance_ctu() 5 1.84e-06 1.84e-06 1.84e-06 0.00% Castro::FluxRegFineAdd() 5 1.782e-06 1.782e-06 1.782e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.225e-06 1.225e-06 1.225e-06 0.00% AmrLevel::AmrLevel() 1 1.06e-06 1.06e-06 1.06e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-26-g0d136ea53aed) finalized