Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-4-gef52970b80e3) initialized Starting run at 08:26:19 UTC on 2022-05-06. Successfully read inputs file ... Castro git describe: 22.05-10-g5caf4517f AMReX git describe: 22.05-4-gef52970b8 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.040887412 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02415315 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04741193 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048261384 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.069177485 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.061488838 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.058553953 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.038755276 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047775528 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050219582 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057918101 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060654789 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06306279 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.038544169 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024523265 seconds Ending run at 08:26:20 UTC on 2022-05-06. Run time = 0.783217981 Run time without initialization = 0.666812536 Average number of zones advanced per microsecond: 3.931 Average number of zones advanced per microsecond per rank: 3.931 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7832 ... 0.7832 ... 0.7832 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1855 0.1855 0.1855 23.68% VisMF::Write(FabArray) 11 0.1604 0.1604 0.1604 20.48% MLCellLinOp::applyBC() 4433 0.07486 0.07486 0.07486 9.56% MLPoisson::Fsmooth() 3280 0.06072 0.06072 0.06072 7.75% StateData::FillBoundary(geom) 328 0.02384 0.02384 0.02384 3.04% MLCGSolver::bicgstab 82 0.02209 0.02209 0.02209 2.82% MultiFab::Dot() 1114 0.02146 0.02146 0.02146 2.74% Castro::normalize_species() 62 0.01965 0.01965 0.01965 2.51% Castro::computeTemp() 63 0.01507 0.01507 0.01507 1.92% FillBoundary_nowait() 4023 0.01446 0.01446 0.01446 1.85% FabArray::ParallelCopy_nowait() 861 0.01346 0.01346 0.01346 1.72% FabArray::setVal() 1144 0.01345 0.01345 0.01345 1.72% MultiFab::LinComb() 1586 0.01275 0.01275 0.01275 1.63% Castro::enforce_min_density() 62 0.01178 0.01178 0.01178 1.50% StateDataPhysBCFunct::() 41 0.01143 0.01143 0.01143 1.46% MLPoisson::Fapply() 1142 0.01082 0.01082 0.01082 1.38% MLCellLinOp::defineAuxData() 11 0.01017 0.01017 0.01017 1.30% Gravity::fill_multipole_BCs() 11 0.008439 0.008439 0.008439 1.08% Castro::estTimeStep() 21 0.007214 0.007214 0.007214 0.92% MLMG::addInterpCorrection() 410 0.006775 0.006775 0.006775 0.86% amrex::average_down 410 0.006342 0.006342 0.006342 0.81% MultiFab::Xpay() 585 0.006198 0.006198 0.006198 0.79% Castro::do_advance_ctu() 10 0.005285 0.005285 0.005285 0.67% Castro::reset_internal_energy(MultiFab) 63 0.004899 0.004899 0.004899 0.63% Amr::checkPoint() 3 0.0041 0.0041 0.0041 0.52% BndryData::define() 11 0.003846 0.003846 0.003846 0.49% Castro::enforce_speed_limit() 62 0.003291 0.003291 0.003291 0.42% Castro::construct_new_gravity_source() 10 0.002593 0.002593 0.002593 0.33% Amr::writePlotFile() 2 0.002411 0.002411 0.002411 0.31% MLMG::ResNormInf() 93 0.00199 0.00199 0.00199 0.25% Gravity::get_new_grav_vector() 11 0.001945 0.001945 0.001945 0.25% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.23% Gravity::get_old_grav_vector() 10 0.001747 0.001747 0.001747 0.22% Castro::expand_state() 10 0.001738 0.001738 0.001738 0.22% Castro::construct_old_gravity_source() 10 0.001728 0.001728 0.001728 0.22% MLMG::oneIter() 82 0.001722 0.001722 0.001722 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001699 0.001699 0.001699 0.22% Castro::reset_internal_energy(Fab) 504 0.001574 0.001574 0.001574 0.20% Gravity::actual_solve_with_mlmg() 11 0.001432 0.001432 0.001432 0.18% MLCellLinOp::setLevelBC() 11 0.001357 0.001357 0.001357 0.17% FabArray::mult() 43 0.001341 0.001341 0.001341 0.17% FabArray::setDomainBndry() 41 0.001329 0.001329 0.001329 0.17% Castro::initData() 1 0.001203 0.001203 0.001203 0.15% MultiFab::contains_nan() 20 0.001178 0.001178 0.001178 0.15% MLCellLinOp::prepareForSolve() 11 0.001116 0.001116 0.001116 0.14% MLCellLinOp::smooth() 1640 0.001079 0.001079 0.001079 0.14% MLCellLinOp::compGrad() 11 0.0009155 0.0009155 0.0009155 0.12% FabArrayBase::getCPC() 1323 0.0008334 0.0008334 0.0008334 0.11% FabArray::FillBoundary() 4023 0.0008247 0.0008247 0.0008247 0.11% MLMG::prepareForSolve() 11 0.0007972 0.0007972 0.0007972 0.10% Castro::subcycle_advance_ctu() 10 0.000765 0.000765 0.000765 0.10% FabArrayBase::getFB() 4023 0.0007141 0.0007141 0.0007141 0.09% FabArrayBase::CPC::define() 454 0.0006906 0.0006906 0.0006906 0.09% MLCellLinOp::apply() 1142 0.0004975 0.0004975 0.0004975 0.06% Amr::InitAmr() 1 0.0004552 0.0004552 0.0004552 0.06% CGSolver::sxay() 1586 0.000422 0.000422 0.000422 0.05% Gravity::update_max_rhs() 11 0.0004165 0.0004165 0.0004165 0.05% Gravity::solve_for_phi() 10 0.0003795 0.0003795 0.0003795 0.05% MLLinOp::defineGrids() 11 0.0003778 0.0003778 0.0003778 0.05% MLMG::mgVcycle() 82 0.0003734 0.0003734 0.0003734 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003231 0.0003231 0.0003231 0.04% FabArray::ParallelCopy() 861 0.0002823 0.0002823 0.0002823 0.04% main() 1 0.0002778 0.0002778 0.0002778 0.04% MultiFab::Copy() 11 0.000271 0.000271 0.000271 0.03% FillPatchIterator::Initialize 41 0.0002608 0.0002608 0.0002608 0.03% MultiFab::max() 11 0.0002599 0.0002599 0.0002599 0.03% MLCellLinOp::correctionResidual() 492 0.000231 0.000231 0.000231 0.03% Castro::construct_new_gravity() 10 0.0002217 0.0002217 0.0002217 0.03% MLMG::MLRhsNormInf() 11 0.0002023 0.0002023 0.0002023 0.03% Amr::coarseTimeStep() 10 0.0001921 0.0001921 0.0001921 0.02% MLCellLinOp::defineBC() 11 0.0001873 0.0001873 0.0001873 0.02% Amr::timeStep() 10 0.0001843 0.0001843 0.0001843 0.02% MLMG:computeResOfCorrection() 410 0.00014 0.00014 0.00014 0.02% StateData::checkPoint() 12 0.0001273 0.0001273 0.0001273 0.02% MLMG::actualBottomSolve() 82 0.0001153 0.0001153 0.0001153 0.01% MLMG::mgVcycle_down::0 82 9.642e-05 9.642e-05 9.642e-05 0.01% FabArrayBase::FB::FB() 56 9.177e-05 9.177e-05 9.177e-05 0.01% MLMG::mgVcycle_down::1 82 8.545e-05 8.545e-05 8.545e-05 0.01% Castro::Castro() 1 8.517e-05 8.517e-05 8.517e-05 0.01% MLMG::mgVcycle_down::2 82 8.307e-05 8.307e-05 8.307e-05 0.01% Castro::advance() 10 8.09e-05 8.09e-05 8.09e-05 0.01% MLMG::mgVcycle_down::4 82 7.86e-05 7.86e-05 7.86e-05 0.01% Castro::initialize_advance() 10 7.856e-05 7.856e-05 7.856e-05 0.01% MLMG::mgVcycle_down::3 82 7.726e-05 7.726e-05 7.726e-05 0.01% MLMG::solve() 11 7.589e-05 7.589e-05 7.589e-05 0.01% Castro::clean_state() 62 7.324e-05 7.324e-05 7.324e-05 0.01% Castro::finalize_advance() 10 7.089e-05 7.089e-05 7.089e-05 0.01% AmrLevel::checkPoint() 3 6.764e-05 6.764e-05 6.764e-05 0.01% MLMG::mgVcycle_up::4 82 6.091e-05 6.091e-05 6.091e-05 0.01% Castro::initialize_do_advance() 10 5.62e-05 5.62e-05 5.62e-05 0.01% MLMG::mgVcycle_up::0 82 5.001e-05 5.001e-05 5.001e-05 0.01% MLCellLinOp::solutionResidual() 93 4.927e-05 4.927e-05 4.927e-05 0.01% MLMG::mgVcycle_up::1 82 4.799e-05 4.799e-05 4.799e-05 0.01% MLMG::mgVcycle_up::3 82 4.798e-05 4.798e-05 4.798e-05 0.01% MLMG::mgVcycle_up::2 82 4.713e-05 4.713e-05 4.713e-05 0.01% StateData::define() 4 4.094e-05 4.094e-05 4.094e-05 0.01% Castro::construct_new_source() 50 4.039e-05 4.039e-05 4.039e-05 0.01% MLMG::mgVcycle_bottom 82 3.818e-05 3.818e-05 3.818e-05 0.00% Castro::swap_state_time_levels() 10 3.666e-05 3.666e-05 3.666e-05 0.00% Castro::finalize_do_advance() 10 3.627e-05 3.627e-05 3.627e-05 0.00% Castro::enforce_consistent_e() 1 3.479e-05 3.479e-05 3.479e-05 0.00% MLMG::computeResidual() 82 3.447e-05 3.447e-05 3.447e-05 0.00% Gravity::actual_multilevel_solve() 1 3.21e-05 3.21e-05 3.21e-05 0.00% Castro::post_timestep() 10 3.122e-05 3.122e-05 3.122e-05 0.00% FillPatchSingleLevel 41 2.819e-05 2.819e-05 2.819e-05 0.00% MLLinOp::define() 11 2.605e-05 2.605e-05 2.605e-05 0.00% makeSFC 55 2.577e-05 2.577e-05 2.577e-05 0.00% Castro::initMFs() 1 2.559e-05 2.559e-05 2.559e-05 0.00% Amr::writeSmallPlotFile() 1 2.425e-05 2.425e-05 2.425e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.416e-05 2.416e-05 2.416e-05 0.00% Amr::FinalizeInit() 1 2.117e-05 2.117e-05 2.117e-05 0.00% MLPoisson::define() 11 2.092e-05 2.092e-05 2.092e-05 0.00% Castro::buildMetrics() 1 1.955e-05 1.955e-05 1.955e-05 0.00% Castro::construct_old_source() 50 1.857e-05 1.857e-05 1.857e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.714e-05 1.714e-05 1.714e-05 0.00% Castro::do_new_sources() 10 1.647e-05 1.647e-05 1.647e-05 0.00% Castro::do_old_sources() 10 1.55e-05 1.55e-05 1.55e-05 0.00% DistributionMapping::Distribute() 56 1.505e-05 1.505e-05 1.505e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.498e-05 1.498e-05 1.498e-05 0.00% Amr::defBaseLevel() 1 1.445e-05 1.445e-05 1.445e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.389e-05 1.389e-05 1.389e-05 0.00% Castro::apply_source_to_state() 20 1.214e-05 1.214e-05 1.214e-05 0.00% Castro::check_for_nan() 20 1.05e-05 1.05e-05 1.05e-05 0.00% Castro::construct_old_gravity() 10 1.002e-05 1.002e-05 1.002e-05 0.00% Amr::initSubcycle() 1 9.339e-06 9.339e-06 9.339e-06 0.00% Gravity::swapTimeLevels() 10 8.706e-06 8.706e-06 8.706e-06 0.00% MLPoisson::prepareForSolve() 11 7.9e-06 7.9e-06 7.9e-06 0.00% Castro::computeNewDt() 9 6.392e-06 6.392e-06 6.392e-06 0.00% MLMG::computeMLResidual() 11 6.301e-06 6.301e-06 6.301e-06 0.00% Amr::InitializeInit() 1 6.247e-06 6.247e-06 6.247e-06 0.00% MLMG::getGradSolution() 11 5.663e-06 5.663e-06 5.663e-06 0.00% MLMG::MLResNormInf() 11 4.923e-06 4.923e-06 4.923e-06 0.00% Castro::create_source_corrector() 10 4.923e-06 4.923e-06 4.923e-06 0.00% AmrLevel::checkPointPost() 3 4.735e-06 4.735e-06 4.735e-06 0.00% MLMG::buildFineMask() 11 4.698e-06 4.698e-06 4.698e-06 0.00% Gravity::set_mass_offset() 11 4.696e-06 4.696e-06 4.696e-06 0.00% Castro::FluxRegFineAdd() 10 4.349e-06 4.349e-06 4.349e-06 0.00% Castro::retry_advance_ctu() 10 4.255e-06 4.255e-06 4.255e-06 0.00% Castro::post_init() 1 4.101e-06 4.101e-06 4.101e-06 0.00% Castro::FluxRegCrseInit 10 3.084e-06 3.084e-06 3.084e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.943e-06 2.943e-06 2.943e-06 0.00% Amr::init() 1 2.924e-06 2.924e-06 2.924e-06 0.00% Castro::computeInitialDt() 2 2.83e-06 2.83e-06 2.83e-06 0.00% AmrLevel::checkPointPre() 3 2.116e-06 2.116e-06 2.116e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.035e-06 2.035e-06 2.035e-06 0.00% Castro::post_regrid() 1 1.349e-06 1.349e-06 1.349e-06 0.00% Amr::initialInit() 1 1.074e-06 1.074e-06 1.074e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7832 0.7832 0.7832 100.00% Amr::coarseTimeStep() 10 0.6421 0.6421 0.6421 81.98% Amr::timeStep() 10 0.5606 0.5606 0.5606 71.57% Castro::advance() 10 0.5497 0.5497 0.5497 70.18% Castro::subcycle_advance_ctu() 10 0.5362 0.5362 0.5362 68.46% Castro::do_advance_ctu() 10 0.5355 0.5355 0.5355 68.36% Gravity::solve_phi_with_mlmg() 11 0.297 0.297 0.297 37.92% Gravity::actual_solve_with_mlmg() 11 0.2883 0.2883 0.2883 36.81% Castro::construct_new_gravity() 10 0.2691 0.2691 0.2691 34.36% MLMG::solve() 11 0.2671 0.2671 0.2671 34.10% Gravity::solve_for_phi() 10 0.2542 0.2542 0.2542 32.45% MLMG::oneIter() 82 0.253 0.253 0.253 32.30% MLMG::mgVcycle() 82 0.2513 0.2513 0.2513 32.08% Castro::construct_ctu_hydro_source() 10 0.1855 0.1855 0.1855 23.68% VisMF::Write(FabArray) 11 0.1604 0.1604 0.1604 20.48% MLCellLinOp::smooth() 1640 0.1292 0.1292 0.1292 16.50% Amr::checkPoint() 3 0.1183 0.1183 0.1183 15.11% Amr::init() 1 0.1158 0.1158 0.1158 14.79% AmrLevel::checkPoint() 3 0.1142 0.1142 0.1142 14.58% StateData::checkPoint() 12 0.1141 0.1141 0.1141 14.57% MLCellLinOp::applyBC() 4433 0.09095 0.09095 0.09095 11.61% MLMG::mgVcycle_bottom 82 0.07619 0.07619 0.07619 9.73% MLMG::actualBottomSolve() 82 0.07615 0.07615 0.07615 9.72% MLCGSolver::bicgstab 82 0.07538 0.07538 0.07538 9.62% MLPoisson::Fsmooth() 3280 0.06072 0.06072 0.06072 7.75% Castro::clean_state() 62 0.05563 0.05563 0.05563 7.10% Amr::initialInit() 1 0.05065 0.05065 0.05065 6.47% Amr::writePlotFile() 2 0.04879 0.04879 0.04879 6.23% Amr::FinalizeInit() 1 0.0466 0.0466 0.0466 5.95% Castro::post_init() 1 0.04519 0.04519 0.04519 5.77% Gravity::multilevel_solve_for_new_phi() 1 0.04325 0.04325 0.04325 5.52% Gravity::actual_multilevel_solve() 1 0.04324 0.04324 0.04324 5.52% FillPatchIterator::Initialize 41 0.04087 0.04087 0.04087 5.22% FillPatchSingleLevel 41 0.03928 0.03928 0.03928 5.01% StateDataPhysBCFunct::() 41 0.03527 0.03527 0.03527 4.50% MLMG::mgVcycle_down::0 82 0.03498 0.03498 0.03498 4.47% MLCellLinOp::apply() 1142 0.0344 0.0344 0.0344 4.39% MLMG::mgVcycle_up::0 82 0.02982 0.02982 0.02982 3.81% StateData::FillBoundary(geom) 328 0.02384 0.02384 0.02384 3.04% Castro::computeTemp() 63 0.02154 0.02154 0.02154 2.75% MultiFab::Dot() 1114 0.02146 0.02146 0.02146 2.74% MLCellLinOp::correctionResidual() 492 0.01997 0.01997 0.01997 2.55% Castro::normalize_species() 62 0.01965 0.01965 0.01965 2.51% Castro::initialize_do_advance() 10 0.01957 0.01957 0.01957 2.50% MLMG:computeResOfCorrection() 410 0.01731 0.01731 0.01731 2.21% MLPoisson::define() 11 0.01703 0.01703 0.01703 2.17% Gravity::get_new_grav_vector() 11 0.0166 0.0166 0.0166 2.12% MLMG::mgVcycle_down::1 82 0.01657 0.01657 0.01657 2.12% FabArray::FillBoundary() 4023 0.01609 0.01609 0.01609 2.05% MLMG::mgVcycle_down::2 82 0.01604 0.01604 0.01604 2.05% FillBoundary_nowait() 4023 0.01527 0.01527 0.01527 1.95% MLMG::mgVcycle_down::3 82 0.01519 0.01519 0.01519 1.94% Castro::construct_old_gravity() 10 0.0147 0.0147 0.0147 1.88% Gravity::get_old_grav_vector() 10 0.01469 0.01469 0.01469 1.88% FabArray::ParallelCopy() 861 0.01461 0.01461 0.01461 1.87% MLMG::mgVcycle_down::4 82 0.01439 0.01439 0.01439 1.84% FabArray::ParallelCopy_nowait() 861 0.01433 0.01433 0.01433 1.83% FabArray::setVal() 1144 0.01345 0.01345 0.01345 1.72% Castro::initialize_advance() 10 0.01328 0.01328 0.01328 1.70% CGSolver::sxay() 1586 0.01317 0.01317 0.01317 1.68% MLCGSolver::ParallelAllReduce 1514 0.01287 0.01287 0.01287 1.64% MultiFab::LinComb() 1586 0.01275 0.01275 0.01275 1.63% MLMG::mgVcycle_up::2 82 0.0124 0.0124 0.0124 1.58% MLMG::mgVcycle_up::1 82 0.01218 0.01218 0.01218 1.56% Castro::do_new_sources() 10 0.01217 0.01217 0.01217 1.55% MLMG::addInterpCorrection() 410 0.0121 0.0121 0.0121 1.55% Castro::enforce_min_density() 62 0.01178 0.01178 0.01178 1.50% MLMG::mgVcycle_up::3 82 0.01167 0.01167 0.01167 1.49% amrex::average_down 410 0.01165 0.01165 0.01165 1.49% MLCellLinOp::defineAuxData() 11 0.01156 0.01156 0.01156 1.48% MLMG::mgVcycle_up::4 82 0.0115 0.0115 0.0115 1.47% Castro::expand_state() 10 0.0111 0.0111 0.0111 1.42% Castro::do_old_sources() 10 0.01094 0.01094 0.01094 1.40% MLPoisson::Fapply() 1142 0.01082 0.01082 0.01082 1.38% Castro::post_timestep() 10 0.01074 0.01074 0.01074 1.37% Gravity::fill_multipole_BCs() 11 0.008439 0.008439 0.008439 1.08% Castro::estTimeStep() 21 0.007214 0.007214 0.007214 0.92% MLCellLinOp::solutionResidual() 93 0.007131 0.007131 0.007131 0.91% Castro::reset_internal_energy(MultiFab) 63 0.006473 0.006473 0.006473 0.83% MultiFab::Xpay() 585 0.006198 0.006198 0.006198 0.79% MLMG::computeResidual() 82 0.006145 0.006145 0.006145 0.78% MLCellLinOp::defineBC() 11 0.004991 0.004991 0.004991 0.64% BndryData::define() 11 0.004804 0.004804 0.004804 0.61% MLMG::prepareForSolve() 11 0.004627 0.004627 0.004627 0.59% Amr::InitializeInit() 1 0.004046 0.004046 0.004046 0.52% Amr::defBaseLevel() 1 0.004039 0.004039 0.004039 0.52% Castro::initData() 1 0.003527 0.003527 0.003527 0.45% Castro::computeNewDt() 9 0.003482 0.003482 0.003482 0.44% Castro::enforce_speed_limit() 62 0.003291 0.003291 0.003291 0.42% Castro::construct_new_source() 50 0.002634 0.002634 0.002634 0.34% Castro::construct_new_gravity_source() 10 0.002593 0.002593 0.002593 0.33% MLMG::ResNormInf() 93 0.00199 0.00199 0.00199 0.25% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.23% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.23% Castro::construct_old_source() 50 0.001747 0.001747 0.001747 0.22% Castro::construct_old_gravity_source() 10 0.001728 0.001728 0.001728 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001699 0.001699 0.001699 0.22% Castro::reset_internal_energy(Fab) 504 0.001574 0.001574 0.001574 0.20% FabArrayBase::getCPC() 1323 0.001524 0.001524 0.001524 0.19% MLMG::getGradSolution() 11 0.001387 0.001387 0.001387 0.18% MLCellLinOp::compGrad() 11 0.001381 0.001381 0.001381 0.18% MLCellLinOp::setLevelBC() 11 0.001357 0.001357 0.001357 0.17% FabArray::mult() 43 0.001341 0.001341 0.001341 0.17% FabArray::setDomainBndry() 41 0.001329 0.001329 0.001329 0.17% Castro::check_for_nan() 20 0.001189 0.001189 0.001189 0.15% MultiFab::contains_nan() 20 0.001178 0.001178 0.001178 0.15% Castro::post_regrid() 1 0.001149 0.001149 0.001149 0.15% MLPoisson::prepareForSolve() 11 0.001124 0.001124 0.001124 0.14% MLCellLinOp::prepareForSolve() 11 0.001116 0.001116 0.001116 0.14% MLMG::computeMLResidual() 11 0.001027 0.001027 0.001027 0.13% Gravity::update_max_rhs() 11 0.0008283 0.0008283 0.0008283 0.11% FabArrayBase::getFB() 4023 0.0008059 0.0008059 0.0008059 0.10% FabArrayBase::CPC::define() 454 0.0006906 0.0006906 0.0006906 0.09% Castro::computeInitialDt() 2 0.0006798 0.0006798 0.0006798 0.09% Amr::InitAmr() 1 0.0004646 0.0004646 0.0004646 0.06% MLLinOp::define() 11 0.0004594 0.0004594 0.0004594 0.06% Gravity::swapTimeLevels() 10 0.0004479 0.0004479 0.0004479 0.06% Castro::Castro() 1 0.0004378 0.0004378 0.0004378 0.06% MLLinOp::defineGrids() 11 0.0004334 0.0004334 0.0004334 0.06% MultiFab::Copy() 11 0.000271 0.000271 0.000271 0.03% MultiFab::max() 11 0.0002599 0.0002599 0.0002599 0.03% MLMG::MLResNormInf() 11 0.0002569 0.0002569 0.0002569 0.03% MLMG::MLRhsNormInf() 11 0.0002023 0.0002023 0.0002023 0.03% Castro::buildMetrics() 1 0.0001601 0.0001601 0.0001601 0.02% FabArrayBase::FB::FB() 56 9.177e-05 9.177e-05 9.177e-05 0.01% Castro::finalize_advance() 10 7.832e-05 7.832e-05 7.832e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.591e-05 5.591e-05 5.591e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.351e-05 5.351e-05 5.351e-05 0.01% StateData::define() 4 4.094e-05 4.094e-05 4.094e-05 0.01% makeSFC 55 3.962e-05 3.962e-05 3.962e-05 0.01% Castro::swap_state_time_levels() 10 3.666e-05 3.666e-05 3.666e-05 0.00% Castro::finalize_do_advance() 10 3.627e-05 3.627e-05 3.627e-05 0.00% Castro::enforce_consistent_e() 1 3.479e-05 3.479e-05 3.479e-05 0.00% Castro::initMFs() 1 2.559e-05 2.559e-05 2.559e-05 0.00% Amr::writeSmallPlotFile() 1 2.425e-05 2.425e-05 2.425e-05 0.00% DistributionMapping::Distribute() 56 1.505e-05 1.505e-05 1.505e-05 0.00% Amr::initSubcycle() 1 9.339e-06 9.339e-06 9.339e-06 0.00% Castro::create_source_corrector() 10 4.923e-06 4.923e-06 4.923e-06 0.00% AmrLevel::checkPointPost() 3 4.735e-06 4.735e-06 4.735e-06 0.00% MLMG::buildFineMask() 11 4.698e-06 4.698e-06 4.698e-06 0.00% Gravity::set_mass_offset() 11 4.696e-06 4.696e-06 4.696e-06 0.00% Castro::FluxRegFineAdd() 10 4.349e-06 4.349e-06 4.349e-06 0.00% Castro::retry_advance_ctu() 10 4.255e-06 4.255e-06 4.255e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.133e-06 4.133e-06 4.133e-06 0.00% Castro::FluxRegCrseInit 10 3.084e-06 3.084e-06 3.084e-06 0.00% AmrLevel::checkPointPre() 3 2.116e-06 2.116e-06 2.116e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.035e-06 2.035e-06 2.035e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-4-gef52970b80e3) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-4-gef52970b80e3) initialized Starting run at 08:26:20 UTC on 2022-05-06. Successfully read inputs file ... Castro git describe: 22.05-10-g5caf4517f AMReX git describe: 22.05-4-gef52970b8 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.439985648 Restart time = 0.045324608 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049800742 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048600757 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.052043175 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060790647 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065571218 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.059282347 seconds Ending run at 08:26:21 UTC on 2022-05-06. Run time = 0.382358527 Run time without initialization = 0.336434437 Average number of zones advanced per microsecond: 3.896 Average number of zones advanced per microsecond per rank: 3.896 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3824 ... 0.3824 ... 0.3824 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0872 0.0872 0.0872 22.82% VisMF::Write(FabArray) 1 0.05794 0.05794 0.05794 15.15% VisMF::Read() 3 0.03826 0.03826 0.03826 10.01% MLCellLinOp::applyBC() 1946 0.03213 0.03213 0.03213 8.40% MLPoisson::Fsmooth() 1440 0.02619 0.02619 0.02619 6.85% StateData::FillBoundary(geom) 160 0.01186 0.01186 0.01186 3.10% MLCGSolver::bicgstab 36 0.009405 0.009405 0.009405 2.46% MultiFab::Dot() 484 0.009128 0.009128 0.009128 2.39% Castro::normalize_species() 30 0.00889 0.00889 0.00889 2.32% Castro::computeTemp() 30 0.007876 0.007876 0.007876 2.06% FabArray::setVal() 537 0.006383 0.006383 0.006383 1.67% FillBoundary_nowait() 1766 0.006265 0.006265 0.006265 1.64% FabArray::ParallelCopy_nowait() 380 0.006072 0.006072 0.006072 1.59% MultiFab::LinComb() 690 0.005566 0.005566 0.005566 1.46% MLCellLinOp::defineAuxData() 6 0.005428 0.005428 0.005428 1.42% StateDataPhysBCFunct::() 20 0.005246 0.005246 0.005246 1.37% Gravity::fill_multipole_BCs() 6 0.004701 0.004701 0.004701 1.23% MLPoisson::Fapply() 500 0.004696 0.004696 0.004696 1.23% Castro::enforce_min_density() 30 0.004672 0.004672 0.004672 1.22% Amr::restart() 1 0.002945 0.002945 0.002945 0.77% MLMG::addInterpCorrection() 180 0.002919 0.002919 0.002919 0.76% Castro::estTimeStep() 10 0.002809 0.002809 0.002809 0.73% amrex::average_down 180 0.002724 0.002724 0.002724 0.71% MultiFab::Xpay() 258 0.002688 0.002688 0.002688 0.70% Castro::do_advance_ctu() 5 0.002356 0.002356 0.002356 0.62% Castro::reset_internal_energy(MultiFab) 30 0.002115 0.002115 0.002115 0.55% BndryData::define() 6 0.002094 0.002094 0.002094 0.55% Amr::writePlotFile() 1 0.001424 0.001424 0.001424 0.37% Castro::construct_new_gravity_source() 5 0.001225 0.001225 0.001225 0.32% Castro::subcycle_advance_ctu() 5 0.001199 0.001199 0.001199 0.31% MultiFab::Saxpy() 10 0.0009217 0.0009217 0.0009217 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009153 0.0009153 0.0009153 0.24% Gravity::get_old_grav_vector() 5 0.0008822 0.0008822 0.0008822 0.23% Castro::expand_state() 5 0.0008738 0.0008738 0.0008738 0.23% MLMG::ResNormInf() 42 0.0008707 0.0008707 0.0008707 0.23% Gravity::get_new_grav_vector() 5 0.0008644 0.0008644 0.0008644 0.23% Castro::construct_old_gravity_source() 5 0.0008465 0.0008465 0.0008465 0.22% Castro::reset_internal_energy(Fab) 240 0.0007943 0.0007943 0.0007943 0.21% MLMG::oneIter() 36 0.0007508 0.0007508 0.0007508 0.20% MLCellLinOp::setLevelBC() 6 0.000733 0.000733 0.000733 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007319 0.0007319 0.0007319 0.19% Castro::enforce_speed_limit() 30 0.0007018 0.0007018 0.0007018 0.18% FabArray::mult() 22 0.0006549 0.0006549 0.0006549 0.17% FabArray::setDomainBndry() 20 0.0006485 0.0006485 0.0006485 0.17% MLCellLinOp::prepareForSolve() 6 0.0006057 0.0006057 0.0006057 0.16% MultiFab::contains_nan() 10 0.0005825 0.0005825 0.0005825 0.15% MLCellLinOp::smooth() 720 0.0004904 0.0004904 0.0004904 0.13% MLCellLinOp::compGrad() 6 0.0004882 0.0004882 0.0004882 0.13% MLMG::prepareForSolve() 6 0.000425 0.000425 0.000425 0.11% FabArrayBase::CPC::define() 244 0.0004103 0.0004103 0.0004103 0.11% Amr::InitAmr() 1 0.0003986 0.0003986 0.0003986 0.10% FabArrayBase::getCPC() 632 0.0003911 0.0003911 0.0003911 0.10% FabArray::FillBoundary() 1766 0.0003702 0.0003702 0.0003702 0.10% main() 1 0.0003003 0.0003003 0.0003003 0.08% FabArrayBase::getFB() 1766 0.0002927 0.0002927 0.0002927 0.08% MLCellLinOp::apply() 500 0.0002292 0.0002292 0.0002292 0.06% Gravity::update_max_rhs() 6 0.0002246 0.0002246 0.0002246 0.06% CGSolver::sxay() 690 0.0002143 0.0002143 0.0002143 0.06% Gravity::solve_for_phi() 5 0.0002059 0.0002059 0.0002059 0.05% MLLinOp::defineGrids() 6 0.0001998 0.0001998 0.0001998 0.05% MLMG::mgVcycle() 36 0.0001647 0.0001647 0.0001647 0.04% MLCGSolver::ParallelAllReduce 659 0.000147 0.000147 0.000147 0.04% MultiFab::Copy() 6 0.000147 0.000147 0.000147 0.04% MultiFab::max() 6 0.0001345 0.0001345 0.0001345 0.04% FabArray::ParallelCopy() 380 0.0001331 0.0001331 0.0001331 0.03% FillPatchIterator::Initialize 20 0.0001219 0.0001219 0.0001219 0.03% Amr::coarseTimeStep() 5 0.0001157 0.0001157 0.0001157 0.03% MLMG::MLRhsNormInf() 6 0.0001077 0.0001077 0.0001077 0.03% Castro::construct_new_gravity() 5 0.0001054 0.0001054 0.0001054 0.03% MLCellLinOp::correctionResidual() 216 0.0001026 0.0001026 0.0001026 0.03% MLCellLinOp::defineBC() 6 0.0001024 0.0001024 0.0001024 0.03% Castro::advance() 5 9.811e-05 9.811e-05 9.811e-05 0.03% Amr::timeStep() 5 8.684e-05 8.684e-05 8.684e-05 0.02% AmrLevel::restart() 1 7.384e-05 7.384e-05 7.384e-05 0.02% StateData::restartDoit() 4 7.061e-05 7.061e-05 7.061e-05 0.02% FabArrayBase::FB::FB() 26 6.134e-05 6.134e-05 6.134e-05 0.02% MLMG:computeResOfCorrection() 180 5.989e-05 5.989e-05 5.989e-05 0.02% MLMG::actualBottomSolve() 36 4.793e-05 4.793e-05 4.793e-05 0.01% Castro::finalize_advance() 5 4.388e-05 4.388e-05 4.388e-05 0.01% Castro::create_source_corrector() 5 4.257e-05 4.257e-05 4.257e-05 0.01% Castro::initialize_do_advance() 5 4.233e-05 4.233e-05 4.233e-05 0.01% MLMG::mgVcycle_down::0 36 3.849e-05 3.849e-05 3.849e-05 0.01% Castro::clean_state() 30 3.618e-05 3.618e-05 3.618e-05 0.01% Castro::construct_old_source() 25 3.618e-05 3.618e-05 3.618e-05 0.01% MLMG::mgVcycle_down::1 36 3.616e-05 3.616e-05 3.616e-05 0.01% Castro::initialize_advance() 5 3.493e-05 3.493e-05 3.493e-05 0.01% MLMG::mgVcycle_down::2 36 3.464e-05 3.464e-05 3.464e-05 0.01% MLMG::mgVcycle_down::4 36 3.379e-05 3.379e-05 3.379e-05 0.01% MLMG::solve() 6 3.374e-05 3.374e-05 3.374e-05 0.01% MLMG::mgVcycle_down::3 36 3.285e-05 3.285e-05 3.285e-05 0.01% Gravity::actual_multilevel_solve() 1 3.025e-05 3.025e-05 3.025e-05 0.01% Castro::buildMetrics() 1 2.914e-05 2.914e-05 2.914e-05 0.01% MLMG::mgVcycle_up::4 36 2.886e-05 2.886e-05 2.886e-05 0.01% Castro::post_restart() 1 2.793e-05 2.793e-05 2.793e-05 0.01% Castro::construct_new_source() 25 2.738e-05 2.738e-05 2.738e-05 0.01% Amr::writeSmallPlotFile() 1 2.543e-05 2.543e-05 2.543e-05 0.01% Castro::initMFs() 1 2.542e-05 2.542e-05 2.542e-05 0.01% Castro::swap_state_time_levels() 5 2.45e-05 2.45e-05 2.45e-05 0.01% MLMG::mgVcycle_up::3 36 2.255e-05 2.255e-05 2.255e-05 0.01% MLMG::mgVcycle_up::0 36 2.252e-05 2.252e-05 2.252e-05 0.01% MLMG::mgVcycle_up::2 36 2.211e-05 2.211e-05 2.211e-05 0.01% MLLinOp::define() 6 2.082e-05 2.082e-05 2.082e-05 0.01% MLCellLinOp::solutionResidual() 42 2.07e-05 2.07e-05 2.07e-05 0.01% MLMG::mgVcycle_up::1 36 2.066e-05 2.066e-05 2.066e-05 0.01% Castro::finalize_do_advance() 5 1.933e-05 1.933e-05 1.933e-05 0.01% MLMG::computeResidual() 36 1.618e-05 1.618e-05 1.618e-05 0.00% MLMG::mgVcycle_bottom 36 1.617e-05 1.617e-05 1.617e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.605e-05 1.605e-05 1.605e-05 0.00% makeSFC 30 1.379e-05 1.379e-05 1.379e-05 0.00% MLPoisson::define() 6 1.376e-05 1.376e-05 1.376e-05 0.00% FillPatchSingleLevel 20 1.362e-05 1.362e-05 1.362e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.308e-05 1.308e-05 1.308e-05 0.00% Amr::initSubcycle() 1 1.017e-05 1.017e-05 1.017e-05 0.00% Castro::do_new_sources() 5 9.158e-06 9.158e-06 9.158e-06 0.00% DistributionMapping::Distribute() 31 8.08e-06 8.08e-06 8.08e-06 0.00% Castro::do_old_sources() 5 8.046e-06 8.046e-06 8.046e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.463e-06 7.463e-06 7.463e-06 0.00% Castro::apply_source_to_state() 10 5.795e-06 5.795e-06 5.795e-06 0.00% Castro::construct_old_gravity() 5 5.713e-06 5.713e-06 5.713e-06 0.00% Castro::check_for_nan() 10 5.601e-06 5.601e-06 5.601e-06 0.00% Castro::post_timestep() 5 4.695e-06 4.695e-06 4.695e-06 0.00% MLPoisson::prepareForSolve() 6 4.3e-06 4.3e-06 4.3e-06 0.00% Gravity::swapTimeLevels() 5 4.184e-06 4.184e-06 4.184e-06 0.00% Castro::computeNewDt() 5 3.126e-06 3.126e-06 3.126e-06 0.00% MLMG::computeMLResidual() 6 3.03e-06 3.03e-06 3.03e-06 0.00% MLMG::getGradSolution() 6 2.96e-06 2.96e-06 2.96e-06 0.00% MLMG::buildFineMask() 6 2.88e-06 2.88e-06 2.88e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.866e-06 2.866e-06 2.866e-06 0.00% MLMG::MLResNormInf() 6 2.768e-06 2.768e-06 2.768e-06 0.00% Castro::retry_advance_ctu() 5 2.121e-06 2.121e-06 2.121e-06 0.00% Gravity::set_mass_offset() 6 2.006e-06 2.006e-06 2.006e-06 0.00% Castro::FluxRegCrseInit 5 1.717e-06 1.717e-06 1.717e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.33e-06 1.33e-06 1.33e-06 0.00% Castro::FluxRegFineAdd() 5 1.289e-06 1.289e-06 1.289e-06 0.00% Amr::init() 1 9.83e-07 9.83e-07 9.83e-07 0.00% AmrLevel::AmrLevel() 1 9.17e-07 9.17e-07 9.17e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3824 0.3824 0.3824 100.00% Amr::coarseTimeStep() 5 0.2769 0.2769 0.2769 72.42% Amr::timeStep() 5 0.2752 0.2752 0.2752 71.96% Castro::advance() 5 0.271 0.271 0.271 70.86% Castro::subcycle_advance_ctu() 5 0.264 0.264 0.264 69.03% Castro::do_advance_ctu() 5 0.2628 0.2628 0.2628 68.72% Castro::construct_new_gravity() 5 0.1361 0.1361 0.1361 35.60% Gravity::solve_phi_with_mlmg() 6 0.1318 0.1318 0.1318 34.46% Gravity::solve_for_phi() 5 0.1284 0.1284 0.1284 33.57% Gravity::actual_solve_with_mlmg() 6 0.1269 0.1269 0.1269 33.20% MLMG::solve() 6 0.1155 0.1155 0.1155 30.21% MLMG::oneIter() 36 0.1088 0.1088 0.1088 28.46% MLMG::mgVcycle() 36 0.1081 0.1081 0.1081 28.26% Castro::construct_ctu_hydro_source() 5 0.08725 0.08725 0.08725 22.82% Amr::writePlotFile() 1 0.05936 0.05936 0.05936 15.52% VisMF::Write(FabArray) 1 0.05794 0.05794 0.05794 15.15% MLCellLinOp::smooth() 720 0.05556 0.05556 0.05556 14.53% Amr::init() 1 0.04536 0.04536 0.04536 11.86% Amr::restart() 1 0.04536 0.04536 0.04536 11.86% MLCellLinOp::applyBC() 1946 0.03912 0.03912 0.03912 10.23% AmrLevel::restart() 1 0.03847 0.03847 0.03847 10.06% StateData::restartDoit() 4 0.03839 0.03839 0.03839 10.04% VisMF::Read() 3 0.03826 0.03826 0.03826 10.01% MLMG::mgVcycle_bottom 36 0.03267 0.03267 0.03267 8.54% MLMG::actualBottomSolve() 36 0.03266 0.03266 0.03266 8.54% MLCGSolver::bicgstab 36 0.03233 0.03233 0.03233 8.46% MLPoisson::Fsmooth() 1440 0.02619 0.02619 0.02619 6.85% Castro::clean_state() 30 0.02509 0.02509 0.02509 6.56% FillPatchIterator::Initialize 20 0.01989 0.01989 0.01989 5.20% FillPatchSingleLevel 20 0.01912 0.01912 0.01912 5.00% StateDataPhysBCFunct::() 20 0.0171 0.0171 0.0171 4.47% MLMG::mgVcycle_down::0 36 0.01509 0.01509 0.01509 3.95% MLCellLinOp::apply() 500 0.01491 0.01491 0.01491 3.90% MLMG::mgVcycle_up::0 36 0.01283 0.01283 0.01283 3.36% StateData::FillBoundary(geom) 160 0.01186 0.01186 0.01186 3.10% Castro::computeTemp() 30 0.01079 0.01079 0.01079 2.82% Castro::initialize_do_advance() 5 0.009719 0.009719 0.009719 2.54% MLPoisson::define() 6 0.009203 0.009203 0.009203 2.41% MultiFab::Dot() 484 0.009128 0.009128 0.009128 2.39% Castro::normalize_species() 30 0.00889 0.00889 0.00889 2.32% MLCellLinOp::correctionResidual() 216 0.008654 0.008654 0.008654 2.26% Gravity::get_new_grav_vector() 5 0.007644 0.007644 0.007644 2.00% MLMG:computeResOfCorrection() 180 0.007495 0.007495 0.007495 1.96% Castro::construct_old_gravity() 5 0.007312 0.007312 0.007312 1.91% Gravity::get_old_grav_vector() 5 0.007306 0.007306 0.007306 1.91% MLMG::mgVcycle_down::1 36 0.007133 0.007133 0.007133 1.87% FabArray::FillBoundary() 1766 0.006989 0.006989 0.006989 1.83% MLMG::mgVcycle_down::2 36 0.006892 0.006892 0.006892 1.80% Castro::initialize_advance() 5 0.006848 0.006848 0.006848 1.79% FillBoundary_nowait() 1766 0.006619 0.006619 0.006619 1.73% FabArray::ParallelCopy() 380 0.006606 0.006606 0.006606 1.73% MLMG::mgVcycle_down::3 36 0.006524 0.006524 0.006524 1.71% FabArray::ParallelCopy_nowait() 380 0.006473 0.006473 0.006473 1.69% FabArray::setVal() 537 0.006383 0.006383 0.006383 1.67% MLMG::mgVcycle_down::4 36 0.006214 0.006214 0.006214 1.63% MLCellLinOp::defineAuxData() 6 0.00619 0.00619 0.00619 1.62% CGSolver::sxay() 690 0.00578 0.00578 0.00578 1.51% Castro::do_new_sources() 5 0.005764 0.005764 0.005764 1.51% MultiFab::LinComb() 690 0.005566 0.005566 0.005566 1.46% Castro::expand_state() 5 0.005551 0.005551 0.005551 1.45% MLCGSolver::ParallelAllReduce 659 0.005503 0.005503 0.005503 1.44% MLMG::mgVcycle_up::2 36 0.005329 0.005329 0.005329 1.39% MLMG::mgVcycle_up::1 36 0.005238 0.005238 0.005238 1.37% MLMG::addInterpCorrection() 180 0.005204 0.005204 0.005204 1.36% amrex::average_down 180 0.005039 0.005039 0.005039 1.32% MLMG::mgVcycle_up::3 36 0.005028 0.005028 0.005028 1.32% MLMG::mgVcycle_up::4 36 0.004952 0.004952 0.004952 1.30% Gravity::fill_multipole_BCs() 6 0.004701 0.004701 0.004701 1.23% MLPoisson::Fapply() 500 0.004696 0.004696 0.004696 1.23% Castro::enforce_min_density() 30 0.004672 0.004672 0.004672 1.22% Castro::do_old_sources() 5 0.004642 0.004642 0.004642 1.21% Castro::post_timestep() 5 0.004125 0.004125 0.004125 1.08% Castro::post_restart() 1 0.003754 0.003754 0.003754 0.98% Gravity::multilevel_solve_for_new_phi() 1 0.003635 0.003635 0.003635 0.95% Gravity::actual_multilevel_solve() 1 0.003619 0.003619 0.003619 0.95% MLCellLinOp::solutionResidual() 42 0.003177 0.003177 0.003177 0.83% Castro::reset_internal_energy(MultiFab) 30 0.002909 0.002909 0.002909 0.76% Castro::estTimeStep() 10 0.002809 0.002809 0.002809 0.73% MLCellLinOp::defineBC() 6 0.00275 0.00275 0.00275 0.72% MultiFab::Xpay() 258 0.002688 0.002688 0.002688 0.70% BndryData::define() 6 0.002648 0.002648 0.002648 0.69% MLMG::computeResidual() 36 0.002637 0.002637 0.002637 0.69% MLMG::prepareForSolve() 6 0.002494 0.002494 0.002494 0.65% Castro::computeNewDt() 5 0.00162 0.00162 0.00162 0.42% Castro::construct_new_source() 25 0.001253 0.001253 0.001253 0.33% Castro::construct_new_gravity_source() 5 0.001225 0.001225 0.001225 0.32% Castro::apply_source_to_state() 10 0.0009275 0.0009275 0.0009275 0.24% MultiFab::Saxpy() 10 0.0009217 0.0009217 0.0009217 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009153 0.0009153 0.0009153 0.24% Castro::construct_old_source() 25 0.0008827 0.0008827 0.0008827 0.23% MLMG::ResNormInf() 42 0.0008707 0.0008707 0.0008707 0.23% Castro::construct_old_gravity_source() 5 0.0008465 0.0008465 0.0008465 0.22% FabArrayBase::getCPC() 632 0.0008014 0.0008014 0.0008014 0.21% Castro::reset_internal_energy(Fab) 240 0.0007943 0.0007943 0.0007943 0.21% MLMG::getGradSolution() 6 0.0007464 0.0007464 0.0007464 0.20% MLCellLinOp::compGrad() 6 0.0007434 0.0007434 0.0007434 0.19% MLCellLinOp::setLevelBC() 6 0.000733 0.000733 0.000733 0.19% Castro::enforce_speed_limit() 30 0.0007018 0.0007018 0.0007018 0.18% FabArray::mult() 22 0.0006549 0.0006549 0.0006549 0.17% FabArray::setDomainBndry() 20 0.0006485 0.0006485 0.0006485 0.17% MLPoisson::prepareForSolve() 6 0.00061 0.00061 0.00061 0.16% MLCellLinOp::prepareForSolve() 6 0.0006057 0.0006057 0.0006057 0.16% Castro::check_for_nan() 10 0.0005881 0.0005881 0.0005881 0.15% MultiFab::contains_nan() 10 0.0005825 0.0005825 0.0005825 0.15% MLMG::computeMLResidual() 6 0.0005591 0.0005591 0.0005591 0.15% Gravity::update_max_rhs() 6 0.0004379 0.0004379 0.0004379 0.11% FabArrayBase::CPC::define() 244 0.0004103 0.0004103 0.0004103 0.11% Amr::InitAmr() 1 0.0004087 0.0004087 0.0004087 0.11% FabArrayBase::getFB() 1766 0.000354 0.000354 0.000354 0.09% MLLinOp::define() 6 0.0002501 0.0002501 0.0002501 0.07% Gravity::swapTimeLevels() 5 0.0002298 0.0002298 0.0002298 0.06% MLLinOp::defineGrids() 6 0.0002293 0.0002293 0.0002293 0.06% Castro::buildMetrics() 1 0.0001603 0.0001603 0.0001603 0.04% MultiFab::Copy() 6 0.000147 0.000147 0.000147 0.04% MLMG::MLResNormInf() 6 0.0001349 0.0001349 0.0001349 0.04% MultiFab::max() 6 0.0001345 0.0001345 0.0001345 0.04% MLMG::MLRhsNormInf() 6 0.0001077 0.0001077 0.0001077 0.03% FabArrayBase::FB::FB() 26 6.134e-05 6.134e-05 6.134e-05 0.02% Castro::finalize_advance() 5 4.689e-05 4.689e-05 4.689e-05 0.01% Castro::create_source_corrector() 5 4.257e-05 4.257e-05 4.257e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.816e-05 2.816e-05 2.816e-05 0.01% Amr::writeSmallPlotFile() 1 2.543e-05 2.543e-05 2.543e-05 0.01% Castro::initMFs() 1 2.542e-05 2.542e-05 2.542e-05 0.01% Castro::swap_state_time_levels() 5 2.45e-05 2.45e-05 2.45e-05 0.01% makeSFC 30 2.07e-05 2.07e-05 2.07e-05 0.01% Castro::finalize_do_advance() 5 1.933e-05 1.933e-05 1.933e-05 0.01% Amr::initSubcycle() 1 1.017e-05 1.017e-05 1.017e-05 0.00% DistributionMapping::Distribute() 31 8.08e-06 8.08e-06 8.08e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.032e-06 4.032e-06 4.032e-06 0.00% MLMG::buildFineMask() 6 2.88e-06 2.88e-06 2.88e-06 0.00% Castro::retry_advance_ctu() 5 2.121e-06 2.121e-06 2.121e-06 0.00% Gravity::set_mass_offset() 6 2.006e-06 2.006e-06 2.006e-06 0.00% Castro::FluxRegCrseInit 5 1.717e-06 1.717e-06 1.717e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.33e-06 1.33e-06 1.33e-06 0.00% Castro::FluxRegFineAdd() 5 1.289e-06 1.289e-06 1.289e-06 0.00% AmrLevel::AmrLevel() 1 9.17e-07 9.17e-07 9.17e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-4-gef52970b80e3) finalized