Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-4-gef52970b80e3) initialized Starting run at 08:24:51 UTC on 2022-05-05. Successfully read inputs file ... Castro git describe: 22.05-9-gf85dde19e AMReX git describe: 22.05-4-gef52970b8 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.040418606 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022666813 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.051101575 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048069856 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.058508826 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064478692 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.057721385 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.036043557 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.056251697 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.04928863 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05979449 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059138994 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06010823 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.036203991 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022518251 seconds Ending run at 08:24:52 UTC on 2022-05-05. Run time = 0.769518595 Run time without initialization = 0.659724307 Average number of zones advanced per microsecond: 3.974 Average number of zones advanced per microsecond per rank: 3.974 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7695 ... 0.7695 ... 0.7695 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1845 0.1845 0.1845 23.97% VisMF::Write(FabArray) 11 0.1513 0.1513 0.1513 19.66% MLCellLinOp::applyBC() 4379 0.0717 0.0717 0.0717 9.32% MLPoisson::Fsmooth() 3240 0.05892 0.05892 0.05892 7.66% StateData::FillBoundary(geom) 328 0.02285 0.02285 0.02285 2.97% StateDataPhysBCFunct::() 41 0.02121 0.02121 0.02121 2.76% MLCGSolver::bicgstab 81 0.02105 0.02105 0.02105 2.74% MultiFab::Dot() 1100 0.02036 0.02036 0.02036 2.65% Castro::normalize_species() 62 0.01731 0.01731 0.01731 2.25% Castro::computeTemp() 63 0.01408 0.01408 0.01408 1.83% FillBoundary_nowait() 3974 0.01403 0.01403 0.01403 1.82% Castro::enforce_min_density() 62 0.0132 0.0132 0.0132 1.72% FabArray::setVal() 1135 0.01301 0.01301 0.01301 1.69% FabArray::ParallelCopy_nowait() 851 0.01278 0.01278 0.01278 1.66% MultiFab::LinComb() 1566 0.01217 0.01217 0.01217 1.58% MLPoisson::Fapply() 1128 0.0104 0.0104 0.0104 1.35% MLCellLinOp::defineAuxData() 11 0.009619 0.009619 0.009619 1.25% Gravity::fill_multipole_BCs() 11 0.00858 0.00858 0.00858 1.11% Castro::estTimeStep() 21 0.006737 0.006737 0.006737 0.88% MLMG::addInterpCorrection() 405 0.006498 0.006498 0.006498 0.84% amrex::average_down 405 0.00609 0.00609 0.00609 0.79% MultiFab::Xpay() 578 0.005997 0.005997 0.005997 0.78% Castro::reset_internal_energy(MultiFab) 63 0.005003 0.005003 0.005003 0.65% Castro::do_advance_ctu() 10 0.004965 0.004965 0.004965 0.65% Amr::checkPoint() 3 0.004141 0.004141 0.004141 0.54% Castro::enforce_speed_limit() 62 0.004003 0.004003 0.004003 0.52% BndryData::define() 11 0.003565 0.003565 0.003565 0.46% Castro::construct_new_gravity_source() 10 0.003171 0.003171 0.003171 0.41% Castro::construct_old_gravity_source() 10 0.002643 0.002643 0.002643 0.34% Amr::writePlotFile() 2 0.00241 0.00241 0.00241 0.31% MLMG::ResNormInf() 92 0.001948 0.001948 0.001948 0.25% Gravity::get_new_grav_vector() 11 0.001913 0.001913 0.001913 0.25% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.23% Castro::expand_state() 10 0.001728 0.001728 0.001728 0.22% Gravity::get_old_grav_vector() 10 0.001709 0.001709 0.001709 0.22% MLMG::oneIter() 81 0.001678 0.001678 0.001678 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001624 0.001624 0.001624 0.21% Castro::reset_internal_energy(Fab) 504 0.001486 0.001486 0.001486 0.19% Gravity::actual_solve_with_mlmg() 11 0.001446 0.001446 0.001446 0.19% FabArray::mult() 43 0.001336 0.001336 0.001336 0.17% MLCellLinOp::setLevelBC() 11 0.001316 0.001316 0.001316 0.17% FabArray::setDomainBndry() 41 0.001278 0.001278 0.001278 0.17% MLCellLinOp::smooth() 1620 0.001173 0.001173 0.001173 0.15% MultiFab::contains_nan() 20 0.001171 0.001171 0.001171 0.15% Castro::initData() 1 0.00117 0.00117 0.00117 0.15% MLCellLinOp::prepareForSolve() 11 0.001076 0.001076 0.001076 0.14% MLCellLinOp::compGrad() 11 0.000901 0.000901 0.000901 0.12% FabArrayBase::getCPC() 1313 0.000819 0.000819 0.000819 0.11% FabArray::FillBoundary() 3974 0.000813 0.000813 0.000813 0.11% MLMG::prepareForSolve() 11 0.0007639 0.0007639 0.0007639 0.10% FabArrayBase::getFB() 3974 0.0007061 0.0007061 0.0007061 0.09% FabArrayBase::CPC::define() 454 0.0007032 0.0007032 0.0007032 0.09% MLCellLinOp::apply() 1128 0.0004999 0.0004999 0.0004999 0.06% Amr::InitAmr() 1 0.0004682 0.0004682 0.0004682 0.06% CGSolver::sxay() 1566 0.0004131 0.0004131 0.0004131 0.05% Gravity::update_max_rhs() 11 0.0004041 0.0004041 0.0004041 0.05% MLLinOp::defineGrids() 11 0.0003854 0.0003854 0.0003854 0.05% Gravity::solve_for_phi() 10 0.0003725 0.0003725 0.0003725 0.05% MLMG::mgVcycle() 81 0.0003572 0.0003572 0.0003572 0.05% MLCGSolver::ParallelAllReduce 1495 0.0003502 0.0003502 0.0003502 0.05% main() 1 0.0002857 0.0002857 0.0002857 0.04% FabArray::ParallelCopy() 851 0.0002706 0.0002706 0.0002706 0.04% MultiFab::Copy() 11 0.0002675 0.0002675 0.0002675 0.03% FillPatchIterator::Initialize 41 0.0002597 0.0002597 0.0002597 0.03% MultiFab::max() 11 0.0002562 0.0002562 0.0002562 0.03% Castro::subcycle_advance_ctu() 10 0.0002464 0.0002464 0.0002464 0.03% MLCellLinOp::correctionResidual() 486 0.0002331 0.0002331 0.0002331 0.03% Castro::construct_new_gravity() 10 0.0002232 0.0002232 0.0002232 0.03% Amr::coarseTimeStep() 10 0.0002025 0.0002025 0.0002025 0.03% MLMG::MLRhsNormInf() 11 0.0001995 0.0001995 0.0001995 0.03% Amr::timeStep() 10 0.0001987 0.0001987 0.0001987 0.03% MLCellLinOp::defineBC() 11 0.0001854 0.0001854 0.0001854 0.02% StateData::checkPoint() 12 0.0001521 0.0001521 0.0001521 0.02% MLMG:computeResOfCorrection() 405 0.0001377 0.0001377 0.0001377 0.02% Castro::advance() 10 0.0001337 0.0001337 0.0001337 0.02% MLMG::actualBottomSolve() 81 0.0001127 0.0001127 0.0001127 0.01% MLMG::mgVcycle_down::0 81 9.021e-05 9.021e-05 9.021e-05 0.01% Castro::Castro() 1 8.79e-05 8.79e-05 8.79e-05 0.01% FabArrayBase::FB::FB() 56 8.628e-05 8.628e-05 8.628e-05 0.01% MLMG::mgVcycle_down::1 81 8.103e-05 8.103e-05 8.103e-05 0.01% MLMG::solve() 11 7.755e-05 7.755e-05 7.755e-05 0.01% Castro::initialize_advance() 10 7.679e-05 7.679e-05 7.679e-05 0.01% MLMG::mgVcycle_down::2 81 7.594e-05 7.594e-05 7.594e-05 0.01% MLMG::mgVcycle_down::4 81 7.358e-05 7.358e-05 7.358e-05 0.01% Castro::clean_state() 62 7.314e-05 7.314e-05 7.314e-05 0.01% MLMG::mgVcycle_down::3 81 7.222e-05 7.222e-05 7.222e-05 0.01% AmrLevel::checkPoint() 3 7.152e-05 7.152e-05 7.152e-05 0.01% MLMG::mgVcycle_up::4 81 5.951e-05 5.951e-05 5.951e-05 0.01% Castro::finalize_advance() 10 5.896e-05 5.896e-05 5.896e-05 0.01% Castro::initialize_do_advance() 10 5.698e-05 5.698e-05 5.698e-05 0.01% MLCellLinOp::solutionResidual() 92 5.185e-05 5.185e-05 5.185e-05 0.01% MLMG::mgVcycle_up::3 81 4.974e-05 4.974e-05 4.974e-05 0.01% MLMG::mgVcycle_up::0 81 4.792e-05 4.792e-05 4.792e-05 0.01% MLMG::mgVcycle_up::1 81 4.744e-05 4.744e-05 4.744e-05 0.01% MLMG::mgVcycle_up::2 81 4.622e-05 4.622e-05 4.622e-05 0.01% StateData::define() 4 4.501e-05 4.501e-05 4.501e-05 0.01% Castro::swap_state_time_levels() 10 3.748e-05 3.748e-05 3.748e-05 0.00% Castro::finalize_do_advance() 10 3.68e-05 3.68e-05 3.68e-05 0.00% MLMG::mgVcycle_bottom 81 3.525e-05 3.525e-05 3.525e-05 0.00% MLMG::computeResidual() 81 3.478e-05 3.478e-05 3.478e-05 0.00% Castro::enforce_consistent_e() 1 3.409e-05 3.409e-05 3.409e-05 0.00% Gravity::actual_multilevel_solve() 1 3.119e-05 3.119e-05 3.119e-05 0.00% FillPatchSingleLevel 41 2.901e-05 2.901e-05 2.901e-05 0.00% MLLinOp::define() 11 2.618e-05 2.618e-05 2.618e-05 0.00% Castro::initMFs() 1 2.523e-05 2.523e-05 2.523e-05 0.00% Castro::construct_old_source() 50 2.497e-05 2.497e-05 2.497e-05 0.00% makeSFC 55 2.482e-05 2.482e-05 2.482e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.476e-05 2.476e-05 2.476e-05 0.00% Amr::writeSmallPlotFile() 1 2.313e-05 2.313e-05 2.313e-05 0.00% MLPoisson::define() 11 2.307e-05 2.307e-05 2.307e-05 0.00% Castro::do_new_sources() 10 2.263e-05 2.263e-05 2.263e-05 0.00% Amr::FinalizeInit() 1 2.208e-05 2.208e-05 2.208e-05 0.00% Castro::buildMetrics() 1 2.089e-05 2.089e-05 2.089e-05 0.00% Castro::construct_new_source() 50 1.728e-05 1.728e-05 1.728e-05 0.00% Amr::defBaseLevel() 1 1.72e-05 1.72e-05 1.72e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.674e-05 1.674e-05 1.674e-05 0.00% Castro::do_old_sources() 10 1.662e-05 1.662e-05 1.662e-05 0.00% DistributionMapping::Distribute() 56 1.553e-05 1.553e-05 1.553e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.547e-05 1.547e-05 1.547e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.455e-05 1.455e-05 1.455e-05 0.00% Castro::check_for_nan() 20 1.262e-05 1.262e-05 1.262e-05 0.00% Castro::apply_source_to_state() 20 1.232e-05 1.232e-05 1.232e-05 0.00% Castro::construct_old_gravity() 10 9.687e-06 9.687e-06 9.687e-06 0.00% Amr::initSubcycle() 1 9.603e-06 9.603e-06 9.603e-06 0.00% Gravity::swapTimeLevels() 10 8.308e-06 8.308e-06 8.308e-06 0.00% MLPoisson::prepareForSolve() 11 8.137e-06 8.137e-06 8.137e-06 0.00% Castro::post_timestep() 10 7.142e-06 7.142e-06 7.142e-06 0.00% MLMG::computeMLResidual() 11 7.068e-06 7.068e-06 7.068e-06 0.00% Castro::computeNewDt() 9 6.619e-06 6.619e-06 6.619e-06 0.00% Amr::InitializeInit() 1 6.541e-06 6.541e-06 6.541e-06 0.00% MLMG::getGradSolution() 11 5.736e-06 5.736e-06 5.736e-06 0.00% AmrLevel::checkPointPost() 3 5.271e-06 5.271e-06 5.271e-06 0.00% MLMG::MLResNormInf() 11 5.054e-06 5.054e-06 5.054e-06 0.00% MLMG::buildFineMask() 11 5.004e-06 5.004e-06 5.004e-06 0.00% Castro::post_init() 1 4.542e-06 4.542e-06 4.542e-06 0.00% Castro::retry_advance_ctu() 10 4.391e-06 4.391e-06 4.391e-06 0.00% Castro::create_source_corrector() 10 4.174e-06 4.174e-06 4.174e-06 0.00% Gravity::set_mass_offset() 11 4.074e-06 4.074e-06 4.074e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.509e-06 3.509e-06 3.509e-06 0.00% Castro::FluxRegCrseInit 10 3.257e-06 3.257e-06 3.257e-06 0.00% Castro::computeInitialDt() 2 2.837e-06 2.837e-06 2.837e-06 0.00% Amr::init() 1 2.663e-06 2.663e-06 2.663e-06 0.00% Castro::FluxRegFineAdd() 10 2.605e-06 2.605e-06 2.605e-06 0.00% AmrLevel::checkPointPre() 3 2.078e-06 2.078e-06 2.078e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.971e-06 1.971e-06 1.971e-06 0.00% Castro::post_regrid() 1 1.339e-06 1.339e-06 1.339e-06 0.00% Amr::initialInit() 1 1.046e-06 1.046e-06 1.046e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7695 0.7695 0.7695 100.00% Amr::coarseTimeStep() 10 0.637 0.637 0.637 82.78% Amr::timeStep() 10 0.5606 0.5606 0.5606 72.85% Castro::advance() 10 0.5502 0.5502 0.5502 71.49% Castro::subcycle_advance_ctu() 10 0.5371 0.5371 0.5371 69.80% Castro::do_advance_ctu() 10 0.5369 0.5369 0.5369 69.76% Gravity::solve_phi_with_mlmg() 11 0.2858 0.2858 0.2858 37.13% Gravity::actual_solve_with_mlmg() 11 0.277 0.277 0.277 35.99% Castro::construct_new_gravity() 10 0.2712 0.2712 0.2712 35.25% MLMG::solve() 11 0.2567 0.2567 0.2567 33.36% Gravity::solve_for_phi() 10 0.2474 0.2474 0.2474 32.15% MLMG::oneIter() 81 0.243 0.243 0.243 31.58% MLMG::mgVcycle() 81 0.2413 0.2413 0.2413 31.36% Castro::construct_ctu_hydro_source() 10 0.1845 0.1845 0.1845 23.97% VisMF::Write(FabArray) 11 0.1513 0.1513 0.1513 19.66% MLCellLinOp::smooth() 1620 0.1248 0.1248 0.1248 16.21% Amr::checkPoint() 3 0.1128 0.1128 0.1128 14.66% Amr::init() 1 0.1092 0.1092 0.1092 14.19% AmrLevel::checkPoint() 3 0.1087 0.1087 0.1087 14.12% StateData::checkPoint() 12 0.1086 0.1086 0.1086 14.11% MLCellLinOp::applyBC() 4379 0.08733 0.08733 0.08733 11.35% MLMG::mgVcycle_bottom 81 0.07257 0.07257 0.07257 9.43% MLMG::actualBottomSolve() 81 0.07254 0.07254 0.07254 9.43% MLCGSolver::bicgstab 81 0.07181 0.07181 0.07181 9.33% MLPoisson::Fsmooth() 3240 0.05892 0.05892 0.05892 7.66% Castro::clean_state() 62 0.05434 0.05434 0.05434 7.06% FillPatchIterator::Initialize 41 0.04955 0.04955 0.04955 6.44% FillPatchSingleLevel 41 0.04802 0.04802 0.04802 6.24% Amr::initialInit() 1 0.04599 0.04599 0.04599 5.98% Amr::writePlotFile() 2 0.0453 0.0453 0.0453 5.89% StateDataPhysBCFunct::() 41 0.04406 0.04406 0.04406 5.73% Amr::FinalizeInit() 1 0.04186 0.04186 0.04186 5.44% Castro::post_init() 1 0.04054 0.04054 0.04054 5.27% Gravity::multilevel_solve_for_new_phi() 1 0.03876 0.03876 0.03876 5.04% Gravity::actual_multilevel_solve() 1 0.03874 0.03874 0.03874 5.03% MLMG::mgVcycle_down::0 81 0.03403 0.03403 0.03403 4.42% MLCellLinOp::apply() 1128 0.03311 0.03311 0.03311 4.30% MLMG::mgVcycle_up::0 81 0.02889 0.02889 0.02889 3.75% Gravity::get_new_grav_vector() 11 0.02528 0.02528 0.02528 3.28% StateData::FillBoundary(geom) 328 0.02285 0.02285 0.02285 2.97% Castro::initialize_do_advance() 10 0.02149 0.02149 0.02149 2.79% Castro::computeTemp() 63 0.02057 0.02057 0.02057 2.67% MultiFab::Dot() 1100 0.02036 0.02036 0.02036 2.65% MLCellLinOp::correctionResidual() 486 0.01933 0.01933 0.01933 2.51% Castro::normalize_species() 62 0.01731 0.01731 0.01731 2.25% MLMG:computeResOfCorrection() 405 0.01679 0.01679 0.01679 2.18% MLPoisson::define() 11 0.01613 0.01613 0.01613 2.10% MLMG::mgVcycle_down::1 81 0.01596 0.01596 0.01596 2.07% FabArray::FillBoundary() 3974 0.01563 0.01563 0.01563 2.03% MLMG::mgVcycle_down::2 81 0.01541 0.01541 0.01541 2.00% FillBoundary_nowait() 3974 0.01482 0.01482 0.01482 1.93% MLMG::mgVcycle_down::3 81 0.01455 0.01455 0.01455 1.89% Castro::construct_old_gravity() 10 0.01421 0.01421 0.01421 1.85% Gravity::get_old_grav_vector() 10 0.0142 0.0142 0.0142 1.85% FabArray::ParallelCopy() 851 0.01393 0.01393 0.01393 1.81% MLMG::mgVcycle_down::4 81 0.01377 0.01377 0.01377 1.79% FabArray::ParallelCopy_nowait() 851 0.01366 0.01366 0.01366 1.77% Castro::enforce_min_density() 62 0.0132 0.0132 0.0132 1.72% FabArray::setVal() 1135 0.01301 0.01301 0.01301 1.69% Castro::initialize_advance() 10 0.01286 0.01286 0.01286 1.67% CGSolver::sxay() 1566 0.01258 0.01258 0.01258 1.63% MLCGSolver::ParallelAllReduce 1495 0.01225 0.01225 0.01225 1.59% MultiFab::LinComb() 1566 0.01217 0.01217 0.01217 1.58% MLMG::mgVcycle_up::2 81 0.01189 0.01189 0.01189 1.55% Castro::do_old_sources() 10 0.01176 0.01176 0.01176 1.53% MLMG::mgVcycle_up::1 81 0.01171 0.01171 0.01171 1.52% Castro::expand_state() 10 0.01167 0.01167 0.01167 1.52% MLMG::addInterpCorrection() 405 0.01148 0.01148 0.01148 1.49% Castro::do_new_sources() 10 0.01145 0.01145 0.01145 1.49% MLMG::mgVcycle_up::3 81 0.01117 0.01117 0.01117 1.45% amrex::average_down 405 0.01111 0.01111 0.01111 1.44% MLMG::mgVcycle_up::4 81 0.01101 0.01101 0.01101 1.43% MLCellLinOp::defineAuxData() 11 0.01098 0.01098 0.01098 1.43% MLPoisson::Fapply() 1128 0.0104 0.0104 0.0104 1.35% Castro::post_timestep() 10 0.01021 0.01021 0.01021 1.33% Gravity::fill_multipole_BCs() 11 0.00858 0.00858 0.00858 1.11% MLCellLinOp::solutionResidual() 92 0.006946 0.006946 0.006946 0.90% Castro::estTimeStep() 21 0.006737 0.006737 0.006737 0.88% Castro::reset_internal_energy(MultiFab) 63 0.006489 0.006489 0.006489 0.84% MultiFab::Xpay() 578 0.005997 0.005997 0.005997 0.78% MLMG::computeResidual() 81 0.005972 0.005972 0.005972 0.78% MLCellLinOp::defineBC() 11 0.004668 0.004668 0.004668 0.61% BndryData::define() 11 0.004483 0.004483 0.004483 0.58% MLMG::prepareForSolve() 11 0.004465 0.004465 0.004465 0.58% Amr::InitializeInit() 1 0.004134 0.004134 0.004134 0.54% Amr::defBaseLevel() 1 0.004128 0.004128 0.004128 0.54% Castro::enforce_speed_limit() 62 0.004003 0.004003 0.004003 0.52% Castro::initData() 1 0.003607 0.003607 0.003607 0.47% Castro::computeNewDt() 9 0.003403 0.003403 0.003403 0.44% Castro::construct_new_source() 50 0.003188 0.003188 0.003188 0.41% Castro::construct_new_gravity_source() 10 0.003171 0.003171 0.003171 0.41% Castro::construct_old_source() 50 0.002668 0.002668 0.002668 0.35% Castro::construct_old_gravity_source() 10 0.002643 0.002643 0.002643 0.34% MLMG::ResNormInf() 92 0.001948 0.001948 0.001948 0.25% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.24% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001624 0.001624 0.001624 0.21% FabArrayBase::getCPC() 1313 0.001522 0.001522 0.001522 0.20% Castro::reset_internal_energy(Fab) 504 0.001486 0.001486 0.001486 0.19% MLMG::getGradSolution() 11 0.001365 0.001365 0.001365 0.18% MLCellLinOp::compGrad() 11 0.00136 0.00136 0.00136 0.18% FabArray::mult() 43 0.001336 0.001336 0.001336 0.17% MLCellLinOp::setLevelBC() 11 0.001316 0.001316 0.001316 0.17% FabArray::setDomainBndry() 41 0.001278 0.001278 0.001278 0.17% Castro::check_for_nan() 20 0.001183 0.001183 0.001183 0.15% MultiFab::contains_nan() 20 0.001171 0.001171 0.001171 0.15% Castro::post_regrid() 1 0.001088 0.001088 0.001088 0.14% MLPoisson::prepareForSolve() 11 0.001084 0.001084 0.001084 0.14% MLCellLinOp::prepareForSolve() 11 0.001076 0.001076 0.001076 0.14% MLMG::computeMLResidual() 11 0.001015 0.001015 0.001015 0.13% Gravity::update_max_rhs() 11 0.0008076 0.0008076 0.0008076 0.10% FabArrayBase::getFB() 3974 0.0007924 0.0007924 0.0007924 0.10% FabArrayBase::CPC::define() 454 0.0007032 0.0007032 0.0007032 0.09% Castro::computeInitialDt() 2 0.0006734 0.0006734 0.0006734 0.09% Amr::InitAmr() 1 0.0004778 0.0004778 0.0004778 0.06% MLLinOp::define() 11 0.0004671 0.0004671 0.0004671 0.06% MLLinOp::defineGrids() 11 0.000441 0.000441 0.000441 0.06% Castro::Castro() 1 0.0004387 0.0004387 0.0004387 0.06% Gravity::swapTimeLevels() 10 0.0004365 0.0004365 0.0004365 0.06% MultiFab::Copy() 11 0.0002675 0.0002675 0.0002675 0.03% MLMG::MLResNormInf() 11 0.0002571 0.0002571 0.0002571 0.03% MultiFab::max() 11 0.0002562 0.0002562 0.0002562 0.03% MLMG::MLRhsNormInf() 11 0.0001995 0.0001995 0.0001995 0.03% Castro::buildMetrics() 1 0.000159 0.000159 0.000159 0.02% FabArrayBase::FB::FB() 56 8.628e-05 8.628e-05 8.628e-05 0.01% Castro::finalize_advance() 10 6.483e-05 6.483e-05 6.483e-05 0.01% AmrLevel::AmrLevel(dm) 1 6.048e-05 6.048e-05 6.048e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.358e-05 5.358e-05 5.358e-05 0.01% StateData::define() 4 4.501e-05 4.501e-05 4.501e-05 0.01% makeSFC 55 3.903e-05 3.903e-05 3.903e-05 0.01% Castro::swap_state_time_levels() 10 3.748e-05 3.748e-05 3.748e-05 0.00% Castro::finalize_do_advance() 10 3.68e-05 3.68e-05 3.68e-05 0.00% Castro::enforce_consistent_e() 1 3.409e-05 3.409e-05 3.409e-05 0.00% Castro::initMFs() 1 2.523e-05 2.523e-05 2.523e-05 0.00% Amr::writeSmallPlotFile() 1 2.313e-05 2.313e-05 2.313e-05 0.00% DistributionMapping::Distribute() 56 1.553e-05 1.553e-05 1.553e-05 0.00% Amr::initSubcycle() 1 9.603e-06 9.603e-06 9.603e-06 0.00% AmrLevel::checkPointPost() 3 5.271e-06 5.271e-06 5.271e-06 0.00% MLMG::buildFineMask() 11 5.004e-06 5.004e-06 5.004e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.832e-06 4.832e-06 4.832e-06 0.00% Castro::retry_advance_ctu() 10 4.391e-06 4.391e-06 4.391e-06 0.00% Castro::create_source_corrector() 10 4.174e-06 4.174e-06 4.174e-06 0.00% Gravity::set_mass_offset() 11 4.074e-06 4.074e-06 4.074e-06 0.00% Castro::FluxRegCrseInit 10 3.257e-06 3.257e-06 3.257e-06 0.00% Castro::FluxRegFineAdd() 10 2.605e-06 2.605e-06 2.605e-06 0.00% AmrLevel::checkPointPre() 3 2.078e-06 2.078e-06 2.078e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.971e-06 1.971e-06 1.971e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-4-gef52970b80e3) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-4-gef52970b80e3) initialized Starting run at 08:24:53 UTC on 2022-05-05. Successfully read inputs file ... Castro git describe: 22.05-9-gf85dde19e AMReX git describe: 22.05-4-gef52970b8 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.425654747 Restart time = 0.047475279 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047961194 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047836812 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.052655632 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054731243 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06088057 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.167485177 seconds Ending run at 08:24:53 UTC on 2022-05-05. Run time = 0.480004942 Run time without initialization = 0.431960103 Average number of zones advanced per microsecond: 3.034 Average number of zones advanced per microsecond per rank: 3.034 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4801 ... 0.4801 ... 0.4801 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 1 0.1660 0.1660 0.1660 34.58% Castro::construct_ctu_hydro_source() 5 0.08102 0.08102 0.08102 16.88% VisMF::Read() 3 0.04035 0.04035 0.04035 8.41% MLCellLinOp::applyBC() 1946 0.03157 0.03157 0.03157 6.58% MLPoisson::Fsmooth() 1440 0.02562 0.02562 0.02562 5.34% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 2.41% MLCGSolver::bicgstab 36 0.009187 0.009187 0.009187 1.91% MultiFab::Dot() 484 0.008865 0.008865 0.008865 1.85% Castro::normalize_species() 30 0.008 0.008 0.008 1.67% Castro::computeTemp() 30 0.006622 0.006622 0.006622 1.38% FabArray::setVal() 537 0.006266 0.006266 0.006266 1.31% FillBoundary_nowait() 1766 0.00624 0.00624 0.00624 1.30% FabArray::ParallelCopy_nowait() 380 0.005909 0.005909 0.005909 1.23% MLCellLinOp::defineAuxData() 6 0.005397 0.005397 0.005397 1.12% MultiFab::LinComb() 690 0.005309 0.005309 0.005309 1.11% StateDataPhysBCFunct::() 20 0.005183 0.005183 0.005183 1.08% Gravity::fill_multipole_BCs() 6 0.004707 0.004707 0.004707 0.98% MLPoisson::Fapply() 500 0.004606 0.004606 0.004606 0.96% Amr::restart() 1 0.003073 0.003073 0.003073 0.64% MLMG::addInterpCorrection() 180 0.002877 0.002877 0.002877 0.60% amrex::average_down 180 0.002676 0.002676 0.002676 0.56% MultiFab::Xpay() 258 0.002642 0.002642 0.002642 0.55% Castro::enforce_min_density() 30 0.002346 0.002346 0.002346 0.49% BndryData::define() 6 0.001982 0.001982 0.001982 0.41% Castro::do_advance_ctu() 5 0.001948 0.001948 0.001948 0.41% Castro::reset_internal_energy(Fab) 240 0.001741 0.001741 0.001741 0.36% Castro::estTimeStep() 10 0.001728 0.001728 0.001728 0.36% Castro::construct_new_gravity_source() 5 0.001703 0.001703 0.001703 0.35% Castro::reset_internal_energy(MultiFab) 30 0.001632 0.001632 0.001632 0.34% Amr::writePlotFile() 1 0.001573 0.001573 0.001573 0.33% Castro::subcycle_advance_ctu() 5 0.001487 0.001487 0.001487 0.31% Castro::construct_old_gravity_source() 5 0.001394 0.001394 0.001394 0.29% MultiFab::Saxpy() 10 0.0009327 0.0009327 0.0009327 0.19% Castro::expand_state() 5 0.000899 0.000899 0.000899 0.19% Gravity::get_old_grav_vector() 5 0.0008966 0.0008966 0.0008966 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008796 0.0008796 0.0008796 0.18% MLMG::ResNormInf() 42 0.0008708 0.0008708 0.0008708 0.18% Gravity::get_new_grav_vector() 5 0.00087 0.00087 0.00087 0.18% MLMG::oneIter() 36 0.0007393 0.0007393 0.0007393 0.15% Gravity::actual_solve_with_mlmg() 6 0.0007369 0.0007369 0.0007369 0.15% MLCellLinOp::setLevelBC() 6 0.0007253 0.0007253 0.0007253 0.15% FabArray::setDomainBndry() 20 0.0006517 0.0006517 0.0006517 0.14% FabArray::mult() 22 0.0006502 0.0006502 0.0006502 0.14% MultiFab::contains_nan() 10 0.000601 0.000601 0.000601 0.13% MLCellLinOp::prepareForSolve() 6 0.0005855 0.0005855 0.0005855 0.12% MLCellLinOp::compGrad() 6 0.0004876 0.0004876 0.0004876 0.10% Castro::enforce_speed_limit() 30 0.0004874 0.0004874 0.0004874 0.10% MLCellLinOp::smooth() 720 0.0004839 0.0004839 0.0004839 0.10% MLMG::prepareForSolve() 6 0.0004248 0.0004248 0.0004248 0.09% FabArrayBase::CPC::define() 244 0.0004235 0.0004235 0.0004235 0.09% FabArrayBase::getCPC() 632 0.0003947 0.0003947 0.0003947 0.08% Amr::InitAmr() 1 0.0003842 0.0003842 0.0003842 0.08% FabArray::FillBoundary() 1766 0.0003756 0.0003756 0.0003756 0.08% main() 1 0.0003282 0.0003282 0.0003282 0.07% FabArrayBase::getFB() 1766 0.000287 0.000287 0.000287 0.06% MLCellLinOp::apply() 500 0.000243 0.000243 0.000243 0.05% Gravity::update_max_rhs() 6 0.0002326 0.0002326 0.0002326 0.05% Gravity::solve_for_phi() 5 0.0002241 0.0002241 0.0002241 0.05% CGSolver::sxay() 690 0.0002129 0.0002129 0.0002129 0.04% MLLinOp::defineGrids() 6 0.0001933 0.0001933 0.0001933 0.04% Castro::create_source_corrector() 5 0.0001637 0.0001637 0.0001637 0.03% MLMG::mgVcycle() 36 0.0001589 0.0001589 0.0001589 0.03% Castro::construct_new_source() 25 0.0001516 0.0001516 0.0001516 0.03% MLCGSolver::ParallelAllReduce 659 0.0001469 0.0001469 0.0001469 0.03% Castro::initialize_advance() 5 0.0001468 0.0001468 0.0001468 0.03% MultiFab::max() 6 0.0001451 0.0001451 0.0001451 0.03% MultiFab::Copy() 6 0.0001406 0.0001406 0.0001406 0.03% FabArray::ParallelCopy() 380 0.0001351 0.0001351 0.0001351 0.03% FillPatchIterator::Initialize 20 0.0001276 0.0001276 0.0001276 0.03% Amr::coarseTimeStep() 5 0.000118 0.000118 0.000118 0.02% MLCellLinOp::correctionResidual() 216 0.0001079 0.0001079 0.0001079 0.02% Castro::construct_new_gravity() 5 0.0001077 0.0001077 0.0001077 0.02% Castro::advance() 5 0.0001076 0.0001076 0.0001076 0.02% Amr::timeStep() 5 0.0001066 0.0001066 0.0001066 0.02% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.02% MLCellLinOp::defineBC() 6 0.0001049 0.0001049 0.0001049 0.02% Castro::initialize_do_advance() 5 8.325e-05 8.325e-05 8.325e-05 0.02% AmrLevel::restart() 1 7.71e-05 7.71e-05 7.71e-05 0.02% Castro::construct_old_source() 25 7.554e-05 7.554e-05 7.554e-05 0.02% Castro::post_timestep() 5 7.471e-05 7.471e-05 7.471e-05 0.02% StateData::restartDoit() 4 7.367e-05 7.367e-05 7.367e-05 0.02% MLMG:computeResOfCorrection() 180 6.461e-05 6.461e-05 6.461e-05 0.01% FabArrayBase::FB::FB() 26 6.196e-05 6.196e-05 6.196e-05 0.01% Castro::construct_old_gravity() 5 5.046e-05 5.046e-05 5.046e-05 0.01% MLMG::actualBottomSolve() 36 4.849e-05 4.849e-05 4.849e-05 0.01% MLMG::mgVcycle_down::0 36 4.266e-05 4.266e-05 4.266e-05 0.01% MLMG::mgVcycle_down::1 36 3.999e-05 3.999e-05 3.999e-05 0.01% Castro::do_new_sources() 5 3.863e-05 3.863e-05 3.863e-05 0.01% Castro::clean_state() 30 3.862e-05 3.862e-05 3.862e-05 0.01% MLMG::mgVcycle_down::2 36 3.584e-05 3.584e-05 3.584e-05 0.01% MLMG::solve() 6 3.555e-05 3.555e-05 3.555e-05 0.01% MLMG::mgVcycle_down::4 36 3.416e-05 3.416e-05 3.416e-05 0.01% MLMG::mgVcycle_down::3 36 3.319e-05 3.319e-05 3.319e-05 0.01% Castro::buildMetrics() 1 3.174e-05 3.174e-05 3.174e-05 0.01% Amr::writeSmallPlotFile() 1 3.164e-05 3.164e-05 3.164e-05 0.01% MLMG::mgVcycle_up::4 36 2.94e-05 2.94e-05 2.94e-05 0.01% Gravity::actual_multilevel_solve() 1 2.939e-05 2.939e-05 2.939e-05 0.01% Castro::swap_state_time_levels() 5 2.769e-05 2.769e-05 2.769e-05 0.01% Castro::post_restart() 1 2.758e-05 2.758e-05 2.758e-05 0.01% Castro::initMFs() 1 2.734e-05 2.734e-05 2.734e-05 0.01% Castro::finalize_advance() 5 2.632e-05 2.632e-05 2.632e-05 0.01% MLMG::mgVcycle_up::0 36 2.415e-05 2.415e-05 2.415e-05 0.01% MLMG::mgVcycle_up::3 36 2.274e-05 2.274e-05 2.274e-05 0.00% MLMG::mgVcycle_up::2 36 2.269e-05 2.269e-05 2.269e-05 0.00% MLCellLinOp::solutionResidual() 42 2.242e-05 2.242e-05 2.242e-05 0.00% MLLinOp::define() 6 2.214e-05 2.214e-05 2.214e-05 0.00% MLMG::mgVcycle_up::1 36 2.164e-05 2.164e-05 2.164e-05 0.00% Castro::finalize_do_advance() 5 2.065e-05 2.065e-05 2.065e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.726e-05 1.726e-05 1.726e-05 0.00% MLMG::computeResidual() 36 1.671e-05 1.671e-05 1.671e-05 0.00% MLMG::mgVcycle_bottom 36 1.666e-05 1.666e-05 1.666e-05 0.00% FillPatchSingleLevel 20 1.504e-05 1.504e-05 1.504e-05 0.00% MLPoisson::define() 6 1.49e-05 1.49e-05 1.49e-05 0.00% makeSFC 30 1.428e-05 1.428e-05 1.428e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.413e-05 1.413e-05 1.413e-05 0.00% Amr::initSubcycle() 1 9.795e-06 9.795e-06 9.795e-06 0.00% Castro::do_old_sources() 5 9.021e-06 9.021e-06 9.021e-06 0.00% DistributionMapping::Distribute() 31 9.018e-06 9.018e-06 9.018e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.952e-06 7.952e-06 7.952e-06 0.00% Castro::apply_source_to_state() 10 6.667e-06 6.667e-06 6.667e-06 0.00% Castro::check_for_nan() 10 6.571e-06 6.571e-06 6.571e-06 0.00% MLPoisson::prepareForSolve() 6 4.844e-06 4.844e-06 4.844e-06 0.00% Gravity::swapTimeLevels() 5 4.467e-06 4.467e-06 4.467e-06 0.00% MLMG::computeMLResidual() 6 3.526e-06 3.526e-06 3.526e-06 0.00% Castro::computeNewDt() 5 3.385e-06 3.385e-06 3.385e-06 0.00% MLMG::getGradSolution() 6 3.376e-06 3.376e-06 3.376e-06 0.00% MLMG::MLResNormInf() 6 3.211e-06 3.211e-06 3.211e-06 0.00% MLMG::buildFineMask() 6 3.168e-06 3.168e-06 3.168e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.828e-06 2.828e-06 2.828e-06 0.00% Gravity::set_mass_offset() 6 2.487e-06 2.487e-06 2.487e-06 0.00% Castro::retry_advance_ctu() 5 2.125e-06 2.125e-06 2.125e-06 0.00% Castro::FluxRegCrseInit 5 1.808e-06 1.808e-06 1.808e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.336e-06 1.336e-06 1.336e-06 0.00% Castro::FluxRegFineAdd() 5 1.306e-06 1.306e-06 1.306e-06 0.00% Amr::init() 1 1.095e-06 1.095e-06 1.095e-06 0.00% AmrLevel::AmrLevel() 1 1.057e-06 1.057e-06 1.057e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.48 0.48 0.48 99.99% Amr::coarseTimeStep() 5 0.2642 0.2642 0.2642 55.03% Amr::timeStep() 5 0.2632 0.2632 0.2632 54.82% Castro::advance() 5 0.2582 0.2582 0.2582 53.78% Castro::subcycle_advance_ctu() 5 0.2514 0.2514 0.2514 52.36% Castro::do_advance_ctu() 5 0.2499 0.2499 0.2499 52.05% Amr::writePlotFile() 1 0.1676 0.1676 0.1676 34.91% VisMF::Write(FabArray) 1 0.166 0.166 0.166 34.58% Castro::construct_new_gravity() 5 0.1335 0.1335 0.1335 27.81% Gravity::solve_phi_with_mlmg() 6 0.1292 0.1292 0.1292 26.91% Gravity::solve_for_phi() 5 0.1259 0.1259 0.1259 26.23% Gravity::actual_solve_with_mlmg() 6 0.1244 0.1244 0.1244 25.91% MLMG::solve() 6 0.1131 0.1131 0.1131 23.57% MLMG::oneIter() 36 0.1065 0.1065 0.1065 22.18% MLMG::mgVcycle() 36 0.1057 0.1057 0.1057 22.03% Castro::construct_ctu_hydro_source() 5 0.08102 0.08102 0.08102 16.88% MLCellLinOp::smooth() 720 0.05458 0.05458 0.05458 11.37% Amr::init() 1 0.04751 0.04751 0.04751 9.90% Amr::restart() 1 0.04751 0.04751 0.04751 9.90% AmrLevel::restart() 1 0.04056 0.04056 0.04056 8.45% StateData::restartDoit() 4 0.04048 0.04048 0.04048 8.43% VisMF::Read() 3 0.04035 0.04035 0.04035 8.41% MLCellLinOp::applyBC() 1946 0.03853 0.03853 0.03853 8.03% MLMG::mgVcycle_bottom 36 0.03172 0.03172 0.03172 6.61% MLMG::actualBottomSolve() 36 0.0317 0.0317 0.0317 6.60% MLCGSolver::bicgstab 36 0.03138 0.03138 0.03138 6.54% MLPoisson::Fsmooth() 1440 0.02562 0.02562 0.02562 5.34% Castro::clean_state() 30 0.02087 0.02087 0.02087 4.35% FillPatchIterator::Initialize 20 0.01956 0.01956 0.01956 4.07% FillPatchSingleLevel 20 0.01878 0.01878 0.01878 3.91% StateDataPhysBCFunct::() 20 0.01674 0.01674 0.01674 3.49% MLMG::mgVcycle_down::0 36 0.01489 0.01489 0.01489 3.10% MLCellLinOp::apply() 500 0.01466 0.01466 0.01466 3.05% MLMG::mgVcycle_up::0 36 0.01263 0.01263 0.01263 2.63% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 2.41% Castro::computeTemp() 30 0.009995 0.009995 0.009995 2.08% Castro::initialize_do_advance() 5 0.009156 0.009156 0.009156 1.91% MLPoisson::define() 6 0.009036 0.009036 0.009036 1.88% MultiFab::Dot() 484 0.008865 0.008865 0.008865 1.85% MLCellLinOp::correctionResidual() 216 0.008498 0.008498 0.008498 1.77% Castro::normalize_species() 30 0.008 0.008 0.008 1.67% Gravity::get_new_grav_vector() 5 0.007503 0.007503 0.007503 1.56% MLMG:computeResOfCorrection() 180 0.007378 0.007378 0.007378 1.54% Castro::construct_old_gravity() 5 0.00725 0.00725 0.00725 1.51% Gravity::get_old_grav_vector() 5 0.0072 0.0072 0.0072 1.50% MLMG::mgVcycle_down::1 36 0.007023 0.007023 0.007023 1.46% FabArray::FillBoundary() 1766 0.006965 0.006965 0.006965 1.45% MLMG::mgVcycle_down::2 36 0.006763 0.006763 0.006763 1.41% Castro::initialize_advance() 5 0.006654 0.006654 0.006654 1.39% FillBoundary_nowait() 1766 0.006589 0.006589 0.006589 1.37% FabArray::ParallelCopy() 380 0.006451 0.006451 0.006451 1.34% MLMG::mgVcycle_down::3 36 0.006382 0.006382 0.006382 1.33% FabArray::ParallelCopy_nowait() 380 0.006316 0.006316 0.006316 1.32% FabArray::setVal() 537 0.006266 0.006266 0.006266 1.31% MLCellLinOp::defineAuxData() 6 0.006151 0.006151 0.006151 1.28% MLMG::mgVcycle_down::4 36 0.006067 0.006067 0.006067 1.26% Castro::expand_state() 5 0.005641 0.005641 0.005641 1.18% CGSolver::sxay() 690 0.005522 0.005522 0.005522 1.15% Castro::do_new_sources() 5 0.005405 0.005405 0.005405 1.13% MLCGSolver::ParallelAllReduce 659 0.005335 0.005335 0.005335 1.11% MultiFab::LinComb() 690 0.005309 0.005309 0.005309 1.11% MLMG::mgVcycle_up::2 36 0.005245 0.005245 0.005245 1.09% MLMG::mgVcycle_up::1 36 0.00513 0.00513 0.00513 1.07% MLMG::addInterpCorrection() 180 0.005085 0.005085 0.005085 1.06% MLMG::mgVcycle_up::3 36 0.004901 0.004901 0.004901 1.02% amrex::average_down 180 0.004901 0.004901 0.004901 1.02% Castro::post_timestep() 5 0.004884 0.004884 0.004884 1.02% MLMG::mgVcycle_up::4 36 0.004823 0.004823 0.004823 1.00% Gravity::fill_multipole_BCs() 6 0.004707 0.004707 0.004707 0.98% MLPoisson::Fapply() 500 0.004606 0.004606 0.004606 0.96% Castro::do_old_sources() 5 0.004437 0.004437 0.004437 0.92% Castro::post_restart() 1 0.003683 0.003683 0.003683 0.77% Gravity::multilevel_solve_for_new_phi() 1 0.003565 0.003565 0.003565 0.74% Gravity::actual_multilevel_solve() 1 0.003548 0.003548 0.003548 0.74% Castro::reset_internal_energy(MultiFab) 30 0.003373 0.003373 0.003373 0.70% MLCellLinOp::solutionResidual() 42 0.003184 0.003184 0.003184 0.66% MultiFab::Xpay() 258 0.002642 0.002642 0.002642 0.55% MLMG::computeResidual() 36 0.002624 0.002624 0.002624 0.55% MLCellLinOp::defineBC() 6 0.002624 0.002624 0.002624 0.55% BndryData::define() 6 0.002519 0.002519 0.002519 0.52% MLMG::prepareForSolve() 6 0.002433 0.002433 0.002433 0.51% Castro::enforce_min_density() 30 0.002346 0.002346 0.002346 0.49% Castro::construct_new_source() 25 0.001855 0.001855 0.001855 0.39% Castro::reset_internal_energy(Fab) 240 0.001741 0.001741 0.001741 0.36% Castro::estTimeStep() 10 0.001728 0.001728 0.001728 0.36% Castro::construct_new_gravity_source() 5 0.001703 0.001703 0.001703 0.35% Castro::construct_old_source() 25 0.00147 0.00147 0.00147 0.31% Castro::construct_old_gravity_source() 5 0.001394 0.001394 0.001394 0.29% Castro::apply_source_to_state() 10 0.0009394 0.0009394 0.0009394 0.20% MultiFab::Saxpy() 10 0.0009327 0.0009327 0.0009327 0.19% Castro::computeNewDt() 5 0.0009004 0.0009004 0.0009004 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008796 0.0008796 0.0008796 0.18% MLMG::ResNormInf() 42 0.0008708 0.0008708 0.0008708 0.18% FabArrayBase::getCPC() 632 0.0008182 0.0008182 0.0008182 0.17% MLMG::getGradSolution() 6 0.000748 0.000748 0.000748 0.16% MLCellLinOp::compGrad() 6 0.0007446 0.0007446 0.0007446 0.16% MLCellLinOp::setLevelBC() 6 0.0007253 0.0007253 0.0007253 0.15% FabArray::setDomainBndry() 20 0.0006517 0.0006517 0.0006517 0.14% FabArray::mult() 22 0.0006502 0.0006502 0.0006502 0.14% Castro::check_for_nan() 10 0.0006076 0.0006076 0.0006076 0.13% MultiFab::contains_nan() 10 0.000601 0.000601 0.000601 0.13% MLPoisson::prepareForSolve() 6 0.0005904 0.0005904 0.0005904 0.12% MLCellLinOp::prepareForSolve() 6 0.0005855 0.0005855 0.0005855 0.12% MLMG::computeMLResidual() 6 0.0005804 0.0005804 0.0005804 0.12% Castro::enforce_speed_limit() 30 0.0004874 0.0004874 0.0004874 0.10% Gravity::update_max_rhs() 6 0.0004564 0.0004564 0.0004564 0.10% FabArrayBase::CPC::define() 244 0.0004235 0.0004235 0.0004235 0.09% Amr::InitAmr() 1 0.000394 0.000394 0.000394 0.08% FabArrayBase::getFB() 1766 0.000349 0.000349 0.000349 0.07% MLLinOp::define() 6 0.0002466 0.0002466 0.0002466 0.05% Gravity::swapTimeLevels() 5 0.0002283 0.0002283 0.0002283 0.05% MLLinOp::defineGrids() 6 0.0002245 0.0002245 0.0002245 0.05% Castro::create_source_corrector() 5 0.0001637 0.0001637 0.0001637 0.03% Castro::buildMetrics() 1 0.000161 0.000161 0.000161 0.03% MultiFab::max() 6 0.0001451 0.0001451 0.0001451 0.03% MultiFab::Copy() 6 0.0001406 0.0001406 0.0001406 0.03% MLMG::MLResNormInf() 6 0.0001369 0.0001369 0.0001369 0.03% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.02% FabArrayBase::FB::FB() 26 6.196e-05 6.196e-05 6.196e-05 0.01% Amr::writeSmallPlotFile() 1 3.164e-05 3.164e-05 3.164e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.984e-05 2.984e-05 2.984e-05 0.01% Castro::finalize_advance() 5 2.944e-05 2.944e-05 2.944e-05 0.01% Castro::swap_state_time_levels() 5 2.769e-05 2.769e-05 2.769e-05 0.01% Castro::initMFs() 1 2.734e-05 2.734e-05 2.734e-05 0.01% makeSFC 30 2.188e-05 2.188e-05 2.188e-05 0.00% Castro::finalize_do_advance() 5 2.065e-05 2.065e-05 2.065e-05 0.00% Amr::initSubcycle() 1 9.795e-06 9.795e-06 9.795e-06 0.00% DistributionMapping::Distribute() 31 9.018e-06 9.018e-06 9.018e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.246e-06 4.246e-06 4.246e-06 0.00% MLMG::buildFineMask() 6 3.168e-06 3.168e-06 3.168e-06 0.00% Gravity::set_mass_offset() 6 2.487e-06 2.487e-06 2.487e-06 0.00% Castro::retry_advance_ctu() 5 2.125e-06 2.125e-06 2.125e-06 0.00% Castro::FluxRegCrseInit 5 1.808e-06 1.808e-06 1.808e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.336e-06 1.336e-06 1.336e-06 0.00% Castro::FluxRegFineAdd() 5 1.306e-06 1.306e-06 1.306e-06 0.00% AmrLevel::AmrLevel() 1 1.057e-06 1.057e-06 1.057e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-4-gef52970b80e3) finalized