Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-20-g46e487c72973) initialized Starting run at 09:37:44 UTC on 2022-11-21. Successfully read inputs file ... Castro git describe: 22.11-13-ga00eb1604 AMReX git describe: 22.11-20-g46e487c72 Microphysics git describe: 22.11-38-g15955539 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053732235 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030943076 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048648561 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051644923 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.053392236 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.052479477 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.063857915 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049642434 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.075295008 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.065737675 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058034523 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054424311 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.058108099 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.079127566 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030868794 seconds Ending run at 09:37:45 UTC on 2022-11-21. Run time = 0.879645681 Run time without initialization = 0.741847776 Average number of zones advanced per microsecond: 3.534 Average number of zones advanced per microsecond per rank: 3.534 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8797 ... 0.8797 ... 0.8797 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2077 0.2077 0.2077 23.61% Castro::construct_ctu_hydro_source() 10 0.1882 0.1882 0.1882 21.40% MLCellLinOp::applyBC() 4433 0.08112 0.08112 0.08112 9.22% MLPoisson::Fsmooth() 3280 0.06507 0.06507 0.06507 7.40% Amr::checkPoint() 3 0.03383 0.03383 0.03383 3.85% MLCGSolver::bicgstab 82 0.02435 0.02435 0.02435 2.77% StateData::FillBoundary(geom) 328 0.02369 0.02369 0.02369 2.69% MultiFab::Dot() 1114 0.02286 0.02286 0.02286 2.60% Castro::normalize_species() 62 0.01644 0.01644 0.01644 1.87% StateDataPhysBCFunct::() 41 0.01472 0.01472 0.01472 1.67% MultiFab::LinComb() 1586 0.01465 0.01465 0.01465 1.67% FabArray::setVal() 1144 0.01451 0.01451 0.01451 1.65% FillBoundary_nowait() 4023 0.0144 0.0144 0.0144 1.64% Castro::computeTemp() 63 0.01439 0.01439 0.01439 1.64% FabArray::ParallelCopy_nowait() 861 0.01347 0.01347 0.01347 1.53% MLPoisson::Fapply() 1142 0.01201 0.01201 0.01201 1.37% MLCellLinOp::defineAuxData() 11 0.01167 0.01167 0.01167 1.33% Castro::enforce_min_density() 62 0.009079 0.009079 0.009079 1.03% Gravity::fill_multipole_BCs() 11 0.008912 0.008912 0.008912 1.01% MLMG::addInterpCorrection() 410 0.007864 0.007864 0.007864 0.89% amrex::average_down 410 0.006894 0.006894 0.006894 0.78% MultiFab::Xpay() 585 0.006701 0.006701 0.006701 0.76% Castro::do_advance_ctu() 10 0.005164 0.005164 0.005164 0.59% Castro::estTimeStep() 21 0.004807 0.004807 0.004807 0.55% Castro::reset_internal_energy(MultiFab) 63 0.004124 0.004124 0.004124 0.47% BndryData::define() 11 0.003971 0.003971 0.003971 0.45% Castro::construct_new_gravity_source() 10 0.003296 0.003296 0.003296 0.37% Amr::writePlotFile() 2 0.002807 0.002807 0.002807 0.32% Castro::construct_old_gravity_source() 10 0.002649 0.002649 0.002649 0.30% MLMG::ResNormInf() 93 0.002121 0.002121 0.002121 0.24% Gravity::get_new_grav_vector() 11 0.001942 0.001942 0.001942 0.22% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.21% Gravity::get_old_grav_vector() 10 0.001755 0.001755 0.001755 0.20% Castro::expand_state() 10 0.001741 0.001741 0.001741 0.20% MultiFab::Add() 82 0.001702 0.001702 0.001702 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001684 0.001684 0.001684 0.19% Castro::reset_internal_energy(Fab) 504 0.001628 0.001628 0.001628 0.19% MLCellLinOp::setLevelBC() 11 0.001559 0.001559 0.001559 0.18% Gravity::actual_solve_with_mlmg() 11 0.001384 0.001384 0.001384 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.15% Castro::enforce_speed_limit() 62 0.001319 0.001319 0.001319 0.15% FabArray::setDomainBndry() 41 0.001317 0.001317 0.001317 0.15% Castro::initData() 1 0.001276 0.001276 0.001276 0.14% MLMG::prepareForSolve() 11 0.001232 0.001232 0.001232 0.14% MLCellLinOp::prepareForSolve() 11 0.001188 0.001188 0.001188 0.14% MultiFab::contains_nan() 20 0.001186 0.001186 0.001186 0.13% MLCellLinOp::smooth() 1640 0.001066 0.001066 0.001066 0.12% MLCellLinOp::compGrad() 11 0.0009276 0.0009276 0.0009276 0.11% FabArray::FillBoundary() 4023 0.0008507 0.0008507 0.0008507 0.10% FabArrayBase::getCPC() 1323 0.000792 0.000792 0.000792 0.09% FabArrayBase::CPC::define() 454 0.0006497 0.0006497 0.0006497 0.07% Castro::finalize_advance() 10 0.0006212 0.0006212 0.0006212 0.07% FabArrayBase::getFB() 4023 0.0005957 0.0005957 0.0005957 0.07% Amr::InitAmr() 1 0.0004804 0.0004804 0.0004804 0.05% Gravity::solve_for_phi() 10 0.0004608 0.0004608 0.0004608 0.05% MLCellLinOp::apply() 1142 0.0004545 0.0004545 0.0004545 0.05% Gravity::update_max_rhs() 11 0.0004173 0.0004173 0.0004173 0.05% CGSolver::sxay() 1586 0.0003771 0.0003771 0.0003771 0.04% MultiFab::Copy() 11 0.0003359 0.0003359 0.0003359 0.04% Amr::coarseTimeStep() 10 0.0003184 0.0003184 0.0003184 0.04% MLCGSolver::ParallelAllReduce 1514 0.000293 0.000293 0.000293 0.03% FillPatchIterator::Initialize 41 0.0002899 0.0002899 0.0002899 0.03% MLCellLinOp::defineBC() 11 0.0002732 0.0002732 0.0002732 0.03% MultiFab::max() 11 0.0002644 0.0002644 0.0002644 0.03% FabArray::ParallelCopy() 861 0.000252 0.000252 0.000252 0.03% main() 1 0.0002485 0.0002485 0.0002485 0.03% MLMG::mgVcycle() 82 0.0002296 0.0002296 0.0002296 0.03% MLCellLinOp::correctionResidual() 492 0.0002266 0.0002266 0.0002266 0.03% MLMG::MLRhsNormInf() 11 0.0002209 0.0002209 0.0002209 0.03% Castro::construct_new_gravity() 10 0.0002041 0.0002041 0.0002041 0.02% Castro::subcycle_advance_ctu() 10 0.000204 0.000204 0.000204 0.02% MLMG:computeResOfCorrection() 410 0.0001623 0.0001623 0.0001623 0.02% Amr::timeStep() 10 0.0001489 0.0001489 0.0001489 0.02% MLLinOp::defineGrids() 11 0.0001344 0.0001344 0.0001344 0.02% StateData::checkPoint() 12 0.0001302 0.0001302 0.0001302 0.01% MLMG::mgVcycle_down::0 82 0.000124 0.000124 0.000124 0.01% MLMG::mgVcycle_down::1 82 0.000106 0.000106 0.000106 0.01% Castro::advance() 10 0.0001055 0.0001055 0.0001055 0.01% MLMG::mgVcycle_down::2 82 0.0001014 0.0001014 0.0001014 0.01% MLMG::mgVcycle_down::3 82 9.351e-05 9.351e-05 9.351e-05 0.01% MLMG::mgVcycle_down::4 82 9.32e-05 9.32e-05 9.32e-05 0.01% Castro::Castro() 1 8.72e-05 8.72e-05 8.72e-05 0.01% MLMG::actualBottomSolve() 82 8.241e-05 8.241e-05 8.241e-05 0.01% FabArrayBase::FB::FB() 56 8.166e-05 8.166e-05 8.166e-05 0.01% Castro::initialize_advance() 10 8.139e-05 8.139e-05 8.139e-05 0.01% Castro::clean_state() 62 7.756e-05 7.756e-05 7.756e-05 0.01% AmrLevel::checkPoint() 3 7.568e-05 7.568e-05 7.568e-05 0.01% MLMG::mgVcycle_up::4 82 7.187e-05 7.187e-05 7.187e-05 0.01% MLMG::solve() 11 6.667e-05 6.667e-05 6.667e-05 0.01% MLMG::oneIter() 82 6.436e-05 6.436e-05 6.436e-05 0.01% Castro::initialize_do_advance() 10 6.243e-05 6.243e-05 6.243e-05 0.01% MLMG::mgVcycle_up::3 82 5.995e-05 5.995e-05 5.995e-05 0.01% MLMG::mgVcycle_up::1 82 5.902e-05 5.902e-05 5.902e-05 0.01% MLMG::mgVcycle_up::0 82 5.823e-05 5.823e-05 5.823e-05 0.01% MLMG::mgVcycle_up::2 82 5.669e-05 5.669e-05 5.669e-05 0.01% MLCellLinOp::solutionResidual() 93 4.972e-05 4.972e-05 4.972e-05 0.01% Castro::finalize_do_advance() 10 4.221e-05 4.221e-05 4.221e-05 0.00% StateData::define() 4 4.154e-05 4.154e-05 4.154e-05 0.00% MLMG::computeResidual() 82 4.126e-05 4.126e-05 4.126e-05 0.00% Castro::enforce_consistent_e() 1 3.912e-05 3.912e-05 3.912e-05 0.00% Castro::swap_state_time_levels() 10 3.632e-05 3.632e-05 3.632e-05 0.00% MLMG::mgVcycle_bottom 82 3.49e-05 3.49e-05 3.49e-05 0.00% Gravity::actual_multilevel_solve() 1 3.113e-05 3.113e-05 3.113e-05 0.00% FillPatchSingleLevel 41 2.82e-05 2.82e-05 2.82e-05 0.00% Amr::defBaseLevel() 1 2.617e-05 2.617e-05 2.617e-05 0.00% Amr::writeSmallPlotFile() 1 2.544e-05 2.544e-05 2.544e-05 0.00% makeSFC 55 2.541e-05 2.541e-05 2.541e-05 0.00% Castro::initMFs() 1 2.497e-05 2.497e-05 2.497e-05 0.00% MLPoisson::define() 11 2.485e-05 2.485e-05 2.485e-05 0.00% Castro::do_new_sources() 10 2.479e-05 2.479e-05 2.479e-05 0.00% Castro::create_source_corrector() 10 2.386e-05 2.386e-05 2.386e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.366e-05 2.366e-05 2.366e-05 0.00% MLLinOp::define() 11 2.354e-05 2.354e-05 2.354e-05 0.00% Castro::buildMetrics() 1 2.234e-05 2.234e-05 2.234e-05 0.00% Amr::FinalizeInit() 1 1.985e-05 1.985e-05 1.985e-05 0.00% Castro::construct_new_source() 50 1.831e-05 1.831e-05 1.831e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.776e-05 1.776e-05 1.776e-05 0.00% Castro::construct_old_source() 50 1.766e-05 1.766e-05 1.766e-05 0.00% Castro::apply_source_to_state() 20 1.765e-05 1.765e-05 1.765e-05 0.00% Castro::do_old_sources() 10 1.589e-05 1.589e-05 1.589e-05 0.00% Castro::post_timestep() 10 1.578e-05 1.578e-05 1.578e-05 0.00% DistributionMapping::Distribute() 56 1.41e-05 1.41e-05 1.41e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.263e-05 1.263e-05 1.263e-05 0.00% Castro::check_for_nan() 20 1.221e-05 1.221e-05 1.221e-05 0.00% Castro::construct_old_gravity() 10 1.015e-05 1.015e-05 1.015e-05 0.00% MLPoisson::prepareForSolve() 11 9.739e-06 9.739e-06 9.739e-06 0.00% MLMG::computeMLResidual() 11 9.69e-06 9.69e-06 9.69e-06 0.00% Gravity::swapTimeLevels() 10 9.05e-06 9.05e-06 9.05e-06 0.00% Amr::initSubcycle() 1 8.269e-06 8.269e-06 8.269e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.865e-06 6.865e-06 6.865e-06 0.00% MLMG::getGradSolution() 11 6.509e-06 6.509e-06 6.509e-06 0.00% Castro::computeNewDt() 9 6.052e-06 6.052e-06 6.052e-06 0.00% AmrLevel::checkPointPost() 3 4.717e-06 4.717e-06 4.717e-06 0.00% Amr::InitializeInit() 1 4.703e-06 4.703e-06 4.703e-06 0.00% Castro::retry_advance_ctu() 10 3.695e-06 3.695e-06 3.695e-06 0.00% Gravity::set_mass_offset() 11 3.545e-06 3.545e-06 3.545e-06 0.00% Castro::post_init() 1 3.426e-06 3.426e-06 3.426e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.323e-06 3.323e-06 3.323e-06 0.00% MLMG::MLResNormInf() 11 3.274e-06 3.274e-06 3.274e-06 0.00% Castro::FluxRegCrseInit 10 2.852e-06 2.852e-06 2.852e-06 0.00% AmrLevel::checkPointPre() 3 2.69e-06 2.69e-06 2.69e-06 0.00% Amr::init() 1 2.659e-06 2.659e-06 2.659e-06 0.00% Castro::computeInitialDt() 2 2.332e-06 2.332e-06 2.332e-06 0.00% Castro::FluxRegFineAdd() 10 2.324e-06 2.324e-06 2.324e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.817e-06 1.817e-06 1.817e-06 0.00% Castro::post_regrid() 1 1.162e-06 1.162e-06 1.162e-06 0.00% Amr::initialInit() 1 9.8e-07 9.8e-07 9.8e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8797 0.8797 0.8797 100.00% Amr::coarseTimeStep() 10 0.7108 0.7108 0.7108 80.80% Amr::timeStep() 10 0.5787 0.5787 0.5787 65.79% Castro::advance() 10 0.5713 0.5713 0.5713 64.94% Castro::subcycle_advance_ctu() 10 0.5592 0.5592 0.5592 63.57% Castro::do_advance_ctu() 10 0.559 0.559 0.559 63.55% Gravity::solve_phi_with_mlmg() 11 0.3199 0.3199 0.3199 36.37% Gravity::actual_solve_with_mlmg() 11 0.3108 0.3108 0.3108 35.33% Castro::construct_new_gravity() 10 0.2912 0.2912 0.2912 33.11% MLMG::solve() 11 0.288 0.288 0.288 32.74% Gravity::solve_for_phi() 10 0.2756 0.2756 0.2756 31.33% MLMG::oneIter() 82 0.2728 0.2728 0.2728 31.01% MLMG::mgVcycle() 82 0.271 0.271 0.271 30.81% VisMF::Write(FabArray) 11 0.2077 0.2077 0.2077 23.61% Castro::construct_ctu_hydro_source() 10 0.1882 0.1882 0.1882 21.40% Amr::checkPoint() 3 0.1826 0.1826 0.1826 20.76% AmrLevel::checkPoint() 3 0.1488 0.1488 0.1488 16.91% StateData::checkPoint() 12 0.1487 0.1487 0.1487 16.91% MLCellLinOp::smooth() 1640 0.1383 0.1383 0.1383 15.72% Amr::init() 1 0.1372 0.1372 0.1372 15.60% MLCellLinOp::applyBC() 4433 0.09705 0.09705 0.09705 11.03% MLMG::mgVcycle_bottom 82 0.08349 0.08349 0.08349 9.49% MLMG::actualBottomSolve() 82 0.08345 0.08345 0.08345 9.49% MLCGSolver::bicgstab 82 0.08263 0.08263 0.08263 9.39% MLPoisson::Fsmooth() 3280 0.06507 0.06507 0.06507 7.40% Amr::writePlotFile() 2 0.06193 0.06193 0.06193 7.04% Amr::initialInit() 1 0.05241 0.05241 0.05241 5.96% Amr::FinalizeInit() 1 0.04812 0.04812 0.04812 5.47% Castro::post_init() 1 0.04672 0.04672 0.04672 5.31% Castro::clean_state() 62 0.04615 0.04615 0.04615 5.25% Gravity::multilevel_solve_for_new_phi() 1 0.04486 0.04486 0.04486 5.10% Gravity::actual_multilevel_solve() 1 0.04485 0.04485 0.04485 5.10% FillPatchIterator::Initialize 41 0.04409 0.04409 0.04409 5.01% FillPatchSingleLevel 41 0.04248 0.04248 0.04248 4.83% StateDataPhysBCFunct::() 41 0.03841 0.03841 0.03841 4.37% MLCellLinOp::apply() 1142 0.03683 0.03683 0.03683 4.19% MLMG::mgVcycle_down::0 82 0.03587 0.03587 0.03587 4.08% MLMG::mgVcycle_up::0 82 0.03087 0.03087 0.03087 3.51% StateData::FillBoundary(geom) 328 0.02369 0.02369 0.02369 2.69% MultiFab::Dot() 1114 0.02286 0.02286 0.02286 2.60% MLCellLinOp::correctionResidual() 492 0.02164 0.02164 0.02164 2.46% Castro::initialize_do_advance() 10 0.02097 0.02097 0.02097 2.38% Castro::computeTemp() 63 0.02014 0.02014 0.02014 2.29% MLMG:computeResOfCorrection() 410 0.0187 0.0187 0.0187 2.13% MLPoisson::define() 11 0.01845 0.01845 0.01845 2.10% MLMG::mgVcycle_down::1 82 0.01805 0.01805 0.01805 2.05% MLMG::mgVcycle_down::2 82 0.01756 0.01756 0.01756 2.00% Gravity::get_new_grav_vector() 11 0.01719 0.01719 0.01719 1.95% MLMG::mgVcycle_down::3 82 0.01666 0.01666 0.01666 1.89% Castro::normalize_species() 62 0.01644 0.01644 0.01644 1.87% FabArray::FillBoundary() 4023 0.01592 0.01592 0.01592 1.81% MLMG::mgVcycle_down::4 82 0.01588 0.01588 0.01588 1.81% FillBoundary_nowait() 4023 0.01507 0.01507 0.01507 1.71% CGSolver::sxay() 1586 0.01503 0.01503 0.01503 1.71% Castro::construct_old_gravity() 10 0.0149 0.0149 0.0149 1.69% Gravity::get_old_grav_vector() 10 0.01489 0.01489 0.01489 1.69% MultiFab::LinComb() 1586 0.01465 0.01465 0.01465 1.67% FabArray::ParallelCopy() 861 0.01455 0.01455 0.01455 1.65% FabArray::setVal() 1144 0.01451 0.01451 0.01451 1.65% FabArray::ParallelCopy_nowait() 861 0.0143 0.0143 0.0143 1.63% MLCGSolver::ParallelAllReduce 1514 0.01361 0.01361 0.01361 1.55% MLMG::mgVcycle_up::2 82 0.01354 0.01354 0.01354 1.54% MLMG::mgVcycle_up::1 82 0.01334 0.01334 0.01334 1.52% MLMG::addInterpCorrection() 410 0.01313 0.01313 0.01313 1.49% Castro::expand_state() 10 0.0131 0.0131 0.0131 1.49% MLCellLinOp::defineAuxData() 11 0.01301 0.01301 0.01301 1.48% MLMG::mgVcycle_up::3 82 0.01285 0.01285 0.01285 1.46% MLMG::mgVcycle_up::4 82 0.01267 0.01267 0.01267 1.44% amrex::average_down 410 0.01213 0.01213 0.01213 1.38% MLPoisson::Fapply() 1142 0.01201 0.01201 0.01201 1.37% Castro::do_new_sources() 10 0.01187 0.01187 0.01187 1.35% Castro::do_old_sources() 10 0.01162 0.01162 0.01162 1.32% Castro::initialize_advance() 10 0.01133 0.01133 0.01133 1.29% Castro::enforce_min_density() 62 0.009079 0.009079 0.009079 1.03% Gravity::fill_multipole_BCs() 11 0.008912 0.008912 0.008912 1.01% Castro::post_timestep() 10 0.007277 0.007277 0.007277 0.83% MLCellLinOp::solutionResidual() 93 0.007236 0.007236 0.007236 0.82% MultiFab::Xpay() 585 0.006701 0.006701 0.006701 0.76% MLMG::computeResidual() 82 0.006256 0.006256 0.006256 0.71% Castro::reset_internal_energy(MultiFab) 63 0.005752 0.005752 0.005752 0.65% MLMG::prepareForSolve() 11 0.005501 0.005501 0.005501 0.63% MLCellLinOp::defineBC() 11 0.005201 0.005201 0.005201 0.59% BndryData::define() 11 0.004928 0.004928 0.004928 0.56% Castro::estTimeStep() 21 0.004807 0.004807 0.004807 0.55% Amr::InitializeInit() 1 0.004291 0.004291 0.004291 0.49% Amr::defBaseLevel() 1 0.004286 0.004286 0.004286 0.49% Castro::initData() 1 0.003763 0.003763 0.003763 0.43% Castro::construct_new_source() 50 0.003315 0.003315 0.003315 0.38% Castro::construct_new_gravity_source() 10 0.003296 0.003296 0.003296 0.37% Castro::construct_old_source() 50 0.002667 0.002667 0.002667 0.30% Castro::construct_old_gravity_source() 10 0.002649 0.002649 0.002649 0.30% Castro::computeNewDt() 9 0.002198 0.002198 0.002198 0.25% MLMG::ResNormInf() 93 0.002121 0.002121 0.002121 0.24% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.21% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.21% MultiFab::Add() 82 0.001702 0.001702 0.001702 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001684 0.001684 0.001684 0.19% Castro::reset_internal_energy(Fab) 504 0.001628 0.001628 0.001628 0.19% MLCellLinOp::setLevelBC() 11 0.001559 0.001559 0.001559 0.18% FabArrayBase::getCPC() 1323 0.001442 0.001442 0.001442 0.16% MLMG::getGradSolution() 11 0.001437 0.001437 0.001437 0.16% MLCellLinOp::compGrad() 11 0.001431 0.001431 0.001431 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.15% Castro::enforce_speed_limit() 62 0.001319 0.001319 0.001319 0.15% FabArray::setDomainBndry() 41 0.001317 0.001317 0.001317 0.15% Castro::post_regrid() 1 0.001205 0.001205 0.001205 0.14% Castro::check_for_nan() 20 0.001199 0.001199 0.001199 0.14% MLPoisson::prepareForSolve() 11 0.001198 0.001198 0.001198 0.14% MLCellLinOp::prepareForSolve() 11 0.001188 0.001188 0.001188 0.14% MultiFab::contains_nan() 20 0.001186 0.001186 0.001186 0.13% MLMG::computeMLResidual() 11 0.001031 0.001031 0.001031 0.12% Castro::computeInitialDt() 2 0.0008785 0.0008785 0.0008785 0.10% Gravity::update_max_rhs() 11 0.0008293 0.0008293 0.0008293 0.09% FabArrayBase::getFB() 4023 0.0006774 0.0006774 0.0006774 0.08% FabArrayBase::CPC::define() 454 0.0006497 0.0006497 0.0006497 0.07% Castro::finalize_advance() 10 0.0006264 0.0006264 0.0006264 0.07% Amr::InitAmr() 1 0.0004886 0.0004886 0.0004886 0.06% Gravity::swapTimeLevels() 10 0.0004492 0.0004492 0.0004492 0.05% Castro::Castro() 1 0.0004436 0.0004436 0.0004436 0.05% MultiFab::Copy() 11 0.0003359 0.0003359 0.0003359 0.04% MLMG::MLResNormInf() 11 0.000283 0.000283 0.000283 0.03% MultiFab::max() 11 0.0002644 0.0002644 0.0002644 0.03% MLMG::MLRhsNormInf() 11 0.0002209 0.0002209 0.0002209 0.03% MLLinOp::define() 11 0.0002108 0.0002108 0.0002108 0.02% MLLinOp::defineGrids() 11 0.0001873 0.0001873 0.0001873 0.02% Castro::buildMetrics() 1 0.0001665 0.0001665 0.0001665 0.02% FabArrayBase::FB::FB() 56 8.166e-05 8.166e-05 8.166e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.107e-05 5.107e-05 5.107e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.841e-05 4.841e-05 4.841e-05 0.01% Castro::finalize_do_advance() 10 4.221e-05 4.221e-05 4.221e-05 0.00% StateData::define() 4 4.154e-05 4.154e-05 4.154e-05 0.00% Castro::enforce_consistent_e() 1 3.912e-05 3.912e-05 3.912e-05 0.00% makeSFC 55 3.845e-05 3.845e-05 3.845e-05 0.00% Castro::swap_state_time_levels() 10 3.632e-05 3.632e-05 3.632e-05 0.00% Amr::writeSmallPlotFile() 1 2.544e-05 2.544e-05 2.544e-05 0.00% Castro::initMFs() 1 2.497e-05 2.497e-05 2.497e-05 0.00% Castro::create_source_corrector() 10 2.386e-05 2.386e-05 2.386e-05 0.00% DistributionMapping::Distribute() 56 1.41e-05 1.41e-05 1.41e-05 0.00% Amr::initSubcycle() 1 8.269e-06 8.269e-06 8.269e-06 0.00% AmrLevel::checkPointPost() 3 4.717e-06 4.717e-06 4.717e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.39e-06 4.39e-06 4.39e-06 0.00% Castro::retry_advance_ctu() 10 3.695e-06 3.695e-06 3.695e-06 0.00% Gravity::set_mass_offset() 11 3.545e-06 3.545e-06 3.545e-06 0.00% Castro::FluxRegCrseInit 10 2.852e-06 2.852e-06 2.852e-06 0.00% AmrLevel::checkPointPre() 3 2.69e-06 2.69e-06 2.69e-06 0.00% Castro::FluxRegFineAdd() 10 2.324e-06 2.324e-06 2.324e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.817e-06 1.817e-06 1.817e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-20-g46e487c72973) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-20-g46e487c72973) initialized Starting run at 09:37:46 UTC on 2022-11-21. Successfully read inputs file ... Castro git describe: 22.11-13-ga00eb1604 AMReX git describe: 22.11-20-g46e487c72 Microphysics git describe: 22.11-38-g15955539 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.457486803 Restart time = 0.047814479 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052382108 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051219871 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061040732 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063249239 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.081467105 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032385812 seconds Ending run at 09:37:46 UTC on 2022-11-21. Run time = 0.39053856 Run time without initialization = 0.342144467 Average number of zones advanced per microsecond: 3.831 Average number of zones advanced per microsecond per rank: 3.831 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3906 ... 0.3906 ... 0.3906 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1105 0.1105 0.1105 28.29% VisMF::Read() 3 0.03993 0.03993 0.03993 10.22% MLCellLinOp::applyBC() 1946 0.03467 0.03467 0.03467 8.88% VisMF::Write(FabArray) 1 0.03085 0.03085 0.03085 7.90% MLPoisson::Fsmooth() 1440 0.02723 0.02723 0.02723 6.97% StateData::FillBoundary(geom) 160 0.01121 0.01121 0.01121 2.87% MLCGSolver::bicgstab 36 0.01031 0.01031 0.01031 2.64% MultiFab::Dot() 484 0.009603 0.009603 0.009603 2.46% Castro::computeTemp() 30 0.008664 0.008664 0.008664 2.22% Castro::normalize_species() 30 0.006996 0.006996 0.006996 1.79% FabArray::setVal() 537 0.00683 0.00683 0.00683 1.75% FillBoundary_nowait() 1766 0.006219 0.006219 0.006219 1.59% MLCellLinOp::defineAuxData() 6 0.006174 0.006174 0.006174 1.58% MultiFab::LinComb() 690 0.006137 0.006137 0.006137 1.57% FabArray::ParallelCopy_nowait() 380 0.006024 0.006024 0.006024 1.54% StateDataPhysBCFunct::() 20 0.005793 0.005793 0.005793 1.48% Castro::enforce_min_density() 30 0.005593 0.005593 0.005593 1.43% MLPoisson::Fapply() 500 0.005099 0.005099 0.005099 1.31% Gravity::fill_multipole_BCs() 6 0.004691 0.004691 0.004691 1.20% Amr::restart() 1 0.003602 0.003602 0.003602 0.92% MLMG::addInterpCorrection() 180 0.003377 0.003377 0.003377 0.86% amrex::average_down 180 0.002952 0.002952 0.002952 0.76% MultiFab::Xpay() 258 0.002893 0.002893 0.002893 0.74% Castro::do_advance_ctu() 5 0.002632 0.002632 0.002632 0.67% Castro::estTimeStep() 10 0.002464 0.002464 0.002464 0.63% BndryData::define() 6 0.002129 0.002129 0.002129 0.55% Castro::construct_new_gravity_source() 5 0.001754 0.001754 0.001754 0.45% Castro::reset_internal_energy(MultiFab) 30 0.001733 0.001733 0.001733 0.44% Amr::writePlotFile() 1 0.001615 0.001615 0.001615 0.41% Castro::construct_old_gravity_source() 5 0.001299 0.001299 0.001299 0.33% Castro::enforce_speed_limit() 30 0.001249 0.001249 0.001249 0.32% Castro::reset_internal_energy(Fab) 240 0.0009616 0.0009616 0.0009616 0.25% Gravity::get_old_grav_vector() 5 0.0009595 0.0009595 0.0009595 0.25% Castro::subcycle_advance_ctu() 5 0.0009178 0.0009178 0.0009178 0.23% MLMG::ResNormInf() 42 0.0009173 0.0009173 0.0009173 0.23% MultiFab::Saxpy() 10 0.0009166 0.0009166 0.0009166 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008835 0.0008835 0.0008835 0.23% Gravity::get_new_grav_vector() 5 0.0008744 0.0008744 0.0008744 0.22% Castro::expand_state() 5 0.0008736 0.0008736 0.0008736 0.22% MLCellLinOp::setLevelBC() 6 0.000813 0.000813 0.000813 0.21% Gravity::actual_solve_with_mlmg() 6 0.000728 0.000728 0.000728 0.19% MultiFab::Add() 36 0.0007264 0.0007264 0.0007264 0.19% FabArray::mult() 22 0.000662 0.000662 0.000662 0.17% MLMG::prepareForSolve() 6 0.0006586 0.0006586 0.0006586 0.17% FabArray::setDomainBndry() 20 0.0006407 0.0006407 0.0006407 0.16% MLCellLinOp::prepareForSolve() 6 0.0006394 0.0006394 0.0006394 0.16% MultiFab::contains_nan() 10 0.0005853 0.0005853 0.0005853 0.15% MLCellLinOp::smooth() 720 0.0004899 0.0004899 0.0004899 0.13% MLCellLinOp::compGrad() 6 0.0004874 0.0004874 0.0004874 0.12% Amr::InitAmr() 1 0.0004011 0.0004011 0.0004011 0.10% FabArray::FillBoundary() 1766 0.0003913 0.0003913 0.0003913 0.10% FabArrayBase::CPC::define() 244 0.0003742 0.0003742 0.0003742 0.10% FabArrayBase::getCPC() 632 0.0003733 0.0003733 0.0003733 0.10% Castro::finalize_advance() 5 0.0003288 0.0003288 0.0003288 0.08% main() 1 0.0002625 0.0002625 0.0002625 0.07% FabArrayBase::getFB() 1766 0.0002493 0.0002493 0.0002493 0.06% Gravity::update_max_rhs() 6 0.0002275 0.0002275 0.0002275 0.06% MLCellLinOp::apply() 500 0.0002107 0.0002107 0.0002107 0.05% Gravity::solve_for_phi() 5 0.0001905 0.0001905 0.0001905 0.05% MultiFab::Copy() 6 0.0001796 0.0001796 0.0001796 0.05% CGSolver::sxay() 690 0.0001796 0.0001796 0.0001796 0.05% Castro::construct_new_gravity() 5 0.0001791 0.0001791 0.0001791 0.05% Amr::coarseTimeStep() 5 0.0001769 0.0001769 0.0001769 0.05% MLCellLinOp::defineBC() 6 0.0001429 0.0001429 0.0001429 0.04% MultiFab::max() 6 0.0001363 0.0001363 0.0001363 0.03% MLCGSolver::ParallelAllReduce 659 0.0001336 0.0001336 0.0001336 0.03% FillPatchIterator::Initialize 20 0.0001314 0.0001314 0.0001314 0.03% Castro::construct_new_source() 25 0.000129 0.000129 0.000129 0.03% MLLinOp::defineGrids() 6 0.0001238 0.0001238 0.0001238 0.03% FabArray::ParallelCopy() 380 0.0001227 0.0001227 0.0001227 0.03% MLMG::MLRhsNormInf() 6 0.0001143 0.0001143 0.0001143 0.03% MLMG::mgVcycle() 36 0.0001065 0.0001065 0.0001065 0.03% MLCellLinOp::correctionResidual() 216 9.853e-05 9.853e-05 9.853e-05 0.03% AmrLevel::restart() 1 8.703e-05 8.703e-05 8.703e-05 0.02% StateData::restartDoit() 4 7.998e-05 7.998e-05 7.998e-05 0.02% Amr::timeStep() 5 7.49e-05 7.49e-05 7.49e-05 0.02% Castro::create_source_corrector() 5 7.393e-05 7.393e-05 7.393e-05 0.02% Castro::advance() 5 6.98e-05 6.98e-05 6.98e-05 0.02% MLMG:computeResOfCorrection() 180 6.979e-05 6.979e-05 6.979e-05 0.02% FabArrayBase::FB::FB() 26 5.447e-05 5.447e-05 5.447e-05 0.01% MLMG::mgVcycle_down::0 36 5.357e-05 5.357e-05 5.357e-05 0.01% MLMG::mgVcycle_down::1 36 4.898e-05 4.898e-05 4.898e-05 0.01% MLMG::mgVcycle_down::2 36 4.658e-05 4.658e-05 4.658e-05 0.01% Castro::initialize_do_advance() 5 4.518e-05 4.518e-05 4.518e-05 0.01% Castro::post_timestep() 5 4.475e-05 4.475e-05 4.475e-05 0.01% MLMG::mgVcycle_down::4 36 4.283e-05 4.283e-05 4.283e-05 0.01% MLMG::mgVcycle_down::3 36 4.241e-05 4.241e-05 4.241e-05 0.01% Castro::clean_state() 30 4.018e-05 4.018e-05 4.018e-05 0.01% Castro::initialize_advance() 5 3.754e-05 3.754e-05 3.754e-05 0.01% Castro::construct_old_source() 25 3.735e-05 3.735e-05 3.735e-05 0.01% MLMG::actualBottomSolve() 36 3.716e-05 3.716e-05 3.716e-05 0.01% MLMG::mgVcycle_up::4 36 3.448e-05 3.448e-05 3.448e-05 0.01% Castro::buildMetrics() 1 3.27e-05 3.27e-05 3.27e-05 0.01% MLMG::solve() 6 3.205e-05 3.205e-05 3.205e-05 0.01% Castro::post_restart() 1 3.056e-05 3.056e-05 3.056e-05 0.01% Gravity::actual_multilevel_solve() 1 3.012e-05 3.012e-05 3.012e-05 0.01% MLMG::oneIter() 36 2.869e-05 2.869e-05 2.869e-05 0.01% MLMG::mgVcycle_up::3 36 2.763e-05 2.763e-05 2.763e-05 0.01% MLMG::mgVcycle_up::0 36 2.719e-05 2.719e-05 2.719e-05 0.01% Castro::initMFs() 1 2.703e-05 2.703e-05 2.703e-05 0.01% MLMG::mgVcycle_up::2 36 2.668e-05 2.668e-05 2.668e-05 0.01% Amr::writeSmallPlotFile() 1 2.631e-05 2.631e-05 2.631e-05 0.01% MLMG::mgVcycle_up::1 36 2.574e-05 2.574e-05 2.574e-05 0.01% Castro::swap_state_time_levels() 5 2.481e-05 2.481e-05 2.481e-05 0.01% MLMG::computeResidual() 36 2.211e-05 2.211e-05 2.211e-05 0.01% MLCellLinOp::solutionResidual() 42 2.183e-05 2.183e-05 2.183e-05 0.01% Castro::computeNewDt() 5 2.167e-05 2.167e-05 2.167e-05 0.01% Castro::construct_old_gravity() 5 2.143e-05 2.143e-05 2.143e-05 0.01% MLLinOp::define() 6 2.016e-05 2.016e-05 2.016e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.692e-05 1.692e-05 1.692e-05 0.00% Castro::finalize_do_advance() 5 1.643e-05 1.643e-05 1.643e-05 0.00% MLPoisson::define() 6 1.544e-05 1.544e-05 1.544e-05 0.00% MLMG::mgVcycle_bottom 36 1.514e-05 1.514e-05 1.514e-05 0.00% FillPatchSingleLevel 20 1.387e-05 1.387e-05 1.387e-05 0.00% makeSFC 30 1.303e-05 1.303e-05 1.303e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.233e-05 1.233e-05 1.233e-05 0.00% Castro::do_new_sources() 5 9.356e-06 9.356e-06 9.356e-06 0.00% Amr::initSubcycle() 1 8.482e-06 8.482e-06 8.482e-06 0.00% DistributionMapping::Distribute() 31 8.229e-06 8.229e-06 8.229e-06 0.00% Castro::do_old_sources() 5 7.955e-06 7.955e-06 7.955e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.26e-06 7.26e-06 7.26e-06 0.00% Castro::check_for_nan() 10 6.266e-06 6.266e-06 6.266e-06 0.00% Castro::apply_source_to_state() 10 6.06e-06 6.06e-06 6.06e-06 0.00% MLPoisson::prepareForSolve() 6 4.989e-06 4.989e-06 4.989e-06 0.00% MLMG::computeMLResidual() 6 4.77e-06 4.77e-06 4.77e-06 0.00% Gravity::swapTimeLevels() 5 4.216e-06 4.216e-06 4.216e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.985e-06 3.985e-06 3.985e-06 0.00% MLMG::getGradSolution() 6 2.872e-06 2.872e-06 2.872e-06 0.00% Gravity::set_mass_offset() 6 2.042e-06 2.042e-06 2.042e-06 0.00% MLMG::MLResNormInf() 6 1.968e-06 1.968e-06 1.968e-06 0.00% Castro::retry_advance_ctu() 5 1.842e-06 1.842e-06 1.842e-06 0.00% Castro::FluxRegCrseInit 5 1.428e-06 1.428e-06 1.428e-06 0.00% Amr::init() 1 1.184e-06 1.184e-06 1.184e-06 0.00% Castro::FluxRegFineAdd() 5 1.163e-06 1.163e-06 1.163e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.099e-06 1.099e-06 1.099e-06 0.00% AmrLevel::AmrLevel() 1 8.8e-07 8.8e-07 8.8e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3905 0.3905 0.3905 100.00% Amr::coarseTimeStep() 5 0.3095 0.3095 0.3095 79.25% Amr::timeStep() 5 0.3081 0.3081 0.3081 78.88% Castro::advance() 5 0.3039 0.3039 0.3039 77.82% Castro::subcycle_advance_ctu() 5 0.2979 0.2979 0.2979 76.27% Castro::do_advance_ctu() 5 0.297 0.297 0.297 76.04% Castro::construct_new_gravity() 5 0.144 0.144 0.144 36.86% Gravity::solve_phi_with_mlmg() 6 0.14 0.14 0.14 35.84% Gravity::solve_for_phi() 5 0.1364 0.1364 0.1364 34.92% Gravity::actual_solve_with_mlmg() 6 0.1352 0.1352 0.1352 34.60% MLMG::solve() 6 0.123 0.123 0.123 31.48% MLMG::oneIter() 36 0.1157 0.1157 0.1157 29.63% MLMG::mgVcycle() 36 0.115 0.115 0.115 29.44% Castro::construct_ctu_hydro_source() 5 0.1105 0.1105 0.1105 28.29% MLCellLinOp::smooth() 720 0.05854 0.05854 0.05854 14.99% Amr::init() 1 0.04786 0.04786 0.04786 12.25% Amr::restart() 1 0.04785 0.04785 0.04785 12.25% MLCellLinOp::applyBC() 1946 0.04158 0.04158 0.04158 10.65% AmrLevel::restart() 1 0.04015 0.04015 0.04015 10.28% StateData::restartDoit() 4 0.04006 0.04006 0.04006 10.26% VisMF::Read() 3 0.03993 0.03993 0.03993 10.22% MLMG::mgVcycle_bottom 36 0.0353 0.0353 0.0353 9.04% MLMG::actualBottomSolve() 36 0.03529 0.03529 0.03529 9.03% MLCGSolver::bicgstab 36 0.03493 0.03493 0.03493 8.94% Amr::writePlotFile() 1 0.03246 0.03246 0.03246 8.31% VisMF::Write(FabArray) 1 0.03085 0.03085 0.03085 7.90% MLPoisson::Fsmooth() 1440 0.02723 0.02723 0.02723 6.97% Castro::clean_state() 30 0.02524 0.02524 0.02524 6.46% FillPatchIterator::Initialize 20 0.01979 0.01979 0.01979 5.07% FillPatchSingleLevel 20 0.01902 0.01902 0.01902 4.87% StateDataPhysBCFunct::() 20 0.017 0.017 0.017 4.35% MLCellLinOp::apply() 500 0.01579 0.01579 0.01579 4.04% MLMG::mgVcycle_down::0 36 0.01534 0.01534 0.01534 3.93% MLMG::mgVcycle_up::0 36 0.01311 0.01311 0.01311 3.36% Castro::computeTemp() 30 0.01136 0.01136 0.01136 2.91% StateData::FillBoundary(geom) 160 0.01121 0.01121 0.01121 2.87% Castro::initialize_do_advance() 5 0.01008 0.01008 0.01008 2.58% MLPoisson::define() 6 0.009893 0.009893 0.009893 2.53% MultiFab::Dot() 484 0.009603 0.009603 0.009603 2.46% MLCellLinOp::correctionResidual() 216 0.009251 0.009251 0.009251 2.37% Castro::do_new_sources() 5 0.008064 0.008064 0.008064 2.06% MLMG:computeResOfCorrection() 180 0.007988 0.007988 0.007988 2.05% MLMG::mgVcycle_down::1 36 0.00768 0.00768 0.00768 1.97% MLMG::mgVcycle_down::2 36 0.007448 0.007448 0.007448 1.91% Castro::construct_old_gravity() 5 0.007374 0.007374 0.007374 1.89% Gravity::get_new_grav_vector() 5 0.007369 0.007369 0.007369 1.89% Gravity::get_old_grav_vector() 5 0.007352 0.007352 0.007352 1.88% MLMG::mgVcycle_down::3 36 0.007052 0.007052 0.007052 1.81% Castro::normalize_species() 30 0.006996 0.006996 0.006996 1.79% FabArray::FillBoundary() 1766 0.006914 0.006914 0.006914 1.77% MLCellLinOp::defineAuxData() 6 0.006901 0.006901 0.006901 1.77% FabArray::setVal() 537 0.00683 0.00683 0.00683 1.75% MLMG::mgVcycle_down::4 36 0.006728 0.006728 0.006728 1.72% FillBoundary_nowait() 1766 0.006523 0.006523 0.006523 1.67% FabArray::ParallelCopy() 380 0.006518 0.006518 0.006518 1.67% FabArray::ParallelCopy_nowait() 380 0.006395 0.006395 0.006395 1.64% CGSolver::sxay() 690 0.006317 0.006317 0.006317 1.62% Castro::do_old_sources() 5 0.006192 0.006192 0.006192 1.59% MultiFab::LinComb() 690 0.006137 0.006137 0.006137 1.57% MLCGSolver::ParallelAllReduce 659 0.005756 0.005756 0.005756 1.47% MLMG::mgVcycle_up::2 36 0.005733 0.005733 0.005733 1.47% MLMG::mgVcycle_up::1 36 0.005636 0.005636 0.005636 1.44% Castro::expand_state() 5 0.005629 0.005629 0.005629 1.44% Castro::initialize_advance() 5 0.00562 0.00562 0.00562 1.44% MLMG::addInterpCorrection() 180 0.005609 0.005609 0.005609 1.44% Castro::enforce_min_density() 30 0.005593 0.005593 0.005593 1.43% MLMG::mgVcycle_up::3 36 0.005429 0.005429 0.005429 1.39% MLMG::mgVcycle_up::4 36 0.005406 0.005406 0.005406 1.38% amrex::average_down 180 0.005231 0.005231 0.005231 1.34% MLPoisson::Fapply() 500 0.005099 0.005099 0.005099 1.31% Gravity::fill_multipole_BCs() 6 0.004691 0.004691 0.004691 1.20% Castro::post_timestep() 5 0.00408 0.00408 0.00408 1.04% Castro::post_restart() 1 0.003921 0.003921 0.003921 1.00% Gravity::multilevel_solve_for_new_phi() 1 0.003795 0.003795 0.003795 0.97% Gravity::actual_multilevel_solve() 1 0.003778 0.003778 0.003778 0.97% MLCellLinOp::solutionResidual() 42 0.003216 0.003216 0.003216 0.82% MLMG::prepareForSolve() 6 0.002921 0.002921 0.002921 0.75% MultiFab::Xpay() 258 0.002893 0.002893 0.002893 0.74% MLCellLinOp::defineBC() 6 0.002805 0.002805 0.002805 0.72% Castro::reset_internal_energy(MultiFab) 30 0.002695 0.002695 0.002695 0.69% MLMG::computeResidual() 36 0.002685 0.002685 0.002685 0.69% BndryData::define() 6 0.002662 0.002662 0.002662 0.68% Castro::estTimeStep() 10 0.002464 0.002464 0.002464 0.63% Castro::construct_new_source() 25 0.001883 0.001883 0.001883 0.48% Castro::construct_new_gravity_source() 5 0.001754 0.001754 0.001754 0.45% Castro::construct_old_source() 25 0.001336 0.001336 0.001336 0.34% Castro::construct_old_gravity_source() 5 0.001299 0.001299 0.001299 0.33% Castro::computeNewDt() 5 0.001274 0.001274 0.001274 0.33% Castro::enforce_speed_limit() 30 0.001249 0.001249 0.001249 0.32% Castro::reset_internal_energy(Fab) 240 0.0009616 0.0009616 0.0009616 0.25% Castro::apply_source_to_state() 10 0.0009226 0.0009226 0.0009226 0.24% MLMG::ResNormInf() 42 0.0009173 0.0009173 0.0009173 0.23% MultiFab::Saxpy() 10 0.0009166 0.0009166 0.0009166 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008835 0.0008835 0.0008835 0.23% MLCellLinOp::setLevelBC() 6 0.000813 0.000813 0.000813 0.21% MLMG::getGradSolution() 6 0.0007636 0.0007636 0.0007636 0.20% MLCellLinOp::compGrad() 6 0.0007607 0.0007607 0.0007607 0.19% FabArrayBase::getCPC() 632 0.0007475 0.0007475 0.0007475 0.19% MultiFab::Add() 36 0.0007264 0.0007264 0.0007264 0.19% FabArray::mult() 22 0.000662 0.000662 0.000662 0.17% MLPoisson::prepareForSolve() 6 0.0006444 0.0006444 0.0006444 0.16% FabArray::setDomainBndry() 20 0.0006407 0.0006407 0.0006407 0.16% MLCellLinOp::prepareForSolve() 6 0.0006394 0.0006394 0.0006394 0.16% Castro::check_for_nan() 10 0.0005915 0.0005915 0.0005915 0.15% MultiFab::contains_nan() 10 0.0005853 0.0005853 0.0005853 0.15% MLMG::computeMLResidual() 6 0.0005584 0.0005584 0.0005584 0.14% Gravity::update_max_rhs() 6 0.00045 0.00045 0.00045 0.12% Amr::InitAmr() 1 0.0004095 0.0004095 0.0004095 0.10% FabArrayBase::CPC::define() 244 0.0003742 0.0003742 0.0003742 0.10% Castro::finalize_advance() 5 0.0003314 0.0003314 0.0003314 0.08% FabArrayBase::getFB() 1766 0.0003038 0.0003038 0.0003038 0.08% Gravity::swapTimeLevels() 5 0.0002239 0.0002239 0.0002239 0.06% MultiFab::Copy() 6 0.0001796 0.0001796 0.0001796 0.05% MLLinOp::define() 6 0.0001721 0.0001721 0.0001721 0.04% MLLinOp::defineGrids() 6 0.000152 0.000152 0.000152 0.04% Castro::buildMetrics() 1 0.0001512 0.0001512 0.0001512 0.04% MLMG::MLResNormInf() 6 0.0001478 0.0001478 0.0001478 0.04% MultiFab::max() 6 0.0001363 0.0001363 0.0001363 0.03% MLMG::MLRhsNormInf() 6 0.0001143 0.0001143 0.0001143 0.03% Castro::create_source_corrector() 5 7.393e-05 7.393e-05 7.393e-05 0.02% FabArrayBase::FB::FB() 26 5.447e-05 5.447e-05 5.447e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.703e-05 2.703e-05 2.703e-05 0.01% Castro::initMFs() 1 2.703e-05 2.703e-05 2.703e-05 0.01% Amr::writeSmallPlotFile() 1 2.631e-05 2.631e-05 2.631e-05 0.01% Castro::swap_state_time_levels() 5 2.481e-05 2.481e-05 2.481e-05 0.01% makeSFC 30 1.977e-05 1.977e-05 1.977e-05 0.01% Castro::finalize_do_advance() 5 1.643e-05 1.643e-05 1.643e-05 0.00% Amr::initSubcycle() 1 8.482e-06 8.482e-06 8.482e-06 0.00% DistributionMapping::Distribute() 31 8.229e-06 8.229e-06 8.229e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.466e-06 5.466e-06 5.466e-06 0.00% Gravity::set_mass_offset() 6 2.042e-06 2.042e-06 2.042e-06 0.00% Castro::retry_advance_ctu() 5 1.842e-06 1.842e-06 1.842e-06 0.00% Castro::FluxRegCrseInit 5 1.428e-06 1.428e-06 1.428e-06 0.00% Castro::FluxRegFineAdd() 5 1.163e-06 1.163e-06 1.163e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.099e-06 1.099e-06 1.099e-06 0.00% AmrLevel::AmrLevel() 1 8.8e-07 8.8e-07 8.8e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-20-g46e487c72973) finalized