Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-37-g3cc4a66759c9) initialized Starting run at 08:22:12 UTC on 2022-04-28. Successfully read inputs file ... Castro git describe: 22.04-37-gb93839ec1 AMReX git describe: 22.04-37-g3cc4a6675 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.041495749 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.023882841 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048735432 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050612319 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.06646853 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059264351 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.053734469 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.038103554 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.058180255 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.060034618 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060046676 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.047528729 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065004 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.038033637 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024184608 seconds Ending run at 08:22:12 UTC on 2022-04-28. Run time = 0.786431189 Run time without initialization = 0.670385567 Average number of zones advanced per microsecond: 3.910 Average number of zones advanced per microsecond per rank: 3.910 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7865 ... 0.7865 ... 0.7865 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1831 0.1831 0.1831 23.28% VisMF::Write(FabArray) 11 0.1593 0.1593 0.1593 20.25% MLCellLinOp::applyBC() 4433 0.07411 0.07411 0.07411 9.42% MLPoisson::Fsmooth() 3280 0.06071 0.06071 0.06071 7.72% StateData::FillBoundary(geom) 328 0.02434 0.02434 0.02434 3.10% MLCGSolver::bicgstab 82 0.02169 0.02169 0.02169 2.76% MultiFab::Dot() 1114 0.02143 0.02143 0.02143 2.73% Castro::normalize_species() 62 0.02045 0.02045 0.02045 2.60% StateDataPhysBCFunct::() 41 0.01691 0.01691 0.01691 2.15% Castro::computeTemp() 63 0.01571 0.01571 0.01571 2.00% FillBoundary_nowait() 4023 0.0143 0.0143 0.0143 1.82% Castro::enforce_min_density() 62 0.01361 0.01361 0.01361 1.73% FabArray::setVal() 1144 0.01345 0.01345 0.01345 1.71% FabArray::ParallelCopy_nowait() 861 0.01341 0.01341 0.01341 1.70% MultiFab::LinComb() 1586 0.01263 0.01263 0.01263 1.61% MLPoisson::Fapply() 1142 0.01078 0.01078 0.01078 1.37% MLCellLinOp::defineAuxData() 11 0.01006 0.01006 0.01006 1.28% Gravity::fill_multipole_BCs() 11 0.008091 0.008091 0.008091 1.03% Castro::estTimeStep() 21 0.007381 0.007381 0.007381 0.94% Castro::reset_internal_energy() 63 0.006992 0.006992 0.006992 0.89% MLMG::addInterpCorrection() 410 0.006809 0.006809 0.006809 0.87% MultiFab::Xpay() 585 0.006212 0.006212 0.006212 0.79% amrex::average_down 410 0.006201 0.006201 0.006201 0.79% Castro::do_advance_ctu() 10 0.00514 0.00514 0.00514 0.65% Amr::checkPoint() 3 0.003996 0.003996 0.003996 0.51% BndryData::define() 11 0.003813 0.003813 0.003813 0.48% Castro::construct_new_gravity_source() 10 0.003152 0.003152 0.003152 0.40% Amr::writePlotFile() 2 0.002442 0.002442 0.002442 0.31% Castro::construct_old_gravity_source() 10 0.0024 0.0024 0.0024 0.31% Castro::enforce_speed_limit() 62 0.002354 0.002354 0.002354 0.30% Gravity::get_new_grav_vector() 11 0.00194 0.00194 0.00194 0.25% MLMG::ResNormInf() 93 0.001907 0.001907 0.001907 0.24% MultiFab::Saxpy() 20 0.001824 0.001824 0.001824 0.23% Castro::expand_state() 10 0.001745 0.001745 0.001745 0.22% Gravity::get_old_grav_vector() 10 0.001726 0.001726 0.001726 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001686 0.001686 0.001686 0.21% MLMG::oneIter() 82 0.001666 0.001666 0.001666 0.21% Gravity::actual_solve_with_mlmg() 11 0.001347 0.001347 0.001347 0.17% MLCellLinOp::setLevelBC() 11 0.001345 0.001345 0.001345 0.17% FabArray::mult() 43 0.001334 0.001334 0.001334 0.17% FabArray::setDomainBndry() 41 0.001301 0.001301 0.001301 0.17% Castro::initData() 1 0.001191 0.001191 0.001191 0.15% MultiFab::contains_nan() 20 0.001168 0.001168 0.001168 0.15% MLCellLinOp::prepareForSolve() 11 0.001098 0.001098 0.001098 0.14% MLCellLinOp::smooth() 1640 0.001087 0.001087 0.001087 0.14% MLCellLinOp::compGrad() 11 0.0009151 0.0009151 0.0009151 0.12% FabArray::FillBoundary() 4023 0.0008424 0.0008424 0.0008424 0.11% FabArrayBase::getCPC() 1323 0.0008004 0.0008004 0.0008004 0.10% MLMG::prepareForSolve() 11 0.0007806 0.0007806 0.0007806 0.10% FabArrayBase::CPC::define() 454 0.0006898 0.0006898 0.0006898 0.09% FabArrayBase::getFB() 4023 0.0006375 0.0006375 0.0006375 0.08% MLCellLinOp::apply() 1142 0.0004767 0.0004767 0.0004767 0.06% CGSolver::sxay() 1586 0.0004494 0.0004494 0.0004494 0.06% Amr::InitAmr() 1 0.0004424 0.0004424 0.0004424 0.06% Gravity::update_max_rhs() 11 0.0004064 0.0004064 0.0004064 0.05% Gravity::solve_for_phi() 10 0.0003854 0.0003854 0.0003854 0.05% MLMG::mgVcycle() 82 0.0003792 0.0003792 0.0003792 0.05% MLLinOp::defineGrids() 11 0.0003483 0.0003483 0.0003483 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003455 0.0003455 0.0003455 0.04% main() 1 0.0002955 0.0002955 0.0002955 0.04% FabArray::ParallelCopy() 861 0.0002749 0.0002749 0.0002749 0.03% MultiFab::Copy() 11 0.0002637 0.0002637 0.0002637 0.03% MultiFab::max() 11 0.0002556 0.0002556 0.0002556 0.03% FillPatchIterator::Initialize 41 0.0002511 0.0002511 0.0002511 0.03% Castro::construct_new_gravity() 10 0.0002087 0.0002087 0.0002087 0.03% MLCellLinOp::correctionResidual() 492 0.0002085 0.0002085 0.0002085 0.03% MLMG::MLRhsNormInf() 11 0.0002015 0.0002015 0.0002015 0.03% Castro::subcycle_advance_ctu() 10 0.0002006 0.0002006 0.0002006 0.03% MLCellLinOp::defineBC() 11 0.000191 0.000191 0.000191 0.02% Amr::coarseTimeStep() 10 0.0001905 0.0001905 0.0001905 0.02% Amr::timeStep() 10 0.0001741 0.0001741 0.0001741 0.02% Amr::defBaseLevel() 1 0.0001378 0.0001378 0.0001378 0.02% MLMG:computeResOfCorrection() 410 0.0001353 0.0001353 0.0001353 0.02% StateData::checkPoint() 12 0.0001291 0.0001291 0.0001291 0.02% MLMG::actualBottomSolve() 82 0.0001049 0.0001049 0.0001049 0.01% MLMG::mgVcycle_down::0 82 9.279e-05 9.279e-05 9.279e-05 0.01% FabArrayBase::FB::FB() 56 8.678e-05 8.678e-05 8.678e-05 0.01% Castro::finalize_advance() 10 8.459e-05 8.459e-05 8.459e-05 0.01% Castro::advance() 10 8.324e-05 8.324e-05 8.324e-05 0.01% MLMG::solve() 11 7.658e-05 7.658e-05 7.658e-05 0.01% MLMG::mgVcycle_down::1 82 7.596e-05 7.596e-05 7.596e-05 0.01% Castro::initialize_advance() 10 7.56e-05 7.56e-05 7.56e-05 0.01% MLMG::mgVcycle_down::2 82 7.555e-05 7.555e-05 7.555e-05 0.01% Castro::clean_state() 62 7.356e-05 7.356e-05 7.356e-05 0.01% MLMG::mgVcycle_down::4 82 7.329e-05 7.329e-05 7.329e-05 0.01% MLMG::mgVcycle_down::3 82 7.212e-05 7.212e-05 7.212e-05 0.01% AmrLevel::checkPoint() 3 7.168e-05 7.168e-05 7.168e-05 0.01% MLMG::mgVcycle_up::4 82 6.385e-05 6.385e-05 6.385e-05 0.01% Castro::initialize_do_advance() 10 5.532e-05 5.532e-05 5.532e-05 0.01% MLMG::mgVcycle_up::0 82 5.156e-05 5.156e-05 5.156e-05 0.01% Castro::post_timestep() 10 5.153e-05 5.153e-05 5.153e-05 0.01% MLMG::mgVcycle_up::3 82 4.835e-05 4.835e-05 4.835e-05 0.01% MLCellLinOp::solutionResidual() 93 4.814e-05 4.814e-05 4.814e-05 0.01% MLMG::mgVcycle_up::1 82 4.747e-05 4.747e-05 4.747e-05 0.01% MLMG::mgVcycle_up::2 82 4.611e-05 4.611e-05 4.611e-05 0.01% StateData::define() 4 4.251e-05 4.251e-05 4.251e-05 0.01% Castro::enforce_consistent_e() 1 3.661e-05 3.661e-05 3.661e-05 0.00% Castro::swap_state_time_levels() 10 3.638e-05 3.638e-05 3.638e-05 0.00% MLMG::mgVcycle_bottom 82 3.487e-05 3.487e-05 3.487e-05 0.00% Castro::finalize_do_advance() 10 3.379e-05 3.379e-05 3.379e-05 0.00% MLMG::computeResidual() 82 3.265e-05 3.265e-05 3.265e-05 0.00% Gravity::actual_multilevel_solve() 1 3.055e-05 3.055e-05 3.055e-05 0.00% FillPatchSingleLevel 41 2.82e-05 2.82e-05 2.82e-05 0.00% makeSFC 55 2.689e-05 2.689e-05 2.689e-05 0.00% Amr::writeSmallPlotFile() 1 2.46e-05 2.46e-05 2.46e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.451e-05 2.451e-05 2.451e-05 0.00% MLLinOp::define() 11 2.255e-05 2.255e-05 2.255e-05 0.00% MLPoisson::define() 11 2.093e-05 2.093e-05 2.093e-05 0.00% Amr::FinalizeInit() 1 1.992e-05 1.992e-05 1.992e-05 0.00% Castro::construct_old_source() 50 1.955e-05 1.955e-05 1.955e-05 0.00% Castro::construct_new_source() 50 1.785e-05 1.785e-05 1.785e-05 0.00% Castro::do_new_sources() 10 1.675e-05 1.675e-05 1.675e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.607e-05 1.607e-05 1.607e-05 0.00% Castro::do_old_sources() 10 1.553e-05 1.553e-05 1.553e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.523e-05 1.523e-05 1.523e-05 0.00% DistributionMapping::Distribute() 56 1.467e-05 1.467e-05 1.467e-05 0.00% Castro::check_for_nan() 20 1.147e-05 1.147e-05 1.147e-05 0.00% Castro::apply_source_to_state() 20 1.106e-05 1.106e-05 1.106e-05 0.00% Amr::InitializeInit() 1 1.097e-05 1.097e-05 1.097e-05 0.00% Castro::construct_old_gravity() 10 1.093e-05 1.093e-05 1.093e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.041e-05 1.041e-05 1.041e-05 0.00% Amr::initSubcycle() 1 9.503e-06 9.503e-06 9.503e-06 0.00% Gravity::swapTimeLevels() 10 8.872e-06 8.872e-06 8.872e-06 0.00% MLPoisson::prepareForSolve() 11 8.807e-06 8.807e-06 8.807e-06 0.00% MLMG::computeMLResidual() 11 7.339e-06 7.339e-06 7.339e-06 0.00% Castro::create_source_corrector() 10 5.94e-06 5.94e-06 5.94e-06 0.00% MLMG::buildFineMask() 11 5.921e-06 5.921e-06 5.921e-06 0.00% Castro::computeNewDt() 9 5.819e-06 5.819e-06 5.819e-06 0.00% MLMG::getGradSolution() 11 5.558e-06 5.558e-06 5.558e-06 0.00% AmrLevel::checkPointPost() 3 5.164e-06 5.164e-06 5.164e-06 0.00% MLMG::MLResNormInf() 11 5.081e-06 5.081e-06 5.081e-06 0.00% Gravity::set_mass_offset() 11 4.096e-06 4.096e-06 4.096e-06 0.00% Castro::post_init() 1 3.777e-06 3.777e-06 3.777e-06 0.00% Castro::retry_advance_ctu() 10 3.658e-06 3.658e-06 3.658e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.169e-06 3.169e-06 3.169e-06 0.00% Amr::init() 1 3.011e-06 3.011e-06 3.011e-06 0.00% Castro::FluxRegFineAdd() 10 2.654e-06 2.654e-06 2.654e-06 0.00% Castro::computeInitialDt() 2 2.277e-06 2.277e-06 2.277e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.204e-06 2.204e-06 2.204e-06 0.00% AmrLevel::checkPointPre() 3 2.009e-06 2.009e-06 2.009e-06 0.00% Castro::post_regrid() 1 1.294e-06 1.294e-06 1.294e-06 0.00% Amr::initialInit() 1 1.042e-06 1.042e-06 1.042e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7864 0.7864 0.7864 100.00% Amr::coarseTimeStep() 10 0.646 0.646 0.646 82.14% Amr::timeStep() 10 0.5658 0.5658 0.5658 71.94% Castro::advance() 10 0.5545 0.5545 0.5545 70.50% Castro::subcycle_advance_ctu() 10 0.5406 0.5406 0.5406 68.74% Castro::do_advance_ctu() 10 0.5404 0.5404 0.5404 68.72% Gravity::solve_phi_with_mlmg() 11 0.2944 0.2944 0.2944 37.43% Gravity::actual_solve_with_mlmg() 11 0.286 0.286 0.286 36.37% Castro::construct_new_gravity() 10 0.2669 0.2669 0.2669 33.93% MLMG::solve() 11 0.2651 0.2651 0.2651 33.71% Gravity::solve_for_phi() 10 0.2517 0.2517 0.2517 32.00% MLMG::oneIter() 82 0.2512 0.2512 0.2512 31.94% MLMG::mgVcycle() 82 0.2495 0.2495 0.2495 31.73% Castro::construct_ctu_hydro_source() 10 0.1831 0.1831 0.1831 23.28% VisMF::Write(FabArray) 11 0.1593 0.1593 0.1593 20.25% MLCellLinOp::smooth() 1640 0.1285 0.1285 0.1285 16.34% Amr::checkPoint() 3 0.1177 0.1177 0.1177 14.97% Amr::init() 1 0.1154 0.1154 0.1154 14.68% AmrLevel::checkPoint() 3 0.1137 0.1137 0.1137 14.46% StateData::checkPoint() 12 0.1137 0.1137 0.1137 14.45% MLCellLinOp::applyBC() 4433 0.08998 0.08998 0.08998 11.44% MLMG::mgVcycle_bottom 82 0.07538 0.07538 0.07538 9.59% MLMG::actualBottomSolve() 82 0.07535 0.07535 0.07535 9.58% MLCGSolver::bicgstab 82 0.07459 0.07459 0.07459 9.48% MLPoisson::Fsmooth() 3280 0.06071 0.06071 0.06071 7.72% Castro::clean_state() 62 0.05847 0.05847 0.05847 7.44% Amr::initialInit() 1 0.04996 0.04996 0.04996 6.35% Amr::writePlotFile() 2 0.04818 0.04818 0.04818 6.13% FillPatchIterator::Initialize 41 0.04683 0.04683 0.04683 5.95% Amr::FinalizeInit() 1 0.04623 0.04623 0.04623 5.88% FillPatchSingleLevel 41 0.04528 0.04528 0.04528 5.76% Castro::post_init() 1 0.04495 0.04495 0.04495 5.72% Gravity::multilevel_solve_for_new_phi() 1 0.04313 0.04313 0.04313 5.48% Gravity::actual_multilevel_solve() 1 0.04311 0.04311 0.04311 5.48% StateDataPhysBCFunct::() 41 0.04125 0.04125 0.04125 5.25% MLMG::mgVcycle_down::0 82 0.03473 0.03473 0.03473 4.42% MLCellLinOp::apply() 1142 0.03405 0.03405 0.03405 4.33% MLMG::mgVcycle_up::0 82 0.02954 0.02954 0.02954 3.76% StateData::FillBoundary(geom) 328 0.02434 0.02434 0.02434 3.10% Castro::computeTemp() 63 0.0227 0.0227 0.0227 2.89% MultiFab::Dot() 1114 0.02143 0.02143 0.02143 2.73% Castro::initialize_do_advance() 10 0.02083 0.02083 0.02083 2.65% Castro::construct_old_gravity() 10 0.02064 0.02064 0.02064 2.62% Gravity::get_old_grav_vector() 10 0.02063 0.02063 0.02063 2.62% Castro::normalize_species() 62 0.02045 0.02045 0.02045 2.60% MLCellLinOp::correctionResidual() 492 0.01989 0.01989 0.01989 2.53% MLMG:computeResOfCorrection() 410 0.01726 0.01726 0.01726 2.20% MLPoisson::define() 11 0.01683 0.01683 0.01683 2.14% Gravity::get_new_grav_vector() 11 0.01672 0.01672 0.01672 2.13% MLMG::mgVcycle_down::1 82 0.01647 0.01647 0.01647 2.09% MLMG::mgVcycle_down::2 82 0.01594 0.01594 0.01594 2.03% FabArray::FillBoundary() 4023 0.01587 0.01587 0.01587 2.02% MLMG::mgVcycle_down::3 82 0.0151 0.0151 0.0151 1.92% FillBoundary_nowait() 4023 0.01503 0.01503 0.01503 1.91% FabArray::ParallelCopy() 861 0.01454 0.01454 0.01454 1.85% Castro::do_new_sources() 10 0.01454 0.01454 0.01454 1.85% MLMG::mgVcycle_down::4 82 0.01433 0.01433 0.01433 1.82% FabArray::ParallelCopy_nowait() 861 0.01426 0.01426 0.01426 1.81% Castro::initialize_advance() 10 0.01366 0.01366 0.01366 1.74% Castro::enforce_min_density() 62 0.01361 0.01361 0.01361 1.73% FabArray::setVal() 1144 0.01345 0.01345 0.01345 1.71% CGSolver::sxay() 1586 0.01308 0.01308 0.01308 1.66% MLCGSolver::ParallelAllReduce 1514 0.01287 0.01287 0.01287 1.64% MultiFab::LinComb() 1586 0.01263 0.01263 0.01263 1.61% MLMG::mgVcycle_up::2 82 0.01235 0.01235 0.01235 1.57% MLMG::mgVcycle_up::1 82 0.0123 0.0123 0.0123 1.56% MLMG::addInterpCorrection() 410 0.01208 0.01208 0.01208 1.54% MLMG::mgVcycle_up::3 82 0.01156 0.01156 0.01156 1.47% amrex::average_down 410 0.01147 0.01147 0.01147 1.46% MLMG::mgVcycle_up::4 82 0.01147 0.01147 0.01147 1.46% MLCellLinOp::defineAuxData() 11 0.01143 0.01143 0.01143 1.45% Castro::post_timestep() 10 0.01117 0.01117 0.01117 1.42% Castro::expand_state() 10 0.01107 0.01107 0.01107 1.41% Castro::do_old_sources() 10 0.01088 0.01088 0.01088 1.38% MLPoisson::Fapply() 1142 0.01078 0.01078 0.01078 1.37% Gravity::fill_multipole_BCs() 11 0.008091 0.008091 0.008091 1.03% Castro::estTimeStep() 21 0.007381 0.007381 0.007381 0.94% MLCellLinOp::solutionResidual() 93 0.0071 0.0071 0.0071 0.90% Castro::reset_internal_energy() 63 0.006992 0.006992 0.006992 0.89% MultiFab::Xpay() 585 0.006212 0.006212 0.006212 0.79% MLMG::computeResidual() 82 0.006118 0.006118 0.006118 0.78% MLCellLinOp::defineBC() 11 0.004958 0.004958 0.004958 0.63% BndryData::define() 11 0.004767 0.004767 0.004767 0.61% MLMG::prepareForSolve() 11 0.004588 0.004588 0.004588 0.58% Amr::InitializeInit() 1 0.003727 0.003727 0.003727 0.47% Amr::defBaseLevel() 1 0.003716 0.003716 0.003716 0.47% Castro::computeNewDt() 9 0.003335 0.003335 0.003335 0.42% Castro::initData() 1 0.003209 0.003209 0.003209 0.41% Castro::construct_new_source() 50 0.00317 0.00317 0.00317 0.40% Castro::construct_new_gravity_source() 10 0.003152 0.003152 0.003152 0.40% Castro::construct_old_source() 50 0.00242 0.00242 0.00242 0.31% Castro::construct_old_gravity_source() 10 0.0024 0.0024 0.0024 0.31% Castro::enforce_speed_limit() 62 0.002354 0.002354 0.002354 0.30% MLMG::ResNormInf() 93 0.001907 0.001907 0.001907 0.24% Castro::apply_source_to_state() 20 0.001835 0.001835 0.001835 0.23% MultiFab::Saxpy() 20 0.001824 0.001824 0.001824 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001686 0.001686 0.001686 0.21% FabArrayBase::getCPC() 1323 0.00149 0.00149 0.00149 0.19% MLMG::getGradSolution() 11 0.001382 0.001382 0.001382 0.18% MLCellLinOp::compGrad() 11 0.001376 0.001376 0.001376 0.17% MLCellLinOp::setLevelBC() 11 0.001345 0.001345 0.001345 0.17% FabArray::mult() 43 0.001334 0.001334 0.001334 0.17% FabArray::setDomainBndry() 41 0.001301 0.001301 0.001301 0.17% Castro::check_for_nan() 20 0.001179 0.001179 0.001179 0.15% MultiFab::contains_nan() 20 0.001168 0.001168 0.001168 0.15% MLPoisson::prepareForSolve() 11 0.001107 0.001107 0.001107 0.14% MLCellLinOp::prepareForSolve() 11 0.001098 0.001098 0.001098 0.14% Castro::post_regrid() 1 0.001077 0.001077 0.001077 0.14% MLMG::computeMLResidual() 11 0.001022 0.001022 0.001022 0.13% Gravity::update_max_rhs() 11 0.0008093 0.0008093 0.0008093 0.10% FabArrayBase::getFB() 4023 0.0007243 0.0007243 0.0007243 0.09% FabArrayBase::CPC::define() 454 0.0006898 0.0006898 0.0006898 0.09% Castro::computeInitialDt() 2 0.0006374 0.0006374 0.0006374 0.08% Amr::InitAmr() 1 0.0004519 0.0004519 0.0004519 0.06% Gravity::swapTimeLevels() 10 0.0004454 0.0004454 0.0004454 0.06% MLLinOp::define() 11 0.0004286 0.0004286 0.0004286 0.05% MLLinOp::defineGrids() 11 0.0004061 0.0004061 0.0004061 0.05% MultiFab::Copy() 11 0.0002637 0.0002637 0.0002637 0.03% MultiFab::max() 11 0.0002556 0.0002556 0.0002556 0.03% MLMG::MLResNormInf() 11 0.0002527 0.0002527 0.0002527 0.03% MLMG::MLRhsNormInf() 11 0.0002015 0.0002015 0.0002015 0.03% Castro::finalize_advance() 10 8.724e-05 8.724e-05 8.724e-05 0.01% FabArrayBase::FB::FB() 56 8.678e-05 8.678e-05 8.678e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.562e-05 5.562e-05 5.562e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.292e-05 5.292e-05 5.292e-05 0.01% StateData::define() 4 4.251e-05 4.251e-05 4.251e-05 0.01% makeSFC 55 4.039e-05 4.039e-05 4.039e-05 0.01% Castro::enforce_consistent_e() 1 3.661e-05 3.661e-05 3.661e-05 0.00% Castro::swap_state_time_levels() 10 3.638e-05 3.638e-05 3.638e-05 0.00% Castro::finalize_do_advance() 10 3.379e-05 3.379e-05 3.379e-05 0.00% Amr::writeSmallPlotFile() 1 2.46e-05 2.46e-05 2.46e-05 0.00% DistributionMapping::Distribute() 56 1.467e-05 1.467e-05 1.467e-05 0.00% Amr::initSubcycle() 1 9.503e-06 9.503e-06 9.503e-06 0.00% Castro::create_source_corrector() 10 5.94e-06 5.94e-06 5.94e-06 0.00% MLMG::buildFineMask() 11 5.921e-06 5.921e-06 5.921e-06 0.00% AmrLevel::checkPointPost() 3 5.164e-06 5.164e-06 5.164e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.346e-06 4.346e-06 4.346e-06 0.00% Gravity::set_mass_offset() 11 4.096e-06 4.096e-06 4.096e-06 0.00% Castro::retry_advance_ctu() 10 3.658e-06 3.658e-06 3.658e-06 0.00% Castro::FluxRegFineAdd() 10 2.654e-06 2.654e-06 2.654e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.204e-06 2.204e-06 2.204e-06 0.00% AmrLevel::checkPointPre() 3 2.009e-06 2.009e-06 2.009e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-37-g3cc4a66759c9) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-37-g3cc4a66759c9) initialized Starting run at 08:22:13 UTC on 2022-04-28. Successfully read inputs file ... Castro git describe: 22.04-37-gb93839ec1 AMReX git describe: 22.04-37-g3cc4a6675 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.432872467 Restart time = 0.045558285 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048767144 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047238445 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057343852 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060905143 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061271439 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.085502056 seconds Ending run at 08:22:14 UTC on 2022-04-28. Run time = 0.407526874 Run time without initialization = 0.361391119 Average number of zones advanced per microsecond: 3.627 Average number of zones advanced per microsecond per rank: 3.627 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4076 ... 0.4076 ... 0.4076 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0839 0.0839 0.0839 20.58% Amr::writePlotFile() 1 0.06201 0.06201 0.06201 15.21% VisMF::Read() 3 0.03858 0.03858 0.03858 9.47% MLCellLinOp::applyBC() 1946 0.03213 0.03213 0.03213 7.88% MLPoisson::Fsmooth() 1440 0.02615 0.02615 0.02615 6.42% VisMF::Write(FabArray) 1 0.02357 0.02357 0.02357 5.78% StateData::FillBoundary(geom) 160 0.01216 0.01216 0.01216 2.98% MLCGSolver::bicgstab 36 0.009366 0.009366 0.009366 2.30% MultiFab::Dot() 484 0.009137 0.009137 0.009137 2.24% Castro::normalize_species() 30 0.009082 0.009082 0.009082 2.23% Castro::enforce_min_density() 30 0.006569 0.006569 0.006569 1.61% Castro::computeTemp() 30 0.006516 0.006516 0.006516 1.60% FabArray::setVal() 537 0.006437 0.006437 0.006437 1.58% FillBoundary_nowait() 1766 0.006351 0.006351 0.006351 1.56% FabArray::ParallelCopy_nowait() 380 0.006125 0.006125 0.006125 1.50% MultiFab::LinComb() 690 0.005473 0.005473 0.005473 1.34% MLCellLinOp::defineAuxData() 6 0.005418 0.005418 0.005418 1.33% StateDataPhysBCFunct::() 20 0.005005 0.005005 0.005005 1.23% Gravity::fill_multipole_BCs() 6 0.004759 0.004759 0.004759 1.17% MLPoisson::Fapply() 500 0.004708 0.004708 0.004708 1.16% Castro::estTimeStep() 10 0.003573 0.003573 0.003573 0.88% MLMG::addInterpCorrection() 180 0.002946 0.002946 0.002946 0.72% Amr::restart() 1 0.002926 0.002926 0.002926 0.72% Castro::do_advance_ctu() 5 0.002791 0.002791 0.002791 0.68% amrex::average_down 180 0.002729 0.002729 0.002729 0.67% MultiFab::Xpay() 258 0.002721 0.002721 0.002721 0.67% Castro::reset_internal_energy() 30 0.002574 0.002574 0.002574 0.63% BndryData::define() 6 0.002083 0.002083 0.002083 0.51% Castro::construct_new_gravity_source() 5 0.001518 0.001518 0.001518 0.37% Castro::enforce_speed_limit() 30 0.001372 0.001372 0.001372 0.34% Castro::construct_old_gravity_source() 5 0.001222 0.001222 0.001222 0.30% MultiFab::Saxpy() 10 0.00092 0.00092 0.00092 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008989 0.0008989 0.0008989 0.22% Gravity::get_old_grav_vector() 5 0.0008899 0.0008899 0.0008899 0.22% Castro::expand_state() 5 0.0008767 0.0008767 0.0008767 0.22% Gravity::get_new_grav_vector() 5 0.0008722 0.0008722 0.0008722 0.21% MLMG::ResNormInf() 42 0.0008541 0.0008541 0.0008541 0.21% MLMG::oneIter() 36 0.0007249 0.0007249 0.0007249 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007218 0.0007218 0.0007218 0.18% MLCellLinOp::setLevelBC() 6 0.0007151 0.0007151 0.0007151 0.18% FabArray::mult() 22 0.0006551 0.0006551 0.0006551 0.16% FabArray::setDomainBndry() 20 0.0006538 0.0006538 0.0006538 0.16% MLCellLinOp::prepareForSolve() 6 0.0006041 0.0006041 0.0006041 0.15% MultiFab::contains_nan() 10 0.0005793 0.0005793 0.0005793 0.14% MLCellLinOp::smooth() 720 0.0005066 0.0005066 0.0005066 0.12% MLCellLinOp::compGrad() 6 0.0004831 0.0004831 0.0004831 0.12% MLMG::prepareForSolve() 6 0.0004369 0.0004369 0.0004369 0.11% FabArrayBase::CPC::define() 244 0.0004238 0.0004238 0.0004238 0.10% FabArrayBase::getCPC() 632 0.0004218 0.0004218 0.0004218 0.10% FabArray::FillBoundary() 1766 0.0004182 0.0004182 0.0004182 0.10% Amr::InitAmr() 1 0.0003987 0.0003987 0.0003987 0.10% FabArrayBase::getFB() 1766 0.0003245 0.0003245 0.0003245 0.08% main() 1 0.0003043 0.0003043 0.0003043 0.07% Gravity::update_max_rhs() 6 0.0002337 0.0002337 0.0002337 0.06% MLCellLinOp::apply() 500 0.0002212 0.0002212 0.0002212 0.05% CGSolver::sxay() 690 0.0002168 0.0002168 0.0002168 0.05% Gravity::solve_for_phi() 5 0.0002055 0.0002055 0.0002055 0.05% MLLinOp::defineGrids() 6 0.00019 0.00019 0.00019 0.05% MLMG::mgVcycle() 36 0.0001819 0.0001819 0.0001819 0.04% MLCGSolver::ParallelAllReduce 659 0.000156 0.000156 0.000156 0.04% MultiFab::Copy() 6 0.0001447 0.0001447 0.0001447 0.04% FabArray::ParallelCopy() 380 0.0001439 0.0001439 0.0001439 0.04% Castro::subcycle_advance_ctu() 5 0.0001392 0.0001392 0.0001392 0.03% MultiFab::max() 6 0.000135 0.000135 0.000135 0.03% FillPatchIterator::Initialize 20 0.0001223 0.0001223 0.0001223 0.03% MLMG::MLRhsNormInf() 6 0.0001078 0.0001078 0.0001078 0.03% Castro::construct_new_gravity() 5 0.0001076 0.0001076 0.0001076 0.03% Amr::coarseTimeStep() 5 0.000107 0.000107 0.000107 0.03% MLCellLinOp::defineBC() 6 0.0001003 0.0001003 0.0001003 0.02% MLCellLinOp::correctionResidual() 216 9.909e-05 9.909e-05 9.909e-05 0.02% Amr::timeStep() 5 9.21e-05 9.21e-05 9.21e-05 0.02% StateData::restartDoit() 4 7.843e-05 7.843e-05 7.843e-05 0.02% MLMG:computeResOfCorrection() 180 7.423e-05 7.423e-05 7.423e-05 0.02% AmrLevel::restart() 1 7.146e-05 7.146e-05 7.146e-05 0.02% FabArrayBase::FB::FB() 26 6.062e-05 6.062e-05 6.062e-05 0.01% Castro::advance() 5 5.929e-05 5.929e-05 5.929e-05 0.01% MLMG::actualBottomSolve() 36 4.972e-05 4.972e-05 4.972e-05 0.01% Castro::clean_state() 30 4.232e-05 4.232e-05 4.232e-05 0.01% MLMG::mgVcycle_down::1 36 3.894e-05 3.894e-05 3.894e-05 0.01% MLMG::mgVcycle_down::0 36 3.864e-05 3.864e-05 3.864e-05 0.01% MLMG::solve() 6 3.836e-05 3.836e-05 3.836e-05 0.01% Castro::initialize_advance() 5 3.667e-05 3.667e-05 3.667e-05 0.01% MLMG::mgVcycle_down::2 36 3.506e-05 3.506e-05 3.506e-05 0.01% Castro::finalize_advance() 5 3.459e-05 3.459e-05 3.459e-05 0.01% MLMG::mgVcycle_down::4 36 3.401e-05 3.401e-05 3.401e-05 0.01% MLMG::mgVcycle_down::3 36 3.358e-05 3.358e-05 3.358e-05 0.01% Castro::post_restart() 1 3.274e-05 3.274e-05 3.274e-05 0.01% Gravity::actual_multilevel_solve() 1 2.917e-05 2.917e-05 2.917e-05 0.01% Castro::initialize_do_advance() 5 2.906e-05 2.906e-05 2.906e-05 0.01% MLMG::mgVcycle_up::4 36 2.704e-05 2.704e-05 2.704e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% Castro::swap_state_time_levels() 5 2.465e-05 2.465e-05 2.465e-05 0.01% MLCellLinOp::solutionResidual() 42 2.307e-05 2.307e-05 2.307e-05 0.01% MLMG::mgVcycle_up::0 36 2.229e-05 2.229e-05 2.229e-05 0.01% MLLinOp::define() 6 2.133e-05 2.133e-05 2.133e-05 0.01% MLMG::mgVcycle_up::3 36 2.073e-05 2.073e-05 2.073e-05 0.01% MLMG::mgVcycle_up::2 36 2.048e-05 2.048e-05 2.048e-05 0.01% MLMG::mgVcycle_up::1 36 2e-05 2e-05 2e-05 0.00% Castro::finalize_do_advance() 5 1.875e-05 1.875e-05 1.875e-05 0.00% MLMG::mgVcycle_bottom 36 1.738e-05 1.738e-05 1.738e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.662e-05 1.662e-05 1.662e-05 0.00% MLMG::computeResidual() 36 1.613e-05 1.613e-05 1.613e-05 0.00% makeSFC 30 1.482e-05 1.482e-05 1.482e-05 0.00% MLPoisson::define() 6 1.472e-05 1.472e-05 1.472e-05 0.00% FillPatchSingleLevel 20 1.45e-05 1.45e-05 1.45e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.402e-05 1.402e-05 1.402e-05 0.00% Castro::construct_old_source() 25 1.063e-05 1.063e-05 1.063e-05 0.00% Amr::initSubcycle() 1 1e-05 1e-05 1e-05 0.00% Castro::construct_new_source() 25 9.798e-06 9.798e-06 9.798e-06 0.00% DistributionMapping::Distribute() 31 9.386e-06 9.386e-06 9.386e-06 0.00% Castro::do_new_sources() 5 8.608e-06 8.608e-06 8.608e-06 0.00% Castro::do_old_sources() 5 8.087e-06 8.087e-06 8.087e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.799e-06 7.799e-06 7.799e-06 0.00% Castro::apply_source_to_state() 10 6.585e-06 6.585e-06 6.585e-06 0.00% Castro::post_timestep() 5 6.201e-06 6.201e-06 6.201e-06 0.00% Castro::check_for_nan() 10 6.124e-06 6.124e-06 6.124e-06 0.00% Castro::construct_old_gravity() 5 5.529e-06 5.529e-06 5.529e-06 0.00% MLPoisson::prepareForSolve() 6 5.08e-06 5.08e-06 5.08e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.672e-06 4.672e-06 4.672e-06 0.00% Gravity::swapTimeLevels() 5 4.165e-06 4.165e-06 4.165e-06 0.00% MLMG::buildFineMask() 6 3.79e-06 3.79e-06 3.79e-06 0.00% Castro::computeNewDt() 5 3.673e-06 3.673e-06 3.673e-06 0.00% MLMG::getGradSolution() 6 3.135e-06 3.135e-06 3.135e-06 0.00% Castro::create_source_corrector() 5 3.11e-06 3.11e-06 3.11e-06 0.00% MLMG::computeMLResidual() 6 3.05e-06 3.05e-06 3.05e-06 0.00% MLMG::MLResNormInf() 6 3.013e-06 3.013e-06 3.013e-06 0.00% Castro::retry_advance_ctu() 5 2.755e-06 2.755e-06 2.755e-06 0.00% Gravity::set_mass_offset() 6 2.579e-06 2.579e-06 2.579e-06 0.00% Castro::FluxRegFineAdd() 5 2.048e-06 2.048e-06 2.048e-06 0.00% Amr::init() 1 1.369e-06 1.369e-06 1.369e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.024e-06 1.024e-06 1.024e-06 0.00% AmrLevel::AmrLevel() 1 9.16e-07 9.16e-07 9.16e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4075 0.4075 0.4075 100.00% Amr::coarseTimeStep() 5 0.2756 0.2756 0.2756 67.63% Amr::timeStep() 5 0.2735 0.2735 0.2735 67.10% Castro::advance() 5 0.2683 0.2683 0.2683 65.82% Castro::subcycle_advance_ctu() 5 0.2609 0.2609 0.2609 64.01% Castro::do_advance_ctu() 5 0.2607 0.2607 0.2607 63.98% Castro::construct_new_gravity() 5 0.1363 0.1363 0.1363 33.43% Gravity::solve_phi_with_mlmg() 6 0.132 0.132 0.132 32.39% Gravity::solve_for_phi() 5 0.1286 0.1286 0.1286 31.56% Gravity::actual_solve_with_mlmg() 6 0.1271 0.1271 0.1271 31.19% MLMG::solve() 6 0.1158 0.1158 0.1158 28.40% MLMG::oneIter() 36 0.109 0.109 0.109 26.75% MLMG::mgVcycle() 36 0.1083 0.1083 0.1083 26.57% Amr::writePlotFile() 1 0.08558 0.08558 0.08558 21.00% Castro::construct_ctu_hydro_source() 5 0.08387 0.08387 0.08387 20.58% MLCellLinOp::smooth() 720 0.05569 0.05569 0.05569 13.66% Amr::init() 1 0.0456 0.0456 0.0456 11.19% Amr::restart() 1 0.04559 0.04559 0.04559 11.19% MLCellLinOp::applyBC() 1946 0.03928 0.03928 0.03928 9.64% AmrLevel::restart() 1 0.03879 0.03879 0.03879 9.52% StateData::restartDoit() 4 0.03872 0.03872 0.03872 9.50% VisMF::Read() 3 0.03858 0.03858 0.03858 9.47% MLMG::mgVcycle_bottom 36 0.03256 0.03256 0.03256 7.99% MLMG::actualBottomSolve() 36 0.03254 0.03254 0.03254 7.98% MLCGSolver::bicgstab 36 0.03221 0.03221 0.03221 7.90% Castro::clean_state() 30 0.02616 0.02616 0.02616 6.42% MLPoisson::Fsmooth() 1440 0.02615 0.02615 0.02615 6.42% VisMF::Write(FabArray) 1 0.02357 0.02357 0.02357 5.78% FillPatchIterator::Initialize 20 0.01996 0.01996 0.01996 4.90% FillPatchSingleLevel 20 0.01919 0.01919 0.01919 4.71% StateDataPhysBCFunct::() 20 0.01717 0.01717 0.01717 4.21% MLMG::mgVcycle_down::0 36 0.0151 0.0151 0.0151 3.71% MLCellLinOp::apply() 500 0.01492 0.01492 0.01492 3.66% MLMG::mgVcycle_up::0 36 0.0129 0.0129 0.0129 3.17% StateData::FillBoundary(geom) 160 0.01216 0.01216 0.01216 2.98% Castro::initialize_do_advance() 5 0.01026 0.01026 0.01026 2.52% MLPoisson::define() 6 0.009178 0.009178 0.009178 2.25% MultiFab::Dot() 484 0.009137 0.009137 0.009137 2.24% Castro::computeTemp() 30 0.009091 0.009091 0.009091 2.23% Castro::normalize_species() 30 0.009082 0.009082 0.009082 2.23% MLCellLinOp::correctionResidual() 216 0.008687 0.008687 0.008687 2.13% MLMG:computeResOfCorrection() 180 0.007531 0.007531 0.007531 1.85% Gravity::get_new_grav_vector() 5 0.007525 0.007525 0.007525 1.85% Castro::construct_old_gravity() 5 0.007319 0.007319 0.007319 1.80% Gravity::get_old_grav_vector() 5 0.007314 0.007314 0.007314 1.79% Castro::initialize_advance() 5 0.007294 0.007294 0.007294 1.79% MLMG::mgVcycle_down::1 36 0.007173 0.007173 0.007173 1.76% FabArray::FillBoundary() 1766 0.007155 0.007155 0.007155 1.76% MLMG::mgVcycle_down::2 36 0.00697 0.00697 0.00697 1.71% FillBoundary_nowait() 1766 0.006736 0.006736 0.006736 1.65% FabArray::ParallelCopy() 380 0.006694 0.006694 0.006694 1.64% Castro::enforce_min_density() 30 0.006569 0.006569 0.006569 1.61% MLMG::mgVcycle_down::3 36 0.006565 0.006565 0.006565 1.61% FabArray::ParallelCopy_nowait() 380 0.00655 0.00655 0.00655 1.61% FabArray::setVal() 537 0.006437 0.006437 0.006437 1.58% MLMG::mgVcycle_down::4 36 0.006253 0.006253 0.006253 1.53% MLCellLinOp::defineAuxData() 6 0.006186 0.006186 0.006186 1.52% Castro::expand_state() 5 0.005821 0.005821 0.005821 1.43% Castro::do_new_sources() 5 0.005783 0.005783 0.005783 1.42% CGSolver::sxay() 690 0.005689 0.005689 0.005689 1.40% MLCGSolver::ParallelAllReduce 659 0.005526 0.005526 0.005526 1.36% MultiFab::LinComb() 690 0.005473 0.005473 0.005473 1.34% MLMG::mgVcycle_up::2 36 0.005329 0.005329 0.005329 1.31% MLMG::addInterpCorrection() 180 0.005286 0.005286 0.005286 1.30% Castro::do_old_sources() 5 0.005272 0.005272 0.005272 1.29% MLMG::mgVcycle_up::1 36 0.005253 0.005253 0.005253 1.29% Castro::post_timestep() 5 0.005089 0.005089 0.005089 1.25% amrex::average_down 180 0.005079 0.005079 0.005079 1.25% MLMG::mgVcycle_up::3 36 0.004998 0.004998 0.004998 1.23% MLMG::mgVcycle_up::4 36 0.004997 0.004997 0.004997 1.23% Gravity::fill_multipole_BCs() 6 0.004759 0.004759 0.004759 1.17% MLPoisson::Fapply() 500 0.004708 0.004708 0.004708 1.16% Castro::post_restart() 1 0.003743 0.003743 0.003743 0.92% Gravity::multilevel_solve_for_new_phi() 1 0.003612 0.003612 0.003612 0.89% Gravity::actual_multilevel_solve() 1 0.003595 0.003595 0.003595 0.88% Castro::estTimeStep() 10 0.003573 0.003573 0.003573 0.88% MLCellLinOp::solutionResidual() 42 0.003215 0.003215 0.003215 0.79% MLCellLinOp::defineBC() 6 0.002735 0.002735 0.002735 0.67% MultiFab::Xpay() 258 0.002721 0.002721 0.002721 0.67% MLMG::computeResidual() 36 0.002666 0.002666 0.002666 0.65% BndryData::define() 6 0.002635 0.002635 0.002635 0.65% Castro::reset_internal_energy() 30 0.002574 0.002574 0.002574 0.63% MLMG::prepareForSolve() 6 0.002512 0.002512 0.002512 0.62% Castro::computeNewDt() 5 0.00206 0.00206 0.00206 0.51% Castro::construct_new_source() 25 0.001528 0.001528 0.001528 0.37% Castro::construct_new_gravity_source() 5 0.001518 0.001518 0.001518 0.37% Castro::enforce_speed_limit() 30 0.001372 0.001372 0.001372 0.34% Castro::construct_old_source() 25 0.001233 0.001233 0.001233 0.30% Castro::construct_old_gravity_source() 5 0.001222 0.001222 0.001222 0.30% Castro::apply_source_to_state() 10 0.0009266 0.0009266 0.0009266 0.23% MultiFab::Saxpy() 10 0.00092 0.00092 0.00092 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008989 0.0008989 0.0008989 0.22% MLMG::ResNormInf() 42 0.0008541 0.0008541 0.0008541 0.21% FabArrayBase::getCPC() 632 0.0008456 0.0008456 0.0008456 0.21% MLMG::getGradSolution() 6 0.0007447 0.0007447 0.0007447 0.18% MLCellLinOp::compGrad() 6 0.0007416 0.0007416 0.0007416 0.18% MLCellLinOp::setLevelBC() 6 0.0007151 0.0007151 0.0007151 0.18% FabArray::mult() 22 0.0006551 0.0006551 0.0006551 0.16% FabArray::setDomainBndry() 20 0.0006538 0.0006538 0.0006538 0.16% MLPoisson::prepareForSolve() 6 0.0006092 0.0006092 0.0006092 0.15% MLCellLinOp::prepareForSolve() 6 0.0006041 0.0006041 0.0006041 0.15% Castro::check_for_nan() 10 0.0005855 0.0005855 0.0005855 0.14% MultiFab::contains_nan() 10 0.0005793 0.0005793 0.0005793 0.14% MLMG::computeMLResidual() 6 0.0005689 0.0005689 0.0005689 0.14% Gravity::update_max_rhs() 6 0.0004471 0.0004471 0.0004471 0.11% FabArrayBase::CPC::define() 244 0.0004238 0.0004238 0.0004238 0.10% Amr::InitAmr() 1 0.0004087 0.0004087 0.0004087 0.10% FabArrayBase::getFB() 1766 0.0003852 0.0003852 0.0003852 0.09% MLLinOp::define() 6 0.000242 0.000242 0.000242 0.06% Gravity::swapTimeLevels() 5 0.0002305 0.0002305 0.0002305 0.06% MLLinOp::defineGrids() 6 0.0002207 0.0002207 0.0002207 0.05% MultiFab::Copy() 6 0.0001447 0.0001447 0.0001447 0.04% MLMG::MLResNormInf() 6 0.0001379 0.0001379 0.0001379 0.03% MultiFab::max() 6 0.000135 0.000135 0.000135 0.03% MLMG::MLRhsNormInf() 6 0.0001078 0.0001078 0.0001078 0.03% FabArrayBase::FB::FB() 26 6.062e-05 6.062e-05 6.062e-05 0.01% Castro::finalize_advance() 5 3.664e-05 3.664e-05 3.664e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.962e-05 2.962e-05 2.962e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% Castro::swap_state_time_levels() 5 2.465e-05 2.465e-05 2.465e-05 0.01% makeSFC 30 2.182e-05 2.182e-05 2.182e-05 0.01% Castro::finalize_do_advance() 5 1.875e-05 1.875e-05 1.875e-05 0.00% Amr::initSubcycle() 1 1e-05 1e-05 1e-05 0.00% DistributionMapping::Distribute() 31 9.386e-06 9.386e-06 9.386e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 7.058e-06 7.058e-06 7.058e-06 0.00% MLMG::buildFineMask() 6 3.79e-06 3.79e-06 3.79e-06 0.00% Castro::create_source_corrector() 5 3.11e-06 3.11e-06 3.11e-06 0.00% Castro::retry_advance_ctu() 5 2.755e-06 2.755e-06 2.755e-06 0.00% Gravity::set_mass_offset() 6 2.579e-06 2.579e-06 2.579e-06 0.00% Castro::FluxRegFineAdd() 5 2.048e-06 2.048e-06 2.048e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.024e-06 1.024e-06 1.024e-06 0.00% AmrLevel::AmrLevel() 1 9.16e-07 9.16e-07 9.16e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-37-g3cc4a66759c9) finalized