Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-24-gea7e8a507e77) initialized Starting run at 10:07:25 UTC on 2022-12-26. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 22.12-24-gea7e8a507 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058523691 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.033431814 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.045470053 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.047773409 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.056966389 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.066324668 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.074473466 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.056797609 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.058460498 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.046095704 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.046210824 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.04772197 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059344313 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057000186 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033431174 seconds Ending run at 10:07:26 UTC on 2022-12-26. Run time = 0.837152779 Run time without initialization = 0.696710396 Average number of zones advanced per microsecond: 3.763 Average number of zones advanced per microsecond per rank: 3.763 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8372 ... 0.8372 ... 0.8372 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2324 0.2324 0.2324 27.76% Castro::construct_ctu_hydro_source() 10 0.1947 0.1947 0.1947 23.26% MLCellLinOp::applyBC() 4433 0.07494 0.07494 0.07494 8.95% MLPoisson::Fsmooth() 3280 0.03222 0.03222 0.03222 3.85% FillBoundary_nowait() 4023 0.03119 0.03119 0.03119 3.73% StateData::FillBoundary(geom) 328 0.0235 0.0235 0.0235 2.81% amrex::Dot() 1114 0.02034 0.02034 0.02034 2.43% Castro::normalize_species() 62 0.01554 0.01554 0.01554 1.86% amrex::Copy() 1029 0.01483 0.01483 0.01483 1.77% FabArray::norminf() 743 0.01432 0.01432 0.01432 1.71% Castro::computeTemp() 63 0.01431 0.01431 0.01431 1.71% FabArray::ParallelCopy_nowait() 861 0.0131 0.0131 0.0131 1.56% FabArray::setVal() 1144 0.01304 0.01304 0.01304 1.56% StateDataPhysBCFunct::() 41 0.01236 0.01236 0.01236 1.48% MLPoisson::Fapply() 1142 0.0104 0.0104 0.0104 1.24% MLCellLinOp::defineAuxData() 11 0.009631 0.009631 0.009631 1.15% FabArray::Saxpy() 813 0.008146 0.008146 0.008146 0.97% FabArray::Xpay() 821 0.00804 0.00804 0.00804 0.96% Castro::enforce_min_density() 62 0.007672 0.007672 0.007672 0.92% MLMG::addInterpCorrection() 410 0.006543 0.006543 0.006543 0.78% Gravity::fill_multipole_BCs() 11 0.006137 0.006137 0.006137 0.73% Castro::estTimeStep() 21 0.006015 0.006015 0.006015 0.72% amrex::average_down 410 0.005802 0.005802 0.005802 0.69% FabArray::LinComb() 557 0.004525 0.004525 0.004525 0.54% amrex::Add() 164 0.004322 0.004322 0.004322 0.52% Castro::reset_internal_energy(MultiFab) 63 0.004301 0.004301 0.004301 0.51% Amr::checkPoint() 3 0.003957 0.003957 0.003957 0.47% BndryData::define() 11 0.003653 0.003653 0.003653 0.44% Castro::construct_new_gravity_source() 10 0.003266 0.003266 0.003266 0.39% Castro::do_advance_ctu() 10 0.002886 0.002886 0.002886 0.34% Castro::construct_old_gravity_source() 10 0.002607 0.002607 0.002607 0.31% Amr::writePlotFile() 2 0.002353 0.002353 0.002353 0.28% Castro::reset_internal_energy(Fab) 504 0.002255 0.002255 0.002255 0.27% MLCGSolver::bicgstab 82 0.002133 0.002133 0.002133 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001636 0.001636 0.001636 0.20% MLCellLinOp::setLevelBC() 11 0.001408 0.001408 0.001408 0.17% Gravity::actual_solve_with_mlmg() 11 0.00136 0.00136 0.00136 0.16% FabArray::mult() 43 0.00135 0.00135 0.00135 0.16% FabArray::setDomainBndry() 41 0.001318 0.001318 0.001318 0.16% Castro::initData() 1 0.001277 0.001277 0.001277 0.15% MLCellLinOp::smooth() 1640 0.001217 0.001217 0.001217 0.15% MultiFab::contains_nan() 20 0.001177 0.001177 0.001177 0.14% MLCellLinOp::prepareForSolve() 11 0.001103 0.001103 0.001103 0.13% Castro::enforce_speed_limit() 62 0.001094 0.001094 0.001094 0.13% MLCellLinOp::compGrad() 11 0.0009184 0.0009184 0.0009184 0.11% MLMG::prepareForSolve() 11 0.0008488 0.0008488 0.0008488 0.10% FabArray::FillBoundary() 4023 0.0007845 0.0007845 0.0007845 0.09% FabArrayBase::getCPC() 1323 0.0007229 0.0007229 0.0007229 0.09% FabArrayBase::CPC::define() 454 0.0006906 0.0006906 0.0006906 0.08% Gravity::get_new_grav_vector() 11 0.0006063 0.0006063 0.0006063 0.07% FabArrayBase::getFB() 4023 0.0005973 0.0005973 0.0005973 0.07% Gravity::get_old_grav_vector() 10 0.0005337 0.0005337 0.0005337 0.06% Amr::InitAmr() 1 0.0005287 0.0005287 0.0005287 0.06% MLCellLinOp::apply() 1142 0.0005129 0.0005129 0.0005129 0.06% MLMG::mgVcycle() 82 0.0004185 0.0004185 0.0004185 0.05% Amr::coarseTimeStep() 10 0.0003569 0.0003569 0.0003569 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002887 0.0002887 0.0002887 0.03% main() 1 0.0002735 0.0002735 0.0002735 0.03% MultiFab::max() 11 0.0002656 0.0002656 0.0002656 0.03% MLCellLinOp::correctionResidual() 492 0.0002427 0.0002427 0.0002427 0.03% FabArray::ParallelCopy() 861 0.0002384 0.0002384 0.0002384 0.03% FillPatchIterator::Initialize 41 0.0002095 0.0002095 0.0002095 0.03% MLCellLinOp::defineBC() 11 0.0001971 0.0001971 0.0001971 0.02% Castro::subcycle_advance_ctu() 10 0.0001845 0.0001845 0.0001845 0.02% MLLinOp::defineGrids() 11 0.0001718 0.0001718 0.0001718 0.02% Gravity::solve_for_phi() 10 0.0001606 0.0001606 0.0001606 0.02% Amr::timeStep() 10 0.0001541 0.0001541 0.0001541 0.02% Castro::advance() 10 0.0001492 0.0001492 0.0001492 0.02% StateData::checkPoint() 12 0.00013 0.00013 0.00013 0.02% MLMG:computeResOfCorrection() 410 0.0001149 0.0001149 0.0001149 0.01% Gravity::update_max_rhs() 11 0.0001138 0.0001138 0.0001138 0.01% MLMG::mgVcycle_down::0 82 9.657e-05 9.657e-05 9.657e-05 0.01% MLMG::actualBottomSolve() 82 9.326e-05 9.326e-05 9.326e-05 0.01% Castro::finalize_advance() 10 9.093e-05 9.093e-05 9.093e-05 0.01% MLMG::mgVcycle_down::1 82 8.758e-05 8.758e-05 8.758e-05 0.01% MLMG::mgVcycle_down::2 82 8.373e-05 8.373e-05 8.373e-05 0.01% FabArrayBase::FB::FB() 56 8.237e-05 8.237e-05 8.237e-05 0.01% MLMG::mgVcycle_down::3 82 8.18e-05 8.18e-05 8.18e-05 0.01% Castro::Castro() 1 7.909e-05 7.909e-05 7.909e-05 0.01% Castro::clean_state() 62 7.816e-05 7.816e-05 7.816e-05 0.01% MLMG::solve() 11 7.742e-05 7.742e-05 7.742e-05 0.01% MLMG::mgVcycle_down::4 82 7.734e-05 7.734e-05 7.734e-05 0.01% Castro::expand_state() 10 7.457e-05 7.457e-05 7.457e-05 0.01% AmrLevel::checkPoint() 3 7.169e-05 7.169e-05 7.169e-05 0.01% Castro::initialize_advance() 10 6.786e-05 6.786e-05 6.786e-05 0.01% MLMG::mgVcycle_up::4 82 6.558e-05 6.558e-05 6.558e-05 0.01% MLMG::mgVcycle_up::0 82 5.782e-05 5.782e-05 5.782e-05 0.01% MLMG::oneIter() 82 5.401e-05 5.401e-05 5.401e-05 0.01% MLMG::mgVcycle_up::1 82 5.374e-05 5.374e-05 5.374e-05 0.01% MLMG::mgVcycle_up::3 82 5.186e-05 5.186e-05 5.186e-05 0.01% MLCellLinOp::solutionResidual() 93 5.179e-05 5.179e-05 5.179e-05 0.01% MLMG::mgVcycle_up::2 82 5.109e-05 5.109e-05 5.109e-05 0.01% Castro::initialize_do_advance() 10 4.564e-05 4.564e-05 4.564e-05 0.01% Castro::swap_state_time_levels() 10 4.342e-05 4.342e-05 4.342e-05 0.01% Castro::construct_new_source() 50 3.84e-05 3.84e-05 3.84e-05 0.00% Castro::finalize_do_advance() 10 3.487e-05 3.487e-05 3.487e-05 0.00% Castro::enforce_consistent_e() 1 3.484e-05 3.484e-05 3.484e-05 0.00% StateData::define() 4 3.364e-05 3.364e-05 3.364e-05 0.00% MLMG::mgVcycle_bottom 82 3.245e-05 3.245e-05 3.245e-05 0.00% MLMG::ResNormInf() 93 3.236e-05 3.236e-05 3.236e-05 0.00% MLMG::computeResidual() 82 3.214e-05 3.214e-05 3.214e-05 0.00% FillPatchSingleLevel 41 2.968e-05 2.968e-05 2.968e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.957e-05 2.957e-05 2.957e-05 0.00% makeSFC 55 2.56e-05 2.56e-05 2.56e-05 0.00% Castro::construct_new_gravity() 10 2.449e-05 2.449e-05 2.449e-05 0.00% Amr::writeSmallPlotFile() 1 2.346e-05 2.346e-05 2.346e-05 0.00% Castro::initMFs() 1 2.044e-05 2.044e-05 2.044e-05 0.00% Amr::FinalizeInit() 1 1.992e-05 1.992e-05 1.992e-05 0.00% MLPoisson::define() 11 1.965e-05 1.965e-05 1.965e-05 0.00% Castro::construct_old_source() 50 1.836e-05 1.836e-05 1.836e-05 0.00% Castro::do_new_sources() 10 1.792e-05 1.792e-05 1.792e-05 0.00% Castro::buildMetrics() 1 1.638e-05 1.638e-05 1.638e-05 0.00% DistributionMapping::Distribute() 56 1.598e-05 1.598e-05 1.598e-05 0.00% Castro::do_old_sources() 10 1.57e-05 1.57e-05 1.57e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.431e-05 1.431e-05 1.431e-05 0.00% Amr::defBaseLevel() 1 1.346e-05 1.346e-05 1.346e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.272e-05 1.272e-05 1.272e-05 0.00% Castro::check_for_nan() 20 1.074e-05 1.074e-05 1.074e-05 0.00% Castro::apply_source_to_state() 20 1.029e-05 1.029e-05 1.029e-05 0.00% MLLinOp::define() 11 1.005e-05 1.005e-05 1.005e-05 0.00% Gravity::swapTimeLevels() 10 9.55e-06 9.55e-06 9.55e-06 0.00% Castro::post_timestep() 10 9.343e-06 9.343e-06 9.343e-06 0.00% AmrLevel::AmrLevel(dm) 1 9.046e-06 9.046e-06 9.046e-06 0.00% Castro::post_init() 1 8.96e-06 8.96e-06 8.96e-06 0.00% Castro::construct_old_gravity() 10 8.901e-06 8.901e-06 8.901e-06 0.00% Amr::initSubcycle() 1 8.795e-06 8.795e-06 8.795e-06 0.00% MLPoisson::prepareForSolve() 11 8.164e-06 8.164e-06 8.164e-06 0.00% MLMG::computeMLResidual() 11 8.017e-06 8.017e-06 8.017e-06 0.00% Gravity::actual_multilevel_solve() 1 7.098e-06 7.098e-06 7.098e-06 0.00% Castro::computeNewDt() 9 7.006e-06 7.006e-06 7.006e-06 0.00% MLMG::getGradSolution() 11 6.092e-06 6.092e-06 6.092e-06 0.00% Amr::InitializeInit() 1 5.214e-06 5.214e-06 5.214e-06 0.00% AmrLevel::checkPointPost() 3 4.333e-06 4.333e-06 4.333e-06 0.00% Gravity::set_mass_offset() 11 4.148e-06 4.148e-06 4.148e-06 0.00% MLMG::MLRhsNormInf() 11 4.103e-06 4.103e-06 4.103e-06 0.00% Castro::create_source_corrector() 10 4.085e-06 4.085e-06 4.085e-06 0.00% Castro::retry_advance_ctu() 10 4.028e-06 4.028e-06 4.028e-06 0.00% MLMG::MLResNormInf() 11 3.446e-06 3.446e-06 3.446e-06 0.00% Castro::FluxRegCrseInit 10 2.882e-06 2.882e-06 2.882e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.692e-06 2.692e-06 2.692e-06 0.00% Amr::init() 1 2.402e-06 2.402e-06 2.402e-06 0.00% Castro::FluxRegFineAdd() 10 2.349e-06 2.349e-06 2.349e-06 0.00% Castro::computeInitialDt() 2 2.288e-06 2.288e-06 2.288e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.907e-06 1.907e-06 1.907e-06 0.00% AmrLevel::checkPointPre() 3 1.755e-06 1.755e-06 1.755e-06 0.00% Castro::post_regrid() 1 1.049e-06 1.049e-06 1.049e-06 0.00% Amr::initialInit() 1 1.012e-06 1.012e-06 1.012e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8372 0.8372 0.8372 100.00% Amr::coarseTimeStep() 10 0.6631 0.6631 0.6631 79.20% Amr::timeStep() 10 0.5451 0.5451 0.5451 65.11% Castro::advance() 10 0.5377 0.5377 0.5377 64.23% Castro::subcycle_advance_ctu() 10 0.5256 0.5256 0.5256 62.78% Castro::do_advance_ctu() 10 0.5254 0.5254 0.5254 62.76% Gravity::solve_phi_with_mlmg() 11 0.2799 0.2799 0.2799 33.43% Gravity::actual_solve_with_mlmg() 11 0.2733 0.2733 0.2733 32.64% Castro::construct_new_gravity() 10 0.2551 0.2551 0.2551 30.48% MLMG::solve() 11 0.2531 0.2531 0.2531 30.24% Gravity::solve_for_phi() 10 0.2399 0.2399 0.2399 28.66% MLMG::oneIter() 82 0.239 0.239 0.239 28.55% MLMG::mgVcycle() 82 0.2354 0.2354 0.2354 28.12% VisMF::Write(FabArray) 11 0.2324 0.2324 0.2324 27.76% Castro::construct_ctu_hydro_source() 10 0.1947 0.1947 0.1947 23.26% Amr::checkPoint() 3 0.1725 0.1725 0.1725 20.60% AmrLevel::checkPoint() 3 0.1685 0.1685 0.1685 20.13% StateData::checkPoint() 12 0.1684 0.1684 0.1684 20.12% Amr::init() 1 0.1398 0.1398 0.1398 16.70% MLCellLinOp::smooth() 1640 0.1155 0.1155 0.1155 13.80% MLCellLinOp::applyBC() 4433 0.1076 0.1076 0.1076 12.85% MLMG::mgVcycle_bottom 82 0.07317 0.07317 0.07317 8.74% MLMG::actualBottomSolve() 82 0.07314 0.07314 0.07314 8.74% MLCGSolver::bicgstab 82 0.07244 0.07244 0.07244 8.65% Amr::writePlotFile() 2 0.06699 0.06699 0.06699 8.00% Amr::initialInit() 1 0.04769 0.04769 0.04769 5.70% Castro::clean_state() 62 0.04428 0.04428 0.04428 5.29% Amr::FinalizeInit() 1 0.04356 0.04356 0.04356 5.20% Castro::post_init() 1 0.04229 0.04229 0.04229 5.05% FillPatchIterator::Initialize 41 0.04141 0.04141 0.04141 4.95% Gravity::multilevel_solve_for_new_phi() 1 0.04042 0.04042 0.04042 4.83% Gravity::actual_multilevel_solve() 1 0.04041 0.04041 0.04041 4.83% FillPatchSingleLevel 41 0.03988 0.03988 0.03988 4.76% MLCellLinOp::apply() 1142 0.03597 0.03597 0.03597 4.30% StateDataPhysBCFunct::() 41 0.03586 0.03586 0.03586 4.28% MLMG::mgVcycle_down::0 82 0.03359 0.03359 0.03359 4.01% FabArray::FillBoundary() 4023 0.03265 0.03265 0.03265 3.90% MLPoisson::Fsmooth() 3280 0.03222 0.03222 0.03222 3.85% FillBoundary_nowait() 4023 0.03187 0.03187 0.03187 3.81% MLMG::mgVcycle_up::0 82 0.02538 0.02538 0.02538 3.03% StateData::FillBoundary(geom) 328 0.0235 0.0235 0.0235 2.81% MLCellLinOp::correctionResidual() 492 0.02211 0.02211 0.02211 2.64% Castro::computeTemp() 63 0.02087 0.02087 0.02087 2.49% amrex::Dot() 1114 0.02034 0.02034 0.02034 2.43% MLMG:computeResOfCorrection() 410 0.01952 0.01952 0.01952 2.33% Castro::initialize_do_advance() 10 0.01819 0.01819 0.01819 2.17% Gravity::get_new_grav_vector() 11 0.01679 0.01679 0.01679 2.01% MLPoisson::define() 11 0.01598 0.01598 0.01598 1.91% MLMG::mgVcycle_down::1 82 0.01557 0.01557 0.01557 1.86% Castro::normalize_species() 62 0.01554 0.01554 0.01554 1.86% amrex::Copy() 1029 0.01483 0.01483 0.01483 1.77% Castro::construct_old_gravity() 10 0.01476 0.01476 0.01476 1.76% Gravity::get_old_grav_vector() 10 0.01476 0.01476 0.01476 1.76% MLMG::mgVcycle_down::2 82 0.01447 0.01447 0.01447 1.73% FabArray::norminf() 743 0.01432 0.01432 0.01432 1.71% MLMG::mgVcycle_down::3 82 0.0142 0.0142 0.0142 1.70% FabArray::ParallelCopy() 861 0.01415 0.01415 0.01415 1.69% MLMG::mgVcycle_down::4 82 0.01399 0.01399 0.01399 1.67% FabArray::ParallelCopy_nowait() 861 0.01391 0.01391 0.01391 1.66% Castro::do_new_sources() 10 0.01338 0.01338 0.01338 1.60% FabArray::setVal() 1144 0.01304 0.01304 0.01304 1.56% MLCGSolver::ParallelAllReduce 1514 0.01222 0.01222 0.01222 1.46% MLMG::addInterpCorrection() 410 0.01164 0.01164 0.01164 1.39% Castro::initialize_advance() 10 0.0114 0.0114 0.0114 1.36% MLMG::mgVcycle_up::4 82 0.01138 0.01138 0.01138 1.36% MLMG::mgVcycle_up::1 82 0.01132 0.01132 0.01132 1.35% Castro::expand_state() 10 0.01112 0.01112 0.01112 1.33% MLMG::mgVcycle_up::2 82 0.01106 0.01106 0.01106 1.32% MLCellLinOp::defineAuxData() 11 0.01094 0.01094 0.01094 1.31% amrex::average_down 410 0.01085 0.01085 0.01085 1.30% MLMG::mgVcycle_up::3 82 0.01082 0.01082 0.01082 1.29% MLPoisson::Fapply() 1142 0.0104 0.0104 0.0104 1.24% Castro::do_old_sources() 10 0.009909 0.009909 0.009909 1.18% FabArray::Saxpy() 813 0.008146 0.008146 0.008146 0.97% FabArray::Xpay() 821 0.00804 0.00804 0.00804 0.96% Castro::enforce_min_density() 62 0.007672 0.007672 0.007672 0.92% Castro::post_timestep() 10 0.007203 0.007203 0.007203 0.86% MLCellLinOp::solutionResidual() 93 0.007089 0.007089 0.007089 0.85% Castro::reset_internal_energy(MultiFab) 63 0.006556 0.006556 0.006556 0.78% Gravity::fill_multipole_BCs() 11 0.006376 0.006376 0.006376 0.76% MLMG::computeResidual() 82 0.006125 0.006125 0.006125 0.73% Castro::estTimeStep() 21 0.006015 0.006015 0.006015 0.72% MLCellLinOp::defineBC() 11 0.004788 0.004788 0.004788 0.57% MLMG::prepareForSolve() 11 0.004601 0.004601 0.004601 0.55% BndryData::define() 11 0.004591 0.004591 0.004591 0.55% FabArray::LinComb() 557 0.004525 0.004525 0.004525 0.54% amrex::Add() 164 0.004322 0.004322 0.004322 0.52% Amr::InitializeInit() 1 0.004129 0.004129 0.004129 0.49% Amr::defBaseLevel() 1 0.004124 0.004124 0.004124 0.49% Castro::initData() 1 0.003643 0.003643 0.003643 0.44% Castro::construct_new_source() 50 0.003304 0.003304 0.003304 0.39% Castro::construct_new_gravity_source() 10 0.003266 0.003266 0.003266 0.39% Castro::computeNewDt() 9 0.003049 0.003049 0.003049 0.36% Castro::construct_old_source() 50 0.002626 0.002626 0.002626 0.31% Castro::construct_old_gravity_source() 10 0.002607 0.002607 0.002607 0.31% Castro::reset_internal_energy(Fab) 504 0.002255 0.002255 0.002255 0.27% MLMG::ResNormInf() 93 0.002109 0.002109 0.002109 0.25% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001636 0.001636 0.001636 0.20% FabArrayBase::getCPC() 1323 0.001414 0.001414 0.001414 0.17% MLCellLinOp::setLevelBC() 11 0.001408 0.001408 0.001408 0.17% MLMG::getGradSolution() 11 0.001385 0.001385 0.001385 0.17% MLCellLinOp::compGrad() 11 0.001379 0.001379 0.001379 0.16% FabArray::mult() 43 0.00135 0.00135 0.00135 0.16% FabArray::setDomainBndry() 41 0.001318 0.001318 0.001318 0.16% Castro::check_for_nan() 20 0.001188 0.001188 0.001188 0.14% MultiFab::contains_nan() 20 0.001177 0.001177 0.001177 0.14% MLPoisson::prepareForSolve() 11 0.001112 0.001112 0.001112 0.13% MLCellLinOp::prepareForSolve() 11 0.001103 0.001103 0.001103 0.13% Castro::post_regrid() 1 0.0011 0.0011 0.0011 0.13% Castro::enforce_speed_limit() 62 0.001094 0.001094 0.001094 0.13% MLMG::computeMLResidual() 11 0.001004 0.001004 0.001004 0.12% Gravity::update_max_rhs() 11 0.0008456 0.0008456 0.0008456 0.10% Castro::computeInitialDt() 2 0.0008437 0.0008437 0.0008437 0.10% FabArrayBase::CPC::define() 454 0.0006906 0.0006906 0.0006906 0.08% FabArrayBase::getFB() 4023 0.0006797 0.0006797 0.0006797 0.08% Castro::finalize_advance() 10 0.0006084 0.0006084 0.0006084 0.07% Amr::InitAmr() 1 0.0005375 0.0005375 0.0005375 0.06% Gravity::swapTimeLevels() 10 0.00044 0.00044 0.00044 0.05% Castro::Castro() 1 0.000421 0.000421 0.000421 0.05% MLMG::MLResNormInf() 11 0.0002783 0.0002783 0.0002783 0.03% MultiFab::max() 11 0.0002656 0.0002656 0.0002656 0.03% MLLinOp::define() 11 0.0002369 0.0002369 0.0002369 0.03% MLLinOp::defineGrids() 11 0.0002268 0.0002268 0.0002268 0.03% MLMG::MLRhsNormInf() 11 0.000218 0.000218 0.000218 0.03% Castro::buildMetrics() 1 0.0001578 0.0001578 0.0001578 0.02% FabArrayBase::FB::FB() 56 8.237e-05 8.237e-05 8.237e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.308e-05 5.308e-05 5.308e-05 0.01% Castro::swap_state_time_levels() 10 4.342e-05 4.342e-05 4.342e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.269e-05 4.269e-05 4.269e-05 0.01% makeSFC 55 4.036e-05 4.036e-05 4.036e-05 0.00% Castro::finalize_do_advance() 10 3.487e-05 3.487e-05 3.487e-05 0.00% Castro::enforce_consistent_e() 1 3.484e-05 3.484e-05 3.484e-05 0.00% StateData::define() 4 3.364e-05 3.364e-05 3.364e-05 0.00% Amr::writeSmallPlotFile() 1 2.346e-05 2.346e-05 2.346e-05 0.00% Castro::initMFs() 1 2.044e-05 2.044e-05 2.044e-05 0.00% DistributionMapping::Distribute() 56 1.598e-05 1.598e-05 1.598e-05 0.00% Amr::initSubcycle() 1 8.795e-06 8.795e-06 8.795e-06 0.00% AmrLevel::checkPointPost() 3 4.333e-06 4.333e-06 4.333e-06 0.00% Gravity::set_mass_offset() 11 4.148e-06 4.148e-06 4.148e-06 0.00% Castro::create_source_corrector() 10 4.085e-06 4.085e-06 4.085e-06 0.00% Castro::retry_advance_ctu() 10 4.028e-06 4.028e-06 4.028e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.912e-06 3.912e-06 3.912e-06 0.00% Castro::FluxRegCrseInit 10 2.882e-06 2.882e-06 2.882e-06 0.00% Castro::FluxRegFineAdd() 10 2.349e-06 2.349e-06 2.349e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.907e-06 1.907e-06 1.907e-06 0.00% AmrLevel::checkPointPre() 3 1.755e-06 1.755e-06 1.755e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-24-gea7e8a507e77) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-24-gea7e8a507e77) initialized Starting run at 10:07:27 UTC on 2022-12-26. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 22.12-24-gea7e8a507 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.488238132 Restart time = 0.044685242 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049591589 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049081201 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054962513 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057421533 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.078431455 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033075967 seconds Ending run at 10:07:27 UTC on 2022-12-26. Run time = 0.368247536 Run time without initialization = 0.322981838 Average number of zones advanced per microsecond: 4.058 Average number of zones advanced per microsecond per rank: 4.058 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3683 ... 0.3683 ... 0.3683 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1063 0.1063 0.1063 28.87% VisMF::Read() 3 0.03849 0.03849 0.03849 10.45% MLCellLinOp::applyBC() 1946 0.03249 0.03249 0.03249 8.82% VisMF::Write(FabArray) 1 0.03149 0.03149 0.03149 8.55% MLPoisson::Fsmooth() 1440 0.01396 0.01396 0.01396 3.79% FillBoundary_nowait() 1766 0.01292 0.01292 0.01292 3.51% StateData::FillBoundary(geom) 160 0.01135 0.01135 0.01135 3.08% amrex::Dot() 484 0.008743 0.008743 0.008743 2.37% Castro::normalize_species() 30 0.008461 0.008461 0.008461 2.30% Castro::computeTemp() 30 0.007153 0.007153 0.007153 1.94% amrex::Copy() 463 0.007 0.007 0.007 1.90% FabArray::setVal() 537 0.006283 0.006283 0.006283 1.71% FabArray::norminf() 326 0.006201 0.006201 0.006201 1.68% FabArray::ParallelCopy_nowait() 380 0.005953 0.005953 0.005953 1.62% StateDataPhysBCFunct::() 20 0.005399 0.005399 0.005399 1.47% Castro::enforce_min_density() 30 0.005263 0.005263 0.005263 1.43% MLCellLinOp::defineAuxData() 6 0.005216 0.005216 0.005216 1.42% MLPoisson::Fapply() 500 0.004538 0.004538 0.004538 1.23% FabArray::Saxpy() 355 0.003646 0.003646 0.003646 0.99% FabArray::Xpay() 361 0.003551 0.003551 0.003551 0.96% MLMG::addInterpCorrection() 180 0.002831 0.002831 0.002831 0.77% Gravity::fill_multipole_BCs() 6 0.002625 0.002625 0.002625 0.71% amrex::average_down 180 0.002559 0.002559 0.002559 0.69% Amr::restart() 1 0.002482 0.002482 0.002482 0.67% Castro::estTimeStep() 10 0.002457 0.002457 0.002457 0.67% BndryData::define() 6 0.002041 0.002041 0.002041 0.55% Castro::reset_internal_energy(MultiFab) 30 0.001995 0.001995 0.001995 0.54% FabArray::LinComb() 242 0.001924 0.001924 0.001924 0.52% amrex::Add() 72 0.001853 0.001853 0.001853 0.50% Castro::construct_new_gravity_source() 5 0.001768 0.001768 0.001768 0.48% Castro::do_advance_ctu() 5 0.001486 0.001486 0.001486 0.40% Amr::writePlotFile() 1 0.001419 0.001419 0.001419 0.39% Castro::construct_old_gravity_source() 5 0.001285 0.001285 0.001285 0.35% Castro::enforce_speed_limit() 30 0.001264 0.001264 0.001264 0.34% MLCGSolver::bicgstab 36 0.000944 0.000944 0.000944 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008798 0.0008798 0.0008798 0.24% Castro::reset_internal_energy(Fab) 240 0.0008662 0.0008662 0.0008662 0.24% Castro::post_timestep() 5 0.0008141 0.0008141 0.0008141 0.22% Castro::create_source_corrector() 5 0.0007959 0.0007959 0.0007959 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007624 0.0007624 0.0007624 0.21% MLCellLinOp::setLevelBC() 6 0.0007453 0.0007453 0.0007453 0.20% FabArray::setDomainBndry() 20 0.0006784 0.0006784 0.0006784 0.18% FabArray::mult() 22 0.0006733 0.0006733 0.0006733 0.18% MultiFab::contains_nan() 10 0.0005945 0.0005945 0.0005945 0.16% MLCellLinOp::prepareForSolve() 6 0.0005858 0.0005858 0.0005858 0.16% MLCellLinOp::smooth() 720 0.0005465 0.0005465 0.0005465 0.15% MLCellLinOp::compGrad() 6 0.0004927 0.0004927 0.0004927 0.13% MLMG::prepareForSolve() 6 0.0004594 0.0004594 0.0004594 0.12% FabArrayBase::CPC::define() 244 0.0004164 0.0004164 0.0004164 0.11% Amr::InitAmr() 1 0.0004008 0.0004008 0.0004008 0.11% FabArrayBase::getCPC() 632 0.0003656 0.0003656 0.0003656 0.10% FabArray::FillBoundary() 1766 0.0003491 0.0003491 0.0003491 0.09% Gravity::get_old_grav_vector() 5 0.0003417 0.0003417 0.0003417 0.09% FabArrayBase::getFB() 1766 0.0003155 0.0003155 0.0003155 0.09% Gravity::get_new_grav_vector() 5 0.0002651 0.0002651 0.0002651 0.07% main() 1 0.0002572 0.0002572 0.0002572 0.07% MLCellLinOp::apply() 500 0.0002292 0.0002292 0.0002292 0.06% Amr::coarseTimeStep() 5 0.0002008 0.0002008 0.0002008 0.05% MLMG::mgVcycle() 36 0.0001789 0.0001789 0.0001789 0.05% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.04% MLCGSolver::ParallelAllReduce 659 0.0001299 0.0001299 0.0001299 0.04% Castro::subcycle_advance_ctu() 5 0.0001285 0.0001285 0.0001285 0.03% Castro::construct_new_source() 25 0.0001249 0.0001249 0.0001249 0.03% FabArray::ParallelCopy() 380 0.0001153 0.0001153 0.0001153 0.03% FillPatchIterator::Initialize 20 0.0001114 0.0001114 0.0001114 0.03% MLCellLinOp::defineBC() 6 0.0001057 0.0001057 0.0001057 0.03% MLCellLinOp::correctionResidual() 216 0.0001036 0.0001036 0.0001036 0.03% MLLinOp::defineGrids() 6 0.0001027 0.0001027 0.0001027 0.03% Amr::timeStep() 5 7.717e-05 7.717e-05 7.717e-05 0.02% AmrLevel::restart() 1 7.646e-05 7.646e-05 7.646e-05 0.02% Castro::computeNewDt() 5 6.627e-05 6.627e-05 6.627e-05 0.02% Gravity::solve_for_phi() 5 6.256e-05 6.256e-05 6.256e-05 0.02% StateData::restartDoit() 4 6.056e-05 6.056e-05 6.056e-05 0.02% FabArrayBase::FB::FB() 26 5.985e-05 5.985e-05 5.985e-05 0.02% Castro::finalize_advance() 5 5.918e-05 5.918e-05 5.918e-05 0.02% Gravity::update_max_rhs() 6 5.906e-05 5.906e-05 5.906e-05 0.02% Castro::construct_new_gravity() 5 5.453e-05 5.453e-05 5.453e-05 0.01% MLMG:computeResOfCorrection() 180 5.109e-05 5.109e-05 5.109e-05 0.01% MLMG::mgVcycle_down::0 36 4.371e-05 4.371e-05 4.371e-05 0.01% Castro::clean_state() 30 4.141e-05 4.141e-05 4.141e-05 0.01% MLMG::actualBottomSolve() 36 4.119e-05 4.119e-05 4.119e-05 0.01% MLMG::mgVcycle_down::1 36 4.115e-05 4.115e-05 4.115e-05 0.01% MLMG::mgVcycle_down::4 36 3.811e-05 3.811e-05 3.811e-05 0.01% Castro::expand_state() 5 3.784e-05 3.784e-05 3.784e-05 0.01% MLMG::mgVcycle_down::2 36 3.76e-05 3.76e-05 3.76e-05 0.01% MLMG::mgVcycle_down::3 36 3.712e-05 3.712e-05 3.712e-05 0.01% MLMG::solve() 6 3.554e-05 3.554e-05 3.554e-05 0.01% Castro::advance() 5 3.477e-05 3.477e-05 3.477e-05 0.01% Castro::initialize_advance() 5 3.344e-05 3.344e-05 3.344e-05 0.01% MLMG::mgVcycle_up::4 36 3.174e-05 3.174e-05 3.174e-05 0.01% MLMG::mgVcycle_up::0 36 2.647e-05 2.647e-05 2.647e-05 0.01% Castro::buildMetrics() 1 2.51e-05 2.51e-05 2.51e-05 0.01% MLMG::mgVcycle_up::3 36 2.424e-05 2.424e-05 2.424e-05 0.01% MLMG::mgVcycle_up::2 36 2.388e-05 2.388e-05 2.388e-05 0.01% Castro::initialize_do_advance() 5 2.359e-05 2.359e-05 2.359e-05 0.01% Amr::writeSmallPlotFile() 1 2.342e-05 2.342e-05 2.342e-05 0.01% MLMG::oneIter() 36 2.335e-05 2.335e-05 2.335e-05 0.01% MLMG::mgVcycle_up::1 36 2.293e-05 2.293e-05 2.293e-05 0.01% MLCellLinOp::solutionResidual() 42 2.239e-05 2.239e-05 2.239e-05 0.01% Castro::swap_state_time_levels() 5 2.132e-05 2.132e-05 2.132e-05 0.01% Castro::initMFs() 1 2.048e-05 2.048e-05 2.048e-05 0.01% Castro::post_restart() 1 2.035e-05 2.035e-05 2.035e-05 0.01% Castro::finalize_do_advance() 5 1.817e-05 1.817e-05 1.817e-05 0.00% MLMG::ResNormInf() 42 1.697e-05 1.697e-05 1.697e-05 0.00% MLMG::mgVcycle_bottom 36 1.515e-05 1.515e-05 1.515e-05 0.00% FillPatchSingleLevel 20 1.495e-05 1.495e-05 1.495e-05 0.00% MLMG::computeResidual() 36 1.38e-05 1.38e-05 1.38e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.318e-05 1.318e-05 1.318e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.308e-05 1.308e-05 1.308e-05 0.00% MLPoisson::define() 6 1.304e-05 1.304e-05 1.304e-05 0.00% makeSFC 30 1.289e-05 1.289e-05 1.289e-05 0.00% Castro::construct_old_source() 25 1.034e-05 1.034e-05 1.034e-05 0.00% Castro::do_new_sources() 5 9.568e-06 9.568e-06 9.568e-06 0.00% DistributionMapping::Distribute() 31 9.042e-06 9.042e-06 9.042e-06 0.00% Amr::initSubcycle() 1 8.168e-06 8.168e-06 8.168e-06 0.00% Castro::do_old_sources() 5 8.141e-06 8.141e-06 8.141e-06 0.00% MLMG::computeMLResidual() 6 7.952e-06 7.952e-06 7.952e-06 0.00% Castro::check_for_nan() 10 7.279e-06 7.279e-06 7.279e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.268e-06 7.268e-06 7.268e-06 0.00% Gravity::actual_multilevel_solve() 1 6.87e-06 6.87e-06 6.87e-06 0.00% Castro::construct_old_gravity() 5 6.238e-06 6.238e-06 6.238e-06 0.00% Castro::apply_source_to_state() 10 5.798e-06 5.798e-06 5.798e-06 0.00% MLLinOp::define() 6 5.16e-06 5.16e-06 5.16e-06 0.00% MLPoisson::prepareForSolve() 6 4.233e-06 4.233e-06 4.233e-06 0.00% Gravity::swapTimeLevels() 5 4.118e-06 4.118e-06 4.118e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.834e-06 3.834e-06 3.834e-06 0.00% MLMG::getGradSolution() 6 2.923e-06 2.923e-06 2.923e-06 0.00% Castro::retry_advance_ctu() 5 2.659e-06 2.659e-06 2.659e-06 0.00% MLMG::MLResNormInf() 6 2.449e-06 2.449e-06 2.449e-06 0.00% MLMG::MLRhsNormInf() 6 2.204e-06 2.204e-06 2.204e-06 0.00% Gravity::set_mass_offset() 6 2.017e-06 2.017e-06 2.017e-06 0.00% Castro::FluxRegCrseInit 5 1.705e-06 1.705e-06 1.705e-06 0.00% Amr::init() 1 1.369e-06 1.369e-06 1.369e-06 0.00% Castro::FluxRegFineAdd() 5 1.236e-06 1.236e-06 1.236e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.049e-06 1.049e-06 1.049e-06 0.00% AmrLevel::AmrLevel() 1 8.26e-07 8.26e-07 8.26e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3683 0.3683 0.3683 100.00% Amr::coarseTimeStep() 5 0.2897 0.2897 0.2897 78.66% Amr::timeStep() 5 0.2882 0.2882 0.2882 78.24% Castro::advance() 5 0.2823 0.2823 0.2823 76.65% Castro::subcycle_advance_ctu() 5 0.2757 0.2757 0.2757 74.86% Castro::do_advance_ctu() 5 0.2756 0.2756 0.2756 74.82% Castro::construct_new_gravity() 5 0.1285 0.1285 0.1285 34.88% Gravity::solve_phi_with_mlmg() 6 0.1237 0.1237 0.1237 33.60% Gravity::actual_solve_with_mlmg() 6 0.1209 0.1209 0.1209 32.82% Gravity::solve_for_phi() 5 0.1207 0.1207 0.1207 32.77% MLMG::solve() 6 0.1098 0.1098 0.1098 29.82% Castro::construct_ctu_hydro_source() 5 0.1063 0.1063 0.1063 28.87% MLMG::oneIter() 36 0.103 0.103 0.103 27.96% MLMG::mgVcycle() 36 0.1014 0.1014 0.1014 27.54% MLCellLinOp::smooth() 720 0.04959 0.04959 0.04959 13.47% MLCellLinOp::applyBC() 1946 0.04614 0.04614 0.04614 12.53% Amr::init() 1 0.04473 0.04473 0.04473 12.15% Amr::restart() 1 0.04473 0.04473 0.04473 12.15% AmrLevel::restart() 1 0.03869 0.03869 0.03869 10.51% StateData::restartDoit() 4 0.03861 0.03861 0.03861 10.48% VisMF::Read() 3 0.03849 0.03849 0.03849 10.45% Amr::writePlotFile() 1 0.03316 0.03316 0.03316 9.00% MLMG::mgVcycle_bottom 36 0.03156 0.03156 0.03156 8.57% MLMG::actualBottomSolve() 36 0.03154 0.03154 0.03154 8.57% VisMF::Write(FabArray) 1 0.03149 0.03149 0.03149 8.55% MLCGSolver::bicgstab 36 0.03124 0.03124 0.03124 8.48% Castro::clean_state() 30 0.02504 0.02504 0.02504 6.80% FillPatchIterator::Initialize 20 0.01952 0.01952 0.01952 5.30% FillPatchSingleLevel 20 0.01873 0.01873 0.01873 5.09% StateDataPhysBCFunct::() 20 0.01675 0.01675 0.01675 4.55% MLCellLinOp::apply() 500 0.01556 0.01556 0.01556 4.22% MLMG::mgVcycle_down::0 36 0.01423 0.01423 0.01423 3.86% MLPoisson::Fsmooth() 1440 0.01396 0.01396 0.01396 3.79% FabArray::FillBoundary() 1766 0.01365 0.01365 0.01365 3.71% FillBoundary_nowait() 1766 0.0133 0.0133 0.0133 3.61% StateData::FillBoundary(geom) 160 0.01135 0.01135 0.01135 3.08% MLMG::mgVcycle_up::0 36 0.01076 0.01076 0.01076 2.92% Castro::computeTemp() 30 0.01001 0.01001 0.01001 2.72% MLCellLinOp::correctionResidual() 216 0.00947 0.00947 0.00947 2.57% Castro::initialize_do_advance() 5 0.009147 0.009147 0.009147 2.48% MLPoisson::define() 6 0.00878 0.00878 0.00878 2.38% amrex::Dot() 484 0.008743 0.008743 0.008743 2.37% Castro::normalize_species() 30 0.008461 0.008461 0.008461 2.30% MLMG:computeResOfCorrection() 180 0.008323 0.008323 0.008323 2.26% Castro::do_new_sources() 5 0.007724 0.007724 0.007724 2.10% Gravity::get_new_grav_vector() 5 0.007616 0.007616 0.007616 2.07% Castro::construct_old_gravity() 5 0.007246 0.007246 0.007246 1.97% Gravity::get_old_grav_vector() 5 0.007239 0.007239 0.007239 1.97% amrex::Copy() 463 0.007 0.007 0.007 1.90% MLMG::mgVcycle_down::1 36 0.006761 0.006761 0.006761 1.84% FabArray::ParallelCopy() 380 0.006462 0.006462 0.006462 1.75% FabArray::ParallelCopy_nowait() 380 0.006347 0.006347 0.006347 1.72% MLMG::mgVcycle_down::2 36 0.006315 0.006315 0.006315 1.71% FabArray::setVal() 537 0.006283 0.006283 0.006283 1.71% Castro::initialize_advance() 5 0.006223 0.006223 0.006223 1.69% FabArray::norminf() 326 0.006201 0.006201 0.006201 1.68% MLMG::mgVcycle_down::3 36 0.006177 0.006177 0.006177 1.68% MLMG::mgVcycle_down::4 36 0.006112 0.006112 0.006112 1.66% MLCellLinOp::defineAuxData() 6 0.005946 0.005946 0.005946 1.61% Castro::post_timestep() 5 0.005801 0.005801 0.005801 1.58% Castro::expand_state() 5 0.005479 0.005479 0.005479 1.49% Castro::do_old_sources() 5 0.005339 0.005339 0.005339 1.45% MLCGSolver::ParallelAllReduce 659 0.005282 0.005282 0.005282 1.43% Castro::enforce_min_density() 30 0.005263 0.005263 0.005263 1.43% MLMG::addInterpCorrection() 180 0.005066 0.005066 0.005066 1.38% MLMG::mgVcycle_up::4 36 0.004969 0.004969 0.004969 1.35% MLMG::mgVcycle_up::1 36 0.004882 0.004882 0.004882 1.33% amrex::average_down 180 0.004815 0.004815 0.004815 1.31% MLMG::mgVcycle_up::2 36 0.00478 0.00478 0.00478 1.30% MLMG::mgVcycle_up::3 36 0.004714 0.004714 0.004714 1.28% MLPoisson::Fapply() 500 0.004538 0.004538 0.004538 1.23% FabArray::Saxpy() 355 0.003646 0.003646 0.003646 0.99% FabArray::Xpay() 361 0.003551 0.003551 0.003551 0.96% Castro::post_restart() 1 0.003383 0.003383 0.003383 0.92% Gravity::multilevel_solve_for_new_phi() 1 0.003273 0.003273 0.003273 0.89% Gravity::actual_multilevel_solve() 1 0.00326 0.00326 0.00326 0.89% MLCellLinOp::solutionResidual() 42 0.003232 0.003232 0.003232 0.88% Castro::reset_internal_energy(MultiFab) 30 0.002861 0.002861 0.002861 0.78% Gravity::fill_multipole_BCs() 6 0.002753 0.002753 0.002753 0.75% MLMG::computeResidual() 36 0.002687 0.002687 0.002687 0.73% MLCellLinOp::defineBC() 6 0.002684 0.002684 0.002684 0.73% BndryData::define() 6 0.002578 0.002578 0.002578 0.70% MLMG::prepareForSolve() 6 0.002478 0.002478 0.002478 0.67% Castro::estTimeStep() 10 0.002457 0.002457 0.002457 0.67% FabArray::LinComb() 242 0.001924 0.001924 0.001924 0.52% Castro::construct_new_source() 25 0.001893 0.001893 0.001893 0.51% amrex::Add() 72 0.001853 0.001853 0.001853 0.50% Castro::construct_new_gravity_source() 5 0.001768 0.001768 0.001768 0.48% Castro::computeNewDt() 5 0.001328 0.001328 0.001328 0.36% Castro::construct_old_source() 25 0.001296 0.001296 0.001296 0.35% Castro::construct_old_gravity_source() 5 0.001285 0.001285 0.001285 0.35% Castro::enforce_speed_limit() 30 0.001264 0.001264 0.001264 0.34% MLMG::ResNormInf() 42 0.0009482 0.0009482 0.0009482 0.26% Castro::apply_source_to_state() 10 0.0009226 0.0009226 0.0009226 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008798 0.0008798 0.0008798 0.24% Castro::reset_internal_energy(Fab) 240 0.0008662 0.0008662 0.0008662 0.24% Castro::create_source_corrector() 5 0.0007959 0.0007959 0.0007959 0.22% FabArrayBase::getCPC() 632 0.0007821 0.0007821 0.0007821 0.21% MLMG::getGradSolution() 6 0.0007559 0.0007559 0.0007559 0.21% MLCellLinOp::compGrad() 6 0.000753 0.000753 0.000753 0.20% MLCellLinOp::setLevelBC() 6 0.0007453 0.0007453 0.0007453 0.20% FabArray::setDomainBndry() 20 0.0006784 0.0006784 0.0006784 0.18% FabArray::mult() 22 0.0006733 0.0006733 0.0006733 0.18% Castro::check_for_nan() 10 0.0006018 0.0006018 0.0006018 0.16% MultiFab::contains_nan() 10 0.0005945 0.0005945 0.0005945 0.16% MLPoisson::prepareForSolve() 6 0.00059 0.00059 0.00059 0.16% MLCellLinOp::prepareForSolve() 6 0.0005858 0.0005858 0.0005858 0.16% MLMG::computeMLResidual() 6 0.0005665 0.0005665 0.0005665 0.15% Gravity::update_max_rhs() 6 0.0004463 0.0004463 0.0004463 0.12% FabArrayBase::CPC::define() 244 0.0004164 0.0004164 0.0004164 0.11% Amr::InitAmr() 1 0.000409 0.000409 0.000409 0.11% FabArrayBase::getFB() 1766 0.0003754 0.0003754 0.0003754 0.10% Castro::finalize_advance() 5 0.0003319 0.0003319 0.0003319 0.09% Gravity::swapTimeLevels() 5 0.0002256 0.0002256 0.0002256 0.06% Castro::buildMetrics() 1 0.0001516 0.0001516 0.0001516 0.04% MLMG::MLResNormInf() 6 0.0001506 0.0001506 0.0001506 0.04% MLLinOp::define() 6 0.0001367 0.0001367 0.0001367 0.04% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.04% MLLinOp::defineGrids() 6 0.0001315 0.0001315 0.0001315 0.04% MLMG::MLRhsNormInf() 6 0.0001154 0.0001154 0.0001154 0.03% FabArrayBase::FB::FB() 26 5.985e-05 5.985e-05 5.985e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.781e-05 2.781e-05 2.781e-05 0.01% Amr::writeSmallPlotFile() 1 2.342e-05 2.342e-05 2.342e-05 0.01% Castro::swap_state_time_levels() 5 2.132e-05 2.132e-05 2.132e-05 0.01% makeSFC 30 2.054e-05 2.054e-05 2.054e-05 0.01% Castro::initMFs() 1 2.048e-05 2.048e-05 2.048e-05 0.01% Castro::finalize_do_advance() 5 1.817e-05 1.817e-05 1.817e-05 0.00% DistributionMapping::Distribute() 31 9.042e-06 9.042e-06 9.042e-06 0.00% Amr::initSubcycle() 1 8.168e-06 8.168e-06 8.168e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.228e-06 5.228e-06 5.228e-06 0.00% Castro::retry_advance_ctu() 5 2.659e-06 2.659e-06 2.659e-06 0.00% Gravity::set_mass_offset() 6 2.017e-06 2.017e-06 2.017e-06 0.00% Castro::FluxRegCrseInit 5 1.705e-06 1.705e-06 1.705e-06 0.00% Castro::FluxRegFineAdd() 5 1.236e-06 1.236e-06 1.236e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.049e-06 1.049e-06 1.049e-06 0.00% AmrLevel::AmrLevel() 1 8.26e-07 8.26e-07 8.26e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-24-gea7e8a507e77) finalized