Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-33-g4d6413c45fa0) initialized Starting run at 09:53:14 UTC on 2022-11-24. Successfully read inputs file ... Castro git describe: 22.11-14-g129e2797f AMReX git describe: 22.11-33-g4d6413c45 Microphysics git describe: 22.11-41-g63248ba9 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.054165631 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.048446353 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049889847 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052343016 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.062716356 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.061116889 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.068656979 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.053013272 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.091175082 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.061262884 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060447051 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060488287 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.057012961 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.050042857 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031236896 seconds Ending run at 09:53:14 UTC on 2022-11-24. Run time = 0.916141413 Run time without initialization = 0.760061016 Average number of zones advanced per microsecond: 3.449 Average number of zones advanced per microsecond per rank: 3.449 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.9162 ... 0.9162 ... 0.9162 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2279 0.2279 0.2279 24.87% Castro::construct_ctu_hydro_source() 10 0.2152 0.2152 0.2152 23.49% MLCellLinOp::applyBC() 4433 0.0822 0.0822 0.0822 8.97% MLPoisson::Fsmooth() 3280 0.06516 0.06516 0.06516 7.11% StateDataPhysBCFunct::() 41 0.02678 0.02678 0.02678 2.92% StateData::FillBoundary(geom) 328 0.02444 0.02444 0.02444 2.67% amrex::Dot() 1114 0.02296 0.02296 0.02296 2.51% FabArray::setVal() 1144 0.01898 0.01898 0.01898 2.07% amrex::Copy() 1029 0.01605 0.01605 0.01605 1.75% FabArray::LinComb() 1586 0.01471 0.01471 0.01471 1.61% Castro::normalize_species() 62 0.01468 0.01468 0.01468 1.60% Castro::computeTemp() 63 0.0146 0.0146 0.0146 1.59% FillBoundary_nowait() 4023 0.01431 0.01431 0.01431 1.56% FabArray::ParallelCopy_nowait() 861 0.01365 0.01365 0.01365 1.49% FabArray::norminf() 639 0.01328 0.01328 0.01328 1.45% MLCellLinOp::defineAuxData() 11 0.01217 0.01217 0.01217 1.33% MLPoisson::Fapply() 1142 0.01207 0.01207 0.01207 1.32% Gravity::fill_multipole_BCs() 11 0.008529 0.008529 0.008529 0.93% MLMG::addInterpCorrection() 410 0.007885 0.007885 0.007885 0.86% Castro::enforce_min_density() 62 0.00714 0.00714 0.00714 0.78% amrex::average_down 410 0.006981 0.006981 0.006981 0.76% FabArray::Xpay() 585 0.006744 0.006744 0.006744 0.74% Amr::checkPoint() 3 0.00616 0.00616 0.00616 0.67% Castro::estTimeStep() 21 0.004654 0.004654 0.004654 0.51% BndryData::define() 11 0.004045 0.004045 0.004045 0.44% Castro::reset_internal_energy(MultiFab) 63 0.00399 0.00399 0.00399 0.44% Castro::construct_new_gravity_source() 10 0.003292 0.003292 0.003292 0.36% MLCGSolver::bicgstab 82 0.003276 0.003276 0.003276 0.36% Castro::do_advance_ctu() 10 0.00301 0.00301 0.00301 0.33% Castro::construct_old_gravity_source() 10 0.002705 0.002705 0.002705 0.30% amrex::Add() 164 0.002573 0.002573 0.002573 0.28% Amr::writePlotFile() 2 0.002424 0.002424 0.002424 0.26% MLMG::ResNormInf() 93 0.00213 0.00213 0.00213 0.23% FabArray::Saxpy() 20 0.001813 0.001813 0.001813 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00173 0.00173 0.00173 0.19% Castro::reset_internal_energy(Fab) 504 0.001714 0.001714 0.001714 0.19% MLCellLinOp::setLevelBC() 11 0.001574 0.001574 0.001574 0.17% Gravity::actual_solve_with_mlmg() 11 0.001402 0.001402 0.001402 0.15% FabArray::setDomainBndry() 41 0.001398 0.001398 0.001398 0.15% Castro::initData() 1 0.001363 0.001363 0.001363 0.15% FabArray::mult() 43 0.001349 0.001349 0.001349 0.15% MLMG::prepareForSolve() 11 0.001252 0.001252 0.001252 0.14% MLCellLinOp::prepareForSolve() 11 0.001203 0.001203 0.001203 0.13% MultiFab::contains_nan() 20 0.001194 0.001194 0.001194 0.13% Castro::enforce_speed_limit() 62 0.001103 0.001103 0.001103 0.12% MLCellLinOp::smooth() 1640 0.00102 0.00102 0.00102 0.11% MLCellLinOp::compGrad() 11 0.000954 0.000954 0.000954 0.10% FabArray::FillBoundary() 4023 0.0008315 0.0008315 0.0008315 0.09% FabArrayBase::getCPC() 1323 0.0007803 0.0007803 0.0007803 0.09% Gravity::get_new_grav_vector() 11 0.0007508 0.0007508 0.0007508 0.08% FabArrayBase::CPC::define() 454 0.0006642 0.0006642 0.0006642 0.07% FabArrayBase::getFB() 4023 0.0006154 0.0006154 0.0006154 0.07% Gravity::get_old_grav_vector() 10 0.0005623 0.0005623 0.0005623 0.06% MLCellLinOp::apply() 1142 0.0004881 0.0004881 0.0004881 0.05% Amr::InitAmr() 1 0.0004721 0.0004721 0.0004721 0.05% MLLinOp::defineGrids() 11 0.0004428 0.0004428 0.0004428 0.05% FillPatchIterator::Initialize 41 0.0003613 0.0003613 0.0003613 0.04% CGSolver::sxay() 1586 0.0003535 0.0003535 0.0003535 0.04% Amr::coarseTimeStep() 10 0.0003447 0.0003447 0.0003447 0.04% MLCellLinOp::defineBC() 11 0.000317 0.000317 0.000317 0.03% MLCGSolver::ParallelAllReduce 1514 0.0003022 0.0003022 0.0003022 0.03% FabArray::ParallelCopy() 861 0.0002729 0.0002729 0.0002729 0.03% main() 1 0.0002716 0.0002716 0.0002716 0.03% MultiFab::max() 11 0.0002711 0.0002711 0.0002711 0.03% MLCellLinOp::correctionResidual() 492 0.0002432 0.0002432 0.0002432 0.03% MLMG::MLRhsNormInf() 11 0.0002207 0.0002207 0.0002207 0.02% MLMG::mgVcycle() 82 0.0002157 0.0002157 0.0002157 0.02% Amr::timeStep() 10 0.0001763 0.0001763 0.0001763 0.02% MLMG:computeResOfCorrection() 410 0.0001637 0.0001637 0.0001637 0.02% Castro::subcycle_advance_ctu() 10 0.0001488 0.0001488 0.0001488 0.02% Castro::finalize_advance() 10 0.0001441 0.0001441 0.0001441 0.02% Gravity::update_max_rhs() 11 0.0001376 0.0001376 0.0001376 0.02% StateData::checkPoint() 12 0.0001305 0.0001305 0.0001305 0.01% Gravity::solve_for_phi() 10 0.0001287 0.0001287 0.0001287 0.01% MLMG::mgVcycle_down::0 82 0.0001192 0.0001192 0.0001192 0.01% MLMG::mgVcycle_down::1 82 0.0001072 0.0001072 0.0001072 0.01% MLMG::mgVcycle_down::2 82 9.864e-05 9.864e-05 9.864e-05 0.01% Castro::advance() 10 9.47e-05 9.47e-05 9.47e-05 0.01% Castro::Castro() 1 9.203e-05 9.203e-05 9.203e-05 0.01% MLMG::mgVcycle_down::3 82 9.106e-05 9.106e-05 9.106e-05 0.01% MLMG::mgVcycle_down::4 82 8.983e-05 8.983e-05 8.983e-05 0.01% FabArrayBase::FB::FB() 56 8.727e-05 8.727e-05 8.727e-05 0.01% Castro::initialize_advance() 10 8.462e-05 8.462e-05 8.462e-05 0.01% MLMG::actualBottomSolve() 82 8.064e-05 8.064e-05 8.064e-05 0.01% Castro::expand_state() 10 7.837e-05 7.837e-05 7.837e-05 0.01% Castro::clean_state() 62 7.734e-05 7.734e-05 7.734e-05 0.01% AmrLevel::checkPoint() 3 7.67e-05 7.67e-05 7.67e-05 0.01% MLMG::mgVcycle_up::4 82 7.574e-05 7.574e-05 7.574e-05 0.01% MLMG::solve() 11 7.204e-05 7.204e-05 7.204e-05 0.01% Castro::initialize_do_advance() 10 6.5e-05 6.5e-05 6.5e-05 0.01% MLMG::mgVcycle_up::0 82 6.33e-05 6.33e-05 6.33e-05 0.01% MLMG::mgVcycle_up::3 82 6.171e-05 6.171e-05 6.171e-05 0.01% MLMG::mgVcycle_up::1 82 6.076e-05 6.076e-05 6.076e-05 0.01% MLMG::mgVcycle_up::2 82 5.993e-05 5.993e-05 5.993e-05 0.01% MLMG::oneIter() 82 5.912e-05 5.912e-05 5.912e-05 0.01% Castro::swap_state_time_levels() 10 5.689e-05 5.689e-05 5.689e-05 0.01% MLCellLinOp::solutionResidual() 93 5.649e-05 5.649e-05 5.649e-05 0.01% StateData::define() 4 5.173e-05 5.173e-05 5.173e-05 0.01% MLMG::computeResidual() 82 4.658e-05 4.658e-05 4.658e-05 0.01% Castro::finalize_do_advance() 10 3.526e-05 3.526e-05 3.526e-05 0.00% Castro::enforce_consistent_e() 1 3.423e-05 3.423e-05 3.423e-05 0.00% MultiFab::Add() 82 3.299e-05 3.299e-05 3.299e-05 0.00% MLMG::mgVcycle_bottom 82 3.096e-05 3.096e-05 3.096e-05 0.00% FillPatchSingleLevel 41 2.9e-05 2.9e-05 2.9e-05 0.00% MLPoisson::define() 11 2.81e-05 2.81e-05 2.81e-05 0.00% Castro::construct_new_gravity() 10 2.777e-05 2.777e-05 2.777e-05 0.00% makeSFC 55 2.669e-05 2.669e-05 2.669e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.648e-05 2.648e-05 2.648e-05 0.00% Castro::initMFs() 1 2.567e-05 2.567e-05 2.567e-05 0.00% Amr::writeSmallPlotFile() 1 2.553e-05 2.553e-05 2.553e-05 0.00% MLLinOp::define() 11 2.487e-05 2.487e-05 2.487e-05 0.00% Castro::create_source_corrector() 10 2.356e-05 2.356e-05 2.356e-05 0.00% Castro::buildMetrics() 1 2.218e-05 2.218e-05 2.218e-05 0.00% Amr::FinalizeInit() 1 2.047e-05 2.047e-05 2.047e-05 0.00% Castro::construct_old_source() 50 1.952e-05 1.952e-05 1.952e-05 0.00% Amr::defBaseLevel() 1 1.944e-05 1.944e-05 1.944e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.889e-05 1.889e-05 1.889e-05 0.00% Castro::construct_new_source() 50 1.814e-05 1.814e-05 1.814e-05 0.00% Castro::do_new_sources() 10 1.641e-05 1.641e-05 1.641e-05 0.00% Castro::do_old_sources() 10 1.554e-05 1.554e-05 1.554e-05 0.00% DistributionMapping::Distribute() 56 1.47e-05 1.47e-05 1.47e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.428e-05 1.428e-05 1.428e-05 0.00% Castro::check_for_nan() 20 1.269e-05 1.269e-05 1.269e-05 0.00% Castro::post_timestep() 10 1.19e-05 1.19e-05 1.19e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.172e-05 1.172e-05 1.172e-05 0.00% Castro::apply_source_to_state() 20 1.104e-05 1.104e-05 1.104e-05 0.00% Castro::post_init() 1 1.013e-05 1.013e-05 1.013e-05 0.00% MLMG::computeMLResidual() 11 1.001e-05 1.001e-05 1.001e-05 0.00% MLPoisson::prepareForSolve() 11 9.968e-06 9.968e-06 9.968e-06 0.00% Castro::construct_old_gravity() 10 9.703e-06 9.703e-06 9.703e-06 0.00% Gravity::swapTimeLevels() 10 9.466e-06 9.466e-06 9.466e-06 0.00% Gravity::actual_multilevel_solve() 1 8.719e-06 8.719e-06 8.719e-06 0.00% Amr::initSubcycle() 1 8.531e-06 8.531e-06 8.531e-06 0.00% Castro::computeNewDt() 9 7.109e-06 7.109e-06 7.109e-06 0.00% MLMG::getGradSolution() 11 6.66e-06 6.66e-06 6.66e-06 0.00% MultiFab::Copy() 11 5.201e-06 5.201e-06 5.201e-06 0.00% Amr::InitializeInit() 1 4.758e-06 4.758e-06 4.758e-06 0.00% Gravity::set_mass_offset() 11 4.34e-06 4.34e-06 4.34e-06 0.00% AmrLevel::checkPointPost() 3 3.963e-06 3.963e-06 3.963e-06 0.00% Castro::retry_advance_ctu() 10 3.82e-06 3.82e-06 3.82e-06 0.00% MLMG::MLResNormInf() 11 3.78e-06 3.78e-06 3.78e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.193e-06 3.193e-06 3.193e-06 0.00% Castro::FluxRegCrseInit 10 2.664e-06 2.664e-06 2.664e-06 0.00% Castro::computeInitialDt() 2 2.563e-06 2.563e-06 2.563e-06 0.00% Amr::init() 1 2.316e-06 2.316e-06 2.316e-06 0.00% Castro::FluxRegFineAdd() 10 2.248e-06 2.248e-06 2.248e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.031e-06 2.031e-06 2.031e-06 0.00% AmrLevel::checkPointPre() 3 1.902e-06 1.902e-06 1.902e-06 0.00% Castro::post_regrid() 1 1.187e-06 1.187e-06 1.187e-06 0.00% Amr::initialInit() 1 1.024e-06 1.024e-06 1.024e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9162 0.9162 0.9162 100.00% Amr::coarseTimeStep() 10 0.7286 0.7286 0.7286 79.53% Amr::timeStep() 10 0.6222 0.6222 0.6222 67.92% Castro::advance() 10 0.616 0.616 0.616 67.23% Castro::subcycle_advance_ctu() 10 0.6039 0.6039 0.6039 65.91% Castro::do_advance_ctu() 10 0.6037 0.6037 0.6037 65.90% Gravity::solve_phi_with_mlmg() 11 0.3231 0.3231 0.3231 35.26% Gravity::actual_solve_with_mlmg() 11 0.3141 0.3141 0.3141 34.28% Castro::construct_new_gravity() 10 0.3129 0.3129 0.3129 34.15% MLMG::solve() 11 0.2902 0.2902 0.2902 31.67% Gravity::solve_for_phi() 10 0.278 0.278 0.278 30.34% MLMG::oneIter() 82 0.2748 0.2748 0.2748 29.99% MLMG::mgVcycle() 82 0.273 0.273 0.273 29.80% VisMF::Write(FabArray) 11 0.2279 0.2279 0.2279 24.87% Castro::construct_ctu_hydro_source() 10 0.2152 0.2152 0.2152 23.49% Amr::checkPoint() 3 0.1574 0.1574 0.1574 17.18% Amr::init() 1 0.1555 0.1555 0.1555 16.97% AmrLevel::checkPoint() 3 0.1512 0.1512 0.1512 16.51% StateData::checkPoint() 12 0.1511 0.1511 0.1511 16.50% MLCellLinOp::smooth() 1640 0.139 0.139 0.139 15.17% MLCellLinOp::applyBC() 4433 0.09805 0.09805 0.09805 10.70% MLMG::mgVcycle_bottom 82 0.08442 0.08442 0.08442 9.21% MLMG::actualBottomSolve() 82 0.08439 0.08439 0.08439 9.21% MLCGSolver::bicgstab 82 0.08354 0.08354 0.08354 9.12% Amr::writePlotFile() 2 0.0798 0.0798 0.0798 8.71% MLPoisson::Fsmooth() 3280 0.06516 0.06516 0.06516 7.11% FillPatchIterator::Initialize 41 0.05719 0.05719 0.05719 6.24% FillPatchSingleLevel 41 0.05543 0.05543 0.05543 6.05% Amr::initialInit() 1 0.05275 0.05275 0.05275 5.76% StateDataPhysBCFunct::() 41 0.05122 0.05122 0.05122 5.59% Amr::FinalizeInit() 1 0.04864 0.04864 0.04864 5.31% Castro::post_init() 1 0.0473 0.0473 0.0473 5.16% Gravity::multilevel_solve_for_new_phi() 1 0.04556 0.04556 0.04556 4.97% Gravity::actual_multilevel_solve() 1 0.04554 0.04554 0.04554 4.97% Castro::clean_state() 62 0.04262 0.04262 0.04262 4.65% MLCellLinOp::apply() 1142 0.03729 0.03729 0.03729 4.07% Gravity::get_new_grav_vector() 11 0.03632 0.03632 0.03632 3.96% MLMG::mgVcycle_down::0 82 0.03587 0.03587 0.03587 3.92% MLMG::mgVcycle_up::0 82 0.03087 0.03087 0.03087 3.37% StateData::FillBoundary(geom) 328 0.02444 0.02444 0.02444 2.67% amrex::Dot() 1114 0.02296 0.02296 0.02296 2.51% MLCellLinOp::correctionResidual() 492 0.02184 0.02184 0.02184 2.38% Castro::computeTemp() 63 0.02031 0.02031 0.02031 2.22% Castro::initialize_do_advance() 10 0.01977 0.01977 0.01977 2.16% MLPoisson::define() 11 0.01944 0.01944 0.01944 2.12% FabArray::setVal() 1144 0.01898 0.01898 0.01898 2.07% MLMG:computeResOfCorrection() 410 0.01887 0.01887 0.01887 2.06% MLMG::mgVcycle_down::1 82 0.01823 0.01823 0.01823 1.99% MLMG::mgVcycle_down::2 82 0.01771 0.01771 0.01771 1.93% MLMG::mgVcycle_down::3 82 0.01685 0.01685 0.01685 1.84% amrex::Copy() 1029 0.01605 0.01605 0.01605 1.75% MLMG::mgVcycle_down::4 82 0.01602 0.01602 0.01602 1.75% FabArray::FillBoundary() 4023 0.01585 0.01585 0.01585 1.73% Castro::construct_old_gravity() 10 0.01512 0.01512 0.01512 1.65% Gravity::get_old_grav_vector() 10 0.01511 0.01511 0.01511 1.65% CGSolver::sxay() 1586 0.01507 0.01507 0.01507 1.64% FillBoundary_nowait() 4023 0.01502 0.01502 0.01502 1.64% FabArray::ParallelCopy() 861 0.01474 0.01474 0.01474 1.61% FabArray::LinComb() 1586 0.01471 0.01471 0.01471 1.61% Castro::normalize_species() 62 0.01468 0.01468 0.01468 1.60% FabArray::ParallelCopy_nowait() 861 0.01447 0.01447 0.01447 1.58% MLCGSolver::ParallelAllReduce 1514 0.0137 0.0137 0.0137 1.50% MLMG::mgVcycle_up::2 82 0.01368 0.01368 0.01368 1.49% MLCellLinOp::defineAuxData() 11 0.01355 0.01355 0.01355 1.48% MLMG::mgVcycle_up::1 82 0.01348 0.01348 0.01348 1.47% FabArray::norminf() 639 0.01328 0.01328 0.01328 1.45% MLMG::addInterpCorrection() 410 0.01319 0.01319 0.01319 1.44% MLMG::mgVcycle_up::3 82 0.01288 0.01288 0.01288 1.41% MLMG::mgVcycle_up::4 82 0.01277 0.01277 0.01277 1.39% Castro::do_new_sources() 10 0.01255 0.01255 0.01255 1.37% amrex::average_down 410 0.01224 0.01224 0.01224 1.34% MLPoisson::Fapply() 1142 0.01207 0.01207 0.01207 1.32% Castro::expand_state() 10 0.01156 0.01156 0.01156 1.26% Castro::initialize_advance() 10 0.01129 0.01129 0.01129 1.23% Castro::do_old_sources() 10 0.009692 0.009692 0.009692 1.06% Gravity::fill_multipole_BCs() 11 0.008766 0.008766 0.008766 0.96% MLCellLinOp::solutionResidual() 93 0.007392 0.007392 0.007392 0.81% Castro::enforce_min_density() 62 0.00714 0.00714 0.00714 0.78% FabArray::Xpay() 585 0.006744 0.006744 0.006744 0.74% MLMG::computeResidual() 82 0.006381 0.006381 0.006381 0.70% Castro::post_timestep() 10 0.00611 0.00611 0.00611 0.67% Castro::reset_internal_energy(MultiFab) 63 0.005704 0.005704 0.005704 0.62% MLMG::prepareForSolve() 11 0.005539 0.005539 0.005539 0.60% MLCellLinOp::defineBC() 11 0.005335 0.005335 0.005335 0.58% BndryData::define() 11 0.005018 0.005018 0.005018 0.55% Castro::estTimeStep() 21 0.004654 0.004654 0.004654 0.51% Amr::InitializeInit() 1 0.004107 0.004107 0.004107 0.45% Amr::defBaseLevel() 1 0.004102 0.004102 0.004102 0.45% Castro::initData() 1 0.003549 0.003549 0.003549 0.39% Castro::construct_new_source() 50 0.00331 0.00331 0.00331 0.36% Castro::construct_new_gravity_source() 10 0.003292 0.003292 0.003292 0.36% Castro::construct_old_source() 50 0.002725 0.002725 0.002725 0.30% Castro::construct_old_gravity_source() 10 0.002705 0.002705 0.002705 0.30% amrex::Add() 164 0.002573 0.002573 0.002573 0.28% MLMG::ResNormInf() 93 0.00213 0.00213 0.00213 0.23% Castro::computeNewDt() 9 0.002096 0.002096 0.002096 0.23% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.20% FabArray::Saxpy() 20 0.001813 0.001813 0.001813 0.20% MultiFab::Add() 82 0.001733 0.001733 0.001733 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00173 0.00173 0.00173 0.19% Castro::reset_internal_energy(Fab) 504 0.001714 0.001714 0.001714 0.19% MLCellLinOp::setLevelBC() 11 0.001574 0.001574 0.001574 0.17% MLMG::getGradSolution() 11 0.001463 0.001463 0.001463 0.16% MLCellLinOp::compGrad() 11 0.001457 0.001457 0.001457 0.16% FabArrayBase::getCPC() 1323 0.001444 0.001444 0.001444 0.16% FabArray::setDomainBndry() 41 0.001398 0.001398 0.001398 0.15% FabArray::mult() 43 0.001349 0.001349 0.001349 0.15% MLPoisson::prepareForSolve() 11 0.001213 0.001213 0.001213 0.13% Castro::check_for_nan() 20 0.001207 0.001207 0.001207 0.13% MLCellLinOp::prepareForSolve() 11 0.001203 0.001203 0.001203 0.13% MultiFab::contains_nan() 20 0.001194 0.001194 0.001194 0.13% Castro::post_regrid() 1 0.001154 0.001154 0.001154 0.13% Castro::enforce_speed_limit() 62 0.001103 0.001103 0.001103 0.12% MLMG::computeMLResidual() 11 0.001067 0.001067 0.001067 0.12% Castro::computeInitialDt() 2 0.0009236 0.0009236 0.0009236 0.10% Gravity::update_max_rhs() 11 0.0008574 0.0008574 0.0008574 0.09% FabArrayBase::getFB() 4023 0.0007027 0.0007027 0.0007027 0.08% Castro::finalize_advance() 10 0.000679 0.000679 0.000679 0.07% FabArrayBase::CPC::define() 454 0.0006642 0.0006642 0.0006642 0.07% MLLinOp::define() 11 0.0005242 0.0005242 0.0005242 0.06% MLLinOp::defineGrids() 11 0.0004994 0.0004994 0.0004994 0.05% Amr::InitAmr() 1 0.0004806 0.0004806 0.0004806 0.05% Castro::Castro() 1 0.0004659 0.0004659 0.0004659 0.05% Gravity::swapTimeLevels() 10 0.0004454 0.0004454 0.0004454 0.05% MultiFab::Copy() 11 0.0003355 0.0003355 0.0003355 0.04% MLMG::MLResNormInf() 11 0.0002905 0.0002905 0.0002905 0.03% MultiFab::max() 11 0.0002711 0.0002711 0.0002711 0.03% MLMG::MLRhsNormInf() 11 0.0002207 0.0002207 0.0002207 0.02% Castro::buildMetrics() 1 0.0001628 0.0001628 0.0001628 0.02% FabArrayBase::FB::FB() 56 8.727e-05 8.727e-05 8.727e-05 0.01% AmrLevel::AmrLevel(dm) 1 6.345e-05 6.345e-05 6.345e-05 0.01% Castro::swap_state_time_levels() 10 5.689e-05 5.689e-05 5.689e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.458e-05 5.458e-05 5.458e-05 0.01% StateData::define() 4 5.173e-05 5.173e-05 5.173e-05 0.01% makeSFC 55 4.03e-05 4.03e-05 4.03e-05 0.00% Castro::finalize_do_advance() 10 3.526e-05 3.526e-05 3.526e-05 0.00% Castro::enforce_consistent_e() 1 3.423e-05 3.423e-05 3.423e-05 0.00% Castro::initMFs() 1 2.567e-05 2.567e-05 2.567e-05 0.00% Amr::writeSmallPlotFile() 1 2.553e-05 2.553e-05 2.553e-05 0.00% Castro::create_source_corrector() 10 2.356e-05 2.356e-05 2.356e-05 0.00% DistributionMapping::Distribute() 56 1.47e-05 1.47e-05 1.47e-05 0.00% Amr::initSubcycle() 1 8.531e-06 8.531e-06 8.531e-06 0.00% Gravity::set_mass_offset() 11 4.34e-06 4.34e-06 4.34e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.291e-06 4.291e-06 4.291e-06 0.00% AmrLevel::checkPointPost() 3 3.963e-06 3.963e-06 3.963e-06 0.00% Castro::retry_advance_ctu() 10 3.82e-06 3.82e-06 3.82e-06 0.00% Castro::FluxRegCrseInit 10 2.664e-06 2.664e-06 2.664e-06 0.00% Castro::FluxRegFineAdd() 10 2.248e-06 2.248e-06 2.248e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.031e-06 2.031e-06 2.031e-06 0.00% AmrLevel::checkPointPre() 3 1.902e-06 1.902e-06 1.902e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-33-g4d6413c45fa0) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-33-g4d6413c45fa0) initialized Starting run at 09:53:15 UTC on 2022-11-24. Successfully read inputs file ... Castro git describe: 22.11-14-g129e2797f AMReX git describe: 22.11-33-g4d6413c45 Microphysics git describe: 22.11-41-g63248ba9 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.503861178 Restart time = 0.049198858 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.054090734 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051773779 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060908396 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062349124 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.081441229 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03254065 seconds Ending run at 09:53:16 UTC on 2022-11-24. Run time = 0.393311305 Run time without initialization = 0.343498279 Average number of zones advanced per microsecond: 3.816 Average number of zones advanced per microsecond per rank: 3.816 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3933 ... 0.3933 ... 0.3933 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1104 0.1104 0.1104 28.07% VisMF::Read() 3 0.04131 0.04131 0.04131 10.50% MLCellLinOp::applyBC() 1946 0.03463 0.03463 0.03463 8.80% VisMF::Write(FabArray) 1 0.03093 0.03093 0.03093 7.86% MLPoisson::Fsmooth() 1440 0.02721 0.02721 0.02721 6.92% StateData::FillBoundary(geom) 160 0.0118 0.0118 0.0118 3.00% Castro::normalize_species() 30 0.009881 0.009881 0.009881 2.51% amrex::Dot() 484 0.009615 0.009615 0.009615 2.44% Castro::computeTemp() 30 0.008354 0.008354 0.008354 2.12% amrex::Copy() 463 0.00738 0.00738 0.00738 1.88% FabArray::setVal() 537 0.006818 0.006818 0.006818 1.73% MLCellLinOp::defineAuxData() 6 0.006324 0.006324 0.006324 1.61% FabArray::LinComb() 690 0.006199 0.006199 0.006199 1.58% FillBoundary_nowait() 1766 0.006195 0.006195 0.006195 1.57% FabArray::ParallelCopy_nowait() 380 0.005925 0.005925 0.005925 1.51% StateDataPhysBCFunct::() 20 0.005596 0.005596 0.005596 1.42% FabArray::norminf() 278 0.005561 0.005561 0.005561 1.41% MLPoisson::Fapply() 500 0.005106 0.005106 0.005106 1.30% Castro::enforce_min_density() 30 0.004754 0.004754 0.004754 1.21% Gravity::fill_multipole_BCs() 6 0.004374 0.004374 0.004374 1.11% Amr::restart() 1 0.003621 0.003621 0.003621 0.92% MLMG::addInterpCorrection() 180 0.003368 0.003368 0.003368 0.86% amrex::average_down 180 0.00299 0.00299 0.00299 0.76% FabArray::Xpay() 258 0.002899 0.002899 0.002899 0.74% Castro::estTimeStep() 10 0.002851 0.002851 0.002851 0.72% BndryData::define() 6 0.002158 0.002158 0.002158 0.55% Castro::do_advance_ctu() 5 0.00174 0.00174 0.00174 0.44% Castro::reset_internal_energy(MultiFab) 30 0.001706 0.001706 0.001706 0.43% Castro::construct_new_gravity_source() 5 0.001567 0.001567 0.001567 0.40% Amr::writePlotFile() 1 0.001435 0.001435 0.001435 0.36% MLCGSolver::bicgstab 36 0.001395 0.001395 0.001395 0.35% Castro::construct_old_gravity_source() 5 0.001329 0.001329 0.001329 0.34% amrex::Add() 72 0.001099 0.001099 0.001099 0.28% Castro::reset_internal_energy(Fab) 240 0.001027 0.001027 0.001027 0.26% MLMG::ResNormInf() 42 0.0009226 0.0009226 0.0009226 0.23% FabArray::Saxpy() 10 0.0009188 0.0009188 0.0009188 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008982 0.0008982 0.0008982 0.23% MLCellLinOp::setLevelBC() 6 0.000814 0.000814 0.000814 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007857 0.0007857 0.0007857 0.20% MLMG::prepareForSolve() 6 0.0006755 0.0006755 0.0006755 0.17% FabArray::mult() 22 0.0006669 0.0006669 0.0006669 0.17% FabArray::setDomainBndry() 20 0.0006483 0.0006483 0.0006483 0.16% MLCellLinOp::prepareForSolve() 6 0.0006308 0.0006308 0.0006308 0.16% MultiFab::contains_nan() 10 0.0005858 0.0005858 0.0005858 0.15% MLCellLinOp::compGrad() 6 0.0004885 0.0004885 0.0004885 0.12% Castro::enforce_speed_limit() 30 0.0004643 0.0004643 0.0004643 0.12% MLCellLinOp::smooth() 720 0.0004358 0.0004358 0.0004358 0.11% Gravity::get_old_grav_vector() 5 0.000427 0.000427 0.000427 0.11% Amr::InitAmr() 1 0.0004109 0.0004109 0.0004109 0.10% FabArrayBase::CPC::define() 244 0.0003872 0.0003872 0.0003872 0.10% Gravity::get_new_grav_vector() 5 0.000368 0.000368 0.000368 0.09% FabArrayBase::getCPC() 632 0.000356 0.000356 0.000356 0.09% FabArray::FillBoundary() 1766 0.0003525 0.0003525 0.0003525 0.09% FabArrayBase::getFB() 1766 0.0002635 0.0002635 0.0002635 0.07% main() 1 0.0002568 0.0002568 0.0002568 0.07% MLCellLinOp::apply() 500 0.0002187 0.0002187 0.0002187 0.06% MLLinOp::defineGrids() 6 0.0001826 0.0001826 0.0001826 0.05% Amr::coarseTimeStep() 5 0.0001758 0.0001758 0.0001758 0.04% CGSolver::sxay() 690 0.0001707 0.0001707 0.0001707 0.04% Castro::subcycle_advance_ctu() 5 0.0001685 0.0001685 0.0001685 0.04% MLCellLinOp::defineBC() 6 0.0001486 0.0001486 0.0001486 0.04% FillPatchIterator::Initialize 20 0.0001394 0.0001394 0.0001394 0.04% MultiFab::max() 6 0.0001341 0.0001341 0.0001341 0.03% MLCGSolver::ParallelAllReduce 659 0.0001311 0.0001311 0.0001311 0.03% FabArray::ParallelCopy() 380 0.0001285 0.0001285 0.0001285 0.03% MLMG::MLRhsNormInf() 6 0.0001143 0.0001143 0.0001143 0.03% Castro::advance() 5 0.0001062 0.0001062 0.0001062 0.03% MLCellLinOp::correctionResidual() 216 0.0001029 0.0001029 0.0001029 0.03% Castro::post_timestep() 5 9.799e-05 9.799e-05 9.799e-05 0.02% MLMG::mgVcycle() 36 8.69e-05 8.69e-05 8.69e-05 0.02% StateData::restartDoit() 4 8.153e-05 8.153e-05 8.153e-05 0.02% Amr::timeStep() 5 7.887e-05 7.887e-05 7.887e-05 0.02% Castro::construct_new_source() 25 7.732e-05 7.732e-05 7.732e-05 0.02% Castro::finalize_advance() 5 7.613e-05 7.613e-05 7.613e-05 0.02% AmrLevel::restart() 1 7.463e-05 7.463e-05 7.463e-05 0.02% MLMG:computeResOfCorrection() 180 7.389e-05 7.389e-05 7.389e-05 0.02% Gravity::update_max_rhs() 6 6.58e-05 6.58e-05 6.58e-05 0.02% Castro::construct_new_gravity() 5 6.406e-05 6.406e-05 6.406e-05 0.02% Gravity::solve_for_phi() 5 6.135e-05 6.135e-05 6.135e-05 0.02% FabArrayBase::FB::FB() 26 5.644e-05 5.644e-05 5.644e-05 0.01% MLMG::mgVcycle_down::0 36 5.514e-05 5.514e-05 5.514e-05 0.01% Castro::create_source_corrector() 5 5.355e-05 5.355e-05 5.355e-05 0.01% Castro::initialize_do_advance() 5 4.635e-05 4.635e-05 4.635e-05 0.01% MLMG::mgVcycle_down::1 36 4.43e-05 4.43e-05 4.43e-05 0.01% MLMG::mgVcycle_down::2 36 4.093e-05 4.093e-05 4.093e-05 0.01% MLMG::mgVcycle_down::4 36 3.984e-05 3.984e-05 3.984e-05 0.01% Castro::expand_state() 5 3.916e-05 3.916e-05 3.916e-05 0.01% Castro::clean_state() 30 3.887e-05 3.887e-05 3.887e-05 0.01% Castro::initialize_advance() 5 3.851e-05 3.851e-05 3.851e-05 0.01% MLMG::mgVcycle_down::3 36 3.844e-05 3.844e-05 3.844e-05 0.01% Castro::construct_old_source() 25 3.61e-05 3.61e-05 3.61e-05 0.01% MLMG::actualBottomSolve() 36 3.478e-05 3.478e-05 3.478e-05 0.01% MLMG::mgVcycle_up::4 36 3.366e-05 3.366e-05 3.366e-05 0.01% Castro::post_restart() 1 3.297e-05 3.297e-05 3.297e-05 0.01% MLMG::solve() 6 3.275e-05 3.275e-05 3.275e-05 0.01% Castro::buildMetrics() 1 3.192e-05 3.192e-05 3.192e-05 0.01% Castro::swap_state_time_levels() 5 2.856e-05 2.856e-05 2.856e-05 0.01% Castro::initMFs() 1 2.851e-05 2.851e-05 2.851e-05 0.01% MLMG::mgVcycle_up::3 36 2.652e-05 2.652e-05 2.652e-05 0.01% MLMG::mgVcycle_up::0 36 2.646e-05 2.646e-05 2.646e-05 0.01% Amr::writeSmallPlotFile() 1 2.602e-05 2.602e-05 2.602e-05 0.01% MLMG::mgVcycle_up::2 36 2.549e-05 2.549e-05 2.549e-05 0.01% MLMG::oneIter() 36 2.502e-05 2.502e-05 2.502e-05 0.01% Castro::do_old_sources() 5 2.451e-05 2.451e-05 2.451e-05 0.01% MLMG::mgVcycle_up::1 36 2.408e-05 2.408e-05 2.408e-05 0.01% MLCellLinOp::solutionResidual() 42 2.387e-05 2.387e-05 2.387e-05 0.01% Castro::construct_old_gravity() 5 2.173e-05 2.173e-05 2.173e-05 0.01% MLMG::computeResidual() 36 2.089e-05 2.089e-05 2.089e-05 0.01% MLLinOp::define() 6 2.013e-05 2.013e-05 2.013e-05 0.01% MLPoisson::define() 6 2.002e-05 2.002e-05 2.002e-05 0.01% Castro::computeNewDt() 5 1.98e-05 1.98e-05 1.98e-05 0.01% Castro::finalize_do_advance() 5 1.802e-05 1.802e-05 1.802e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.737e-05 1.737e-05 1.737e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.426e-05 1.426e-05 1.426e-05 0.00% makeSFC 30 1.413e-05 1.413e-05 1.413e-05 0.00% MLMG::mgVcycle_bottom 36 1.402e-05 1.402e-05 1.402e-05 0.00% FillPatchSingleLevel 20 1.319e-05 1.319e-05 1.319e-05 0.00% MultiFab::Add() 36 1.215e-05 1.215e-05 1.215e-05 0.00% Castro::do_new_sources() 5 9.231e-06 9.231e-06 9.231e-06 0.00% DistributionMapping::Distribute() 31 8.515e-06 8.515e-06 8.515e-06 0.00% Gravity::actual_multilevel_solve() 1 7.948e-06 7.948e-06 7.948e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.9e-06 7.9e-06 7.9e-06 0.00% Amr::initSubcycle() 1 7.815e-06 7.815e-06 7.815e-06 0.00% Castro::apply_source_to_state() 10 5.804e-06 5.804e-06 5.804e-06 0.00% Castro::check_for_nan() 10 5.657e-06 5.657e-06 5.657e-06 0.00% MLPoisson::prepareForSolve() 6 5.134e-06 5.134e-06 5.134e-06 0.00% MLMG::computeMLResidual() 6 4.739e-06 4.739e-06 4.739e-06 0.00% Gravity::swapTimeLevels() 5 4.215e-06 4.215e-06 4.215e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.425e-06 3.425e-06 3.425e-06 0.00% MLMG::getGradSolution() 6 3.197e-06 3.197e-06 3.197e-06 0.00% MultiFab::Copy() 6 2.682e-06 2.682e-06 2.682e-06 0.00% Gravity::set_mass_offset() 6 2.32e-06 2.32e-06 2.32e-06 0.00% MLMG::MLResNormInf() 6 2.318e-06 2.318e-06 2.318e-06 0.00% Castro::retry_advance_ctu() 5 1.815e-06 1.815e-06 1.815e-06 0.00% Castro::FluxRegCrseInit 5 1.564e-06 1.564e-06 1.564e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.207e-06 1.207e-06 1.207e-06 0.00% Castro::FluxRegFineAdd() 5 1.057e-06 1.057e-06 1.057e-06 0.00% Amr::init() 1 1.042e-06 1.042e-06 1.042e-06 0.00% AmrLevel::AmrLevel() 1 9.29e-07 9.29e-07 9.29e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3933 0.3933 0.3933 100.00% Amr::coarseTimeStep() 5 0.3107 0.3107 0.3107 79.00% Amr::timeStep() 5 0.3088 0.3088 0.3088 78.51% Castro::advance() 5 0.3035 0.3035 0.3035 77.16% Castro::subcycle_advance_ctu() 5 0.298 0.298 0.298 75.76% Castro::do_advance_ctu() 5 0.2978 0.2978 0.2978 75.72% Castro::construct_new_gravity() 5 0.1445 0.1445 0.1445 36.74% Gravity::solve_phi_with_mlmg() 6 0.1401 0.1401 0.1401 35.62% Gravity::solve_for_phi() 5 0.1365 0.1365 0.1365 34.71% Gravity::actual_solve_with_mlmg() 6 0.1355 0.1355 0.1355 34.45% MLMG::solve() 6 0.123 0.123 0.123 31.26% MLMG::oneIter() 36 0.1157 0.1157 0.1157 29.42% MLMG::mgVcycle() 36 0.115 0.115 0.115 29.23% Castro::construct_ctu_hydro_source() 5 0.1104 0.1104 0.1104 28.07% MLCellLinOp::smooth() 720 0.05834 0.05834 0.05834 14.83% Amr::init() 1 0.04927 0.04927 0.04927 12.53% Amr::restart() 1 0.04927 0.04927 0.04927 12.53% AmrLevel::restart() 1 0.04152 0.04152 0.04152 10.56% MLCellLinOp::applyBC() 1946 0.04149 0.04149 0.04149 10.55% StateData::restartDoit() 4 0.04144 0.04144 0.04144 10.54% VisMF::Read() 3 0.04131 0.04131 0.04131 10.50% MLMG::mgVcycle_bottom 36 0.03557 0.03557 0.03557 9.04% MLMG::actualBottomSolve() 36 0.03556 0.03556 0.03556 9.04% MLCGSolver::bicgstab 36 0.0352 0.0352 0.0352 8.95% Amr::writePlotFile() 1 0.03262 0.03262 0.03262 8.29% VisMF::Write(FabArray) 1 0.03093 0.03093 0.03093 7.86% MLPoisson::Fsmooth() 1440 0.02721 0.02721 0.02721 6.92% Castro::clean_state() 30 0.02623 0.02623 0.02623 6.67% FillPatchIterator::Initialize 20 0.02018 0.02018 0.02018 5.13% FillPatchSingleLevel 20 0.01939 0.01939 0.01939 4.93% StateDataPhysBCFunct::() 20 0.0174 0.0174 0.0174 4.42% MLCellLinOp::apply() 500 0.01586 0.01586 0.01586 4.03% MLMG::mgVcycle_down::0 36 0.01526 0.01526 0.01526 3.88% MLMG::mgVcycle_up::0 36 0.01303 0.01303 0.01303 3.31% StateData::FillBoundary(geom) 160 0.0118 0.0118 0.0118 3.00% Castro::computeTemp() 30 0.01109 0.01109 0.01109 2.82% Castro::initialize_do_advance() 5 0.01029 0.01029 0.01029 2.62% MLPoisson::define() 6 0.01016 0.01016 0.01016 2.58% Castro::normalize_species() 30 0.009881 0.009881 0.009881 2.51% amrex::Dot() 484 0.009615 0.009615 0.009615 2.44% MLCellLinOp::correctionResidual() 216 0.009302 0.009302 0.009302 2.36% MLMG:computeResOfCorrection() 180 0.00801 0.00801 0.00801 2.04% Gravity::get_new_grav_vector() 5 0.007822 0.007822 0.007822 1.99% MLMG::mgVcycle_down::1 36 0.007628 0.007628 0.007628 1.94% Castro::construct_old_gravity() 5 0.007525 0.007525 0.007525 1.91% Gravity::get_old_grav_vector() 5 0.007503 0.007503 0.007503 1.91% Castro::do_new_sources() 5 0.00742 0.00742 0.00742 1.89% MLMG::mgVcycle_down::2 36 0.007417 0.007417 0.007417 1.89% amrex::Copy() 463 0.00738 0.00738 0.00738 1.88% MLMG::mgVcycle_down::3 36 0.007075 0.007075 0.007075 1.80% MLCellLinOp::defineAuxData() 6 0.007062 0.007062 0.007062 1.80% FabArray::FillBoundary() 1766 0.006867 0.006867 0.006867 1.75% FabArray::setVal() 537 0.006818 0.006818 0.006818 1.73% MLMG::mgVcycle_down::4 36 0.006755 0.006755 0.006755 1.72% FillBoundary_nowait() 1766 0.006515 0.006515 0.006515 1.66% FabArray::ParallelCopy() 380 0.006419 0.006419 0.006419 1.63% CGSolver::sxay() 690 0.00637 0.00637 0.00637 1.62% FabArray::ParallelCopy_nowait() 380 0.006291 0.006291 0.006291 1.60% FabArray::LinComb() 690 0.006199 0.006199 0.006199 1.58% Castro::do_old_sources() 5 0.006094 0.006094 0.006094 1.55% MLCGSolver::ParallelAllReduce 659 0.005774 0.005774 0.005774 1.47% MLMG::mgVcycle_up::2 36 0.005701 0.005701 0.005701 1.45% Castro::expand_state() 5 0.0057 0.0057 0.0057 1.45% MLMG::mgVcycle_up::1 36 0.005637 0.005637 0.005637 1.43% MLMG::addInterpCorrection() 180 0.005589 0.005589 0.005589 1.42% FabArray::norminf() 278 0.005561 0.005561 0.005561 1.41% MLMG::mgVcycle_up::3 36 0.005399 0.005399 0.005399 1.37% MLMG::mgVcycle_up::4 36 0.005399 0.005399 0.005399 1.37% Castro::post_timestep() 5 0.005246 0.005246 0.005246 1.33% amrex::average_down 180 0.005208 0.005208 0.005208 1.32% MLPoisson::Fapply() 500 0.005106 0.005106 0.005106 1.30% Castro::initialize_advance() 5 0.00506 0.00506 0.00506 1.29% Castro::enforce_min_density() 30 0.004754 0.004754 0.004754 1.21% Gravity::fill_multipole_BCs() 6 0.004492 0.004492 0.004492 1.14% Castro::post_restart() 1 0.003936 0.003936 0.003936 1.00% Gravity::multilevel_solve_for_new_phi() 1 0.003816 0.003816 0.003816 0.97% Gravity::actual_multilevel_solve() 1 0.003799 0.003799 0.003799 0.97% MLCellLinOp::solutionResidual() 42 0.003227 0.003227 0.003227 0.82% MLMG::prepareForSolve() 6 0.002924 0.002924 0.002924 0.74% FabArray::Xpay() 258 0.002899 0.002899 0.002899 0.74% Castro::estTimeStep() 10 0.002851 0.002851 0.002851 0.72% MLCellLinOp::defineBC() 6 0.002845 0.002845 0.002845 0.72% Castro::reset_internal_energy(MultiFab) 30 0.002733 0.002733 0.002733 0.69% BndryData::define() 6 0.002696 0.002696 0.002696 0.69% MLMG::computeResidual() 36 0.002687 0.002687 0.002687 0.68% Castro::computeNewDt() 5 0.001737 0.001737 0.001737 0.44% Castro::construct_new_source() 25 0.001644 0.001644 0.001644 0.42% Castro::construct_new_gravity_source() 5 0.001567 0.001567 0.001567 0.40% Castro::construct_old_source() 25 0.001365 0.001365 0.001365 0.35% Castro::construct_old_gravity_source() 5 0.001329 0.001329 0.001329 0.34% amrex::Add() 72 0.001099 0.001099 0.001099 0.28% Castro::reset_internal_energy(Fab) 240 0.001027 0.001027 0.001027 0.26% Castro::apply_source_to_state() 10 0.0009246 0.0009246 0.0009246 0.24% MLMG::ResNormInf() 42 0.0009226 0.0009226 0.0009226 0.23% FabArray::Saxpy() 10 0.0009188 0.0009188 0.0009188 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008982 0.0008982 0.0008982 0.23% MLCellLinOp::setLevelBC() 6 0.000814 0.000814 0.000814 0.21% MLMG::getGradSolution() 6 0.0007626 0.0007626 0.0007626 0.19% MLCellLinOp::compGrad() 6 0.0007594 0.0007594 0.0007594 0.19% FabArrayBase::getCPC() 632 0.0007433 0.0007433 0.0007433 0.19% MultiFab::Add() 36 0.0007412 0.0007412 0.0007412 0.19% FabArray::mult() 22 0.0006669 0.0006669 0.0006669 0.17% FabArray::setDomainBndry() 20 0.0006483 0.0006483 0.0006483 0.16% MLPoisson::prepareForSolve() 6 0.0006359 0.0006359 0.0006359 0.16% MLCellLinOp::prepareForSolve() 6 0.0006308 0.0006308 0.0006308 0.16% Castro::check_for_nan() 10 0.0005914 0.0005914 0.0005914 0.15% MultiFab::contains_nan() 10 0.0005858 0.0005858 0.0005858 0.15% MLMG::computeMLResidual() 6 0.0005652 0.0005652 0.0005652 0.14% Castro::enforce_speed_limit() 30 0.0004643 0.0004643 0.0004643 0.12% Gravity::update_max_rhs() 6 0.0004432 0.0004432 0.0004432 0.11% Amr::InitAmr() 1 0.0004187 0.0004187 0.0004187 0.11% FabArrayBase::CPC::define() 244 0.0003872 0.0003872 0.0003872 0.10% Castro::finalize_advance() 5 0.00034 0.00034 0.00034 0.09% FabArrayBase::getFB() 1766 0.00032 0.00032 0.00032 0.08% MLLinOp::define() 6 0.0002329 0.0002329 0.0002329 0.06% Gravity::swapTimeLevels() 5 0.0002252 0.0002252 0.0002252 0.06% MLLinOp::defineGrids() 6 0.0002128 0.0002128 0.0002128 0.05% MultiFab::Copy() 6 0.0001792 0.0001792 0.0001792 0.05% Castro::buildMetrics() 1 0.0001599 0.0001599 0.0001599 0.04% MLMG::MLResNormInf() 6 0.0001491 0.0001491 0.0001491 0.04% MultiFab::max() 6 0.0001341 0.0001341 0.0001341 0.03% MLMG::MLRhsNormInf() 6 0.0001143 0.0001143 0.0001143 0.03% FabArrayBase::FB::FB() 26 5.644e-05 5.644e-05 5.644e-05 0.01% Castro::create_source_corrector() 5 5.355e-05 5.355e-05 5.355e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.903e-05 2.903e-05 2.903e-05 0.01% Castro::swap_state_time_levels() 5 2.856e-05 2.856e-05 2.856e-05 0.01% Castro::initMFs() 1 2.851e-05 2.851e-05 2.851e-05 0.01% Amr::writeSmallPlotFile() 1 2.602e-05 2.602e-05 2.602e-05 0.01% makeSFC 30 2.113e-05 2.113e-05 2.113e-05 0.01% Castro::finalize_do_advance() 5 1.802e-05 1.802e-05 1.802e-05 0.00% DistributionMapping::Distribute() 31 8.515e-06 8.515e-06 8.515e-06 0.00% Amr::initSubcycle() 1 7.815e-06 7.815e-06 7.815e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.946e-06 4.946e-06 4.946e-06 0.00% Gravity::set_mass_offset() 6 2.32e-06 2.32e-06 2.32e-06 0.00% Castro::retry_advance_ctu() 5 1.815e-06 1.815e-06 1.815e-06 0.00% Castro::FluxRegCrseInit 5 1.564e-06 1.564e-06 1.564e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.207e-06 1.207e-06 1.207e-06 0.00% Castro::FluxRegFineAdd() 5 1.057e-06 1.057e-06 1.057e-06 0.00% AmrLevel::AmrLevel() 1 9.29e-07 9.29e-07 9.29e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-33-g4d6413c45fa0) finalized