Initializing AMReX (23.11-17-g9e35dc19489d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (23.11-17-g9e35dc19489d) initialized Starting run at 09:18:39 UTC on 2023-11-21. Successfully read inputs file ... Castro git describe: 23.11-6-gb720d68fa AMReX git describe: 23.11-17-g9e35dc194 Microphysics git describe: 23.11-9-ge9e31f40 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051265046 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.028608157 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.052551355 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050357839 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.075101743 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.075876189 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.071766729 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.062906683 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.061074011 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047569202 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.076749808 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.072642833 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062090747 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048920029 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.028606707 seconds Ending run at 09:18:40 UTC on 2023-11-21. Run time = 0.922796779 Run time without initialization = 0.78692987 Average number of zones advanced per microsecond: 3.331 Average number of zones advanced per microsecond per rank: 3.331 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9228 ... 0.9228 ... 0.9228 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2657 0.2657 0.2657 28.79% VisMF::Write(FabArray) 11 0.1987 0.1987 0.1987 21.54% MLCellLinOp::applyBC() 4433 0.08032 0.08032 0.08032 8.70% MLPoisson::Fsmooth() 3280 0.03353 0.03353 0.03353 3.63% FillBoundary_nowait() 4023 0.03099 0.03099 0.03099 3.36% StateData::FillBoundary(geom) 328 0.0267 0.0267 0.0267 2.89% amrex::Dot() 1114 0.02126 0.02126 0.02126 2.30% Castro::normalize_species() 62 0.02108 0.02108 0.02108 2.28% FabArray::norminf() 1061 0.01966 0.01966 0.01966 2.13% Amr::checkPoint() 3 0.01886 0.01886 0.01886 2.04% Castro::computeTemp() 63 0.0154 0.0154 0.0154 1.67% FabArray::setVal() 1144 0.01359 0.01359 0.01359 1.47% FabArray::ParallelCopy_nowait() 861 0.01359 0.01359 0.01359 1.47% FabArray::Saxpy() 1370 0.01297 0.01297 0.01297 1.41% Castro::enforce_min_density() 62 0.01255 0.01255 0.01255 1.36% StateDataPhysBCFunct::() 41 0.01124 0.01124 0.01124 1.22% MLPoisson::Fapply() 1142 0.01079 0.01079 0.01079 1.17% amrex::Copy() 472 0.01076 0.01076 0.01076 1.17% MLCellLinOp::defineAuxData() 11 0.0104 0.0104 0.0104 1.13% Gravity::fill_multipole_BCs() 11 0.009314 0.009314 0.009314 1.01% FabArray::Xpay() 821 0.008405 0.008405 0.008405 0.91% MLMG::addInterpCorrection() 410 0.006944 0.006944 0.006944 0.75% Castro::estTimeStep() 21 0.006375 0.006375 0.006375 0.69% amrex::average_down 410 0.006169 0.006169 0.006169 0.67% Castro::reset_internal_energy(MultiFab) 63 0.005225 0.005225 0.005225 0.57% amrex::Add() 164 0.004433 0.004433 0.004433 0.48% BndryData::define() 11 0.003877 0.003877 0.003877 0.42% Castro::construct_new_gravity_source() 10 0.003368 0.003368 0.003368 0.36% Castro::enforce_speed_limit() 62 0.003107 0.003107 0.003107 0.34% Castro::construct_old_gravity_source() 10 0.002887 0.002887 0.002887 0.31% Amr::writePlotFile() 2 0.002255 0.002255 0.002255 0.24% check_for_negative_density() 10 0.001916 0.001916 0.001916 0.21% MLCGSolver::bicgstab 82 0.001816 0.001816 0.001816 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001769 0.001769 0.001769 0.19% Castro::reset_internal_energy(Fab) 504 0.001666 0.001666 0.001666 0.18% Gravity::actual_solve_with_mlmg() 11 0.001588 0.001588 0.001588 0.17% MLCellLinOp::setLevelBC() 11 0.001539 0.001539 0.001539 0.17% Castro::initData() 1 0.001537 0.001537 0.001537 0.17% FabArray::mult() 43 0.001386 0.001386 0.001386 0.15% FabArray::setDomainBndry() 41 0.001349 0.001349 0.001349 0.15% MLCellLinOp::prepareForSolve() 11 0.001319 0.001319 0.001319 0.14% MultiFab::contains_nan() 20 0.001281 0.001281 0.001281 0.14% MLCellLinOp::smooth() 1640 0.001144 0.001144 0.001144 0.12% MLCellLinOp::compGrad() 11 0.001065 0.001065 0.001065 0.12% MLMG::prepareForSolve() 11 0.0009715 0.0009715 0.0009715 0.11% FabArray::FillBoundary() 4023 0.0008788 0.0008788 0.0008788 0.10% FabArrayBase::getCPC() 1323 0.0007661 0.0007661 0.0007661 0.08% FabArrayBase::CPC::define() 454 0.0006793 0.0006793 0.0006793 0.07% FabArrayBase::getFB() 4023 0.0006744 0.0006744 0.0006744 0.07% Gravity::get_new_grav_vector() 11 0.000602 0.000602 0.000602 0.07% Amr::InitAmr() 1 0.0004952 0.0004952 0.0004952 0.05% Gravity::get_old_grav_vector() 10 0.0004744 0.0004744 0.0004744 0.05% MLCellLinOp::apply() 1142 0.0004584 0.0004584 0.0004584 0.05% AmrLevel::FillPatch() 41 0.0003714 0.0003714 0.0003714 0.04% Amr::coarseTimeStep() 10 0.000358 0.000358 0.000358 0.04% MLCGSolver::ParallelAllReduce 1832 0.0003557 0.0003557 0.0003557 0.04% MultiFab::max() 11 0.0003273 0.0003273 0.0003273 0.04% main() 1 0.0003152 0.0003152 0.0003152 0.03% MLCellLinOp::defineBC() 11 0.0002589 0.0002589 0.0002589 0.03% FabArray::ParallelCopy() 861 0.000244 0.000244 0.000244 0.03% Castro::subcycle_advance_ctu() 10 0.0002355 0.0002355 0.0002355 0.03% MLCellLinOp::correctionResidual() 492 0.0002249 0.0002249 0.0002249 0.02% FillPatchIterator::Initialize 41 0.0002229 0.0002229 0.0002229 0.02% MLMG::mgVcycle() 82 0.000205 0.000205 0.000205 0.02% Gravity::solve_for_phi() 10 0.0001769 0.0001769 0.0001769 0.02% Castro::create_source_corrector() 10 0.0001761 0.0001761 0.0001761 0.02% MLLinOp::defineGrids() 11 0.0001587 0.0001587 0.0001587 0.02% Amr::timeStep() 10 0.000158 0.000158 0.000158 0.02% Gravity::update_max_rhs() 11 0.0001445 0.0001445 0.0001445 0.02% Castro::advance() 10 0.0001426 0.0001426 0.0001426 0.02% Castro::do_new_sources() 10 0.0001384 0.0001384 0.0001384 0.01% StateData::checkPoint() 12 0.0001338 0.0001338 0.0001338 0.01% MLMG:computeResOfCorrection() 410 0.0001152 0.0001152 0.0001152 0.01% MLMG::mgVcycle_down::0 82 9.939e-05 9.939e-05 9.939e-05 0.01% FabArrayBase::FB::FB() 56 9.852e-05 9.852e-05 9.852e-05 0.01% Castro::Castro() 1 9.186e-05 9.186e-05 9.186e-05 0.01% MLMG::mgVcycle_down::1 82 8.726e-05 8.726e-05 8.726e-05 0.01% MLMG::actualBottomSolve() 82 8.555e-05 8.555e-05 8.555e-05 0.01% MLMG::mgVcycle_down::2 82 8.549e-05 8.549e-05 8.549e-05 0.01% MLMG::mgVcycle_down::4 82 8.279e-05 8.279e-05 8.279e-05 0.01% MLMG::mgVcycle_down::3 82 8.011e-05 8.011e-05 8.011e-05 0.01% Castro::enforce_consistent_e() 1 7.838e-05 7.838e-05 7.838e-05 0.01% Castro::clean_state() 62 7.673e-05 7.673e-05 7.673e-05 0.01% AmrLevel::checkPoint() 3 7.561e-05 7.561e-05 7.561e-05 0.01% Castro::initialize_advance() 10 7.543e-05 7.543e-05 7.543e-05 0.01% MLMG::solve() 11 7.164e-05 7.164e-05 7.164e-05 0.01% Castro::finalize_advance() 10 6.678e-05 6.678e-05 6.678e-05 0.01% Castro::initialize_do_advance() 10 6.193e-05 6.193e-05 6.193e-05 0.01% MLMG::mgVcycle_up::4 82 6.082e-05 6.082e-05 6.082e-05 0.01% MLMG::oneIter() 82 5.53e-05 5.53e-05 5.53e-05 0.01% MLMG::mgVcycle_up::0 82 5.121e-05 5.121e-05 5.121e-05 0.01% MLMG::mgVcycle_up::3 82 4.947e-05 4.947e-05 4.947e-05 0.01% MLMG::mgVcycle_up::1 82 4.836e-05 4.836e-05 4.836e-05 0.01% MLCellLinOp::solutionResidual() 93 4.782e-05 4.782e-05 4.782e-05 0.01% Castro::do_advance_ctu() 10 4.758e-05 4.758e-05 4.758e-05 0.01% MLMG::mgVcycle_up::2 82 4.693e-05 4.693e-05 4.693e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.632e-05 4.632e-05 4.632e-05 0.01% Castro::finalize_do_advance() 10 4.297e-05 4.297e-05 4.297e-05 0.00% FillPatchSingleLevel 41 4.074e-05 4.074e-05 4.074e-05 0.00% StateData::define() 4 3.969e-05 3.969e-05 3.969e-05 0.00% Castro::swap_state_time_levels() 10 3.938e-05 3.938e-05 3.938e-05 0.00% MLMG::ResNormInf() 93 3.79e-05 3.79e-05 3.79e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.278e-05 3.278e-05 3.278e-05 0.00% MLMG::mgVcycle_bottom 82 3.254e-05 3.254e-05 3.254e-05 0.00% Amr::writeSmallPlotFile() 1 3.174e-05 3.174e-05 3.174e-05 0.00% Castro::initMFs() 1 3.092e-05 3.092e-05 3.092e-05 0.00% MLMG::computeResidual() 82 2.999e-05 2.999e-05 2.999e-05 0.00% Castro::construct_new_gravity() 10 2.969e-05 2.969e-05 2.969e-05 0.00% Castro::buildMetrics() 1 2.542e-05 2.542e-05 2.542e-05 0.00% makeSFC 55 2.467e-05 2.467e-05 2.467e-05 0.00% Amr::defBaseLevel() 1 2.358e-05 2.358e-05 2.358e-05 0.00% MLPoisson::define() 11 2.249e-05 2.249e-05 2.249e-05 0.00% AmrLevel::AmrLevel(dm) 1 2.216e-05 2.216e-05 2.216e-05 0.00% Castro::do_old_sources() 10 2.194e-05 2.194e-05 2.194e-05 0.00% Amr::FinalizeInit() 1 1.988e-05 1.988e-05 1.988e-05 0.00% Castro::construct_old_source() 50 1.903e-05 1.903e-05 1.903e-05 0.00% Castro::construct_new_source() 50 1.861e-05 1.861e-05 1.861e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.847e-05 1.847e-05 1.847e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.64e-05 1.64e-05 1.64e-05 0.00% DistributionMapping::Distribute() 56 1.516e-05 1.516e-05 1.516e-05 0.00% Castro::check_for_nan() 20 1.459e-05 1.459e-05 1.459e-05 0.00% MLMG::computeMLResidual() 11 1.418e-05 1.418e-05 1.418e-05 0.00% Castro::post_init() 1 1.377e-05 1.377e-05 1.377e-05 0.00% MLPoisson::prepareForSolve() 11 1.368e-05 1.368e-05 1.368e-05 0.00% Castro::apply_source_to_state() 20 1.197e-05 1.197e-05 1.197e-05 0.00% Castro::construct_old_gravity() 10 1.126e-05 1.126e-05 1.126e-05 0.00% MLLinOp::define() 11 9.711e-06 9.711e-06 9.711e-06 0.00% Gravity::swapTimeLevels() 10 9.523e-06 9.523e-06 9.523e-06 0.00% Amr::initSubcycle() 1 8.691e-06 8.691e-06 8.691e-06 0.00% Gravity::actual_multilevel_solve() 1 8.53e-06 8.53e-06 8.53e-06 0.00% Castro::post_timestep() 10 8.24e-06 8.24e-06 8.24e-06 0.00% Castro::computeNewDt() 9 7.352e-06 7.352e-06 7.352e-06 0.00% MLMG::getGradSolution() 11 6.201e-06 6.201e-06 6.201e-06 0.00% AmrLevel::checkPointPost() 3 5.819e-06 5.819e-06 5.819e-06 0.00% Castro::expand_state() 10 5.672e-06 5.672e-06 5.672e-06 0.00% Castro::retry_advance_ctu() 10 4.52e-06 4.52e-06 4.52e-06 0.00% Amr::InitializeInit() 1 4.413e-06 4.413e-06 4.413e-06 0.00% Castro::computeInitialDt() 2 4.345e-06 4.345e-06 4.345e-06 0.00% MLMG::MLRhsNormInf() 11 4.023e-06 4.023e-06 4.023e-06 0.00% Gravity::set_mass_offset() 11 3.995e-06 3.995e-06 3.995e-06 0.00% MLMG::MLResNormInf() 11 3.714e-06 3.714e-06 3.714e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.386e-06 3.386e-06 3.386e-06 0.00% Castro::FluxRegCrseInit 10 3.22e-06 3.22e-06 3.22e-06 0.00% AmrLevel::checkPointPre() 3 2.596e-06 2.596e-06 2.596e-06 0.00% Castro::FluxRegFineAdd() 10 2.194e-06 2.194e-06 2.194e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.099e-06 2.099e-06 2.099e-06 0.00% Amr::init() 1 2.097e-06 2.097e-06 2.097e-06 0.00% Castro::post_regrid() 1 1.294e-06 1.294e-06 1.294e-06 0.00% Amr::initialInit() 1 8.98e-07 8.98e-07 8.98e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9228 0.9228 0.9228 100.00% Amr::coarseTimeStep() 10 0.7581 0.7581 0.7581 82.15% Amr::timeStep() 10 0.6421 0.6421 0.6421 69.58% Castro::advance() 10 0.6307 0.6307 0.6307 68.34% Castro::subcycle_advance_ctu() 10 0.6156 0.6156 0.6156 66.70% Castro::do_advance_ctu() 10 0.6153 0.6153 0.6153 66.68% Gravity::solve_phi_with_mlmg() 11 0.296 0.296 0.296 32.07% Gravity::actual_solve_with_mlmg() 11 0.2862 0.2862 0.2862 31.01% Castro::construct_ctu_hydro_source() 10 0.2775 0.2775 0.2775 30.07% Castro::construct_new_gravity() 10 0.2664 0.2664 0.2664 28.87% MLMG::solve() 11 0.2643 0.2643 0.2643 28.64% Gravity::solve_for_phi() 10 0.2504 0.2504 0.2504 27.13% MLMG::oneIter() 82 0.2491 0.2491 0.2491 26.99% MLMG::mgVcycle() 82 0.2454 0.2454 0.2454 26.59% VisMF::Write(FabArray) 11 0.1987 0.1987 0.1987 21.54% Amr::checkPoint() 3 0.1633 0.1633 0.1633 17.69% AmrLevel::checkPoint() 3 0.1444 0.1444 0.1444 15.65% StateData::checkPoint() 12 0.1443 0.1443 0.1443 15.64% Amr::init() 1 0.1352 0.1352 0.1352 14.65% MLCellLinOp::smooth() 1640 0.1206 0.1206 0.1206 13.06% MLCellLinOp::applyBC() 4433 0.113 0.113 0.113 12.24% MLMG::mgVcycle_bottom 82 0.07632 0.07632 0.07632 8.27% MLMG::actualBottomSolve() 82 0.07629 0.07629 0.07629 8.27% MLCGSolver::bicgstab 82 0.07557 0.07557 0.07557 8.19% Castro::clean_state() 62 0.05797 0.05797 0.05797 6.28% Amr::writePlotFile() 2 0.05733 0.05733 0.05733 6.21% Amr::initialInit() 1 0.05524 0.05524 0.05524 5.99% Amr::FinalizeInit() 1 0.04995 0.04995 0.04995 5.41% Castro::post_init() 1 0.04847 0.04847 0.04847 5.25% AmrLevel::FillPatch() 41 0.04785 0.04785 0.04785 5.18% Gravity::multilevel_solve_for_new_phi() 1 0.04614 0.04614 0.04614 5.00% Gravity::actual_multilevel_solve() 1 0.04612 0.04612 0.04612 5.00% FillPatchIterator::Initialize 41 0.04362 0.04362 0.04362 4.73% FillPatchIterator::FillFromLevel0() 41 0.04205 0.04205 0.04205 4.56% FillPatchSingleLevel 41 0.042 0.042 0.042 4.55% StateDataPhysBCFunct::() 41 0.03794 0.03794 0.03794 4.11% MLCellLinOp::apply() 1142 0.03783 0.03783 0.03783 4.10% MLMG::mgVcycle_down::0 82 0.03438 0.03438 0.03438 3.73% MLPoisson::Fsmooth() 3280 0.03353 0.03353 0.03353 3.63% FabArray::FillBoundary() 4023 0.03264 0.03264 0.03264 3.54% FillBoundary_nowait() 4023 0.03177 0.03177 0.03177 3.44% StateData::FillBoundary(geom) 328 0.0267 0.0267 0.0267 2.89% MLMG::mgVcycle_up::0 82 0.02581 0.02581 0.02581 2.80% MLCellLinOp::correctionResidual() 492 0.02297 0.02297 0.02297 2.49% Castro::computeTemp() 63 0.02229 0.02229 0.02229 2.42% Castro::initialize_do_advance() 10 0.02186 0.02186 0.02186 2.37% amrex::Dot() 1114 0.02126 0.02126 0.02126 2.30% Castro::normalize_species() 62 0.02108 0.02108 0.02108 2.28% MLMG:computeResOfCorrection() 410 0.02021 0.02021 0.02021 2.19% FabArray::norminf() 1061 0.01966 0.01966 0.01966 2.13% Castro::do_old_sources() 10 0.0188 0.0188 0.0188 2.04% Gravity::get_new_grav_vector() 11 0.01793 0.01793 0.01793 1.94% MLPoisson::define() 11 0.01721 0.01721 0.01721 1.86% MLMG::mgVcycle_down::1 82 0.01649 0.01649 0.01649 1.79% MLMG::mgVcycle_down::2 82 0.01536 0.01536 0.01536 1.66% Castro::construct_old_gravity() 10 0.01514 0.01514 0.01514 1.64% Gravity::get_old_grav_vector() 10 0.01513 0.01513 0.01513 1.64% MLMG::mgVcycle_down::3 82 0.015 0.015 0.015 1.63% MLMG::mgVcycle_down::4 82 0.01484 0.01484 0.01484 1.61% FabArray::ParallelCopy() 861 0.01462 0.01462 0.01462 1.58% FabArray::ParallelCopy_nowait() 861 0.01438 0.01438 0.01438 1.56% Castro::initialize_advance() 10 0.01437 0.01437 0.01437 1.56% FabArray::setVal() 1144 0.01359 0.01359 0.01359 1.47% FabArray::Saxpy() 1370 0.01297 0.01297 0.01297 1.41% MLCGSolver::ParallelAllReduce 1832 0.01278 0.01278 0.01278 1.38% Castro::enforce_min_density() 62 0.01255 0.01255 0.01255 1.36% MLMG::addInterpCorrection() 410 0.0122 0.0122 0.0122 1.32% Castro::do_new_sources() 10 0.01219 0.01219 0.01219 1.32% MLMG::mgVcycle_up::1 82 0.01195 0.01195 0.01195 1.30% Castro::expand_state() 10 0.01191 0.01191 0.01191 1.29% MLMG::mgVcycle_up::4 82 0.01189 0.01189 0.01189 1.29% MLCellLinOp::defineAuxData() 11 0.01181 0.01181 0.01181 1.28% MLMG::mgVcycle_up::2 82 0.01171 0.01171 0.01171 1.27% amrex::average_down 410 0.01151 0.01151 0.01151 1.25% MLMG::mgVcycle_up::3 82 0.01146 0.01146 0.01146 1.24% Castro::post_timestep() 10 0.01122 0.01122 0.01122 1.22% MLPoisson::Fapply() 1142 0.01079 0.01079 0.01079 1.17% amrex::Copy() 472 0.01076 0.01076 0.01076 1.17% Gravity::fill_multipole_BCs() 11 0.009541 0.009541 0.009541 1.03% FabArray::Xpay() 821 0.008405 0.008405 0.008405 0.91% MLCellLinOp::solutionResidual() 93 0.007597 0.007597 0.007597 0.82% Castro::reset_internal_energy(MultiFab) 63 0.006891 0.006891 0.006891 0.75% Castro::estTimeStep() 21 0.006375 0.006375 0.006375 0.69% MLMG::computeResidual() 82 0.006303 0.006303 0.006303 0.68% Amr::InitializeInit() 1 0.00528 0.00528 0.00528 0.57% Amr::defBaseLevel() 1 0.005276 0.005276 0.005276 0.57% MLCellLinOp::defineBC() 11 0.005152 0.005152 0.005152 0.56% MLMG::prepareForSolve() 11 0.005107 0.005107 0.005107 0.55% BndryData::define() 11 0.004893 0.004893 0.004893 0.53% Castro::initData() 1 0.004616 0.004616 0.004616 0.50% amrex::Add() 164 0.004433 0.004433 0.004433 0.48% Castro::construct_new_source() 50 0.003386 0.003386 0.003386 0.37% Castro::construct_new_gravity_source() 10 0.003368 0.003368 0.003368 0.36% Castro::enforce_speed_limit() 62 0.003107 0.003107 0.003107 0.34% Castro::computeNewDt() 9 0.002931 0.002931 0.002931 0.32% Castro::construct_old_source() 50 0.002906 0.002906 0.002906 0.31% Castro::construct_old_gravity_source() 10 0.002887 0.002887 0.002887 0.31% Castro::finalize_do_advance() 10 0.002439 0.002439 0.002439 0.26% MLMG::ResNormInf() 93 0.002162 0.002162 0.002162 0.23% check_for_negative_density() 10 0.001916 0.001916 0.001916 0.21% Castro::apply_source_to_state() 20 0.001866 0.001866 0.001866 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001769 0.001769 0.001769 0.19% Castro::reset_internal_energy(Fab) 504 0.001666 0.001666 0.001666 0.18% MLMG::getGradSolution() 11 0.001558 0.001558 0.001558 0.17% MLCellLinOp::compGrad() 11 0.001552 0.001552 0.001552 0.17% MLCellLinOp::setLevelBC() 11 0.001539 0.001539 0.001539 0.17% FabArrayBase::getCPC() 1323 0.001445 0.001445 0.001445 0.16% FabArray::mult() 43 0.001386 0.001386 0.001386 0.15% FabArray::setDomainBndry() 41 0.001349 0.001349 0.001349 0.15% MLMG::computeMLResidual() 11 0.001337 0.001337 0.001337 0.14% MLPoisson::prepareForSolve() 11 0.001333 0.001333 0.001333 0.14% MLCellLinOp::prepareForSolve() 11 0.001319 0.001319 0.001319 0.14% Castro::check_for_nan() 20 0.001296 0.001296 0.001296 0.14% MultiFab::contains_nan() 20 0.001281 0.001281 0.001281 0.14% Castro::post_regrid() 1 0.001175 0.001175 0.001175 0.13% Castro::computeInitialDt() 2 0.00106 0.00106 0.00106 0.11% Gravity::update_max_rhs() 11 0.0009857 0.0009857 0.0009857 0.11% FabArrayBase::getFB() 4023 0.000773 0.000773 0.000773 0.08% FabArrayBase::CPC::define() 454 0.0006793 0.0006793 0.0006793 0.07% Castro::finalize_advance() 10 0.0006008 0.0006008 0.0006008 0.07% Castro::Castro() 1 0.00057 0.00057 0.00057 0.06% Amr::InitAmr() 1 0.0005039 0.0005039 0.0005039 0.05% Gravity::swapTimeLevels() 10 0.0004405 0.0004405 0.0004405 0.05% MLMG::MLResNormInf() 11 0.0003313 0.0003313 0.0003313 0.04% MultiFab::max() 11 0.0003273 0.0003273 0.0003273 0.04% Castro::buildMetrics() 1 0.0002796 0.0002796 0.0002796 0.03% MLLinOp::define() 11 0.0002253 0.0002253 0.0002253 0.02% MLMG::MLRhsNormInf() 11 0.0002236 0.0002236 0.0002236 0.02% MLLinOp::defineGrids() 11 0.0002156 0.0002156 0.0002156 0.02% Castro::create_source_corrector() 10 0.0001761 0.0001761 0.0001761 0.02% FabArrayBase::FB::FB() 56 9.852e-05 9.852e-05 9.852e-05 0.01% Castro::enforce_consistent_e() 1 7.838e-05 7.838e-05 7.838e-05 0.01% AmrLevel::AmrLevel(dm) 1 6.185e-05 6.185e-05 6.185e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.484e-05 5.484e-05 5.484e-05 0.01% StateData::define() 4 3.969e-05 3.969e-05 3.969e-05 0.00% Castro::swap_state_time_levels() 10 3.938e-05 3.938e-05 3.938e-05 0.00% makeSFC 55 3.845e-05 3.845e-05 3.845e-05 0.00% Amr::writeSmallPlotFile() 1 3.174e-05 3.174e-05 3.174e-05 0.00% Castro::initMFs() 1 3.092e-05 3.092e-05 3.092e-05 0.00% DistributionMapping::Distribute() 56 1.516e-05 1.516e-05 1.516e-05 0.00% Amr::initSubcycle() 1 8.691e-06 8.691e-06 8.691e-06 0.00% AmrLevel::checkPointPost() 3 5.819e-06 5.819e-06 5.819e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.765e-06 4.765e-06 4.765e-06 0.00% Castro::retry_advance_ctu() 10 4.52e-06 4.52e-06 4.52e-06 0.00% Gravity::set_mass_offset() 11 3.995e-06 3.995e-06 3.995e-06 0.00% Castro::FluxRegCrseInit 10 3.22e-06 3.22e-06 3.22e-06 0.00% AmrLevel::checkPointPre() 3 2.596e-06 2.596e-06 2.596e-06 0.00% Castro::FluxRegFineAdd() 10 2.194e-06 2.194e-06 2.194e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.099e-06 2.099e-06 2.099e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 95 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 136 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1011 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1740 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7620 KiB 14 MiB MLMG::prepareForSolve() 660 660 3522 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 201 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 167 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7516 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 17 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2032 B 2048 KiB Gravity::solve_for_phi() 80 80 554 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 102 KiB 2048 KiB BndryData::define() 1056 1056 322 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 205 KiB 671 KiB Castro::estTimeStep() 21 21 3369 B 480 KiB VisMF::Write(FabArray) 656 656 3338 B 320 KiB Castro::normalize_species() 62 62 7442 B 320 KiB amrex::average_down 1067 1067 1237 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1120 B 257 KiB amrex::Dot() 1360 1360 3390 B 160 KiB FabArray::norminf() 1143 1143 3291 B 160 KiB check_for_negative_density() 10 10 322 B 160 KiB Castro::initData() 1 1 53 B 160 KiB MultiFab::max() 11 11 55 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3439 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 294 B 9648 B MLCellLinOp::applyBC() 8866 8866 218 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3896 B 6144 B StateData::FillBoundary(geom) 1992 1992 39 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 492 492 103 B 1280 B MLCellLinOp::defineBC() 66 66 362 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 551 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 39 KiB 8192 KiB VisMF::Write(FabArray) 744 744 490 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3439 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 294 B 9648 B MLCellLinOp::applyBC() 4433 4433 216 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3896 B 6144 B Gravity::get_new_grav_vector() 3 3 2890 B 3072 B StateData::FillBoundary(geom) 1992 1992 40 B 3024 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 270 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 293 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 1143 1143 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.11-17-g9e35dc19489d) finalized Initializing AMReX (23.11-17-g9e35dc19489d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (23.11-17-g9e35dc19489d) initialized Starting run at 09:18:41 UTC on 2023-11-21. Successfully read inputs file ... Castro git describe: 23.11-6-gb720d68fa AMReX git describe: 23.11-17-g9e35dc194 Microphysics git describe: 23.11-9-ge9e31f40 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.5243752 Restart time = 0.071524673 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053221217 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049195644 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.070058862 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.070550351 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.075431553 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.037266692 seconds Ending run at 09:18:41 UTC on 2023-11-21. Run time = 0.428396299 Run time without initialization = 0.356258273 Average number of zones advanced per microsecond: 3.679 Average number of zones advanced per microsecond per rank: 3.679 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.4284 ... 0.4284 ... 0.4284 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1265 0.1265 0.1265 29.53% VisMF::Read() 3 0.06006 0.06006 0.06006 14.02% VisMF::Write(FabArray) 1 0.03569 0.03569 0.03569 8.33% MLCellLinOp::applyBC() 1946 0.0349 0.0349 0.0349 8.15% MLPoisson::Fsmooth() 1440 0.01435 0.01435 0.01435 3.35% FillBoundary_nowait() 1766 0.01295 0.01295 0.01295 3.02% StateData::FillBoundary(geom) 160 0.01261 0.01261 0.01261 2.94% Castro::normalize_species() 30 0.009455 0.009455 0.009455 2.21% amrex::Dot() 484 0.009174 0.009174 0.009174 2.14% FabArray::norminf() 465 0.008878 0.008878 0.008878 2.07% Castro::computeTemp() 30 0.007699 0.007699 0.007699 1.80% FabArray::setVal() 537 0.00659 0.00659 0.00659 1.54% Castro::enforce_min_density() 30 0.006133 0.006133 0.006133 1.43% FabArray::ParallelCopy_nowait() 380 0.006114 0.006114 0.006114 1.43% FabArray::Saxpy() 597 0.005664 0.005664 0.005664 1.32% MLCellLinOp::defineAuxData() 6 0.005553 0.005553 0.005553 1.30% Gravity::fill_multipole_BCs() 6 0.005499 0.005499 0.005499 1.28% amrex::Copy() 221 0.005307 0.005307 0.005307 1.24% StateDataPhysBCFunct::() 20 0.005166 0.005166 0.005166 1.21% Amr::restart() 1 0.004715 0.004715 0.004715 1.10% MLPoisson::Fapply() 500 0.004651 0.004651 0.004651 1.09% FabArray::Xpay() 361 0.003655 0.003655 0.003655 0.85% Castro::estTimeStep() 10 0.00323 0.00323 0.00323 0.75% MLMG::addInterpCorrection() 180 0.003076 0.003076 0.003076 0.72% amrex::average_down 180 0.002739 0.002739 0.002739 0.64% BndryData::define() 6 0.00212 0.00212 0.00212 0.49% Castro::reset_internal_energy(MultiFab) 30 0.002041 0.002041 0.002041 0.48% amrex::Add() 72 0.001881 0.001881 0.001881 0.44% Castro::construct_new_gravity_source() 5 0.00176 0.00176 0.00176 0.41% Castro::construct_old_gravity_source() 5 0.00141 0.00141 0.00141 0.33% Amr::writePlotFile() 1 0.00141 0.00141 0.00141 0.33% Castro::enforce_speed_limit() 30 0.001162 0.001162 0.001162 0.27% check_for_negative_density() 5 0.001031 0.001031 0.001031 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009656 0.0009656 0.0009656 0.23% Gravity::actual_solve_with_mlmg() 6 0.0008484 0.0008484 0.0008484 0.20% MLCellLinOp::setLevelBC() 6 0.0008412 0.0008412 0.0008412 0.20% Castro::reset_internal_energy(Fab) 240 0.0008281 0.0008281 0.0008281 0.19% MLCGSolver::bicgstab 36 0.0008241 0.0008241 0.0008241 0.19% MLCellLinOp::prepareForSolve() 6 0.0007422 0.0007422 0.0007422 0.17% FabArray::mult() 22 0.0006835 0.0006835 0.0006835 0.16% FabArray::setDomainBndry() 20 0.0006753 0.0006753 0.0006753 0.16% MultiFab::contains_nan() 10 0.0006429 0.0006429 0.0006429 0.15% MLCellLinOp::compGrad() 6 0.0005802 0.0005802 0.0005802 0.14% MLMG::prepareForSolve() 6 0.0005395 0.0005395 0.0005395 0.13% MLCellLinOp::smooth() 720 0.0005249 0.0005249 0.0005249 0.12% FabArrayBase::CPC::define() 244 0.0004183 0.0004183 0.0004183 0.10% Amr::InitAmr() 1 0.0004143 0.0004143 0.0004143 0.10% FabArray::FillBoundary() 1766 0.0003837 0.0003837 0.0003837 0.09% FabArrayBase::getCPC() 632 0.0003632 0.0003632 0.0003632 0.08% main() 1 0.0003629 0.0003629 0.0003629 0.08% Gravity::get_old_grav_vector() 5 0.0003287 0.0003287 0.0003287 0.08% FabArrayBase::getFB() 1766 0.0002816 0.0002816 0.0002816 0.07% Gravity::get_new_grav_vector() 5 0.0002606 0.0002606 0.0002606 0.06% Amr::coarseTimeStep() 5 0.0002157 0.0002157 0.0002157 0.05% MultiFab::max() 6 0.0002124 0.0002124 0.0002124 0.05% MLCellLinOp::apply() 500 0.0002003 0.0002003 0.0002003 0.05% AmrLevel::FillPatch() 20 0.0001982 0.0001982 0.0001982 0.05% MLCGSolver::ParallelAllReduce 798 0.0001503 0.0001503 0.0001503 0.04% MLCellLinOp::defineBC() 6 0.0001373 0.0001373 0.0001373 0.03% Castro::subcycle_advance_ctu() 5 0.0001336 0.0001336 0.0001336 0.03% FillPatchIterator::Initialize 20 0.0001101 0.0001101 0.0001101 0.03% FabArray::ParallelCopy() 380 0.0001081 0.0001081 0.0001081 0.03% MLCellLinOp::correctionResidual() 216 9.651e-05 9.651e-05 9.651e-05 0.02% MLMG::mgVcycle() 36 8.787e-05 8.787e-05 8.787e-05 0.02% MLLinOp::defineGrids() 6 8.353e-05 8.353e-05 8.353e-05 0.02% Gravity::solve_for_phi() 5 8.336e-05 8.336e-05 8.336e-05 0.02% Castro::advance() 5 8.264e-05 8.264e-05 8.264e-05 0.02% Castro::create_source_corrector() 5 8.211e-05 8.211e-05 8.211e-05 0.02% AmrLevel::restart() 1 7.963e-05 7.963e-05 7.963e-05 0.02% Amr::timeStep() 5 7.904e-05 7.904e-05 7.904e-05 0.02% StateData::restartDoit() 4 7.367e-05 7.367e-05 7.367e-05 0.02% Castro::finalize_do_advance() 5 7.235e-05 7.235e-05 7.235e-05 0.02% Castro::construct_new_source() 25 7.067e-05 7.067e-05 7.067e-05 0.02% Castro::initialize_do_advance() 5 6.761e-05 6.761e-05 6.761e-05 0.02% Gravity::update_max_rhs() 6 6.426e-05 6.426e-05 6.426e-05 0.01% FabArrayBase::FB::FB() 26 6.199e-05 6.199e-05 6.199e-05 0.01% Castro::initialize_advance() 5 5.732e-05 5.732e-05 5.732e-05 0.01% Castro::do_new_sources() 5 5.477e-05 5.477e-05 5.477e-05 0.01% MLMG:computeResOfCorrection() 180 5.455e-05 5.455e-05 5.455e-05 0.01% MLMG::mgVcycle_down::0 36 4.896e-05 4.896e-05 4.896e-05 0.01% Castro::do_advance_ctu() 5 4.379e-05 4.379e-05 4.379e-05 0.01% MLMG::actualBottomSolve() 36 3.94e-05 3.94e-05 3.94e-05 0.01% MLMG::mgVcycle_down::1 36 3.915e-05 3.915e-05 3.915e-05 0.01% MLMG::mgVcycle_down::2 36 3.642e-05 3.642e-05 3.642e-05 0.01% MLMG::mgVcycle_down::4 36 3.574e-05 3.574e-05 3.574e-05 0.01% MLMG::mgVcycle_down::3 36 3.471e-05 3.471e-05 3.471e-05 0.01% MLMG::solve() 6 3.361e-05 3.361e-05 3.361e-05 0.01% Castro::clean_state() 30 3.326e-05 3.326e-05 3.326e-05 0.01% FillPatchIterator::FillFromLevel0() 20 3.155e-05 3.155e-05 3.155e-05 0.01% Amr::writeSmallPlotFile() 1 3.087e-05 3.087e-05 3.087e-05 0.01% Castro::post_restart() 1 3.071e-05 3.071e-05 3.071e-05 0.01% Castro::buildMetrics() 1 2.945e-05 2.945e-05 2.945e-05 0.01% Castro::finalize_advance() 5 2.88e-05 2.88e-05 2.88e-05 0.01% MLMG::mgVcycle_up::4 36 2.799e-05 2.799e-05 2.799e-05 0.01% Castro::construct_old_source() 25 2.71e-05 2.71e-05 2.71e-05 0.01% Castro::initMFs() 1 2.544e-05 2.544e-05 2.544e-05 0.01% MLMG::oneIter() 36 2.451e-05 2.451e-05 2.451e-05 0.01% Castro::swap_state_time_levels() 5 2.381e-05 2.381e-05 2.381e-05 0.01% MLCellLinOp::solutionResidual() 42 2.277e-05 2.277e-05 2.277e-05 0.01% MLMG::mgVcycle_up::3 36 2.196e-05 2.196e-05 2.196e-05 0.01% MLMG::mgVcycle_up::0 36 2.183e-05 2.183e-05 2.183e-05 0.01% MLMG::mgVcycle_up::2 36 2.064e-05 2.064e-05 2.064e-05 0.00% MLMG::mgVcycle_up::1 36 2.017e-05 2.017e-05 2.017e-05 0.00% MLMG::ResNormInf() 42 1.975e-05 1.975e-05 1.975e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.757e-05 1.757e-05 1.757e-05 0.00% MLMG::mgVcycle_bottom 36 1.58e-05 1.58e-05 1.58e-05 0.00% FillPatchSingleLevel 20 1.544e-05 1.544e-05 1.544e-05 0.00% MLPoisson::define() 6 1.43e-05 1.43e-05 1.43e-05 0.00% makeSFC 30 1.412e-05 1.412e-05 1.412e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.32e-05 1.32e-05 1.32e-05 0.00% Castro::construct_new_gravity() 5 1.302e-05 1.302e-05 1.302e-05 0.00% MLMG::computeResidual() 36 1.263e-05 1.263e-05 1.263e-05 0.00% Castro::do_old_sources() 5 1.253e-05 1.253e-05 1.253e-05 0.00% MLMG::computeMLResidual() 6 9.784e-06 9.784e-06 9.784e-06 0.00% MLPoisson::prepareForSolve() 6 9.642e-06 9.642e-06 9.642e-06 0.00% DistributionMapping::Distribute() 31 8.944e-06 8.944e-06 8.944e-06 0.00% Amr::initSubcycle() 1 8.291e-06 8.291e-06 8.291e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.123e-06 8.123e-06 8.123e-06 0.00% Gravity::actual_multilevel_solve() 1 7.968e-06 7.968e-06 7.968e-06 0.00% Castro::check_for_nan() 10 7.81e-06 7.81e-06 7.81e-06 0.00% Castro::construct_old_gravity() 5 5.808e-06 5.808e-06 5.808e-06 0.00% Castro::apply_source_to_state() 10 5.697e-06 5.697e-06 5.697e-06 0.00% MLLinOp::define() 6 5.311e-06 5.311e-06 5.311e-06 0.00% Castro::post_timestep() 5 4.388e-06 4.388e-06 4.388e-06 0.00% Gravity::swapTimeLevels() 5 3.988e-06 3.988e-06 3.988e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.9e-06 3.9e-06 3.9e-06 0.00% MLMG::getGradSolution() 6 3.6e-06 3.6e-06 3.6e-06 0.00% Castro::computeNewDt() 5 3.349e-06 3.349e-06 3.349e-06 0.00% Castro::expand_state() 5 2.954e-06 2.954e-06 2.954e-06 0.00% MLMG::MLResNormInf() 6 2.564e-06 2.564e-06 2.564e-06 0.00% Gravity::set_mass_offset() 6 2.257e-06 2.257e-06 2.257e-06 0.00% MLMG::MLRhsNormInf() 6 2.214e-06 2.214e-06 2.214e-06 0.00% Castro::retry_advance_ctu() 5 2.017e-06 2.017e-06 2.017e-06 0.00% Castro::FluxRegCrseInit 5 1.671e-06 1.671e-06 1.671e-06 0.00% Castro::FluxRegFineAdd() 5 1.285e-06 1.285e-06 1.285e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.085e-06 1.085e-06 1.085e-06 0.00% Amr::init() 1 9.51e-07 9.51e-07 9.51e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4284 0.4284 0.4284 99.99% Amr::coarseTimeStep() 5 0.3187 0.3187 0.3187 74.38% Amr::timeStep() 5 0.3164 0.3164 0.3164 73.85% Castro::advance() 5 0.3104 0.3104 0.3104 72.45% Castro::subcycle_advance_ctu() 5 0.3035 0.3035 0.3035 70.84% Castro::do_advance_ctu() 5 0.3034 0.3034 0.3034 70.81% Castro::construct_new_gravity() 5 0.1346 0.1346 0.1346 31.41% Gravity::solve_phi_with_mlmg() 6 0.1325 0.1325 0.1325 30.92% Castro::construct_ctu_hydro_source() 5 0.132 0.132 0.132 30.81% Gravity::solve_for_phi() 5 0.1268 0.1268 0.1268 29.61% Gravity::actual_solve_with_mlmg() 6 0.1267 0.1267 0.1267 29.58% MLMG::solve() 6 0.1149 0.1149 0.1149 26.81% MLMG::oneIter() 36 0.1074 0.1074 0.1074 25.07% MLMG::mgVcycle() 36 0.1059 0.1059 0.1059 24.72% Amr::init() 1 0.07157 0.07157 0.07157 16.71% Amr::restart() 1 0.07157 0.07157 0.07157 16.71% AmrLevel::restart() 1 0.06042 0.06042 0.06042 14.10% StateData::restartDoit() 4 0.06034 0.06034 0.06034 14.08% VisMF::Read() 3 0.06006 0.06006 0.06006 14.02% MLCellLinOp::smooth() 720 0.0516 0.0516 0.0516 12.04% MLCellLinOp::applyBC() 1946 0.04858 0.04858 0.04858 11.34% Amr::writePlotFile() 1 0.03735 0.03735 0.03735 8.72% VisMF::Write(FabArray) 1 0.03569 0.03569 0.03569 8.33% MLMG::mgVcycle_bottom 36 0.03319 0.03319 0.03319 7.75% MLMG::actualBottomSolve() 36 0.03318 0.03318 0.03318 7.74% MLCGSolver::bicgstab 36 0.03286 0.03286 0.03286 7.67% Castro::clean_state() 30 0.02735 0.02735 0.02735 6.38% AmrLevel::FillPatch() 20 0.02272 0.02272 0.02272 5.30% FillPatchIterator::Initialize 20 0.02063 0.02063 0.02063 4.82% FillPatchIterator::FillFromLevel0() 20 0.01985 0.01985 0.01985 4.63% FillPatchSingleLevel 20 0.01981 0.01981 0.01981 4.62% StateDataPhysBCFunct::() 20 0.01777 0.01777 0.01777 4.15% MLCellLinOp::apply() 500 0.01643 0.01643 0.01643 3.84% MLMG::mgVcycle_down::0 36 0.01474 0.01474 0.01474 3.44% MLPoisson::Fsmooth() 1440 0.01435 0.01435 0.01435 3.35% FabArray::FillBoundary() 1766 0.01368 0.01368 0.01368 3.19% FillBoundary_nowait() 1766 0.01329 0.01329 0.01329 3.10% StateData::FillBoundary(geom) 160 0.01261 0.01261 0.01261 2.94% MLMG::mgVcycle_up::0 36 0.01111 0.01111 0.01111 2.59% Castro::initialize_do_advance() 5 0.01094 0.01094 0.01094 2.55% Castro::computeTemp() 30 0.01057 0.01057 0.01057 2.47% MLCellLinOp::correctionResidual() 216 0.009883 0.009883 0.009883 2.31% Castro::do_old_sources() 5 0.00969 0.00969 0.00969 2.26% Castro::normalize_species() 30 0.009455 0.009455 0.009455 2.21% MLPoisson::define() 6 0.009293 0.009293 0.009293 2.17% amrex::Dot() 484 0.009174 0.009174 0.009174 2.14% FabArray::norminf() 465 0.008878 0.008878 0.008878 2.07% MLMG:computeResOfCorrection() 180 0.008671 0.008671 0.008671 2.02% Castro::construct_old_gravity() 5 0.007671 0.007671 0.007671 1.79% Gravity::get_old_grav_vector() 5 0.007666 0.007666 0.007666 1.79% Gravity::get_new_grav_vector() 5 0.007599 0.007599 0.007599 1.77% MLMG::mgVcycle_down::1 36 0.007164 0.007164 0.007164 1.67% Castro::do_new_sources() 5 0.00671 0.00671 0.00671 1.57% FabArray::ParallelCopy() 380 0.006621 0.006621 0.006621 1.55% MLMG::mgVcycle_down::2 36 0.006595 0.006595 0.006595 1.54% FabArray::setVal() 537 0.00659 0.00659 0.00659 1.54% Castro::initialize_advance() 5 0.006516 0.006516 0.006516 1.52% FabArray::ParallelCopy_nowait() 380 0.006513 0.006513 0.006513 1.52% MLMG::mgVcycle_down::3 36 0.00643 0.00643 0.00643 1.50% MLMG::mgVcycle_down::4 36 0.006384 0.006384 0.006384 1.49% MLCellLinOp::defineAuxData() 6 0.006324 0.006324 0.006324 1.48% Castro::post_restart() 1 0.006251 0.006251 0.006251 1.46% Castro::enforce_min_density() 30 0.006133 0.006133 0.006133 1.43% Castro::expand_state() 5 0.006001 0.006001 0.006001 1.40% Castro::post_timestep() 5 0.005941 0.005941 0.005941 1.39% Gravity::multilevel_solve_for_new_phi() 1 0.005877 0.005877 0.005877 1.37% Gravity::actual_multilevel_solve() 1 0.005859 0.005859 0.005859 1.37% FabArray::Saxpy() 597 0.005664 0.005664 0.005664 1.32% Gravity::fill_multipole_BCs() 6 0.005626 0.005626 0.005626 1.31% MLCGSolver::ParallelAllReduce 798 0.005568 0.005568 0.005568 1.30% MLMG::addInterpCorrection() 180 0.005351 0.005351 0.005351 1.25% amrex::Copy() 221 0.005307 0.005307 0.005307 1.24% MLMG::mgVcycle_up::4 36 0.005146 0.005146 0.005146 1.20% MLMG::mgVcycle_up::1 36 0.005101 0.005101 0.005101 1.19% amrex::average_down 180 0.005058 0.005058 0.005058 1.18% MLMG::mgVcycle_up::2 36 0.005004 0.005004 0.005004 1.17% MLMG::mgVcycle_up::3 36 0.00494 0.00494 0.00494 1.15% MLPoisson::Fapply() 500 0.004651 0.004651 0.004651 1.09% FabArray::Xpay() 361 0.003655 0.003655 0.003655 0.85% MLCellLinOp::solutionResidual() 42 0.003486 0.003486 0.003486 0.81% Castro::estTimeStep() 10 0.00323 0.00323 0.00323 0.75% Castro::reset_internal_energy(MultiFab) 30 0.002869 0.002869 0.002869 0.67% MLCellLinOp::defineBC() 6 0.002835 0.002835 0.002835 0.66% MLMG::prepareForSolve() 6 0.002801 0.002801 0.002801 0.65% MLMG::computeResidual() 36 0.002729 0.002729 0.002729 0.64% BndryData::define() 6 0.002698 0.002698 0.002698 0.63% Castro::computeNewDt() 5 0.002048 0.002048 0.002048 0.48% amrex::Add() 72 0.001881 0.001881 0.001881 0.44% Castro::construct_new_source() 25 0.00183 0.00183 0.00183 0.43% Castro::construct_new_gravity_source() 5 0.00176 0.00176 0.00176 0.41% Castro::construct_old_source() 25 0.001438 0.001438 0.001438 0.34% Castro::construct_old_gravity_source() 5 0.00141 0.00141 0.00141 0.33% Castro::finalize_do_advance() 5 0.001258 0.001258 0.001258 0.29% Castro::enforce_speed_limit() 30 0.001162 0.001162 0.001162 0.27% check_for_negative_density() 5 0.001031 0.001031 0.001031 0.24% MLMG::ResNormInf() 42 0.0009878 0.0009878 0.0009878 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009656 0.0009656 0.0009656 0.23% Castro::apply_source_to_state() 10 0.0009548 0.0009548 0.0009548 0.22% MLMG::getGradSolution() 6 0.0008591 0.0008591 0.0008591 0.20% MLCellLinOp::compGrad() 6 0.0008555 0.0008555 0.0008555 0.20% MLCellLinOp::setLevelBC() 6 0.0008412 0.0008412 0.0008412 0.20% Castro::reset_internal_energy(Fab) 240 0.0008281 0.0008281 0.0008281 0.19% FabArrayBase::getCPC() 632 0.0007815 0.0007815 0.0007815 0.18% MLMG::computeMLResidual() 6 0.0007794 0.0007794 0.0007794 0.18% MLPoisson::prepareForSolve() 6 0.0007518 0.0007518 0.0007518 0.18% MLCellLinOp::prepareForSolve() 6 0.0007422 0.0007422 0.0007422 0.17% Gravity::update_max_rhs() 6 0.0007149 0.0007149 0.0007149 0.17% FabArray::mult() 22 0.0006835 0.0006835 0.0006835 0.16% FabArray::setDomainBndry() 20 0.0006753 0.0006753 0.0006753 0.16% Castro::check_for_nan() 10 0.0006508 0.0006508 0.0006508 0.15% MultiFab::contains_nan() 10 0.0006429 0.0006429 0.0006429 0.15% Amr::InitAmr() 1 0.0004226 0.0004226 0.0004226 0.10% FabArrayBase::CPC::define() 244 0.0004183 0.0004183 0.0004183 0.10% FabArrayBase::getFB() 1766 0.0003435 0.0003435 0.0003435 0.08% Castro::finalize_advance() 5 0.0002931 0.0002931 0.0002931 0.07% Gravity::swapTimeLevels() 5 0.0002281 0.0002281 0.0002281 0.05% MultiFab::max() 6 0.0002124 0.0002124 0.0002124 0.05% MLMG::MLResNormInf() 6 0.0001981 0.0001981 0.0001981 0.05% Castro::buildMetrics() 1 0.0001559 0.0001559 0.0001559 0.04% MLLinOp::define() 6 0.0001197 0.0001197 0.0001197 0.03% MLMG::MLRhsNormInf() 6 0.0001179 0.0001179 0.0001179 0.03% MLLinOp::defineGrids() 6 0.0001144 0.0001144 0.0001144 0.03% Castro::create_source_corrector() 5 8.211e-05 8.211e-05 8.211e-05 0.02% FabArrayBase::FB::FB() 26 6.199e-05 6.199e-05 6.199e-05 0.01% Amr::writeSmallPlotFile() 1 3.087e-05 3.087e-05 3.087e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.976e-05 2.976e-05 2.976e-05 0.01% Castro::initMFs() 1 2.544e-05 2.544e-05 2.544e-05 0.01% Castro::swap_state_time_levels() 5 2.381e-05 2.381e-05 2.381e-05 0.01% makeSFC 30 2.163e-05 2.163e-05 2.163e-05 0.01% DistributionMapping::Distribute() 31 8.944e-06 8.944e-06 8.944e-06 0.00% Amr::initSubcycle() 1 8.291e-06 8.291e-06 8.291e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.332e-06 5.332e-06 5.332e-06 0.00% Gravity::set_mass_offset() 6 2.257e-06 2.257e-06 2.257e-06 0.00% Castro::retry_advance_ctu() 5 2.017e-06 2.017e-06 2.017e-06 0.00% Castro::FluxRegCrseInit 5 1.671e-06 1.671e-06 1.671e-06 0.00% Castro::FluxRegFineAdd() 5 1.285e-06 1.285e-06 1.285e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.085e-06 1.085e-06 1.085e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 196 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 137 MiB 692 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1061 KiB 39 MiB Castro::initialize_do_advance() 40 40 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2445 KiB 28 MiB Castro::initialize_advance() 40 40 16 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6413 KiB 14 MiB MLMG::prepareForSolve() 361 361 3291 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 185 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 181 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6399 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 22 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3312 B 2048 KiB Gravity::solve_for_phi() 40 40 605 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 27 KiB 2048 KiB BndryData::define() 576 576 303 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 194 KiB 671 KiB Castro::estTimeStep() 10 10 3619 B 480 KiB VisMF::Write(FabArray) 112 112 2469 B 320 KiB Castro::normalize_species() 30 30 7184 B 320 KiB amrex::average_down 469 469 1143 B 257 KiB MLMG::addInterpCorrection() 468 468 1065 B 257 KiB amrex::Dot() 592 592 3146 B 160 KiB FabArray::norminf() 501 501 3200 B 160 KiB check_for_negative_density() 5 5 350 B 160 KiB MultiFab::max() 6 6 77 B 160 KiB FabArray::setVal() 66 66 17 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3191 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 262 B 9648 B MLCellLinOp::applyBC() 3892 3892 205 B 9344 B amrex::Copy() 56 56 5655 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 42 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 216 216 97 B 1280 B MLCellLinOp::defineBC() 36 36 340 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1148 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 83 KiB 8192 KiB VisMF::Write(FabArray) 120 120 239 KiB 3584 KiB VisMF::Read() 24 24 212 KiB 3000 KiB FabArray::setVal() 66 66 17 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3191 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 262 B 9648 B MLCellLinOp::applyBC() 1946 1946 203 B 9328 B amrex::Copy() 56 56 5655 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2511 B 3072 B StateData::FillBoundary(geom) 960 960 42 B 3024 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 501 B 1296 B amrex::average_down 37 37 212 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.11-17-g9e35dc19489d) finalized