Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-2-g6a7da9909378) initialized Starting run at 10:10:50 UTC on 2023-03-06. Successfully read inputs file ... Castro git describe: 23.03-1-g11cc0c2de AMReX git describe: 23.03-2-g6a7da9909 Microphysics git describe: 23.03-7-g751f2f5f reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056692225 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032454835 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047199892 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048658591 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.059501709 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.055234447 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.065535547 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.0567323 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.063529395 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.056779425 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056456899 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.055199274 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.057255301 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.055956387 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03251145 seconds Ending run at 10:10:51 UTC on 2023-03-06. Run time = 0.847924543 Run time without initialization = 0.711180297 Average number of zones advanced per microsecond: 3.686 Average number of zones advanced per microsecond per rank: 3.686 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.848 ... 0.848 ... 0.848 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2275 0.2275 0.2275 26.83% Castro::construct_ctu_hydro_source() 10 0.2189 0.2189 0.2189 25.82% MLCellLinOp::applyBC() 4433 0.07439 0.07439 0.07439 8.77% MLPoisson::Fsmooth() 3280 0.03133 0.03133 0.03133 3.69% FillBoundary_nowait() 4023 0.03006 0.03006 0.03006 3.54% StateData::FillBoundary(geom) 328 0.0242 0.0242 0.0242 2.85% amrex::Dot() 1114 0.01992 0.01992 0.01992 2.35% amrex::Copy() 1029 0.01449 0.01449 0.01449 1.71% FabArray::norminf() 743 0.01381 0.01381 0.01381 1.63% Castro::normalize_species() 62 0.01329 0.01329 0.01329 1.57% FabArray::ParallelCopy_nowait() 861 0.01303 0.01303 0.01303 1.54% FabArray::setVal() 1144 0.01273 0.01273 0.01273 1.50% Castro::computeTemp() 63 0.01272 0.01272 0.01272 1.50% StateDataPhysBCFunct::() 41 0.01252 0.01252 0.01252 1.48% MLPoisson::Fapply() 1142 0.01005 0.01005 0.01005 1.19% MLCellLinOp::defineAuxData() 11 0.009678 0.009678 0.009678 1.14% Castro::enforce_min_density() 62 0.008727 0.008727 0.008727 1.03% FabArray::Saxpy() 813 0.007905 0.007905 0.007905 0.93% FabArray::Xpay() 821 0.00783 0.00783 0.00783 0.92% MLMG::addInterpCorrection() 410 0.006651 0.006651 0.006651 0.78% Gravity::fill_multipole_BCs() 11 0.006074 0.006074 0.006074 0.72% amrex::average_down 410 0.005779 0.005779 0.005779 0.68% Castro::estTimeStep() 21 0.004886 0.004886 0.004886 0.58% FabArray::LinComb() 557 0.004362 0.004362 0.004362 0.51% amrex::Add() 164 0.004303 0.004303 0.004303 0.51% Castro::reset_internal_energy(MultiFab) 63 0.004023 0.004023 0.004023 0.47% Amr::checkPoint() 3 0.003987 0.003987 0.003987 0.47% BndryData::define() 11 0.003563 0.003563 0.003563 0.42% Castro::construct_new_gravity_source() 10 0.003255 0.003255 0.003255 0.38% Castro::do_advance_ctu() 10 0.002713 0.002713 0.002713 0.32% Castro::construct_old_gravity_source() 10 0.002556 0.002556 0.002556 0.30% Amr::writePlotFile() 2 0.002339 0.002339 0.002339 0.28% MLCGSolver::bicgstab 82 0.002206 0.002206 0.002206 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001628 0.001628 0.001628 0.19% Gravity::actual_solve_with_mlmg() 11 0.001513 0.001513 0.001513 0.18% Castro::reset_internal_energy(Fab) 504 0.00145 0.00145 0.00145 0.17% MLCellLinOp::setLevelBC() 11 0.001342 0.001342 0.001342 0.16% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% FabArray::setDomainBndry() 41 0.001285 0.001285 0.001285 0.15% Castro::initData() 1 0.001269 0.001269 0.001269 0.15% Castro::enforce_speed_limit() 62 0.001257 0.001257 0.001257 0.15% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% MLCellLinOp::smooth() 1640 0.001137 0.001137 0.001137 0.13% MLCellLinOp::prepareForSolve() 11 0.001087 0.001087 0.001087 0.13% Castro::subcycle_advance_ctu() 10 0.0008959 0.0008959 0.0008959 0.11% MLCellLinOp::compGrad() 11 0.0008836 0.0008836 0.0008836 0.10% FabArray::FillBoundary() 4023 0.0008723 0.0008723 0.0008723 0.10% MLMG::prepareForSolve() 11 0.0008692 0.0008692 0.0008692 0.10% FabArrayBase::getCPC() 1323 0.0007356 0.0007356 0.0007356 0.09% FabArrayBase::CPC::define() 454 0.0006751 0.0006751 0.0006751 0.08% Gravity::get_new_grav_vector() 11 0.0006186 0.0006186 0.0006186 0.07% FabArrayBase::getFB() 4023 0.0005849 0.0005849 0.0005849 0.07% Gravity::get_old_grav_vector() 10 0.0005401 0.0005401 0.0005401 0.06% MLCellLinOp::apply() 1142 0.0004877 0.0004877 0.0004877 0.06% Amr::InitAmr() 1 0.0004871 0.0004871 0.0004871 0.06% MLMG::mgVcycle() 82 0.0003921 0.0003921 0.0003921 0.05% MLLinOp::defineGrids() 11 0.0003664 0.0003664 0.0003664 0.04% Amr::coarseTimeStep() 10 0.0003345 0.0003345 0.0003345 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003201 0.0003201 0.0003201 0.04% main() 1 0.0003082 0.0003082 0.0003082 0.04% FabArray::ParallelCopy() 861 0.0002652 0.0002652 0.0002652 0.03% MultiFab::max() 11 0.0002592 0.0002592 0.0002592 0.03% FillPatchIterator::Initialize 41 0.0002427 0.0002427 0.0002427 0.03% MLCellLinOp::correctionResidual() 492 0.000216 0.000216 0.000216 0.03% MLCellLinOp::defineBC() 11 0.0002056 0.0002056 0.0002056 0.02% Amr::timeStep() 10 0.0001604 0.0001604 0.0001604 0.02% StateData::checkPoint() 12 0.0001348 0.0001348 0.0001348 0.02% Gravity::update_max_rhs() 11 0.000129 0.000129 0.000129 0.02% MLMG:computeResOfCorrection() 410 0.0001203 0.0001203 0.0001203 0.01% Gravity::solve_for_phi() 10 0.0001119 0.0001119 0.0001119 0.01% Castro::finalize_advance() 10 0.0001083 0.0001083 0.0001083 0.01% MLMG::actualBottomSolve() 82 9.137e-05 9.137e-05 9.137e-05 0.01% MLMG::mgVcycle_down::0 82 8.783e-05 8.783e-05 8.783e-05 0.01% FabArrayBase::FB::FB() 56 8.497e-05 8.497e-05 8.497e-05 0.01% MLMG::mgVcycle_down::1 82 8.112e-05 8.112e-05 8.112e-05 0.01% Castro::expand_state() 10 7.946e-05 7.946e-05 7.946e-05 0.01% Castro::Castro() 1 7.898e-05 7.898e-05 7.898e-05 0.01% Castro::clean_state() 62 7.719e-05 7.719e-05 7.719e-05 0.01% MLMG::mgVcycle_down::2 82 7.615e-05 7.615e-05 7.615e-05 0.01% MLMG::solve() 11 7.543e-05 7.543e-05 7.543e-05 0.01% AmrLevel::checkPoint() 3 7.482e-05 7.482e-05 7.482e-05 0.01% MLMG::mgVcycle_down::3 82 7.238e-05 7.238e-05 7.238e-05 0.01% Castro::initialize_advance() 10 7.219e-05 7.219e-05 7.219e-05 0.01% MLMG::mgVcycle_down::4 82 7.2e-05 7.2e-05 7.2e-05 0.01% Castro::advance() 10 6.614e-05 6.614e-05 6.614e-05 0.01% MLMG::mgVcycle_up::4 82 6.361e-05 6.361e-05 6.361e-05 0.01% Castro::initialize_do_advance() 10 5.42e-05 5.42e-05 5.42e-05 0.01% MLMG::mgVcycle_up::0 82 5.327e-05 5.327e-05 5.327e-05 0.01% MLMG::mgVcycle_up::3 82 5.156e-05 5.156e-05 5.156e-05 0.01% MLMG::mgVcycle_up::1 82 5.145e-05 5.145e-05 5.145e-05 0.01% MLMG::mgVcycle_up::2 82 5.104e-05 5.104e-05 5.104e-05 0.01% MLMG::oneIter() 82 5.091e-05 5.091e-05 5.091e-05 0.01% MLCellLinOp::solutionResidual() 93 4.874e-05 4.874e-05 4.874e-05 0.01% Castro::swap_state_time_levels() 10 4.032e-05 4.032e-05 4.032e-05 0.00% Castro::finalize_do_advance() 10 3.657e-05 3.657e-05 3.657e-05 0.00% MLMG::computeResidual() 82 3.519e-05 3.519e-05 3.519e-05 0.00% Castro::enforce_consistent_e() 1 3.384e-05 3.384e-05 3.384e-05 0.00% StateData::define() 4 3.26e-05 3.26e-05 3.26e-05 0.00% MLMG::ResNormInf() 93 3.244e-05 3.244e-05 3.244e-05 0.00% MLPoisson::define() 11 3.2e-05 3.2e-05 3.2e-05 0.00% MLMG::mgVcycle_bottom 82 3.083e-05 3.083e-05 3.083e-05 0.00% FillPatchSingleLevel 41 3.04e-05 3.04e-05 3.04e-05 0.00% Castro::construct_old_source() 50 2.964e-05 2.964e-05 2.964e-05 0.00% Amr::writeSmallPlotFile() 1 2.546e-05 2.546e-05 2.546e-05 0.00% Castro::construct_new_gravity() 10 2.538e-05 2.538e-05 2.538e-05 0.00% makeSFC 55 2.519e-05 2.519e-05 2.519e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.309e-05 2.309e-05 2.309e-05 0.00% Castro::create_source_corrector() 10 2.283e-05 2.283e-05 2.283e-05 0.00% Castro::initMFs() 1 2.144e-05 2.144e-05 2.144e-05 0.00% Amr::FinalizeInit() 1 1.868e-05 1.868e-05 1.868e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.784e-05 1.784e-05 1.784e-05 0.00% Castro::construct_new_source() 50 1.768e-05 1.768e-05 1.768e-05 0.00% Castro::buildMetrics() 1 1.757e-05 1.757e-05 1.757e-05 0.00% Castro::do_new_sources() 10 1.678e-05 1.678e-05 1.678e-05 0.00% MLLinOp::define() 11 1.6e-05 1.6e-05 1.6e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.592e-05 1.592e-05 1.592e-05 0.00% Castro::do_old_sources() 10 1.536e-05 1.536e-05 1.536e-05 0.00% DistributionMapping::Distribute() 56 1.506e-05 1.506e-05 1.506e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.464e-05 1.464e-05 1.464e-05 0.00% Amr::defBaseLevel() 1 1.317e-05 1.317e-05 1.317e-05 0.00% Castro::check_for_nan() 20 1.109e-05 1.109e-05 1.109e-05 0.00% Castro::post_init() 1 1.08e-05 1.08e-05 1.08e-05 0.00% Castro::apply_source_to_state() 20 1.071e-05 1.071e-05 1.071e-05 0.00% Castro::construct_old_gravity() 10 1.045e-05 1.045e-05 1.045e-05 0.00% Gravity::swapTimeLevels() 10 9.621e-06 9.621e-06 9.621e-06 0.00% Amr::initSubcycle() 1 8.851e-06 8.851e-06 8.851e-06 0.00% Gravity::actual_multilevel_solve() 1 8.299e-06 8.299e-06 8.299e-06 0.00% MLPoisson::prepareForSolve() 11 7.956e-06 7.956e-06 7.956e-06 0.00% Castro::post_timestep() 10 7.923e-06 7.923e-06 7.923e-06 0.00% MLMG::computeMLResidual() 11 7.114e-06 7.114e-06 7.114e-06 0.00% Castro::computeNewDt() 9 6.387e-06 6.387e-06 6.387e-06 0.00% MLMG::getGradSolution() 11 5.657e-06 5.657e-06 5.657e-06 0.00% Amr::InitializeInit() 1 5.171e-06 5.171e-06 5.171e-06 0.00% AmrLevel::checkPointPost() 3 4.833e-06 4.833e-06 4.833e-06 0.00% Gravity::set_mass_offset() 11 4.435e-06 4.435e-06 4.435e-06 0.00% Castro::retry_advance_ctu() 10 4.22e-06 4.22e-06 4.22e-06 0.00% Castro::FluxRegCrseInit 10 3.663e-06 3.663e-06 3.663e-06 0.00% MLMG::MLRhsNormInf() 11 3.579e-06 3.579e-06 3.579e-06 0.00% MLMG::MLResNormInf() 11 3.369e-06 3.369e-06 3.369e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.313e-06 3.313e-06 3.313e-06 0.00% Amr::init() 1 2.415e-06 2.415e-06 2.415e-06 0.00% Castro::computeInitialDt() 2 2.308e-06 2.308e-06 2.308e-06 0.00% Castro::FluxRegFineAdd() 10 2.004e-06 2.004e-06 2.004e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.932e-06 1.932e-06 1.932e-06 0.00% AmrLevel::checkPointPre() 3 1.73e-06 1.73e-06 1.73e-06 0.00% Amr::initialInit() 1 1.099e-06 1.099e-06 1.099e-06 0.00% Castro::post_regrid() 1 9.52e-07 9.52e-07 9.52e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8479 0.8479 0.8479 100.00% Amr::coarseTimeStep() 10 0.6784 0.6784 0.6784 80.01% Amr::timeStep() 10 0.562 0.562 0.562 66.28% Castro::advance() 10 0.5563 0.5563 0.5563 65.61% Castro::subcycle_advance_ctu() 10 0.5447 0.5447 0.5447 64.23% Castro::do_advance_ctu() 10 0.5438 0.5438 0.5438 64.13% Gravity::solve_phi_with_mlmg() 11 0.275 0.275 0.275 32.43% Gravity::actual_solve_with_mlmg() 11 0.2684 0.2684 0.2684 31.66% Castro::construct_new_gravity() 10 0.2511 0.2511 0.2511 29.61% MLMG::solve() 11 0.2481 0.2481 0.2481 29.26% Gravity::solve_for_phi() 10 0.2355 0.2355 0.2355 27.78% MLMG::oneIter() 82 0.2341 0.2341 0.2341 27.61% MLMG::mgVcycle() 82 0.2305 0.2305 0.2305 27.18% VisMF::Write(FabArray) 11 0.2275 0.2275 0.2275 26.83% Castro::construct_ctu_hydro_source() 10 0.2189 0.2189 0.2189 25.82% Amr::checkPoint() 3 0.1695 0.1695 0.1695 19.99% AmrLevel::checkPoint() 3 0.1655 0.1655 0.1655 19.52% StateData::checkPoint() 12 0.1654 0.1654 0.1654 19.51% Amr::init() 1 0.1361 0.1361 0.1361 16.05% MLCellLinOp::smooth() 1640 0.1132 0.1132 0.1132 13.35% MLCellLinOp::applyBC() 4433 0.106 0.106 0.106 12.50% MLMG::mgVcycle_bottom 82 0.07118 0.07118 0.07118 8.39% MLMG::actualBottomSolve() 82 0.07115 0.07115 0.07115 8.39% MLCGSolver::bicgstab 82 0.07048 0.07048 0.07048 8.31% Amr::writePlotFile() 2 0.06509 0.06509 0.06509 7.68% Amr::initialInit() 1 0.04684 0.04684 0.04684 5.52% Amr::FinalizeInit() 1 0.04294 0.04294 0.04294 5.06% FillPatchIterator::Initialize 41 0.04232 0.04232 0.04232 4.99% Castro::post_init() 1 0.04172 0.04172 0.04172 4.92% Castro::clean_state() 62 0.04087 0.04087 0.04087 4.82% FillPatchSingleLevel 41 0.0408 0.0408 0.0408 4.81% Gravity::multilevel_solve_for_new_phi() 1 0.03987 0.03987 0.03987 4.70% Gravity::actual_multilevel_solve() 1 0.03985 0.03985 0.03985 4.70% StateDataPhysBCFunct::() 41 0.03672 0.03672 0.03672 4.33% MLCellLinOp::apply() 1142 0.03532 0.03532 0.03532 4.17% MLMG::mgVcycle_down::0 82 0.03292 0.03292 0.03292 3.88% FabArray::FillBoundary() 4023 0.0316 0.0316 0.0316 3.73% MLPoisson::Fsmooth() 3280 0.03133 0.03133 0.03133 3.69% FillBoundary_nowait() 4023 0.03073 0.03073 0.03073 3.62% MLMG::mgVcycle_up::0 82 0.02497 0.02497 0.02497 2.94% StateData::FillBoundary(geom) 328 0.0242 0.0242 0.0242 2.85% MLCellLinOp::correctionResidual() 492 0.02162 0.02162 0.02162 2.55% amrex::Dot() 1114 0.01992 0.01992 0.01992 2.35% MLMG:computeResOfCorrection() 410 0.01908 0.01908 0.01908 2.25% Castro::initialize_do_advance() 10 0.01825 0.01825 0.01825 2.15% Castro::computeTemp() 63 0.01819 0.01819 0.01819 2.15% Gravity::get_new_grav_vector() 11 0.01711 0.01711 0.01711 2.02% MLPoisson::define() 11 0.01614 0.01614 0.01614 1.90% MLMG::mgVcycle_down::1 82 0.01528 0.01528 0.01528 1.80% Castro::construct_old_gravity() 10 0.01486 0.01486 0.01486 1.75% Gravity::get_old_grav_vector() 10 0.01485 0.01485 0.01485 1.75% amrex::Copy() 1029 0.01449 0.01449 0.01449 1.71% MLMG::mgVcycle_down::2 82 0.01427 0.01427 0.01427 1.68% FabArray::ParallelCopy() 861 0.01411 0.01411 0.01411 1.66% MLMG::mgVcycle_down::3 82 0.01394 0.01394 0.01394 1.64% FabArray::ParallelCopy_nowait() 861 0.01384 0.01384 0.01384 1.63% FabArray::norminf() 743 0.01381 0.01381 0.01381 1.63% MLMG::mgVcycle_down::4 82 0.01375 0.01375 0.01375 1.62% Castro::normalize_species() 62 0.01329 0.01329 0.01329 1.57% FabArray::setVal() 1144 0.01273 0.01273 0.01273 1.50% MLCGSolver::ParallelAllReduce 1514 0.01197 0.01197 0.01197 1.41% Castro::do_new_sources() 10 0.01186 0.01186 0.01186 1.40% MLMG::addInterpCorrection() 410 0.01169 0.01169 0.01169 1.38% Castro::expand_state() 10 0.0113 0.0113 0.0113 1.33% MLMG::mgVcycle_up::4 82 0.01116 0.01116 0.01116 1.32% MLMG::mgVcycle_up::1 82 0.0111 0.0111 0.0111 1.31% MLCellLinOp::defineAuxData() 11 0.01098 0.01098 0.01098 1.29% Castro::initialize_advance() 10 0.01096 0.01096 0.01096 1.29% MLMG::mgVcycle_up::2 82 0.01089 0.01089 0.01089 1.28% amrex::average_down 410 0.01081 0.01081 0.01081 1.27% MLMG::mgVcycle_up::3 82 0.01065 0.01065 0.01065 1.26% Castro::do_old_sources() 10 0.01023 0.01023 0.01023 1.21% MLPoisson::Fapply() 1142 0.01005 0.01005 0.01005 1.19% Castro::enforce_min_density() 62 0.008727 0.008727 0.008727 1.03% FabArray::Saxpy() 813 0.007905 0.007905 0.007905 0.93% FabArray::Xpay() 821 0.00783 0.00783 0.00783 0.92% MLCellLinOp::solutionResidual() 93 0.007051 0.007051 0.007051 0.83% Gravity::fill_multipole_BCs() 11 0.006325 0.006325 0.006325 0.75% MLMG::computeResidual() 82 0.0061 0.0061 0.0061 0.72% Castro::post_timestep() 10 0.00551 0.00551 0.00551 0.65% Castro::reset_internal_energy(MultiFab) 63 0.005473 0.005473 0.005473 0.65% Castro::estTimeStep() 21 0.004886 0.004886 0.004886 0.58% MLCellLinOp::defineBC() 11 0.00469 0.00469 0.00469 0.55% MLMG::prepareForSolve() 11 0.004522 0.004522 0.004522 0.53% BndryData::define() 11 0.004485 0.004485 0.004485 0.53% FabArray::LinComb() 557 0.004362 0.004362 0.004362 0.51% amrex::Add() 164 0.004303 0.004303 0.004303 0.51% Amr::InitializeInit() 1 0.003901 0.003901 0.003901 0.46% Amr::defBaseLevel() 1 0.003896 0.003896 0.003896 0.46% Castro::initData() 1 0.003416 0.003416 0.003416 0.40% Castro::construct_new_source() 50 0.003272 0.003272 0.003272 0.39% Castro::construct_new_gravity_source() 10 0.003255 0.003255 0.003255 0.38% Castro::construct_old_source() 50 0.002585 0.002585 0.002585 0.30% Castro::computeNewDt() 9 0.002582 0.002582 0.002582 0.30% Castro::construct_old_gravity_source() 10 0.002556 0.002556 0.002556 0.30% MLMG::ResNormInf() 93 0.002037 0.002037 0.002037 0.24% Castro::apply_source_to_state() 20 0.001815 0.001815 0.001815 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001628 0.001628 0.001628 0.19% Castro::reset_internal_energy(Fab) 504 0.00145 0.00145 0.00145 0.17% FabArrayBase::getCPC() 1323 0.001411 0.001411 0.001411 0.17% MLMG::getGradSolution() 11 0.001355 0.001355 0.001355 0.16% MLCellLinOp::compGrad() 11 0.001349 0.001349 0.001349 0.16% MLCellLinOp::setLevelBC() 11 0.001342 0.001342 0.001342 0.16% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% FabArray::setDomainBndry() 41 0.001285 0.001285 0.001285 0.15% Castro::enforce_speed_limit() 62 0.001257 0.001257 0.001257 0.15% Castro::check_for_nan() 20 0.001204 0.001204 0.001204 0.14% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% MLPoisson::prepareForSolve() 11 0.001095 0.001095 0.001095 0.13% MLCellLinOp::prepareForSolve() 11 0.001087 0.001087 0.001087 0.13% Castro::post_regrid() 1 0.001056 0.001056 0.001056 0.12% MLMG::computeMLResidual() 11 0.0009933 0.0009933 0.0009933 0.12% Castro::computeInitialDt() 2 0.0009045 0.0009045 0.0009045 0.11% Gravity::update_max_rhs() 11 0.0008334 0.0008334 0.0008334 0.10% FabArrayBase::CPC::define() 454 0.0006751 0.0006751 0.0006751 0.08% FabArrayBase::getFB() 4023 0.0006699 0.0006699 0.0006699 0.08% Castro::finalize_advance() 10 0.0006268 0.0006268 0.0006268 0.07% Amr::InitAmr() 1 0.0004959 0.0004959 0.0004959 0.06% MLLinOp::define() 11 0.0004378 0.0004378 0.0004378 0.05% Gravity::swapTimeLevels() 10 0.0004298 0.0004298 0.0004298 0.05% MLLinOp::defineGrids() 11 0.0004218 0.0004218 0.0004218 0.05% Castro::Castro() 1 0.0004129 0.0004129 0.0004129 0.05% MLMG::MLResNormInf() 11 0.0002809 0.0002809 0.0002809 0.03% MultiFab::max() 11 0.0002592 0.0002592 0.0002592 0.03% MLMG::MLRhsNormInf() 11 0.0002167 0.0002167 0.0002167 0.03% Castro::buildMetrics() 1 0.0001532 0.0001532 0.0001532 0.02% FabArrayBase::FB::FB() 56 8.497e-05 8.497e-05 8.497e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.342e-05 5.342e-05 5.342e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.852e-05 4.852e-05 4.852e-05 0.01% Castro::swap_state_time_levels() 10 4.032e-05 4.032e-05 4.032e-05 0.00% makeSFC 55 3.878e-05 3.878e-05 3.878e-05 0.00% Castro::finalize_do_advance() 10 3.657e-05 3.657e-05 3.657e-05 0.00% Castro::enforce_consistent_e() 1 3.384e-05 3.384e-05 3.384e-05 0.00% StateData::define() 4 3.26e-05 3.26e-05 3.26e-05 0.00% Amr::writeSmallPlotFile() 1 2.546e-05 2.546e-05 2.546e-05 0.00% Castro::create_source_corrector() 10 2.283e-05 2.283e-05 2.283e-05 0.00% Castro::initMFs() 1 2.144e-05 2.144e-05 2.144e-05 0.00% DistributionMapping::Distribute() 56 1.506e-05 1.506e-05 1.506e-05 0.00% Amr::initSubcycle() 1 8.851e-06 8.851e-06 8.851e-06 0.00% AmrLevel::checkPointPost() 3 4.833e-06 4.833e-06 4.833e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.789e-06 4.789e-06 4.789e-06 0.00% Gravity::set_mass_offset() 11 4.435e-06 4.435e-06 4.435e-06 0.00% Castro::retry_advance_ctu() 10 4.22e-06 4.22e-06 4.22e-06 0.00% Castro::FluxRegCrseInit 10 3.663e-06 3.663e-06 3.663e-06 0.00% Castro::FluxRegFineAdd() 10 2.004e-06 2.004e-06 2.004e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.932e-06 1.932e-06 1.932e-06 0.00% AmrLevel::checkPointPre() 3 1.73e-06 1.73e-06 1.73e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 102 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 118 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1066 KiB 39 MiB Castro::initialize_do_advance() 80 80 25 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2156 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7610 KiB 14 MiB MLMG::prepareForSolve() 660 660 3599 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 209 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 179 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7526 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 11 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1865 B 2048 KiB Gravity::solve_for_phi() 80 80 568 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 96 KiB 2048 KiB BndryData::define() 1056 1056 328 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 209 KiB 671 KiB Castro::estTimeStep() 21 21 2786 B 480 KiB VisMF::Write(FabArray) 656 656 3449 B 320 KiB Castro::normalize_species() 62 62 5099 B 320 KiB amrex::average_down 1067 1067 1271 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1188 B 257 KiB amrex::Dot() 1360 1360 3458 B 160 KiB FabArray::norminf() 907 907 2410 B 160 KiB Castro::do_advance_ctu() 10 10 485 B 160 KiB MultiFab::max() 11 11 47 B 160 KiB Castro::initData() 1 1 27 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3519 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 310 B 9648 B MLCellLinOp::applyBC() 8866 8866 228 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3855 B 6144 B StateData::FillBoundary(geom) 1992 1992 51 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 120 B 1472 B MLCellLinOp::defineBC() 66 66 370 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 561 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 51 KiB 8192 KiB VisMF::Write(FabArray) 744 744 624 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3519 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 309 B 9648 B MLCellLinOp::applyBC() 4433 4433 226 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3855 B 6144 B Gravity::get_new_grav_vector() 3 3 2903 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B StateData::FillBoundary(geom) 1992 1992 52 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 299 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 907 907 17 B 272 B Castro::estTimeStep() 21 21 0 B 32 B MultiFab::contains_nan() 20 20 0 B 16 B MultiFab::max() 11 11 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B Castro::do_advance_ctu() 10 10 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-2-g6a7da9909378) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-2-g6a7da9909378) initialized Starting run at 10:10:52 UTC on 2023-03-06. Successfully read inputs file ... Castro git describe: 23.03-1-g11cc0c2de AMReX git describe: 23.03-2-g6a7da9909 Microphysics git describe: 23.03-7-g751f2f5f reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.46960449 Restart time = 0.095583901 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048366596 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048894866 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054660241 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056673694 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059174349 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032510646 seconds Ending run at 10:10:52 UTC on 2023-03-06. Run time = 0.39686231 Run time without initialization = 0.300692998 Average number of zones advanced per microsecond: 4.359 Average number of zones advanced per microsecond per rank: 4.359 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3969 ... 0.3969 ... 0.3969 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0936 0.0936 0.0936 23.58% Amr::restart() 1 0.05038 0.05038 0.05038 12.69% VisMF::Read() 3 0.04148 0.04148 0.04148 10.45% MLCellLinOp::applyBC() 1946 0.03241 0.03241 0.03241 8.17% VisMF::Write(FabArray) 1 0.03095 0.03095 0.03095 7.80% MLPoisson::Fsmooth() 1440 0.01357 0.01357 0.01357 3.42% FillBoundary_nowait() 1766 0.0135 0.0135 0.0135 3.40% StateData::FillBoundary(geom) 160 0.01189 0.01189 0.01189 3.00% amrex::Dot() 484 0.008486 0.008486 0.008486 2.14% amrex::Copy() 463 0.006768 0.006768 0.006768 1.71% Castro::normalize_species() 30 0.00636 0.00636 0.00636 1.60% FabArray::setVal() 537 0.006067 0.006067 0.006067 1.53% FabArray::norminf() 326 0.005989 0.005989 0.005989 1.51% Castro::computeTemp() 30 0.005943 0.005943 0.005943 1.50% FabArray::ParallelCopy_nowait() 380 0.00594 0.00594 0.00594 1.50% StateDataPhysBCFunct::() 20 0.005492 0.005492 0.005492 1.38% MLCellLinOp::defineAuxData() 6 0.005188 0.005188 0.005188 1.31% MLPoisson::Fapply() 500 0.004359 0.004359 0.004359 1.10% Castro::estTimeStep() 10 0.003676 0.003676 0.003676 0.93% FabArray::Saxpy() 355 0.003515 0.003515 0.003515 0.89% FabArray::Xpay() 361 0.003396 0.003396 0.003396 0.86% Gravity::fill_multipole_BCs() 6 0.003078 0.003078 0.003078 0.78% MLMG::addInterpCorrection() 180 0.002899 0.002899 0.002899 0.73% amrex::average_down 180 0.002526 0.002526 0.002526 0.64% Castro::enforce_min_density() 30 0.00227 0.00227 0.00227 0.57% BndryData::define() 6 0.001939 0.001939 0.001939 0.49% FabArray::LinComb() 242 0.001857 0.001857 0.001857 0.47% amrex::Add() 72 0.001845 0.001845 0.001845 0.46% Castro::reset_internal_energy(MultiFab) 30 0.001833 0.001833 0.001833 0.46% Castro::construct_new_gravity_source() 5 0.00163 0.00163 0.00163 0.41% Amr::writePlotFile() 1 0.00139 0.00139 0.00139 0.35% Castro::construct_old_gravity_source() 5 0.001228 0.001228 0.001228 0.31% Castro::do_advance_ctu() 5 0.001075 0.001075 0.001075 0.27% MLCGSolver::bicgstab 36 0.0009941 0.0009941 0.0009941 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000885 0.000885 0.000885 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007924 0.0007924 0.0007924 0.20% MLCellLinOp::setLevelBC() 6 0.0007307 0.0007307 0.0007307 0.18% Castro::reset_internal_energy(Fab) 240 0.0006827 0.0006827 0.0006827 0.17% FabArray::mult() 22 0.0006464 0.0006464 0.0006464 0.16% FabArray::setDomainBndry() 20 0.0006373 0.0006373 0.0006373 0.16% MLCellLinOp::prepareForSolve() 6 0.0005958 0.0005958 0.0005958 0.15% MultiFab::contains_nan() 10 0.0005865 0.0005865 0.0005865 0.15% Castro::enforce_speed_limit() 30 0.0005702 0.0005702 0.0005702 0.14% MLCellLinOp::smooth() 720 0.0005267 0.0005267 0.0005267 0.13% MLCellLinOp::compGrad() 6 0.0004809 0.0004809 0.0004809 0.12% MLMG::prepareForSolve() 6 0.0004768 0.0004768 0.0004768 0.12% Amr::InitAmr() 1 0.0004066 0.0004066 0.0004066 0.10% FabArrayBase::CPC::define() 244 0.000399 0.000399 0.000399 0.10% FabArray::FillBoundary() 1766 0.0003699 0.0003699 0.0003699 0.09% FabArrayBase::getCPC() 632 0.0003511 0.0003511 0.0003511 0.09% Gravity::get_old_grav_vector() 5 0.0002995 0.0002995 0.0002995 0.08% main() 1 0.0002795 0.0002795 0.0002795 0.07% Gravity::get_new_grav_vector() 5 0.0002706 0.0002706 0.0002706 0.07% FabArrayBase::getFB() 1766 0.000248 0.000248 0.000248 0.06% MLCellLinOp::apply() 500 0.000227 0.000227 0.000227 0.06% MLMG::mgVcycle() 36 0.0001741 0.0001741 0.0001741 0.04% Amr::coarseTimeStep() 5 0.000169 0.000169 0.000169 0.04% MLLinOp::defineGrids() 6 0.0001672 0.0001672 0.0001672 0.04% MLCGSolver::ParallelAllReduce 659 0.0001531 0.0001531 0.0001531 0.04% MultiFab::max() 6 0.0001357 0.0001357 0.0001357 0.03% FabArray::ParallelCopy() 380 0.0001187 0.0001187 0.0001187 0.03% FillPatchIterator::Initialize 20 0.0001183 0.0001183 0.0001183 0.03% MLCellLinOp::defineBC() 6 0.0001102 0.0001102 0.0001102 0.03% MLCellLinOp::correctionResidual() 216 9.598e-05 9.598e-05 9.598e-05 0.02% Amr::timeStep() 5 8.578e-05 8.578e-05 8.578e-05 0.02% AmrLevel::restart() 1 7.575e-05 7.575e-05 7.575e-05 0.02% StateData::restartDoit() 4 7.485e-05 7.485e-05 7.485e-05 0.02% Castro::subcycle_advance_ctu() 5 6.906e-05 6.906e-05 6.906e-05 0.02% Gravity::solve_for_phi() 5 6.48e-05 6.48e-05 6.48e-05 0.02% FabArrayBase::FB::FB() 26 6.34e-05 6.34e-05 6.34e-05 0.02% Gravity::update_max_rhs() 6 6.125e-05 6.125e-05 6.125e-05 0.02% MLMG:computeResOfCorrection() 180 4.937e-05 4.937e-05 4.937e-05 0.01% Castro::advance() 5 4.901e-05 4.901e-05 4.901e-05 0.01% Castro::finalize_advance() 5 4.564e-05 4.564e-05 4.564e-05 0.01% Castro::expand_state() 5 4.203e-05 4.203e-05 4.203e-05 0.01% MLMG::actualBottomSolve() 36 4.012e-05 4.012e-05 4.012e-05 0.01% MLMG::mgVcycle_down::0 36 3.987e-05 3.987e-05 3.987e-05 0.01% Castro::clean_state() 30 3.929e-05 3.929e-05 3.929e-05 0.01% MLMG::mgVcycle_down::1 36 3.794e-05 3.794e-05 3.794e-05 0.01% Castro::initialize_advance() 5 3.585e-05 3.585e-05 3.585e-05 0.01% MLMG::solve() 6 3.555e-05 3.555e-05 3.555e-05 0.01% MLMG::mgVcycle_down::2 36 3.426e-05 3.426e-05 3.426e-05 0.01% MLMG::mgVcycle_down::4 36 3.398e-05 3.398e-05 3.398e-05 0.01% MLMG::mgVcycle_down::3 36 3.296e-05 3.296e-05 3.296e-05 0.01% MLMG::mgVcycle_up::4 36 3.035e-05 3.035e-05 3.035e-05 0.01% Castro::initialize_do_advance() 5 2.687e-05 2.687e-05 2.687e-05 0.01% Amr::writeSmallPlotFile() 1 2.648e-05 2.648e-05 2.648e-05 0.01% Castro::swap_state_time_levels() 5 2.535e-05 2.535e-05 2.535e-05 0.01% Castro::buildMetrics() 1 2.524e-05 2.524e-05 2.524e-05 0.01% Castro::post_restart() 1 2.517e-05 2.517e-05 2.517e-05 0.01% MLMG::mgVcycle_up::0 36 2.498e-05 2.498e-05 2.498e-05 0.01% MLMG::mgVcycle_up::3 36 2.395e-05 2.395e-05 2.395e-05 0.01% Castro::initMFs() 1 2.353e-05 2.353e-05 2.353e-05 0.01% MLMG::oneIter() 36 2.328e-05 2.328e-05 2.328e-05 0.01% MLMG::mgVcycle_up::2 36 2.288e-05 2.288e-05 2.288e-05 0.01% MLCellLinOp::solutionResidual() 42 2.285e-05 2.285e-05 2.285e-05 0.01% MLMG::mgVcycle_up::1 36 2.21e-05 2.21e-05 2.21e-05 0.01% MLPoisson::define() 6 1.997e-05 1.997e-05 1.997e-05 0.01% Castro::finalize_do_advance() 5 1.955e-05 1.955e-05 1.955e-05 0.00% MLMG::ResNormInf() 42 1.683e-05 1.683e-05 1.683e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.606e-05 1.606e-05 1.606e-05 0.00% FillPatchSingleLevel 20 1.496e-05 1.496e-05 1.496e-05 0.00% makeSFC 30 1.418e-05 1.418e-05 1.418e-05 0.00% MLMG::computeResidual() 36 1.414e-05 1.414e-05 1.414e-05 0.00% MLMG::mgVcycle_bottom 36 1.39e-05 1.39e-05 1.39e-05 0.00% Castro::construct_new_gravity() 5 1.259e-05 1.259e-05 1.259e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.194e-05 1.194e-05 1.194e-05 0.00% MLLinOp::define() 6 1.095e-05 1.095e-05 1.095e-05 0.00% Castro::construct_old_source() 25 9.831e-06 9.831e-06 9.831e-06 0.00% Castro::construct_new_source() 25 9.522e-06 9.522e-06 9.522e-06 0.00% DistributionMapping::Distribute() 31 8.825e-06 8.825e-06 8.825e-06 0.00% Amr::initSubcycle() 1 8.629e-06 8.629e-06 8.629e-06 0.00% Castro::do_new_sources() 5 8.417e-06 8.417e-06 8.417e-06 0.00% Castro::do_old_sources() 5 8.226e-06 8.226e-06 8.226e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.863e-06 7.863e-06 7.863e-06 0.00% Gravity::actual_multilevel_solve() 1 7.583e-06 7.583e-06 7.583e-06 0.00% Castro::check_for_nan() 10 5.997e-06 5.997e-06 5.997e-06 0.00% Castro::construct_old_gravity() 5 5.669e-06 5.669e-06 5.669e-06 0.00% Castro::post_timestep() 5 5.074e-06 5.074e-06 5.074e-06 0.00% Castro::apply_source_to_state() 10 5.048e-06 5.048e-06 5.048e-06 0.00% Gravity::swapTimeLevels() 5 4.522e-06 4.522e-06 4.522e-06 0.00% MLPoisson::prepareForSolve() 6 4.208e-06 4.208e-06 4.208e-06 0.00% MLMG::computeMLResidual() 6 4.01e-06 4.01e-06 4.01e-06 0.00% Castro::computeNewDt() 5 3.339e-06 3.339e-06 3.339e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.168e-06 3.168e-06 3.168e-06 0.00% MLMG::getGradSolution() 6 3.104e-06 3.104e-06 3.104e-06 0.00% Gravity::set_mass_offset() 6 2.394e-06 2.394e-06 2.394e-06 0.00% MLMG::MLResNormInf() 6 2.303e-06 2.303e-06 2.303e-06 0.00% Castro::create_source_corrector() 5 2.075e-06 2.075e-06 2.075e-06 0.00% MLMG::MLRhsNormInf() 6 1.939e-06 1.939e-06 1.939e-06 0.00% Castro::retry_advance_ctu() 5 1.803e-06 1.803e-06 1.803e-06 0.00% Castro::FluxRegCrseInit 5 1.365e-06 1.365e-06 1.365e-06 0.00% Castro::FluxRegFineAdd() 5 1.304e-06 1.304e-06 1.304e-06 0.00% AmrLevel::AmrLevel() 1 1.261e-06 1.261e-06 1.261e-06 0.00% Amr::init() 1 1.175e-06 1.175e-06 1.175e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.094e-06 1.094e-06 1.094e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3969 0.3969 0.3969 100.00% Amr::coarseTimeStep() 5 0.2679 0.2679 0.2679 67.51% Amr::timeStep() 5 0.2648 0.2648 0.2648 66.72% Castro::advance() 5 0.2624 0.2624 0.2624 66.12% Castro::subcycle_advance_ctu() 5 0.2563 0.2563 0.2563 64.59% Castro::do_advance_ctu() 5 0.2563 0.2563 0.2563 64.57% Castro::construct_new_gravity() 5 0.1275 0.1275 0.1275 32.12% Gravity::solve_phi_with_mlmg() 6 0.1228 0.1228 0.1228 30.95% Gravity::solve_for_phi() 5 0.1198 0.1198 0.1198 30.19% Gravity::actual_solve_with_mlmg() 6 0.1195 0.1195 0.1195 30.11% MLMG::solve() 6 0.1085 0.1085 0.1085 27.35% MLMG::oneIter() 36 0.1018 0.1018 0.1018 25.66% MLMG::mgVcycle() 36 0.1003 0.1003 0.1003 25.27% Amr::init() 1 0.09563 0.09563 0.09563 24.09% Amr::restart() 1 0.09562 0.09562 0.09562 24.09% Castro::construct_ctu_hydro_source() 5 0.0936 0.0936 0.0936 23.58% MLCellLinOp::smooth() 720 0.04967 0.04967 0.04967 12.51% MLCellLinOp::applyBC() 1946 0.0466 0.0466 0.0466 11.74% AmrLevel::restart() 1 0.04169 0.04169 0.04169 10.50% StateData::restartDoit() 4 0.04161 0.04161 0.04161 10.48% VisMF::Read() 3 0.04148 0.04148 0.04148 10.45% Amr::writePlotFile() 1 0.03259 0.03259 0.03259 8.21% VisMF::Write(FabArray) 1 0.03095 0.03095 0.03095 7.80% MLMG::mgVcycle_bottom 36 0.0306 0.0306 0.0306 7.71% MLMG::actualBottomSolve() 36 0.03058 0.03058 0.03058 7.71% MLCGSolver::bicgstab 36 0.03029 0.03029 0.03029 7.63% FillPatchIterator::Initialize 20 0.02018 0.02018 0.02018 5.08% FillPatchSingleLevel 20 0.01942 0.01942 0.01942 4.89% Castro::clean_state() 30 0.0177 0.0177 0.0177 4.46% StateDataPhysBCFunct::() 20 0.01739 0.01739 0.01739 4.38% MLCellLinOp::apply() 500 0.01535 0.01535 0.01535 3.87% MLMG::mgVcycle_down::0 36 0.01452 0.01452 0.01452 3.66% FabArray::FillBoundary() 1766 0.01418 0.01418 0.01418 3.57% FillBoundary_nowait() 1766 0.01381 0.01381 0.01381 3.48% MLPoisson::Fsmooth() 1440 0.01357 0.01357 0.01357 3.42% StateData::FillBoundary(geom) 160 0.01189 0.01189 0.01189 3.00% MLMG::mgVcycle_up::0 36 0.01108 0.01108 0.01108 2.79% MLCellLinOp::correctionResidual() 216 0.009326 0.009326 0.009326 2.35% Castro::initialize_do_advance() 5 0.009085 0.009085 0.009085 2.29% MLPoisson::define() 6 0.008722 0.008722 0.008722 2.20% amrex::Dot() 484 0.008486 0.008486 0.008486 2.14% Castro::computeTemp() 30 0.008459 0.008459 0.008459 2.13% MLMG:computeResOfCorrection() 180 0.008196 0.008196 0.008196 2.07% Castro::construct_old_gravity() 5 0.007549 0.007549 0.007549 1.90% Gravity::get_old_grav_vector() 5 0.007543 0.007543 0.007543 1.90% Gravity::get_new_grav_vector() 5 0.007541 0.007541 0.007541 1.90% amrex::Copy() 463 0.006768 0.006768 0.006768 1.71% MLMG::mgVcycle_down::1 36 0.006644 0.006644 0.006644 1.67% Castro::do_new_sources() 5 0.006561 0.006561 0.006561 1.65% FabArray::ParallelCopy() 380 0.006438 0.006438 0.006438 1.62% Castro::normalize_species() 30 0.00636 0.00636 0.00636 1.60% FabArray::ParallelCopy_nowait() 380 0.00632 0.00632 0.00632 1.59% MLMG::mgVcycle_down::2 36 0.006192 0.006192 0.006192 1.56% FabArray::setVal() 537 0.006067 0.006067 0.006067 1.53% MLMG::mgVcycle_down::3 36 0.006017 0.006017 0.006017 1.52% FabArray::norminf() 326 0.005989 0.005989 0.005989 1.51% MLMG::mgVcycle_down::4 36 0.005972 0.005972 0.005972 1.50% MLCellLinOp::defineAuxData() 6 0.005913 0.005913 0.005913 1.49% Castro::expand_state() 5 0.005764 0.005764 0.005764 1.45% Castro::initialize_advance() 5 0.005711 0.005711 0.005711 1.44% MLCGSolver::ParallelAllReduce 659 0.005157 0.005157 0.005157 1.30% MLMG::addInterpCorrection() 180 0.005123 0.005123 0.005123 1.29% MLMG::mgVcycle_up::4 36 0.004876 0.004876 0.004876 1.23% MLMG::mgVcycle_up::1 36 0.004835 0.004835 0.004835 1.22% MLMG::mgVcycle_up::2 36 0.004729 0.004729 0.004729 1.19% amrex::average_down 180 0.004718 0.004718 0.004718 1.19% MLMG::mgVcycle_up::3 36 0.004641 0.004641 0.004641 1.17% MLPoisson::Fapply() 500 0.004359 0.004359 0.004359 1.10% Castro::do_old_sources() 5 0.00398 0.00398 0.00398 1.00% Castro::estTimeStep() 10 0.003676 0.003676 0.003676 0.93% FabArray::Saxpy() 355 0.003515 0.003515 0.003515 0.89% FabArray::Xpay() 361 0.003396 0.003396 0.003396 0.86% Castro::post_restart() 1 0.003378 0.003378 0.003378 0.85% Gravity::multilevel_solve_for_new_phi() 1 0.003262 0.003262 0.003262 0.82% Gravity::actual_multilevel_solve() 1 0.003246 0.003246 0.003246 0.82% Gravity::fill_multipole_BCs() 6 0.0032 0.0032 0.0032 0.81% MLCellLinOp::solutionResidual() 42 0.003174 0.003174 0.003174 0.80% Castro::computeNewDt() 5 0.00294 0.00294 0.00294 0.74% MLMG::computeResidual() 36 0.002638 0.002638 0.002638 0.66% MLCellLinOp::defineBC() 6 0.00258 0.00258 0.00258 0.65% Castro::reset_internal_energy(MultiFab) 30 0.002516 0.002516 0.002516 0.63% BndryData::define() 6 0.00247 0.00247 0.00247 0.62% MLMG::prepareForSolve() 6 0.002451 0.002451 0.002451 0.62% Castro::post_timestep() 5 0.002329 0.002329 0.002329 0.59% Castro::enforce_min_density() 30 0.00227 0.00227 0.00227 0.57% FabArray::LinComb() 242 0.001857 0.001857 0.001857 0.47% amrex::Add() 72 0.001845 0.001845 0.001845 0.46% Castro::construct_new_source() 25 0.001639 0.001639 0.001639 0.41% Castro::construct_new_gravity_source() 5 0.00163 0.00163 0.00163 0.41% Castro::construct_old_source() 25 0.001238 0.001238 0.001238 0.31% Castro::construct_old_gravity_source() 5 0.001228 0.001228 0.001228 0.31% Castro::apply_source_to_state() 10 0.0009152 0.0009152 0.0009152 0.23% MLMG::ResNormInf() 42 0.0009069 0.0009069 0.0009069 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000885 0.000885 0.000885 0.22% FabArrayBase::getCPC() 632 0.0007501 0.0007501 0.0007501 0.19% MLMG::getGradSolution() 6 0.0007403 0.0007403 0.0007403 0.19% MLCellLinOp::compGrad() 6 0.0007372 0.0007372 0.0007372 0.19% MLCellLinOp::setLevelBC() 6 0.0007307 0.0007307 0.0007307 0.18% Castro::reset_internal_energy(Fab) 240 0.0006827 0.0006827 0.0006827 0.17% FabArray::mult() 22 0.0006464 0.0006464 0.0006464 0.16% FabArray::setDomainBndry() 20 0.0006373 0.0006373 0.0006373 0.16% MLPoisson::prepareForSolve() 6 0.0006 0.0006 0.0006 0.15% MLCellLinOp::prepareForSolve() 6 0.0005958 0.0005958 0.0005958 0.15% Castro::check_for_nan() 10 0.0005925 0.0005925 0.0005925 0.15% MultiFab::contains_nan() 10 0.0005865 0.0005865 0.0005865 0.15% Castro::enforce_speed_limit() 30 0.0005702 0.0005702 0.0005702 0.14% MLMG::computeMLResidual() 6 0.0005542 0.0005542 0.0005542 0.14% Gravity::update_max_rhs() 6 0.0004446 0.0004446 0.0004446 0.11% Amr::InitAmr() 1 0.0004153 0.0004153 0.0004153 0.10% FabArrayBase::CPC::define() 244 0.000399 0.000399 0.000399 0.10% FabArrayBase::getFB() 1766 0.0003114 0.0003114 0.0003114 0.08% Castro::finalize_advance() 5 0.0003017 0.0003017 0.0003017 0.08% Gravity::swapTimeLevels() 5 0.0002223 0.0002223 0.0002223 0.06% MLLinOp::define() 6 0.0002088 0.0002088 0.0002088 0.05% MLLinOp::defineGrids() 6 0.0001978 0.0001978 0.0001978 0.05% Castro::buildMetrics() 1 0.0001505 0.0001505 0.0001505 0.04% MLMG::MLResNormInf() 6 0.0001498 0.0001498 0.0001498 0.04% MultiFab::max() 6 0.0001357 0.0001357 0.0001357 0.03% MLMG::MLRhsNormInf() 6 0.0001144 0.0001144 0.0001144 0.03% FabArrayBase::FB::FB() 26 6.34e-05 6.34e-05 6.34e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.95e-05 2.95e-05 2.95e-05 0.01% Amr::writeSmallPlotFile() 1 2.648e-05 2.648e-05 2.648e-05 0.01% Castro::swap_state_time_levels() 5 2.535e-05 2.535e-05 2.535e-05 0.01% Castro::initMFs() 1 2.353e-05 2.353e-05 2.353e-05 0.01% makeSFC 30 2.163e-05 2.163e-05 2.163e-05 0.01% Castro::finalize_do_advance() 5 1.955e-05 1.955e-05 1.955e-05 0.00% DistributionMapping::Distribute() 31 8.825e-06 8.825e-06 8.825e-06 0.00% Amr::initSubcycle() 1 8.629e-06 8.629e-06 8.629e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.543e-06 4.543e-06 4.543e-06 0.00% Gravity::set_mass_offset() 6 2.394e-06 2.394e-06 2.394e-06 0.00% Castro::create_source_corrector() 5 2.075e-06 2.075e-06 2.075e-06 0.00% Castro::retry_advance_ctu() 5 1.803e-06 1.803e-06 1.803e-06 0.00% Castro::FluxRegCrseInit 5 1.365e-06 1.365e-06 1.365e-06 0.00% Castro::FluxRegFineAdd() 5 1.304e-06 1.304e-06 1.304e-06 0.00% AmrLevel::AmrLevel() 1 1.261e-06 1.261e-06 1.261e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.094e-06 1.094e-06 1.094e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 219 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 107 MiB 692 MiB Castro::initMFs() 48 48 52 MiB 68 MiB StateData::restartDoit() 32 32 46 MiB 55 MiB Castro::swap_state_time_levels() 32 32 42 MiB 55 MiB Castro::initialize_do_advance() 40 40 25 MiB 39 MiB FillPatchIterator::Initialize 160 160 1110 KiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2298 KiB 28 MiB Castro::initialize_advance() 40 40 15 MiB 23 MiB Castro::buildMetrics() 32 32 12 MiB 15 MiB Castro::post_restart() 48 48 5802 KiB 14 MiB MLMG::prepareForSolve() 361 361 3358 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 196 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 194 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 5798 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 11 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2165 B 2048 KiB Gravity::solve_for_phi() 40 40 616 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 16 KiB 2048 KiB BndryData::define() 576 576 308 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 198 KiB 671 KiB Castro::estTimeStep() 10 10 2519 B 480 KiB VisMF::Write(FabArray) 112 112 1291 B 320 KiB Castro::normalize_species() 30 30 5206 B 320 KiB amrex::average_down 469 469 1138 B 257 KiB MLMG::addInterpCorrection() 468 468 1117 B 257 KiB amrex::Dot() 592 592 3144 B 160 KiB FabArray::norminf() 398 398 2225 B 160 KiB Castro::do_advance_ctu() 5 5 407 B 160 KiB MultiFab::max() 6 6 52 B 160 KiB FabArray::setVal() 66 66 16 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3268 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 297 B 9648 B MLCellLinOp::applyBC() 3892 3892 212 B 9344 B amrex::Copy() 56 56 5188 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 47 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 324 324 110 B 1472 B MLCellLinOp::defineBC() 36 36 347 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1229 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 111 KiB 8192 KiB VisMF::Write(FabArray) 120 120 238 KiB 3584 KiB VisMF::Read() 24 24 194 KiB 3000 KiB FabArray::setVal() 66 66 16 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3268 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 297 B 9648 B MLCellLinOp::applyBC() 1946 1946 211 B 9328 B amrex::Copy() 56 56 5188 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2283 B 3072 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B StateData::FillBoundary(geom) 960 960 48 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 486 B 1296 B amrex::average_down 37 37 192 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 398 398 16 B 272 B Castro::estTimeStep() 10 10 0 B 32 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::do_advance_ctu() 5 5 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-2-g6a7da9909378) finalized