Initializing CUDA... CUDA initialized with 1 device. AMReX (23.05-27-gd65dd906bd53) initialized Starting run at 09:01:59 UTC on 2023-05-24. Successfully read inputs file ... Castro git describe: 23.05-25-g99ae92958 AMReX git describe: 23.05-27-gd65dd906b Microphysics git describe: 23.05-12-g1f8fd85f reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.059323752 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.034118421 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04684588 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051096526 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048412349 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059135487 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.068393157 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.058096095 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.076350532 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051676466 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057634445 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058306262 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069452322 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057970381 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033737924 seconds Ending run at 09:01:59 UTC on 2023-05-24. Run time = 0.876627814 Run time without initialization = 0.737758479 Average number of zones advanced per microsecond: 3.553 Average number of zones advanced per microsecond per rank: 3.553 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8767 ... 0.8767 ... 0.8767 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2363 0.2363 0.2363 26.96% Castro::construct_ctu_hydro_source() 10 0.222 0.222 0.222 25.32% MLCellLinOp::applyBC() 4379 0.07608 0.07608 0.07608 8.68% MLPoisson::Fsmooth() 3240 0.03225 0.03225 0.03225 3.68% FillBoundary_nowait() 3974 0.03103 0.03103 0.03103 3.54% StateData::FillBoundary(geom) 328 0.02544 0.02544 0.02544 2.90% amrex::Dot() 1100 0.02045 0.02045 0.02045 2.33% amrex::Copy() 1019 0.01475 0.01475 0.01475 1.68% Castro::normalize_species() 62 0.01475 0.01475 0.01475 1.68% FabArray::norminf() 734 0.01441 0.01441 0.01441 1.64% Castro::computeTemp() 63 0.01427 0.01427 0.01427 1.63% Castro::enforce_min_density() 62 0.01381 0.01381 0.01381 1.57% FabArray::ParallelCopy_nowait() 851 0.01322 0.01322 0.01322 1.51% FabArray::setVal() 1135 0.01314 0.01314 0.01314 1.50% StateDataPhysBCFunct::() 41 0.01129 0.01129 0.01129 1.29% MLPoisson::Fapply() 1128 0.01027 0.01027 0.01027 1.17% MLCellLinOp::defineAuxData() 11 0.009748 0.009748 0.009748 1.11% FabArray::Saxpy() 803 0.008145 0.008145 0.008145 0.93% FabArray::Xpay() 811 0.008005 0.008005 0.008005 0.91% Gravity::fill_multipole_BCs() 11 0.006568 0.006568 0.006568 0.75% MLMG::addInterpCorrection() 405 0.006551 0.006551 0.006551 0.75% amrex::average_down 405 0.005891 0.005891 0.005891 0.67% Castro::estTimeStep() 21 0.005861 0.005861 0.005861 0.67% FabArray::LinComb() 550 0.004516 0.004516 0.004516 0.52% amrex::Add() 162 0.00431 0.00431 0.00431 0.49% Amr::checkPoint() 3 0.004078 0.004078 0.004078 0.47% Castro::reset_internal_energy(MultiFab) 63 0.004026 0.004026 0.004026 0.46% BndryData::define() 11 0.003751 0.003751 0.003751 0.43% Castro::do_advance_ctu() 10 0.003657 0.003657 0.003657 0.42% Castro::construct_new_gravity_source() 10 0.003225 0.003225 0.003225 0.37% Castro::construct_old_gravity_source() 10 0.002733 0.002733 0.002733 0.31% Amr::writePlotFile() 2 0.002368 0.002368 0.002368 0.27% MLCGSolver::bicgstab 81 0.002114 0.002114 0.002114 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001695 0.001695 0.001695 0.19% Castro::reset_internal_energy(Fab) 504 0.001558 0.001558 0.001558 0.18% Gravity::actual_solve_with_mlmg() 11 0.00147 0.00147 0.00147 0.17% MLCellLinOp::setLevelBC() 11 0.001402 0.001402 0.001402 0.16% FabArray::mult() 43 0.001328 0.001328 0.001328 0.15% Castro::initData() 1 0.001321 0.001321 0.001321 0.15% FabArray::setDomainBndry() 41 0.001296 0.001296 0.001296 0.15% MultiFab::contains_nan() 20 0.0012 0.0012 0.0012 0.14% Castro::enforce_speed_limit() 62 0.001149 0.001149 0.001149 0.13% MLCellLinOp::prepareForSolve() 11 0.001147 0.001147 0.001147 0.13% MLCellLinOp::smooth() 1620 0.001045 0.001045 0.001045 0.12% MLCellLinOp::compGrad() 11 0.0009165 0.0009165 0.0009165 0.10% FabArray::FillBoundary() 3974 0.00088 0.00088 0.00088 0.10% MLMG::prepareForSolve() 11 0.0008599 0.0008599 0.0008599 0.10% FabArrayBase::getCPC() 1313 0.0007102 0.0007102 0.0007102 0.08% FabArrayBase::CPC::define() 454 0.0006642 0.0006642 0.0006642 0.08% Gravity::get_new_grav_vector() 11 0.0006264 0.0006264 0.0006264 0.07% FabArrayBase::getFB() 3974 0.0006142 0.0006142 0.0006142 0.07% Gravity::get_old_grav_vector() 10 0.0005478 0.0005478 0.0005478 0.06% Amr::InitAmr() 1 0.0004802 0.0004802 0.0004802 0.05% MLCellLinOp::apply() 1128 0.0004334 0.0004334 0.0004334 0.05% main() 1 0.0003825 0.0003825 0.0003825 0.04% MLLinOp::defineGrids() 11 0.0003737 0.0003737 0.0003737 0.04% Amr::coarseTimeStep() 10 0.0003408 0.0003408 0.0003408 0.04% MLMG::mgVcycle() 81 0.0003389 0.0003389 0.0003389 0.04% MLCGSolver::ParallelAllReduce 1495 0.0002799 0.0002799 0.0002799 0.03% MultiFab::max() 11 0.0002669 0.0002669 0.0002669 0.03% Castro::create_source_corrector() 10 0.0002421 0.0002421 0.0002421 0.03% FabArray::ParallelCopy() 851 0.0002331 0.0002331 0.0002331 0.03% FillPatchIterator::Initialize 41 0.0002281 0.0002281 0.0002281 0.03% MLCellLinOp::defineBC() 11 0.0002243 0.0002243 0.0002243 0.03% MLCellLinOp::correctionResidual() 486 0.0002114 0.0002114 0.0002114 0.02% Amr::timeStep() 10 0.0001599 0.0001599 0.0001599 0.02% Castro::subcycle_advance_ctu() 10 0.0001584 0.0001584 0.0001584 0.02% Gravity::update_max_rhs() 11 0.0001355 0.0001355 0.0001355 0.02% StateData::checkPoint() 12 0.000131 0.000131 0.000131 0.01% MLMG:computeResOfCorrection() 405 0.0001143 0.0001143 0.0001143 0.01% Gravity::solve_for_phi() 10 0.0001045 0.0001045 0.0001045 0.01% Castro::initialize_advance() 10 9.355e-05 9.355e-05 9.355e-05 0.01% Castro::advance() 10 9.13e-05 9.13e-05 9.13e-05 0.01% FabArrayBase::FB::FB() 56 8.567e-05 8.567e-05 8.567e-05 0.01% Castro::Castro() 1 8.453e-05 8.453e-05 8.453e-05 0.01% MLMG::actualBottomSolve() 81 8.34e-05 8.34e-05 8.34e-05 0.01% MLMG::mgVcycle_down::0 81 7.938e-05 7.938e-05 7.938e-05 0.01% Castro::expand_state() 10 7.731e-05 7.731e-05 7.731e-05 0.01% AmrLevel::checkPoint() 3 7.506e-05 7.506e-05 7.506e-05 0.01% MLMG::mgVcycle_down::1 81 7.156e-05 7.156e-05 7.156e-05 0.01% MLMG::mgVcycle_down::2 81 7.145e-05 7.145e-05 7.145e-05 0.01% Castro::clean_state() 62 6.99e-05 6.99e-05 6.99e-05 0.01% MLMG::solve() 11 6.882e-05 6.882e-05 6.882e-05 0.01% MLMG::mgVcycle_down::3 81 6.843e-05 6.843e-05 6.843e-05 0.01% MLMG::mgVcycle_down::4 81 6.787e-05 6.787e-05 6.787e-05 0.01% Castro::finalize_advance() 10 5.922e-05 5.922e-05 5.922e-05 0.01% MLMG::mgVcycle_up::4 81 5.613e-05 5.613e-05 5.613e-05 0.01% MLMG::oneIter() 81 5.501e-05 5.501e-05 5.501e-05 0.01% Castro::initialize_do_advance() 10 5.114e-05 5.114e-05 5.114e-05 0.01% MLMG::mgVcycle_up::1 81 5.058e-05 5.058e-05 5.058e-05 0.01% MLCellLinOp::solutionResidual() 92 5.021e-05 5.021e-05 5.021e-05 0.01% MLMG::mgVcycle_up::3 81 4.967e-05 4.967e-05 4.967e-05 0.01% MLMG::mgVcycle_up::2 81 4.939e-05 4.939e-05 4.939e-05 0.01% MLMG::mgVcycle_up::0 81 4.867e-05 4.867e-05 4.867e-05 0.01% Castro::construct_new_source() 50 4.578e-05 4.578e-05 4.578e-05 0.01% Castro::swap_state_time_levels() 10 3.895e-05 3.895e-05 3.895e-05 0.00% Castro::finalize_do_advance() 10 3.62e-05 3.62e-05 3.62e-05 0.00% Castro::enforce_consistent_e() 1 3.422e-05 3.422e-05 3.422e-05 0.00% StateData::define() 4 3.35e-05 3.35e-05 3.35e-05 0.00% MLMG::ResNormInf() 92 3.342e-05 3.342e-05 3.342e-05 0.00% MLMG::mgVcycle_bottom 81 3.32e-05 3.32e-05 3.32e-05 0.00% MLMG::computeResidual() 81 3.113e-05 3.113e-05 3.113e-05 0.00% Amr::writeSmallPlotFile() 1 3.015e-05 3.015e-05 3.015e-05 0.00% FillPatchSingleLevel 41 2.731e-05 2.731e-05 2.731e-05 0.00% MLPoisson::define() 11 2.438e-05 2.438e-05 2.438e-05 0.00% Castro::construct_new_gravity() 10 2.399e-05 2.399e-05 2.399e-05 0.00% makeSFC 55 2.342e-05 2.342e-05 2.342e-05 0.00% Castro::initMFs() 1 2.31e-05 2.31e-05 2.31e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.152e-05 2.152e-05 2.152e-05 0.00% Amr::FinalizeInit() 1 1.975e-05 1.975e-05 1.975e-05 0.00% Castro::construct_old_source() 50 1.839e-05 1.839e-05 1.839e-05 0.00% Amr::defBaseLevel() 1 1.805e-05 1.805e-05 1.805e-05 0.00% Castro::buildMetrics() 1 1.755e-05 1.755e-05 1.755e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.678e-05 1.678e-05 1.678e-05 0.00% Castro::do_new_sources() 10 1.557e-05 1.557e-05 1.557e-05 0.00% MLLinOp::define() 11 1.474e-05 1.474e-05 1.474e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.46e-05 1.46e-05 1.46e-05 0.00% Castro::do_old_sources() 10 1.458e-05 1.458e-05 1.458e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.42e-05 1.42e-05 1.42e-05 0.00% DistributionMapping::Distribute() 56 1.42e-05 1.42e-05 1.42e-05 0.00% Castro::apply_source_to_state() 20 1.096e-05 1.096e-05 1.096e-05 0.00% Castro::check_for_nan() 20 1.093e-05 1.093e-05 1.093e-05 0.00% MLPoisson::prepareForSolve() 11 9.228e-06 9.228e-06 9.228e-06 0.00% Gravity::swapTimeLevels() 10 9.023e-06 9.023e-06 9.023e-06 0.00% Castro::construct_old_gravity() 10 9.008e-06 9.008e-06 9.008e-06 0.00% Gravity::actual_multilevel_solve() 1 8.591e-06 8.591e-06 8.591e-06 0.00% Amr::initSubcycle() 1 8.166e-06 8.166e-06 8.166e-06 0.00% MLMG::computeMLResidual() 11 7.622e-06 7.622e-06 7.622e-06 0.00% Castro::post_timestep() 10 7.193e-06 7.193e-06 7.193e-06 0.00% Castro::computeNewDt() 9 7.145e-06 7.145e-06 7.145e-06 0.00% MLMG::getGradSolution() 11 6.076e-06 6.076e-06 6.076e-06 0.00% AmrLevel::checkPointPost() 3 4.815e-06 4.815e-06 4.815e-06 0.00% Amr::InitializeInit() 1 4.486e-06 4.486e-06 4.486e-06 0.00% Castro::post_init() 1 3.901e-06 3.901e-06 3.901e-06 0.00% Gravity::set_mass_offset() 11 3.847e-06 3.847e-06 3.847e-06 0.00% MLMG::MLRhsNormInf() 11 3.602e-06 3.602e-06 3.602e-06 0.00% Castro::retry_advance_ctu() 10 3.499e-06 3.499e-06 3.499e-06 0.00% Amr::init() 1 3.435e-06 3.435e-06 3.435e-06 0.00% MLMG::MLResNormInf() 11 3.4e-06 3.4e-06 3.4e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.065e-06 3.065e-06 3.065e-06 0.00% Castro::computeInitialDt() 2 2.752e-06 2.752e-06 2.752e-06 0.00% Castro::FluxRegCrseInit 10 2.697e-06 2.697e-06 2.697e-06 0.00% AmrLevel::checkPointPre() 3 1.957e-06 1.957e-06 1.957e-06 0.00% Castro::FluxRegFineAdd() 10 1.906e-06 1.906e-06 1.906e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.795e-06 1.795e-06 1.795e-06 0.00% Amr::initialInit() 1 1.42e-06 1.42e-06 1.42e-06 0.00% Castro::post_regrid() 1 9.75e-07 9.75e-07 9.75e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8766 0.8766 0.8766 100.00% Amr::coarseTimeStep() 10 0.7038 0.7038 0.7038 80.28% Amr::timeStep() 10 0.5832 0.5832 0.5832 66.53% Castro::advance() 10 0.5757 0.5757 0.5757 65.67% Castro::subcycle_advance_ctu() 10 0.5627 0.5627 0.5627 64.19% Castro::do_advance_ctu() 10 0.5625 0.5625 0.5625 64.17% Gravity::solve_phi_with_mlmg() 11 0.2817 0.2817 0.2817 32.13% Gravity::actual_solve_with_mlmg() 11 0.2747 0.2747 0.2747 31.33% Castro::construct_new_gravity() 10 0.2604 0.2604 0.2604 29.70% MLMG::solve() 11 0.2539 0.2539 0.2539 28.96% Gravity::solve_for_phi() 10 0.2447 0.2447 0.2447 27.91% MLMG::oneIter() 81 0.2397 0.2397 0.2397 27.34% VisMF::Write(FabArray) 11 0.2363 0.2363 0.2363 26.96% MLMG::mgVcycle() 81 0.2361 0.2361 0.2361 26.93% Castro::construct_ctu_hydro_source() 10 0.222 0.222 0.222 25.32% Amr::checkPoint() 3 0.1755 0.1755 0.1755 20.02% AmrLevel::checkPoint() 3 0.1714 0.1714 0.1714 19.56% StateData::checkPoint() 12 0.1714 0.1714 0.1714 19.55% Amr::init() 1 0.1382 0.1382 0.1382 15.76% MLCellLinOp::smooth() 1620 0.1163 0.1163 0.1163 13.27% MLCellLinOp::applyBC() 4379 0.1087 0.1087 0.1087 12.40% MLMG::mgVcycle_bottom 81 0.0732 0.0732 0.0732 8.35% MLMG::actualBottomSolve() 81 0.07317 0.07317 0.07317 8.35% MLCGSolver::bicgstab 81 0.07247 0.07247 0.07247 8.27% Amr::writePlotFile() 2 0.06798 0.06798 0.06798 7.75% Castro::clean_state() 62 0.04901 0.04901 0.04901 5.59% Amr::initialInit() 1 0.04459 0.04459 0.04459 5.09% FillPatchIterator::Initialize 41 0.04232 0.04232 0.04232 4.83% FillPatchSingleLevel 41 0.0408 0.0408 0.0408 4.65% Amr::FinalizeInit() 1 0.04065 0.04065 0.04065 4.64% Castro::post_init() 1 0.03931 0.03931 0.03931 4.48% Gravity::multilevel_solve_for_new_phi() 1 0.03742 0.03742 0.03742 4.27% Gravity::actual_multilevel_solve() 1 0.0374 0.0374 0.0374 4.27% StateDataPhysBCFunct::() 41 0.03673 0.03673 0.03673 4.19% MLCellLinOp::apply() 1128 0.03588 0.03588 0.03588 4.09% MLMG::mgVcycle_down::0 81 0.03375 0.03375 0.03375 3.85% FabArray::FillBoundary() 3974 0.03261 0.03261 0.03261 3.72% MLPoisson::Fsmooth() 3240 0.03225 0.03225 0.03225 3.68% FillBoundary_nowait() 3974 0.03173 0.03173 0.03173 3.62% StateData::FillBoundary(geom) 328 0.02544 0.02544 0.02544 2.90% MLMG::mgVcycle_up::0 81 0.02537 0.02537 0.02537 2.89% MLCellLinOp::correctionResidual() 486 0.02202 0.02202 0.02202 2.51% amrex::Dot() 1100 0.02045 0.02045 0.02045 2.33% Castro::computeTemp() 63 0.01986 0.01986 0.01986 2.27% Castro::initialize_do_advance() 10 0.01979 0.01979 0.01979 2.26% MLMG:computeResOfCorrection() 405 0.01938 0.01938 0.01938 2.21% Gravity::get_new_grav_vector() 11 0.0173 0.0173 0.0173 1.97% MLPoisson::define() 11 0.01647 0.01647 0.01647 1.88% MLMG::mgVcycle_down::1 81 0.01561 0.01561 0.01561 1.78% Castro::construct_old_gravity() 10 0.01526 0.01526 0.01526 1.74% Gravity::get_old_grav_vector() 10 0.01525 0.01525 0.01525 1.74% amrex::Copy() 1019 0.01475 0.01475 0.01475 1.68% Castro::normalize_species() 62 0.01475 0.01475 0.01475 1.68% MLMG::mgVcycle_down::2 81 0.01461 0.01461 0.01461 1.67% FabArray::norminf() 734 0.01441 0.01441 0.01441 1.64% MLMG::mgVcycle_down::3 81 0.01426 0.01426 0.01426 1.63% FabArray::ParallelCopy() 851 0.01424 0.01424 0.01424 1.62% MLMG::mgVcycle_down::4 81 0.01408 0.01408 0.01408 1.61% FabArray::ParallelCopy_nowait() 851 0.01401 0.01401 0.01401 1.60% Castro::enforce_min_density() 62 0.01381 0.01381 0.01381 1.57% FabArray::setVal() 1135 0.01314 0.01314 0.01314 1.50% Castro::do_new_sources() 10 0.01259 0.01259 0.01259 1.44% Castro::initialize_advance() 10 0.0123 0.0123 0.0123 1.40% MLCGSolver::ParallelAllReduce 1495 0.01224 0.01224 0.01224 1.40% MLMG::addInterpCorrection() 405 0.01163 0.01163 0.01163 1.33% MLMG::mgVcycle_up::1 81 0.01142 0.01142 0.01142 1.30% MLMG::mgVcycle_up::4 81 0.01131 0.01131 0.01131 1.29% MLMG::mgVcycle_up::2 81 0.0112 0.0112 0.0112 1.28% Castro::expand_state() 10 0.01119 0.01119 0.01119 1.28% Castro::do_old_sources() 10 0.01117 0.01117 0.01117 1.27% MLCellLinOp::defineAuxData() 11 0.01106 0.01106 0.01106 1.26% amrex::average_down 405 0.01101 0.01101 0.01101 1.26% MLMG::mgVcycle_up::3 81 0.01091 0.01091 0.01091 1.24% MLPoisson::Fapply() 1128 0.01027 0.01027 0.01027 1.17% FabArray::Saxpy() 803 0.008145 0.008145 0.008145 0.93% FabArray::Xpay() 811 0.008005 0.008005 0.008005 0.91% Castro::post_timestep() 10 0.007423 0.007423 0.007423 0.85% MLCellLinOp::solutionResidual() 92 0.007072 0.007072 0.007072 0.81% Gravity::fill_multipole_BCs() 11 0.006816 0.006816 0.006816 0.78% MLMG::computeResidual() 81 0.006082 0.006082 0.006082 0.69% Castro::estTimeStep() 21 0.005861 0.005861 0.005861 0.67% Castro::reset_internal_energy(MultiFab) 63 0.005585 0.005585 0.005585 0.64% MLCellLinOp::defineBC() 11 0.004952 0.004952 0.004952 0.56% BndryData::define() 11 0.004728 0.004728 0.004728 0.54% MLMG::prepareForSolve() 11 0.004707 0.004707 0.004707 0.54% FabArray::LinComb() 550 0.004516 0.004516 0.004516 0.52% amrex::Add() 162 0.00431 0.00431 0.00431 0.49% Amr::InitializeInit() 1 0.003937 0.003937 0.003937 0.45% Amr::defBaseLevel() 1 0.003932 0.003932 0.003932 0.45% Castro::initData() 1 0.00343 0.00343 0.00343 0.39% Castro::computeNewDt() 9 0.003335 0.003335 0.003335 0.38% Castro::construct_new_source() 50 0.003271 0.003271 0.003271 0.37% Castro::construct_new_gravity_source() 10 0.003225 0.003225 0.003225 0.37% Castro::construct_old_source() 50 0.002752 0.002752 0.002752 0.31% Castro::construct_old_gravity_source() 10 0.002733 0.002733 0.002733 0.31% MLMG::ResNormInf() 92 0.002105 0.002105 0.002105 0.24% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001695 0.001695 0.001695 0.19% Castro::reset_internal_energy(Fab) 504 0.001558 0.001558 0.001558 0.18% MLCellLinOp::setLevelBC() 11 0.001402 0.001402 0.001402 0.16% MLMG::getGradSolution() 11 0.001394 0.001394 0.001394 0.16% MLCellLinOp::compGrad() 11 0.001388 0.001388 0.001388 0.16% FabArrayBase::getCPC() 1313 0.001374 0.001374 0.001374 0.16% FabArray::mult() 43 0.001328 0.001328 0.001328 0.15% FabArray::setDomainBndry() 41 0.001296 0.001296 0.001296 0.15% Castro::check_for_nan() 20 0.001211 0.001211 0.001211 0.14% MultiFab::contains_nan() 20 0.0012 0.0012 0.0012 0.14% MLPoisson::prepareForSolve() 11 0.001156 0.001156 0.001156 0.13% Castro::enforce_speed_limit() 62 0.001149 0.001149 0.001149 0.13% MLCellLinOp::prepareForSolve() 11 0.001147 0.001147 0.001147 0.13% Castro::post_regrid() 1 0.0011 0.0011 0.0011 0.13% MLMG::computeMLResidual() 11 0.001029 0.001029 0.001029 0.12% Castro::computeInitialDt() 2 0.0009301 0.0009301 0.0009301 0.11% Gravity::update_max_rhs() 11 0.0008606 0.0008606 0.0008606 0.10% FabArrayBase::getFB() 3974 0.0006998 0.0006998 0.0006998 0.08% FabArrayBase::CPC::define() 454 0.0006642 0.0006642 0.0006642 0.08% Castro::finalize_advance() 10 0.000576 0.000576 0.000576 0.07% Amr::InitAmr() 1 0.0004883 0.0004883 0.0004883 0.06% Gravity::swapTimeLevels() 10 0.0004602 0.0004602 0.0004602 0.05% MLLinOp::define() 11 0.0004413 0.0004413 0.0004413 0.05% Castro::Castro() 1 0.0004322 0.0004322 0.0004322 0.05% MLLinOp::defineGrids() 11 0.0004266 0.0004266 0.0004266 0.05% MLMG::MLResNormInf() 11 0.0002872 0.0002872 0.0002872 0.03% MultiFab::max() 11 0.0002669 0.0002669 0.0002669 0.03% Castro::create_source_corrector() 10 0.0002421 0.0002421 0.0002421 0.03% MLMG::MLRhsNormInf() 11 0.0002212 0.0002212 0.0002212 0.03% Castro::buildMetrics() 1 0.0001608 0.0001608 0.0001608 0.02% FabArrayBase::FB::FB() 56 8.567e-05 8.567e-05 8.567e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.104e-05 5.104e-05 5.104e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.77e-05 4.77e-05 4.77e-05 0.01% Castro::swap_state_time_levels() 10 3.895e-05 3.895e-05 3.895e-05 0.00% makeSFC 55 3.644e-05 3.644e-05 3.644e-05 0.00% Castro::finalize_do_advance() 10 3.62e-05 3.62e-05 3.62e-05 0.00% Castro::enforce_consistent_e() 1 3.422e-05 3.422e-05 3.422e-05 0.00% StateData::define() 4 3.35e-05 3.35e-05 3.35e-05 0.00% Amr::writeSmallPlotFile() 1 3.015e-05 3.015e-05 3.015e-05 0.00% Castro::initMFs() 1 2.31e-05 2.31e-05 2.31e-05 0.00% DistributionMapping::Distribute() 56 1.42e-05 1.42e-05 1.42e-05 0.00% Amr::initSubcycle() 1 8.166e-06 8.166e-06 8.166e-06 0.00% AmrLevel::checkPointPost() 3 4.815e-06 4.815e-06 4.815e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.242e-06 4.242e-06 4.242e-06 0.00% Gravity::set_mass_offset() 11 3.847e-06 3.847e-06 3.847e-06 0.00% Castro::retry_advance_ctu() 10 3.499e-06 3.499e-06 3.499e-06 0.00% Castro::FluxRegCrseInit 10 2.697e-06 2.697e-06 2.697e-06 0.00% AmrLevel::checkPointPre() 3 1.957e-06 1.957e-06 1.957e-06 0.00% Castro::FluxRegFineAdd() 10 1.906e-06 1.906e-06 1.906e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.795e-06 1.795e-06 1.795e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 91 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 120 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1022 KiB 39 MiB Castro::initialize_do_advance() 80 80 25 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2179 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7610 KiB 14 MiB MLMG::prepareForSolve() 660 660 3563 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 205 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 178 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7526 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1866 B 2048 KiB Gravity::solve_for_phi() 80 80 570 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 87 KiB 2048 KiB BndryData::define() 1056 1056 325 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 207 KiB 671 KiB Castro::estTimeStep() 21 21 2360 B 480 KiB VisMF::Write(FabArray) 656 656 3462 B 320 KiB Castro::normalize_species() 62 62 5475 B 320 KiB amrex::average_down 1054 1054 1262 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1144 B 257 KiB amrex::Dot() 1343 1343 3441 B 160 KiB FabArray::norminf() 896 896 2435 B 160 KiB Castro::do_advance_ctu() 10 10 634 B 160 KiB MultiFab::max() 11 11 46 B 160 KiB Castro::initData() 1 1 28 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3483 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 45 B 10 KiB FillBoundary_nowait() 751 751 311 B 9648 B MLCellLinOp::applyBC() 8758 8758 223 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3863 B 6144 B StateData::FillBoundary(geom) 1992 1992 44 B 2688 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 729 729 120 B 1472 B MLCellLinOp::defineBC() 66 66 366 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 571 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 49 KiB 8192 KiB VisMF::Write(FabArray) 744 744 630 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3483 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 45 B 10 KiB FillBoundary_nowait() 751 751 311 B 9648 B MLCellLinOp::applyBC() 4379 4379 221 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3863 B 6144 B Gravity::get_new_grav_vector() 3 3 2916 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B StateData::FillBoundary(geom) 1992 1992 44 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 82 82 272 B 1296 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLMG::prepareForSolve() 11 11 296 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 25 B 400 B FabArray::norminf() 896 896 17 B 272 B Castro::estTimeStep() 21 21 0 B 32 B MultiFab::contains_nan() 20 20 0 B 16 B MultiFab::max() 11 11 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B Castro::do_advance_ctu() 10 10 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2424 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.05-27-gd65dd906bd53) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.05-27-gd65dd906bd53) initialized Starting run at 09:02:00 UTC on 2023-05-24. Successfully read inputs file ... Castro git describe: 23.05-25-g99ae92958 AMReX git describe: 23.05-27-gd65dd906b Microphysics git describe: 23.05-12-g1f8fd85f reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.470786915 Restart time = 0.086026494 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049912232 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048832155 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063561913 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.067541579 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069897833 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.038852642 seconds Ending run at 09:02:01 UTC on 2023-05-24. Run time = 0.425668436 Run time without initialization = 0.339065171 Average number of zones advanced per microsecond: 3.866 Average number of zones advanced per microsecond per rank: 3.866 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.4257 ... 0.4257 ... 0.4257 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1180 0.1180 0.1180 27.72% Amr::restart() 1 0.04527 0.04527 0.04527 10.63% VisMF::Read() 3 0.03707 0.03707 0.03707 8.71% MLCellLinOp::applyBC() 1946 0.0332 0.0332 0.0332 7.80% VisMF::Write(FabArray) 1 0.03236 0.03236 0.03236 7.60% MLPoisson::Fsmooth() 1440 0.01399 0.01399 0.01399 3.29% FillBoundary_nowait() 1766 0.01278 0.01278 0.01278 3.00% StateData::FillBoundary(geom) 160 0.01218 0.01218 0.01218 2.86% amrex::Copy() 463 0.01185 0.01185 0.01185 2.78% amrex::Dot() 484 0.008804 0.008804 0.008804 2.07% Castro::normalize_species() 30 0.007002 0.007002 0.007002 1.64% FabArray::setVal() 537 0.006247 0.006247 0.006247 1.47% FabArray::norminf() 326 0.006238 0.006238 0.006238 1.47% FabArray::ParallelCopy_nowait() 380 0.006035 0.006035 0.006035 1.42% Castro::enforce_min_density() 30 0.005742 0.005742 0.005742 1.35% StateDataPhysBCFunct::() 20 0.005653 0.005653 0.005653 1.33% Castro::computeTemp() 30 0.00565 0.00565 0.00565 1.33% MLCellLinOp::defineAuxData() 6 0.00524 0.00524 0.00524 1.23% MLPoisson::Fapply() 500 0.004466 0.004466 0.004466 1.05% FabArray::Saxpy() 355 0.00365 0.00365 0.00365 0.86% FabArray::Xpay() 361 0.003493 0.003493 0.003493 0.82% Gravity::fill_multipole_BCs() 6 0.003354 0.003354 0.003354 0.79% Castro::estTimeStep() 10 0.003112 0.003112 0.003112 0.73% MLMG::addInterpCorrection() 180 0.002857 0.002857 0.002857 0.67% amrex::average_down 180 0.002574 0.002574 0.002574 0.60% Castro::reset_internal_energy(MultiFab) 30 0.002436 0.002436 0.002436 0.57% BndryData::define() 6 0.00204 0.00204 0.00204 0.48% FabArray::LinComb() 242 0.001929 0.001929 0.001929 0.45% amrex::Add() 72 0.001846 0.001846 0.001846 0.43% Castro::construct_new_gravity_source() 5 0.001703 0.001703 0.001703 0.40% Castro::do_advance_ctu() 5 0.001591 0.001591 0.001591 0.37% Amr::writePlotFile() 1 0.001439 0.001439 0.001439 0.34% Castro::construct_old_gravity_source() 5 0.001421 0.001421 0.001421 0.33% MLCGSolver::bicgstab 36 0.0009286 0.0009286 0.0009286 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009109 0.0009109 0.0009109 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007678 0.0007678 0.0007678 0.18% Castro::enforce_speed_limit() 30 0.0007404 0.0007404 0.0007404 0.17% MLCellLinOp::setLevelBC() 6 0.0007399 0.0007399 0.0007399 0.17% Castro::reset_internal_energy(Fab) 240 0.0007078 0.0007078 0.0007078 0.17% FabArray::mult() 22 0.0006444 0.0006444 0.0006444 0.15% FabArray::setDomainBndry() 20 0.0006414 0.0006414 0.0006414 0.15% MLCellLinOp::prepareForSolve() 6 0.000609 0.000609 0.000609 0.14% MultiFab::contains_nan() 10 0.0005991 0.0005991 0.0005991 0.14% MLCellLinOp::compGrad() 6 0.0004865 0.0004865 0.0004865 0.11% MLCellLinOp::smooth() 720 0.0004783 0.0004783 0.0004783 0.11% MLMG::prepareForSolve() 6 0.0004657 0.0004657 0.0004657 0.11% FabArrayBase::CPC::define() 244 0.0004211 0.0004211 0.0004211 0.10% FabArray::FillBoundary() 1766 0.0003948 0.0003948 0.0003948 0.09% Amr::InitAmr() 1 0.0003907 0.0003907 0.0003907 0.09% FabArrayBase::getCPC() 632 0.0003486 0.0003486 0.0003486 0.08% main() 1 0.000327 0.000327 0.000327 0.08% Gravity::get_old_grav_vector() 5 0.0003013 0.0003013 0.0003013 0.07% Gravity::get_new_grav_vector() 5 0.0002738 0.0002738 0.0002738 0.06% FabArrayBase::getFB() 1766 0.0002597 0.0002597 0.0002597 0.06% MLCellLinOp::apply() 500 0.0001997 0.0001997 0.0001997 0.05% MLLinOp::defineGrids() 6 0.0001748 0.0001748 0.0001748 0.04% Amr::coarseTimeStep() 5 0.0001584 0.0001584 0.0001584 0.04% MLMG::mgVcycle() 36 0.0001423 0.0001423 0.0001423 0.03% MultiFab::max() 6 0.0001375 0.0001375 0.0001375 0.03% Castro::create_source_corrector() 5 0.0001331 0.0001331 0.0001331 0.03% MLCGSolver::ParallelAllReduce 659 0.000128 0.000128 0.000128 0.03% FillPatchIterator::Initialize 20 0.0001225 0.0001225 0.0001225 0.03% FabArray::ParallelCopy() 380 0.0001108 0.0001108 0.0001108 0.03% MLCellLinOp::defineBC() 6 0.0001101 0.0001101 0.0001101 0.03% MLCellLinOp::correctionResidual() 216 9.464e-05 9.464e-05 9.464e-05 0.02% Castro::subcycle_advance_ctu() 5 8.188e-05 8.188e-05 8.188e-05 0.02% AmrLevel::restart() 1 8.103e-05 8.103e-05 8.103e-05 0.02% Amr::timeStep() 5 7.882e-05 7.882e-05 7.882e-05 0.02% Gravity::solve_for_phi() 5 6.119e-05 6.119e-05 6.119e-05 0.01% Gravity::update_max_rhs() 6 5.98e-05 5.98e-05 5.98e-05 0.01% StateData::restartDoit() 4 5.966e-05 5.966e-05 5.966e-05 0.01% FabArrayBase::FB::FB() 26 5.882e-05 5.882e-05 5.882e-05 0.01% MLMG:computeResOfCorrection() 180 5.315e-05 5.315e-05 5.315e-05 0.01% Castro::clean_state() 30 4.125e-05 4.125e-05 4.125e-05 0.01% Castro::expand_state() 5 4.044e-05 4.044e-05 4.044e-05 0.01% MLMG::actualBottomSolve() 36 3.765e-05 3.765e-05 3.765e-05 0.01% MLMG::mgVcycle_down::0 36 3.567e-05 3.567e-05 3.567e-05 0.01% Castro::finalize_advance() 5 3.441e-05 3.441e-05 3.441e-05 0.01% Castro::initialize_advance() 5 3.302e-05 3.302e-05 3.302e-05 0.01% MLMG::mgVcycle_down::1 36 3.264e-05 3.264e-05 3.264e-05 0.01% MLMG::mgVcycle_down::4 36 3.174e-05 3.174e-05 3.174e-05 0.01% MLMG::mgVcycle_down::2 36 3.125e-05 3.125e-05 3.125e-05 0.01% MLMG::solve() 6 3.094e-05 3.094e-05 3.094e-05 0.01% MLMG::mgVcycle_down::3 36 3.062e-05 3.062e-05 3.062e-05 0.01% MLMG::oneIter() 36 2.844e-05 2.844e-05 2.844e-05 0.01% Castro::initMFs() 1 2.746e-05 2.746e-05 2.746e-05 0.01% Amr::writeSmallPlotFile() 1 2.713e-05 2.713e-05 2.713e-05 0.01% Castro::construct_new_source() 25 2.67e-05 2.67e-05 2.67e-05 0.01% MLMG::mgVcycle_up::4 36 2.657e-05 2.657e-05 2.657e-05 0.01% Castro::initialize_do_advance() 5 2.565e-05 2.565e-05 2.565e-05 0.01% Castro::buildMetrics() 1 2.522e-05 2.522e-05 2.522e-05 0.01% Castro::swap_state_time_levels() 5 2.304e-05 2.304e-05 2.304e-05 0.01% MLMG::mgVcycle_up::3 36 2.225e-05 2.225e-05 2.225e-05 0.01% MLMG::mgVcycle_up::0 36 2.223e-05 2.223e-05 2.223e-05 0.01% MLMG::mgVcycle_up::2 36 2.194e-05 2.194e-05 2.194e-05 0.01% MLMG::mgVcycle_up::1 36 2.176e-05 2.176e-05 2.176e-05 0.01% MLCellLinOp::solutionResidual() 42 2.168e-05 2.168e-05 2.168e-05 0.01% Castro::post_restart() 1 1.939e-05 1.939e-05 1.939e-05 0.00% Castro::finalize_do_advance() 5 1.915e-05 1.915e-05 1.915e-05 0.00% MLPoisson::define() 6 1.83e-05 1.83e-05 1.83e-05 0.00% MLMG::ResNormInf() 42 1.683e-05 1.683e-05 1.683e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.55e-05 1.55e-05 1.55e-05 0.00% MLMG::mgVcycle_bottom 36 1.464e-05 1.464e-05 1.464e-05 0.00% Castro::advance() 5 1.427e-05 1.427e-05 1.427e-05 0.00% FillPatchSingleLevel 20 1.375e-05 1.375e-05 1.375e-05 0.00% MLMG::computeResidual() 36 1.366e-05 1.366e-05 1.366e-05 0.00% makeSFC 30 1.195e-05 1.195e-05 1.195e-05 0.00% Castro::construct_new_gravity() 5 1.186e-05 1.186e-05 1.186e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.079e-05 1.079e-05 1.079e-05 0.00% Castro::construct_old_source() 25 9.975e-06 9.975e-06 9.975e-06 0.00% MLLinOp::define() 6 9.778e-06 9.778e-06 9.778e-06 0.00% DistributionMapping::Distribute() 31 8.314e-06 8.314e-06 8.314e-06 0.00% Castro::do_new_sources() 5 8.184e-06 8.184e-06 8.184e-06 0.00% Amr::initSubcycle() 1 8.018e-06 8.018e-06 8.018e-06 0.00% Gravity::actual_multilevel_solve() 1 7.346e-06 7.346e-06 7.346e-06 0.00% Castro::do_old_sources() 5 7.286e-06 7.286e-06 7.286e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.875e-06 6.875e-06 6.875e-06 0.00% Castro::apply_source_to_state() 10 5.626e-06 5.626e-06 5.626e-06 0.00% Castro::check_for_nan() 10 5.49e-06 5.49e-06 5.49e-06 0.00% Castro::construct_old_gravity() 5 4.982e-06 4.982e-06 4.982e-06 0.00% Castro::post_timestep() 5 4.68e-06 4.68e-06 4.68e-06 0.00% MLPoisson::prepareForSolve() 6 4.41e-06 4.41e-06 4.41e-06 0.00% Gravity::swapTimeLevels() 5 3.539e-06 3.539e-06 3.539e-06 0.00% Castro::computeNewDt() 5 3.424e-06 3.424e-06 3.424e-06 0.00% MLMG::computeMLResidual() 6 3.409e-06 3.409e-06 3.409e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.104e-06 3.104e-06 3.104e-06 0.00% MLMG::getGradSolution() 6 2.887e-06 2.887e-06 2.887e-06 0.00% MLMG::MLResNormInf() 6 2.224e-06 2.224e-06 2.224e-06 0.00% MLMG::MLRhsNormInf() 6 2.116e-06 2.116e-06 2.116e-06 0.00% Gravity::set_mass_offset() 6 2.008e-06 2.008e-06 2.008e-06 0.00% Castro::retry_advance_ctu() 5 1.813e-06 1.813e-06 1.813e-06 0.00% Castro::FluxRegCrseInit 5 1.436e-06 1.436e-06 1.436e-06 0.00% Castro::FluxRegFineAdd() 5 1.159e-06 1.159e-06 1.159e-06 0.00% Amr::init() 1 9.42e-07 9.42e-07 9.42e-07 0.00% MLLinOp::makeSubCommunicator() 6 9.21e-07 9.21e-07 9.21e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4257 0.4257 0.4257 100.00% Amr::coarseTimeStep() 5 0.2999 0.2999 0.2999 70.45% Amr::timeStep() 5 0.2974 0.2974 0.2974 69.87% Castro::advance() 5 0.2939 0.2939 0.2939 69.05% Castro::subcycle_advance_ctu() 5 0.2876 0.2876 0.2876 67.55% Castro::do_advance_ctu() 5 0.2875 0.2875 0.2875 67.53% Castro::construct_new_gravity() 5 0.13 0.13 0.13 30.54% Gravity::solve_phi_with_mlmg() 6 0.1249 0.1249 0.1249 29.35% Gravity::solve_for_phi() 5 0.1219 0.1219 0.1219 28.64% Gravity::actual_solve_with_mlmg() 6 0.1213 0.1213 0.1213 28.50% Castro::construct_ctu_hydro_source() 5 0.118 0.118 0.118 27.72% MLMG::solve() 6 0.1102 0.1102 0.1102 25.88% MLMG::oneIter() 36 0.1034 0.1034 0.1034 24.28% MLMG::mgVcycle() 36 0.1018 0.1018 0.1018 23.92% Amr::init() 1 0.08607 0.08607 0.08607 20.22% Amr::restart() 1 0.08607 0.08607 0.08607 20.22% MLCellLinOp::smooth() 720 0.04991 0.04991 0.04991 11.72% MLCellLinOp::applyBC() 1946 0.04669 0.04669 0.04669 10.97% Amr::writePlotFile() 1 0.03896 0.03896 0.03896 9.15% AmrLevel::restart() 1 0.03726 0.03726 0.03726 8.75% StateData::restartDoit() 4 0.03718 0.03718 0.03718 8.73% VisMF::Read() 3 0.03707 0.03707 0.03707 8.71% VisMF::Write(FabArray) 1 0.03236 0.03236 0.03236 7.60% MLMG::mgVcycle_bottom 36 0.03165 0.03165 0.03165 7.43% MLMG::actualBottomSolve() 36 0.03164 0.03164 0.03164 7.43% MLCGSolver::bicgstab 36 0.03133 0.03133 0.03133 7.36% Castro::clean_state() 30 0.02232 0.02232 0.02232 5.24% FillPatchIterator::Initialize 20 0.02064 0.02064 0.02064 4.85% FillPatchSingleLevel 20 0.01988 0.01988 0.01988 4.67% StateDataPhysBCFunct::() 20 0.01783 0.01783 0.01783 4.19% MLCellLinOp::apply() 500 0.01565 0.01565 0.01565 3.68% MLMG::mgVcycle_down::0 36 0.01435 0.01435 0.01435 3.37% MLPoisson::Fsmooth() 1440 0.01399 0.01399 0.01399 3.29% FabArray::FillBoundary() 1766 0.0135 0.0135 0.0135 3.17% FillBoundary_nowait() 1766 0.0131 0.0131 0.0131 3.08% StateData::FillBoundary(geom) 160 0.01218 0.01218 0.01218 2.86% amrex::Copy() 463 0.01185 0.01185 0.01185 2.78% MLMG::mgVcycle_up::0 36 0.01076 0.01076 0.01076 2.53% Castro::initialize_do_advance() 5 0.009718 0.009718 0.009718 2.28% MLCellLinOp::correctionResidual() 216 0.009498 0.009498 0.009498 2.23% MLPoisson::define() 6 0.008909 0.008909 0.008909 2.09% amrex::Dot() 484 0.008804 0.008804 0.008804 2.07% Castro::computeTemp() 30 0.008794 0.008794 0.008794 2.07% MLMG:computeResOfCorrection() 180 0.008336 0.008336 0.008336 1.96% Gravity::get_new_grav_vector() 5 0.00797 0.00797 0.00797 1.87% Castro::construct_old_gravity() 5 0.007429 0.007429 0.007429 1.75% Gravity::get_old_grav_vector() 5 0.007424 0.007424 0.007424 1.74% Castro::normalize_species() 30 0.007002 0.007002 0.007002 1.64% MLMG::mgVcycle_down::1 36 0.006782 0.006782 0.006782 1.59% FabArray::ParallelCopy() 380 0.006536 0.006536 0.006536 1.54% Castro::do_new_sources() 5 0.006512 0.006512 0.006512 1.53% FabArray::ParallelCopy_nowait() 380 0.006425 0.006425 0.006425 1.51% MLMG::mgVcycle_down::2 36 0.006342 0.006342 0.006342 1.49% FabArray::setVal() 537 0.006247 0.006247 0.006247 1.47% FabArray::norminf() 326 0.006238 0.006238 0.006238 1.47% MLMG::mgVcycle_down::3 36 0.006205 0.006205 0.006205 1.46% MLMG::mgVcycle_down::4 36 0.006113 0.006113 0.006113 1.44% Castro::initialize_advance() 5 0.006061 0.006061 0.006061 1.42% MLCellLinOp::defineAuxData() 6 0.005959 0.005959 0.005959 1.40% Castro::expand_state() 5 0.005763 0.005763 0.005763 1.35% Castro::enforce_min_density() 30 0.005742 0.005742 0.005742 1.35% MLCGSolver::ParallelAllReduce 659 0.005296 0.005296 0.005296 1.24% MLMG::addInterpCorrection() 180 0.005069 0.005069 0.005069 1.19% Castro::do_old_sources() 5 0.005005 0.005005 0.005005 1.18% MLMG::mgVcycle_up::1 36 0.004962 0.004962 0.004962 1.17% MLMG::mgVcycle_up::4 36 0.004899 0.004899 0.004899 1.15% amrex::average_down 180 0.004863 0.004863 0.004863 1.14% MLMG::mgVcycle_up::2 36 0.004842 0.004842 0.004842 1.14% MLMG::mgVcycle_up::3 36 0.004759 0.004759 0.004759 1.12% MLPoisson::Fapply() 500 0.004466 0.004466 0.004466 1.05% FabArray::Saxpy() 355 0.00365 0.00365 0.00365 0.86% FabArray::Xpay() 361 0.003493 0.003493 0.003493 0.82% Gravity::fill_multipole_BCs() 6 0.003476 0.003476 0.003476 0.82% Castro::post_timestep() 5 0.003408 0.003408 0.003408 0.80% Castro::post_restart() 1 0.003367 0.003367 0.003367 0.79% Gravity::multilevel_solve_for_new_phi() 1 0.003256 0.003256 0.003256 0.76% Gravity::actual_multilevel_solve() 1 0.003241 0.003241 0.003241 0.76% MLCellLinOp::solutionResidual() 42 0.003205 0.003205 0.003205 0.75% Castro::reset_internal_energy(MultiFab) 30 0.003144 0.003144 0.003144 0.74% Castro::estTimeStep() 10 0.003112 0.003112 0.003112 0.73% MLCellLinOp::defineBC() 6 0.00272 0.00272 0.00272 0.64% MLMG::computeResidual() 36 0.002659 0.002659 0.002659 0.62% BndryData::define() 6 0.00261 0.00261 0.00261 0.61% MLMG::prepareForSolve() 6 0.002502 0.002502 0.002502 0.59% Castro::computeNewDt() 5 0.002304 0.002304 0.002304 0.54% FabArray::LinComb() 242 0.001929 0.001929 0.001929 0.45% amrex::Add() 72 0.001846 0.001846 0.001846 0.43% Castro::construct_new_source() 25 0.00173 0.00173 0.00173 0.41% Castro::construct_new_gravity_source() 5 0.001703 0.001703 0.001703 0.40% Castro::construct_old_source() 25 0.001431 0.001431 0.001431 0.34% Castro::construct_old_gravity_source() 5 0.001421 0.001421 0.001421 0.33% MLMG::ResNormInf() 42 0.0009315 0.0009315 0.0009315 0.22% Castro::apply_source_to_state() 10 0.0009193 0.0009193 0.0009193 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009109 0.0009109 0.0009109 0.21% FabArrayBase::getCPC() 632 0.0007697 0.0007697 0.0007697 0.18% MLMG::getGradSolution() 6 0.0007515 0.0007515 0.0007515 0.18% MLCellLinOp::compGrad() 6 0.0007487 0.0007487 0.0007487 0.18% Castro::enforce_speed_limit() 30 0.0007404 0.0007404 0.0007404 0.17% MLCellLinOp::setLevelBC() 6 0.0007399 0.0007399 0.0007399 0.17% Castro::reset_internal_energy(Fab) 240 0.0007078 0.0007078 0.0007078 0.17% FabArray::mult() 22 0.0006444 0.0006444 0.0006444 0.15% FabArray::setDomainBndry() 20 0.0006414 0.0006414 0.0006414 0.15% MLPoisson::prepareForSolve() 6 0.0006134 0.0006134 0.0006134 0.14% MLCellLinOp::prepareForSolve() 6 0.000609 0.000609 0.000609 0.14% Castro::check_for_nan() 10 0.0006045 0.0006045 0.0006045 0.14% MultiFab::contains_nan() 10 0.0005991 0.0005991 0.0005991 0.14% MLMG::computeMLResidual() 6 0.0005631 0.0005631 0.0005631 0.13% Gravity::update_max_rhs() 6 0.0004451 0.0004451 0.0004451 0.10% FabArrayBase::CPC::define() 244 0.0004211 0.0004211 0.0004211 0.10% Amr::InitAmr() 1 0.0003988 0.0003988 0.0003988 0.09% FabArrayBase::getFB() 1766 0.0003185 0.0003185 0.0003185 0.07% Castro::finalize_advance() 5 0.0002952 0.0002952 0.0002952 0.07% Gravity::swapTimeLevels() 5 0.0002236 0.0002236 0.0002236 0.05% MLLinOp::define() 6 0.0002112 0.0002112 0.0002112 0.05% MLLinOp::defineGrids() 6 0.0002014 0.0002014 0.0002014 0.05% MLMG::MLResNormInf() 6 0.0001506 0.0001506 0.0001506 0.04% Castro::buildMetrics() 1 0.0001424 0.0001424 0.0001424 0.03% MultiFab::max() 6 0.0001375 0.0001375 0.0001375 0.03% Castro::create_source_corrector() 5 0.0001331 0.0001331 0.0001331 0.03% MLMG::MLRhsNormInf() 6 0.0001182 0.0001182 0.0001182 0.03% FabArrayBase::FB::FB() 26 5.882e-05 5.882e-05 5.882e-05 0.01% Castro::initMFs() 1 2.746e-05 2.746e-05 2.746e-05 0.01% Amr::writeSmallPlotFile() 1 2.713e-05 2.713e-05 2.713e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.576e-05 2.576e-05 2.576e-05 0.01% Castro::swap_state_time_levels() 5 2.304e-05 2.304e-05 2.304e-05 0.01% Castro::finalize_do_advance() 5 1.915e-05 1.915e-05 1.915e-05 0.00% makeSFC 30 1.889e-05 1.889e-05 1.889e-05 0.00% DistributionMapping::Distribute() 31 8.314e-06 8.314e-06 8.314e-06 0.00% Amr::initSubcycle() 1 8.018e-06 8.018e-06 8.018e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.48e-06 4.48e-06 4.48e-06 0.00% Gravity::set_mass_offset() 6 2.008e-06 2.008e-06 2.008e-06 0.00% Castro::retry_advance_ctu() 5 1.813e-06 1.813e-06 1.813e-06 0.00% Castro::FluxRegCrseInit 5 1.436e-06 1.436e-06 1.436e-06 0.00% Castro::FluxRegFineAdd() 5 1.159e-06 1.159e-06 1.159e-06 0.00% MLLinOp::makeSubCommunicator() 6 9.21e-07 9.21e-07 9.21e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 196 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 131 MiB 692 MiB Castro::initMFs() 48 48 54 MiB 68 MiB StateData::restartDoit() 32 32 48 MiB 55 MiB Castro::swap_state_time_levels() 32 32 44 MiB 55 MiB Castro::initialize_do_advance() 40 40 26 MiB 39 MiB FillPatchIterator::Initialize 160 160 1052 KiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2568 KiB 28 MiB Castro::initialize_advance() 40 40 16 MiB 23 MiB Castro::buildMetrics() 32 32 12 MiB 15 MiB Castro::post_restart() 48 48 6092 KiB 14 MiB MLMG::prepareForSolve() 361 361 3178 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 180 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 191 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6088 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 12 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2024 B 2048 KiB Gravity::solve_for_phi() 40 40 585 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 15 KiB 2048 KiB BndryData::define() 576 576 292 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 187 KiB 671 KiB Castro::estTimeStep() 10 10 2406 B 480 KiB VisMF::Write(FabArray) 112 112 1207 B 320 KiB Castro::normalize_species() 30 30 5345 B 320 KiB amrex::average_down 469 469 1121 B 257 KiB MLMG::addInterpCorrection() 468 468 1027 B 257 KiB amrex::Dot() 592 592 3046 B 160 KiB FabArray::norminf() 398 398 2166 B 160 KiB Castro::do_advance_ctu() 5 5 582 B 160 KiB MultiFab::max() 6 6 50 B 160 KiB FabArray::setVal() 66 66 16 KiB 23 KiB MultiFab::contains_nan() 10 10 28 B 20 KiB MLPoisson::Fsmooth() 60 60 3091 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 261 B 9648 B MLCellLinOp::applyBC() 3892 3892 202 B 9344 B amrex::Copy() 56 56 5413 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 45 B 2464 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 324 324 107 B 1472 B MLCellLinOp::defineBC() 36 36 328 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1234 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 101 KiB 8192 KiB VisMF::Write(FabArray) 120 120 227 KiB 3584 KiB VisMF::Read() 24 24 171 KiB 3000 KiB FabArray::setVal() 66 66 16 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3091 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 261 B 9648 B MLCellLinOp::applyBC() 1946 1946 200 B 9328 B amrex::Copy() 56 56 5413 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2405 B 3072 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B StateData::FillBoundary(geom) 960 960 46 B 2544 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 481 B 1296 B amrex::average_down 37 37 203 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 22 B 400 B FabArray::norminf() 398 398 15 B 272 B Castro::estTimeStep() 10 10 0 B 32 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::do_advance_ctu() 5 5 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2424 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.05-27-gd65dd906bd53) finalized