Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-30-g3396b1df1175) initialized Starting run at 08:10:21 UTC on 2023-08-16. Successfully read inputs file ... Castro git describe: 23.08-5-g5303a2cd5 AMReX git describe: 23.08-30-g3396b1df1 Microphysics git describe: 23.08-11-g9ecfd8d2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.050441664 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.028076378 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.051304127 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050579594 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072295692 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.074459748 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.050281891 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.048642169 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.056519806 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.055073129 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.08211137 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.083366585 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.055215124 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048186322 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027910224 seconds Ending run at 08:10:22 UTC on 2023-08-16. Run time = 0.88977858 Run time without initialization = 0.756591632 Average number of zones advanced per microsecond: 3.465 Average number of zones advanced per microsecond per rank: 3.465 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8898 ... 0.8898 ... 0.8898 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2549 0.2549 0.2549 28.65% VisMF::Write(FabArray) 11 0.196 0.196 0.196 22.03% MLCellLinOp::applyBC() 4433 0.07774 0.07774 0.07774 8.74% MLPoisson::Fsmooth() 3280 0.03281 0.03281 0.03281 3.69% FillBoundary_nowait() 4023 0.03086 0.03086 0.03086 3.47% StateData::FillBoundary(geom) 328 0.02688 0.02688 0.02688 3.02% amrex::Dot() 1114 0.02102 0.02102 0.02102 2.36% Castro::normalize_species() 62 0.02034 0.02034 0.02034 2.29% Castro::computeTemp() 63 0.01657 0.01657 0.01657 1.86% amrex::Copy() 1029 0.01505 0.01505 0.01505 1.69% FabArray::norminf() 743 0.01464 0.01464 0.01464 1.65% FabArray::ParallelCopy_nowait() 861 0.0135 0.0135 0.0135 1.52% FabArray::setVal() 1144 0.01347 0.01347 0.01347 1.51% Castro::enforce_min_density() 62 0.01146 0.01146 0.01146 1.29% StateDataPhysBCFunct::() 41 0.0108 0.0108 0.0108 1.21% MLPoisson::Fapply() 1142 0.01057 0.01057 0.01057 1.19% MLCellLinOp::defineAuxData() 11 0.0103 0.0103 0.0103 1.16% Gravity::fill_multipole_BCs() 11 0.009325 0.009325 0.009325 1.05% FabArray::Saxpy() 813 0.008363 0.008363 0.008363 0.94% FabArray::Xpay() 821 0.00826 0.00826 0.00826 0.93% MLMG::addInterpCorrection() 410 0.006879 0.006879 0.006879 0.77% Castro::estTimeStep() 21 0.006263 0.006263 0.006263 0.70% amrex::average_down 410 0.006144 0.006144 0.006144 0.69% Castro::reset_internal_energy(MultiFab) 63 0.0052 0.0052 0.0052 0.58% Amr::checkPoint() 3 0.004758 0.004758 0.004758 0.53% FabArray::LinComb() 557 0.004606 0.004606 0.004606 0.52% Castro::enforce_speed_limit() 62 0.00446 0.00446 0.00446 0.50% amrex::Add() 164 0.004401 0.004401 0.004401 0.49% BndryData::define() 11 0.003921 0.003921 0.003921 0.44% Castro::construct_new_gravity_source() 10 0.003344 0.003344 0.003344 0.38% Castro::construct_old_gravity_source() 10 0.0029 0.0029 0.0029 0.33% MLCGSolver::bicgstab 82 0.002278 0.002278 0.002278 0.26% Amr::writePlotFile() 2 0.001988 0.001988 0.001988 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001806 0.001806 0.001806 0.20% check_for_negative_density() 10 0.001755 0.001755 0.001755 0.20% Castro::reset_internal_energy(Fab) 504 0.001728 0.001728 0.001728 0.19% Castro::initData() 1 0.001522 0.001522 0.001522 0.17% Gravity::actual_solve_with_mlmg() 11 0.001512 0.001512 0.001512 0.17% MLCellLinOp::setLevelBC() 11 0.001468 0.001468 0.001468 0.17% FabArray::mult() 43 0.001368 0.001368 0.001368 0.15% FabArray::setDomainBndry() 41 0.001352 0.001352 0.001352 0.15% MLCellLinOp::prepareForSolve() 11 0.001281 0.001281 0.001281 0.14% MultiFab::contains_nan() 20 0.001277 0.001277 0.001277 0.14% MLCellLinOp::smooth() 1640 0.001153 0.001153 0.001153 0.13% MLCellLinOp::compGrad() 11 0.001046 0.001046 0.001046 0.12% MLMG::prepareForSolve() 11 0.0009475 0.0009475 0.0009475 0.11% FabArray::FillBoundary() 4023 0.0008441 0.0008441 0.0008441 0.09% FabArrayBase::getCPC() 1323 0.0007514 0.0007514 0.0007514 0.08% FabArrayBase::CPC::define() 454 0.0006818 0.0006818 0.0006818 0.08% FabArrayBase::getFB() 4023 0.0006364 0.0006364 0.0006364 0.07% Gravity::get_new_grav_vector() 11 0.000594 0.000594 0.000594 0.07% Amr::InitAmr() 1 0.0004889 0.0004889 0.0004889 0.05% MLCellLinOp::apply() 1142 0.0004794 0.0004794 0.0004794 0.05% Gravity::get_old_grav_vector() 10 0.0004775 0.0004775 0.0004775 0.05% Amr::coarseTimeStep() 10 0.0003763 0.0003763 0.0003763 0.04% AmrLevel::FillPatch() 41 0.0003395 0.0003395 0.0003395 0.04% MultiFab::max() 11 0.0003303 0.0003303 0.0003303 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003022 0.0003022 0.0003022 0.03% main() 1 0.0002815 0.0002815 0.0002815 0.03% FabArray::ParallelCopy() 861 0.0002539 0.0002539 0.0002539 0.03% MLCellLinOp::defineBC() 11 0.0002385 0.0002385 0.0002385 0.03% FillPatchIterator::Initialize 41 0.0002122 0.0002122 0.0002122 0.02% Castro::subcycle_advance_ctu() 10 0.0002101 0.0002101 0.0002101 0.02% MLCellLinOp::correctionResidual() 492 0.0002097 0.0002097 0.0002097 0.02% MLMG::mgVcycle() 82 0.0001971 0.0001971 0.0001971 0.02% Amr::timeStep() 10 0.0001547 0.0001547 0.0001547 0.02% Gravity::solve_for_phi() 10 0.0001544 0.0001544 0.0001544 0.02% MLLinOp::defineGrids() 11 0.0001536 0.0001536 0.0001536 0.02% Castro::create_source_corrector() 10 0.0001416 0.0001416 0.0001416 0.02% Gravity::update_max_rhs() 11 0.0001302 0.0001302 0.0001302 0.01% MLMG:computeResOfCorrection() 410 0.0001209 0.0001209 0.0001209 0.01% StateData::checkPoint() 12 0.0001134 0.0001134 0.0001134 0.01% Castro::advance() 10 0.0001092 0.0001092 0.0001092 0.01% Castro::do_new_sources() 10 0.0001077 0.0001077 0.0001077 0.01% Castro::Castro() 1 9.766e-05 9.766e-05 9.766e-05 0.01% FabArrayBase::FB::FB() 56 9.197e-05 9.197e-05 9.197e-05 0.01% MLMG::actualBottomSolve() 82 8.65e-05 8.65e-05 8.65e-05 0.01% Castro::initialize_advance() 10 8.591e-05 8.591e-05 8.591e-05 0.01% MLMG::mgVcycle_down::0 82 8.527e-05 8.527e-05 8.527e-05 0.01% Castro::clean_state() 62 8.195e-05 8.195e-05 8.195e-05 0.01% MLMG::solve() 11 7.878e-05 7.878e-05 7.878e-05 0.01% MLMG::mgVcycle_down::1 82 7.874e-05 7.874e-05 7.874e-05 0.01% MLMG::mgVcycle_down::2 82 7.791e-05 7.791e-05 7.791e-05 0.01% MLMG::mgVcycle_down::4 82 7.527e-05 7.527e-05 7.527e-05 0.01% MLMG::mgVcycle_down::3 82 7.342e-05 7.342e-05 7.342e-05 0.01% Castro::enforce_consistent_e() 1 7.185e-05 7.185e-05 7.185e-05 0.01% AmrLevel::checkPoint() 3 7.018e-05 7.018e-05 7.018e-05 0.01% Castro::finalize_advance() 10 6.701e-05 6.701e-05 6.701e-05 0.01% MLMG::mgVcycle_up::4 82 6.057e-05 6.057e-05 6.057e-05 0.01% Castro::initialize_do_advance() 10 6.032e-05 6.032e-05 6.032e-05 0.01% MLCellLinOp::solutionResidual() 93 5.106e-05 5.106e-05 5.106e-05 0.01% MLMG::mgVcycle_up::0 82 5.071e-05 5.071e-05 5.071e-05 0.01% MLMG::mgVcycle_up::3 82 5.057e-05 5.057e-05 5.057e-05 0.01% Castro::do_advance_ctu() 10 4.846e-05 4.846e-05 4.846e-05 0.01% MLMG::oneIter() 82 4.82e-05 4.82e-05 4.82e-05 0.01% MLMG::mgVcycle_up::1 82 4.742e-05 4.742e-05 4.742e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.709e-05 4.709e-05 4.709e-05 0.01% MLMG::mgVcycle_up::2 82 4.592e-05 4.592e-05 4.592e-05 0.01% Castro::finalize_do_advance() 10 4.459e-05 4.459e-05 4.459e-05 0.01% FillPatchSingleLevel 41 4.025e-05 4.025e-05 4.025e-05 0.00% Amr::defBaseLevel() 1 3.985e-05 3.985e-05 3.985e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.676e-05 3.676e-05 3.676e-05 0.00% Castro::swap_state_time_levels() 10 3.627e-05 3.627e-05 3.627e-05 0.00% StateData::define() 4 3.592e-05 3.592e-05 3.592e-05 0.00% MLMG::mgVcycle_bottom 82 3.489e-05 3.489e-05 3.489e-05 0.00% MLMG::computeResidual() 82 3.471e-05 3.471e-05 3.471e-05 0.00% MLMG::ResNormInf() 93 3.208e-05 3.208e-05 3.208e-05 0.00% Castro::initMFs() 1 2.94e-05 2.94e-05 2.94e-05 0.00% Castro::buildMetrics() 1 2.738e-05 2.738e-05 2.738e-05 0.00% Amr::writeSmallPlotFile() 1 2.637e-05 2.637e-05 2.637e-05 0.00% makeSFC 55 2.57e-05 2.57e-05 2.57e-05 0.00% Castro::construct_new_gravity() 10 2.57e-05 2.57e-05 2.57e-05 0.00% MLPoisson::define() 11 2.358e-05 2.358e-05 2.358e-05 0.00% Amr::FinalizeInit() 1 2.343e-05 2.343e-05 2.343e-05 0.00% Castro::do_old_sources() 10 2.14e-05 2.14e-05 2.14e-05 0.00% Castro::construct_new_source() 50 2.001e-05 2.001e-05 2.001e-05 0.00% Castro::construct_old_source() 50 1.815e-05 1.815e-05 1.815e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.787e-05 1.787e-05 1.787e-05 0.00% DistributionMapping::Distribute() 56 1.693e-05 1.693e-05 1.693e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.496e-05 1.496e-05 1.496e-05 0.00% Castro::check_for_nan() 20 1.357e-05 1.357e-05 1.357e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.357e-05 1.357e-05 1.357e-05 0.00% MLPoisson::prepareForSolve() 11 1.273e-05 1.273e-05 1.273e-05 0.00% Castro::construct_old_gravity() 10 1.186e-05 1.186e-05 1.186e-05 0.00% Castro::apply_source_to_state() 20 1.163e-05 1.163e-05 1.163e-05 0.00% MLLinOp::define() 11 1.119e-05 1.119e-05 1.119e-05 0.00% Castro::post_init() 1 9.831e-06 9.831e-06 9.831e-06 0.00% Amr::initSubcycle() 1 9.597e-06 9.597e-06 9.597e-06 0.00% Gravity::swapTimeLevels() 10 9.186e-06 9.186e-06 9.186e-06 0.00% MLMG::computeMLResidual() 11 8.475e-06 8.475e-06 8.475e-06 0.00% Gravity::actual_multilevel_solve() 1 8.183e-06 8.183e-06 8.183e-06 0.00% Castro::post_timestep() 10 7.978e-06 7.978e-06 7.978e-06 0.00% Castro::computeNewDt() 9 7.067e-06 7.067e-06 7.067e-06 0.00% Castro::expand_state() 10 5.759e-06 5.759e-06 5.759e-06 0.00% MLMG::getGradSolution() 11 5.685e-06 5.685e-06 5.685e-06 0.00% Castro::retry_advance_ctu() 10 5.597e-06 5.597e-06 5.597e-06 0.00% AmrLevel::checkPointPost() 3 4.769e-06 4.769e-06 4.769e-06 0.00% Amr::InitializeInit() 1 4.723e-06 4.723e-06 4.723e-06 0.00% MLMG::MLRhsNormInf() 11 4.108e-06 4.108e-06 4.108e-06 0.00% Gravity::set_mass_offset() 11 3.796e-06 3.796e-06 3.796e-06 0.00% MLMG::MLResNormInf() 11 3.475e-06 3.475e-06 3.475e-06 0.00% Castro::FluxRegCrseInit 10 3.439e-06 3.439e-06 3.439e-06 0.00% Castro::computeInitialDt() 2 3.072e-06 3.072e-06 3.072e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.858e-06 2.858e-06 2.858e-06 0.00% Castro::FluxRegFineAdd() 10 2.746e-06 2.746e-06 2.746e-06 0.00% Amr::init() 1 2.635e-06 2.635e-06 2.635e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.201e-06 2.201e-06 2.201e-06 0.00% AmrLevel::checkPointPre() 3 2.044e-06 2.044e-06 2.044e-06 0.00% Castro::post_regrid() 1 1.326e-06 1.326e-06 1.326e-06 0.00% Amr::initialInit() 1 1.212e-06 1.212e-06 1.212e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8898 0.8898 0.8898 100.00% Amr::coarseTimeStep() 10 0.7285 0.7285 0.7285 81.87% Amr::timeStep() 10 0.6275 0.6275 0.6275 70.52% Castro::advance() 10 0.6164 0.6164 0.6164 69.27% Castro::subcycle_advance_ctu() 10 0.601 0.601 0.601 67.54% Castro::do_advance_ctu() 10 0.6008 0.6008 0.6008 67.52% Gravity::solve_phi_with_mlmg() 11 0.2909 0.2909 0.2909 32.69% Gravity::actual_solve_with_mlmg() 11 0.2811 0.2811 0.2811 31.59% Castro::construct_ctu_hydro_source() 10 0.266 0.266 0.266 29.90% Castro::construct_new_gravity() 10 0.2622 0.2622 0.2622 29.47% MLMG::solve() 11 0.2594 0.2594 0.2594 29.15% Gravity::solve_for_phi() 10 0.2462 0.2462 0.2462 27.67% MLMG::oneIter() 82 0.2444 0.2444 0.2444 27.47% MLMG::mgVcycle() 82 0.2408 0.2408 0.2408 27.06% VisMF::Write(FabArray) 11 0.196 0.196 0.196 22.03% Amr::checkPoint() 3 0.1474 0.1474 0.1474 16.56% AmrLevel::checkPoint() 3 0.1426 0.1426 0.1426 16.03% StateData::checkPoint() 12 0.1425 0.1425 0.1425 16.02% Amr::init() 1 0.1326 0.1326 0.1326 14.90% MLCellLinOp::smooth() 1640 0.1177 0.1177 0.1177 13.23% MLCellLinOp::applyBC() 4433 0.1102 0.1102 0.1102 12.38% MLMG::mgVcycle_bottom 82 0.07522 0.07522 0.07522 8.45% MLMG::actualBottomSolve() 82 0.07518 0.07518 0.07518 8.45% MLCGSolver::bicgstab 82 0.07447 0.07447 0.07447 8.37% Castro::clean_state() 62 0.05897 0.05897 0.05897 6.63% Amr::writePlotFile() 2 0.0561 0.0561 0.0561 6.30% Amr::initialInit() 1 0.05393 0.05393 0.05393 6.06% Amr::FinalizeInit() 1 0.04892 0.04892 0.04892 5.50% AmrLevel::FillPatch() 41 0.04755 0.04755 0.04755 5.34% Castro::post_init() 1 0.04748 0.04748 0.04748 5.34% Gravity::multilevel_solve_for_new_phi() 1 0.04516 0.04516 0.04516 5.08% Gravity::actual_multilevel_solve() 1 0.04514 0.04514 0.04514 5.07% FillPatchIterator::Initialize 41 0.04335 0.04335 0.04335 4.87% FillPatchIterator::FillFromLevel0() 41 0.04179 0.04179 0.04179 4.70% FillPatchSingleLevel 41 0.04174 0.04174 0.04174 4.69% StateDataPhysBCFunct::() 41 0.03768 0.03768 0.03768 4.23% MLCellLinOp::apply() 1142 0.03698 0.03698 0.03698 4.16% MLMG::mgVcycle_down::0 82 0.03394 0.03394 0.03394 3.81% MLPoisson::Fsmooth() 3280 0.03281 0.03281 0.03281 3.69% FabArray::FillBoundary() 4023 0.03243 0.03243 0.03243 3.64% FillBoundary_nowait() 4023 0.03159 0.03159 0.03159 3.55% StateData::FillBoundary(geom) 328 0.02688 0.02688 0.02688 3.02% MLMG::mgVcycle_up::0 82 0.0255 0.0255 0.0255 2.87% Castro::computeTemp() 63 0.0235 0.0235 0.0235 2.64% MLCellLinOp::correctionResidual() 492 0.02248 0.02248 0.02248 2.53% Castro::initialize_do_advance() 10 0.02215 0.02215 0.02215 2.49% amrex::Dot() 1114 0.02102 0.02102 0.02102 2.36% Castro::normalize_species() 62 0.02034 0.02034 0.02034 2.29% MLMG:computeResOfCorrection() 410 0.01982 0.01982 0.01982 2.23% Castro::do_old_sources() 10 0.01965 0.01965 0.01965 2.21% Gravity::get_new_grav_vector() 11 0.01792 0.01792 0.01792 2.01% MLPoisson::define() 11 0.01716 0.01716 0.01716 1.93% MLMG::mgVcycle_down::1 82 0.01618 0.01618 0.01618 1.82% Castro::construct_old_gravity() 10 0.01529 0.01529 0.01529 1.72% Gravity::get_old_grav_vector() 10 0.01527 0.01527 0.01527 1.72% amrex::Copy() 1029 0.01505 0.01505 0.01505 1.69% MLMG::mgVcycle_down::2 82 0.01491 0.01491 0.01491 1.68% Castro::initialize_advance() 10 0.0147 0.0147 0.0147 1.65% FabArray::norminf() 743 0.01464 0.01464 0.01464 1.65% MLMG::mgVcycle_down::3 82 0.01461 0.01461 0.01461 1.64% FabArray::ParallelCopy() 861 0.01454 0.01454 0.01454 1.63% MLMG::mgVcycle_down::4 82 0.01442 0.01442 0.01442 1.62% FabArray::ParallelCopy_nowait() 861 0.01428 0.01428 0.01428 1.61% FabArray::setVal() 1144 0.01347 0.01347 0.01347 1.51% MLCGSolver::ParallelAllReduce 1514 0.01258 0.01258 0.01258 1.41% Castro::do_new_sources() 10 0.01212 0.01212 0.01212 1.36% MLMG::addInterpCorrection() 410 0.01211 0.01211 0.01211 1.36% Castro::expand_state() 10 0.01182 0.01182 0.01182 1.33% MLCellLinOp::defineAuxData() 11 0.01175 0.01175 0.01175 1.32% MLMG::mgVcycle_up::1 82 0.01166 0.01166 0.01166 1.31% MLMG::mgVcycle_up::4 82 0.01163 0.01163 0.01163 1.31% Castro::enforce_min_density() 62 0.01146 0.01146 0.01146 1.29% amrex::average_down 410 0.01143 0.01143 0.01143 1.28% MLMG::mgVcycle_up::2 82 0.01139 0.01139 0.01139 1.28% MLMG::mgVcycle_up::3 82 0.01115 0.01115 0.01115 1.25% Castro::post_timestep() 10 0.01091 0.01091 0.01091 1.23% MLPoisson::Fapply() 1142 0.01057 0.01057 0.01057 1.19% Gravity::fill_multipole_BCs() 11 0.009552 0.009552 0.009552 1.07% FabArray::Saxpy() 813 0.008363 0.008363 0.008363 0.94% FabArray::Xpay() 821 0.00826 0.00826 0.00826 0.93% MLCellLinOp::solutionResidual() 93 0.007504 0.007504 0.007504 0.84% Castro::reset_internal_energy(MultiFab) 63 0.006929 0.006929 0.006929 0.78% Castro::estTimeStep() 21 0.006263 0.006263 0.006263 0.70% MLMG::computeResidual() 82 0.006231 0.006231 0.006231 0.70% MLCellLinOp::defineBC() 11 0.005167 0.005167 0.005167 0.58% Amr::InitializeInit() 1 0.005013 0.005013 0.005013 0.56% Amr::defBaseLevel() 1 0.005008 0.005008 0.005008 0.56% MLMG::prepareForSolve() 11 0.004974 0.004974 0.004974 0.56% BndryData::define() 11 0.004928 0.004928 0.004928 0.55% FabArray::LinComb() 557 0.004606 0.004606 0.004606 0.52% Castro::enforce_speed_limit() 62 0.00446 0.00446 0.00446 0.50% amrex::Add() 164 0.004401 0.004401 0.004401 0.49% Castro::initData() 1 0.0043 0.0043 0.0043 0.48% Castro::construct_new_source() 50 0.003364 0.003364 0.003364 0.38% Castro::construct_new_gravity_source() 10 0.003344 0.003344 0.003344 0.38% Castro::construct_old_source() 50 0.002918 0.002918 0.002918 0.33% Castro::construct_old_gravity_source() 10 0.0029 0.0029 0.0029 0.33% Castro::computeNewDt() 9 0.002832 0.002832 0.002832 0.32% Castro::finalize_do_advance() 10 0.002323 0.002323 0.002323 0.26% MLMG::ResNormInf() 93 0.002159 0.002159 0.002159 0.24% Castro::apply_source_to_state() 20 0.001858 0.001858 0.001858 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001806 0.001806 0.001806 0.20% check_for_negative_density() 10 0.001755 0.001755 0.001755 0.20% Castro::reset_internal_energy(Fab) 504 0.001728 0.001728 0.001728 0.19% MLMG::getGradSolution() 11 0.001531 0.001531 0.001531 0.17% MLCellLinOp::compGrad() 11 0.001525 0.001525 0.001525 0.17% MLCellLinOp::setLevelBC() 11 0.001468 0.001468 0.001468 0.17% FabArrayBase::getCPC() 1323 0.001433 0.001433 0.001433 0.16% FabArray::mult() 43 0.001368 0.001368 0.001368 0.15% FabArray::setDomainBndry() 41 0.001352 0.001352 0.001352 0.15% MLMG::computeMLResidual() 11 0.001316 0.001316 0.001316 0.15% MLPoisson::prepareForSolve() 11 0.001294 0.001294 0.001294 0.15% Castro::check_for_nan() 20 0.00129 0.00129 0.00129 0.14% MLCellLinOp::prepareForSolve() 11 0.001281 0.001281 0.001281 0.14% MultiFab::contains_nan() 20 0.001277 0.001277 0.001277 0.14% Castro::computeInitialDt() 2 0.001163 0.001163 0.001163 0.13% Castro::post_regrid() 1 0.001132 0.001132 0.001132 0.13% Gravity::update_max_rhs() 11 0.001001 0.001001 0.001001 0.11% FabArrayBase::getFB() 4023 0.0007284 0.0007284 0.0007284 0.08% FabArrayBase::CPC::define() 454 0.0006818 0.0006818 0.0006818 0.08% Castro::finalize_advance() 10 0.0006139 0.0006139 0.0006139 0.07% Castro::Castro() 1 0.0006127 0.0006127 0.0006127 0.07% Amr::InitAmr() 1 0.0004985 0.0004985 0.0004985 0.06% Gravity::swapTimeLevels() 10 0.0004464 0.0004464 0.0004464 0.05% MLMG::MLResNormInf() 11 0.0003402 0.0003402 0.0003402 0.04% MultiFab::max() 11 0.0003303 0.0003303 0.0003303 0.04% Castro::buildMetrics() 1 0.0002895 0.0002895 0.0002895 0.03% MLLinOp::define() 11 0.000222 0.000222 0.000222 0.02% MLMG::MLRhsNormInf() 11 0.0002199 0.0002199 0.0002199 0.02% MLLinOp::defineGrids() 11 0.0002108 0.0002108 0.0002108 0.02% Castro::create_source_corrector() 10 0.0001416 0.0001416 0.0001416 0.02% FabArrayBase::FB::FB() 56 9.197e-05 9.197e-05 9.197e-05 0.01% Castro::enforce_consistent_e() 1 7.185e-05 7.185e-05 7.185e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.495e-05 5.495e-05 5.495e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.088e-05 5.088e-05 5.088e-05 0.01% makeSFC 55 4.139e-05 4.139e-05 4.139e-05 0.00% Castro::swap_state_time_levels() 10 3.627e-05 3.627e-05 3.627e-05 0.00% StateData::define() 4 3.592e-05 3.592e-05 3.592e-05 0.00% Castro::initMFs() 1 2.94e-05 2.94e-05 2.94e-05 0.00% Amr::writeSmallPlotFile() 1 2.637e-05 2.637e-05 2.637e-05 0.00% DistributionMapping::Distribute() 56 1.693e-05 1.693e-05 1.693e-05 0.00% Amr::initSubcycle() 1 9.597e-06 9.597e-06 9.597e-06 0.00% Castro::retry_advance_ctu() 10 5.597e-06 5.597e-06 5.597e-06 0.00% AmrLevel::checkPointPost() 3 4.769e-06 4.769e-06 4.769e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.102e-06 4.102e-06 4.102e-06 0.00% Gravity::set_mass_offset() 11 3.796e-06 3.796e-06 3.796e-06 0.00% Castro::FluxRegCrseInit 10 3.439e-06 3.439e-06 3.439e-06 0.00% Castro::FluxRegFineAdd() 10 2.746e-06 2.746e-06 2.746e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.201e-06 2.201e-06 2.201e-06 0.00% AmrLevel::checkPointPre() 3 2.044e-06 2.044e-06 2.044e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 94 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 134 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1037 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1774 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7618 KiB 14 MiB MLMG::prepareForSolve() 660 660 3586 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 209 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 175 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7516 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 18 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2179 B 2048 KiB Gravity::solve_for_phi() 80 80 565 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 103 KiB 2048 KiB BndryData::define() 1056 1056 327 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 208 KiB 671 KiB Castro::estTimeStep() 21 21 3434 B 480 KiB VisMF::Write(FabArray) 656 656 3438 B 320 KiB Castro::normalize_species() 62 62 7448 B 320 KiB amrex::average_down 1067 1067 1280 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1157 B 257 KiB amrex::Dot() 1360 1360 3471 B 160 KiB FabArray::norminf() 907 907 2430 B 160 KiB check_for_negative_density() 10 10 310 B 160 KiB Castro::initData() 1 1 53 B 160 KiB MultiFab::max() 11 11 57 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 28 B 20 KiB MLPoisson::Fsmooth() 132 132 3500 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 303 B 9648 B MLCellLinOp::applyBC() 8866 8866 225 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3909 B 6144 B StateData::FillBoundary(geom) 1992 1992 40 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 121 B 1472 B MLCellLinOp::defineBC() 66 66 369 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 534 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 38 KiB 8192 KiB VisMF::Write(FabArray) 744 744 500 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3500 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 303 B 9648 B MLCellLinOp::applyBC() 4433 4433 223 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3909 B 6144 B Gravity::get_new_grav_vector() 3 3 2887 B 3072 B StateData::FillBoundary(geom) 1992 1992 41 B 3024 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 271 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 298 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 907 907 17 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-30-g3396b1df1175) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-30-g3396b1df1175) initialized Starting run at 08:10:23 UTC on 2023-08-16. Successfully read inputs file ... Castro git describe: 23.08-5-g5303a2cd5 AMReX git describe: 23.08-30-g3396b1df1 Microphysics git describe: 23.08-11-g9ecfd8d2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.480782994 Restart time = 0.048567505 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.054220868 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.058357445 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072732493 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.082625376 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.073212008 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.047688358 seconds Ending run at 08:10:24 UTC on 2023-08-16. Run time = 0.43842095 Run time without initialization = 0.389288586 Average number of zones advanced per microsecond: 3.367 Average number of zones advanced per microsecond per rank: 3.367 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.4384 ... 0.4384 ... 0.4384 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1495 0.1495 0.1495 34.10% VisMF::Read() 3 0.03736 0.03736 0.03736 8.52% MLCellLinOp::applyBC() 1946 0.03417 0.03417 0.03417 7.79% VisMF::Write(FabArray) 1 0.02796 0.02796 0.02796 6.38% Amr::writePlotFile() 1 0.01954 0.01954 0.01954 4.46% MLPoisson::Fsmooth() 1440 0.01436 0.01436 0.01436 3.28% StateData::FillBoundary(geom) 160 0.01301 0.01301 0.01301 2.97% FillBoundary_nowait() 1766 0.01291 0.01291 0.01291 2.94% Castro::normalize_species() 30 0.00934 0.00934 0.00934 2.13% amrex::Dot() 484 0.009184 0.009184 0.009184 2.09% amrex::Copy() 463 0.007285 0.007285 0.007285 1.66% Castro::computeTemp() 30 0.007251 0.007251 0.007251 1.65% FabArray::setVal() 537 0.006593 0.006593 0.006593 1.50% FabArray::norminf() 326 0.006485 0.006485 0.006485 1.48% FabArray::ParallelCopy_nowait() 380 0.006186 0.006186 0.006186 1.41% Castro::enforce_min_density() 30 0.005842 0.005842 0.005842 1.33% Gravity::fill_multipole_BCs() 6 0.005618 0.005618 0.005618 1.28% MLCellLinOp::defineAuxData() 6 0.005594 0.005594 0.005594 1.28% StateDataPhysBCFunct::() 20 0.004707 0.004707 0.004707 1.07% MLPoisson::Fapply() 500 0.004656 0.004656 0.004656 1.06% Amr::restart() 1 0.00459 0.00459 0.00459 1.05% FabArray::Saxpy() 355 0.003788 0.003788 0.003788 0.86% FabArray::Xpay() 361 0.003643 0.003643 0.003643 0.83% Castro::estTimeStep() 10 0.003163 0.003163 0.003163 0.72% MLMG::addInterpCorrection() 180 0.00309 0.00309 0.00309 0.70% amrex::average_down 180 0.002741 0.002741 0.002741 0.63% FabArray::LinComb() 242 0.002275 0.002275 0.002275 0.52% Castro::reset_internal_energy(MultiFab) 30 0.002156 0.002156 0.002156 0.49% BndryData::define() 6 0.002131 0.002131 0.002131 0.49% amrex::Add() 72 0.001896 0.001896 0.001896 0.43% Castro::reset_internal_energy(Fab) 240 0.001646 0.001646 0.001646 0.38% Castro::construct_new_gravity_source() 5 0.001547 0.001547 0.001547 0.35% Castro::construct_old_gravity_source() 5 0.001211 0.001211 0.001211 0.28% Castro::enforce_speed_limit() 30 0.001118 0.001118 0.001118 0.25% MLCGSolver::bicgstab 36 0.001024 0.001024 0.001024 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009705 0.0009705 0.0009705 0.22% check_for_negative_density() 5 0.0009401 0.0009401 0.0009401 0.21% Castro::initialize_do_advance() 5 0.0008333 0.0008333 0.0008333 0.19% Gravity::actual_solve_with_mlmg() 6 0.0008167 0.0008167 0.0008167 0.19% MLCellLinOp::setLevelBC() 6 0.0008049 0.0008049 0.0008049 0.18% MLCellLinOp::prepareForSolve() 6 0.0007114 0.0007114 0.0007114 0.16% FabArray::setDomainBndry() 20 0.0006906 0.0006906 0.0006906 0.16% FabArray::mult() 22 0.0006894 0.0006894 0.0006894 0.16% MultiFab::contains_nan() 10 0.000653 0.000653 0.000653 0.15% MLCellLinOp::compGrad() 6 0.0005792 0.0005792 0.0005792 0.13% MLMG::prepareForSolve() 6 0.0005377 0.0005377 0.0005377 0.12% MLCellLinOp::smooth() 720 0.0004918 0.0004918 0.0004918 0.11% FabArrayBase::CPC::define() 244 0.0004108 0.0004108 0.0004108 0.09% Amr::InitAmr() 1 0.000407 0.000407 0.000407 0.09% FabArray::FillBoundary() 1766 0.0003574 0.0003574 0.0003574 0.08% FabArrayBase::getCPC() 632 0.0003492 0.0003492 0.0003492 0.08% Gravity::get_old_grav_vector() 5 0.0003358 0.0003358 0.0003358 0.08% FabArrayBase::getFB() 1766 0.0002964 0.0002964 0.0002964 0.07% Gravity::get_new_grav_vector() 5 0.0002619 0.0002619 0.0002619 0.06% main() 1 0.0002523 0.0002523 0.0002523 0.06% MLCellLinOp::apply() 500 0.0002087 0.0002087 0.0002087 0.05% MultiFab::max() 6 0.0002059 0.0002059 0.0002059 0.05% Amr::coarseTimeStep() 5 0.0001829 0.0001829 0.0001829 0.04% AmrLevel::FillPatch() 20 0.0001644 0.0001644 0.0001644 0.04% MLCGSolver::ParallelAllReduce 659 0.0001364 0.0001364 0.0001364 0.03% MLCellLinOp::defineBC() 6 0.0001201 0.0001201 0.0001201 0.03% FabArray::ParallelCopy() 380 0.0001153 0.0001153 0.0001153 0.03% FillPatchIterator::Initialize 20 0.0001053 0.0001053 0.0001053 0.02% Castro::subcycle_advance_ctu() 5 9.826e-05 9.826e-05 9.826e-05 0.02% MLCellLinOp::correctionResidual() 216 9.414e-05 9.414e-05 9.414e-05 0.02% MLMG::mgVcycle() 36 8.824e-05 8.824e-05 8.824e-05 0.02% MLLinOp::defineGrids() 6 8.754e-05 8.754e-05 8.754e-05 0.02% Castro::finalize_do_advance() 5 8.292e-05 8.292e-05 8.292e-05 0.02% Amr::timeStep() 5 8.017e-05 8.017e-05 8.017e-05 0.02% Castro::create_source_corrector() 5 7.565e-05 7.565e-05 7.565e-05 0.02% Castro::advance() 5 7.525e-05 7.525e-05 7.525e-05 0.02% Castro::construct_new_source() 25 7.515e-05 7.515e-05 7.515e-05 0.02% AmrLevel::restart() 1 6.873e-05 6.873e-05 6.873e-05 0.02% Gravity::solve_for_phi() 5 6.821e-05 6.821e-05 6.821e-05 0.02% Gravity::update_max_rhs() 6 6.394e-05 6.394e-05 6.394e-05 0.01% Castro::do_advance_ctu() 5 6.142e-05 6.142e-05 6.142e-05 0.01% StateData::restartDoit() 4 6.003e-05 6.003e-05 6.003e-05 0.01% FabArrayBase::FB::FB() 26 5.684e-05 5.684e-05 5.684e-05 0.01% MLMG:computeResOfCorrection() 180 5.535e-05 5.535e-05 5.535e-05 0.01% MLMG::mgVcycle_down::0 36 4.301e-05 4.301e-05 4.301e-05 0.01% MLMG::actualBottomSolve() 36 4.201e-05 4.201e-05 4.201e-05 0.01% MLMG::solve() 6 4.11e-05 4.11e-05 4.11e-05 0.01% MLMG::mgVcycle_down::1 36 3.632e-05 3.632e-05 3.632e-05 0.01% Castro::initialize_advance() 5 3.548e-05 3.548e-05 3.548e-05 0.01% MLMG::mgVcycle_down::2 36 3.521e-05 3.521e-05 3.521e-05 0.01% MLMG::mgVcycle_down::4 36 3.435e-05 3.435e-05 3.435e-05 0.01% Castro::clean_state() 30 3.371e-05 3.371e-05 3.371e-05 0.01% MLMG::mgVcycle_down::3 36 3.261e-05 3.261e-05 3.261e-05 0.01% Castro::do_old_sources() 5 3.152e-05 3.152e-05 3.152e-05 0.01% Castro::finalize_advance() 5 3.052e-05 3.052e-05 3.052e-05 0.01% Castro::post_timestep() 5 2.954e-05 2.954e-05 2.954e-05 0.01% MLMG::mgVcycle_up::4 36 2.91e-05 2.91e-05 2.91e-05 0.01% Amr::writeSmallPlotFile() 1 2.905e-05 2.905e-05 2.905e-05 0.01% Castro::construct_old_source() 25 2.773e-05 2.773e-05 2.773e-05 0.01% Castro::post_restart() 1 2.729e-05 2.729e-05 2.729e-05 0.01% Castro::swap_state_time_levels() 5 2.54e-05 2.54e-05 2.54e-05 0.01% Castro::buildMetrics() 1 2.536e-05 2.536e-05 2.536e-05 0.01% MLMG::mgVcycle_up::2 36 2.476e-05 2.476e-05 2.476e-05 0.01% Castro::initMFs() 1 2.454e-05 2.454e-05 2.454e-05 0.01% MLCellLinOp::solutionResidual() 42 2.317e-05 2.317e-05 2.317e-05 0.01% MLMG::mgVcycle_up::3 36 2.268e-05 2.268e-05 2.268e-05 0.01% MLMG::mgVcycle_up::0 36 2.248e-05 2.248e-05 2.248e-05 0.01% MLMG::oneIter() 36 2.238e-05 2.238e-05 2.238e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.097e-05 2.097e-05 2.097e-05 0.00% MLMG::mgVcycle_up::1 36 2.08e-05 2.08e-05 2.08e-05 0.00% MLMG::ResNormInf() 42 1.66e-05 1.66e-05 1.66e-05 0.00% MLMG::computeResidual() 36 1.606e-05 1.606e-05 1.606e-05 0.00% FillPatchSingleLevel 20 1.602e-05 1.602e-05 1.602e-05 0.00% MLMG::mgVcycle_bottom 36 1.583e-05 1.583e-05 1.583e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.566e-05 1.566e-05 1.566e-05 0.00% MLPoisson::define() 6 1.44e-05 1.44e-05 1.44e-05 0.00% makeSFC 30 1.381e-05 1.381e-05 1.381e-05 0.00% Castro::construct_new_gravity() 5 1.347e-05 1.347e-05 1.347e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.284e-05 1.284e-05 1.284e-05 0.00% DistributionMapping::Distribute() 31 1.041e-05 1.041e-05 1.041e-05 0.00% Castro::do_new_sources() 5 1.032e-05 1.032e-05 1.032e-05 0.00% Amr::initSubcycle() 1 9.14e-06 9.14e-06 9.14e-06 0.00% MLPoisson::prepareForSolve() 6 9.133e-06 9.133e-06 9.133e-06 0.00% Gravity::actual_multilevel_solve() 1 8.007e-06 8.007e-06 8.007e-06 0.00% Castro::check_for_nan() 10 7.318e-06 7.318e-06 7.318e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.954e-06 6.954e-06 6.954e-06 0.00% Castro::apply_source_to_state() 10 6.055e-06 6.055e-06 6.055e-06 0.00% Castro::construct_old_gravity() 5 6.005e-06 6.005e-06 6.005e-06 0.00% MLLinOp::define() 6 5.607e-06 5.607e-06 5.607e-06 0.00% Castro::computeNewDt() 5 4.054e-06 4.054e-06 4.054e-06 0.00% Gravity::swapTimeLevels() 5 4.021e-06 4.021e-06 4.021e-06 0.00% MLMG::computeMLResidual() 6 3.84e-06 3.84e-06 3.84e-06 0.00% MLMG::getGradSolution() 6 3.007e-06 3.007e-06 3.007e-06 0.00% Castro::expand_state() 5 2.946e-06 2.946e-06 2.946e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.854e-06 2.854e-06 2.854e-06 0.00% MLMG::MLResNormInf() 6 2.387e-06 2.387e-06 2.387e-06 0.00% MLMG::MLRhsNormInf() 6 2.251e-06 2.251e-06 2.251e-06 0.00% Castro::retry_advance_ctu() 5 2.191e-06 2.191e-06 2.191e-06 0.00% Gravity::set_mass_offset() 6 1.993e-06 1.993e-06 1.993e-06 0.00% Castro::FluxRegCrseInit 5 1.481e-06 1.481e-06 1.481e-06 0.00% Castro::FluxRegFineAdd() 5 1.363e-06 1.363e-06 1.363e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.203e-06 1.203e-06 1.203e-06 0.00% Amr::init() 1 8.83e-07 8.83e-07 8.83e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4384 0.4384 0.4384 100.00% Amr::coarseTimeStep() 5 0.3413 0.3413 0.3413 77.85% Amr::timeStep() 5 0.3391 0.3391 0.3391 77.35% Castro::advance() 5 0.3335 0.3335 0.3335 76.07% Castro::subcycle_advance_ctu() 5 0.3262 0.3262 0.3262 74.41% Castro::do_advance_ctu() 5 0.3261 0.3261 0.3261 74.39% Castro::construct_ctu_hydro_source() 5 0.1555 0.1555 0.1555 35.47% Castro::construct_new_gravity() 5 0.1344 0.1344 0.1344 30.66% Gravity::solve_phi_with_mlmg() 6 0.1319 0.1319 0.1319 30.08% Gravity::solve_for_phi() 5 0.1264 0.1264 0.1264 28.82% Gravity::actual_solve_with_mlmg() 6 0.126 0.126 0.126 28.75% MLMG::solve() 6 0.1142 0.1142 0.1142 26.05% MLMG::oneIter() 36 0.1068 0.1068 0.1068 24.36% MLMG::mgVcycle() 36 0.1053 0.1053 0.1053 24.01% MLCellLinOp::smooth() 720 0.05093 0.05093 0.05093 11.62% Amr::init() 1 0.04862 0.04862 0.04862 11.09% Amr::restart() 1 0.04861 0.04861 0.04861 11.09% Amr::writePlotFile() 1 0.04779 0.04779 0.04779 10.90% MLCellLinOp::applyBC() 1946 0.04779 0.04779 0.04779 10.90% AmrLevel::restart() 1 0.03769 0.03769 0.03769 8.60% StateData::restartDoit() 4 0.03762 0.03762 0.03762 8.58% VisMF::Read() 3 0.03736 0.03736 0.03736 8.52% MLMG::mgVcycle_bottom 36 0.03322 0.03322 0.03322 7.58% MLMG::actualBottomSolve() 36 0.0332 0.0332 0.0332 7.57% MLCGSolver::bicgstab 36 0.03289 0.03289 0.03289 7.50% VisMF::Write(FabArray) 1 0.02796 0.02796 0.02796 6.38% Castro::clean_state() 30 0.02739 0.02739 0.02739 6.25% AmrLevel::FillPatch() 20 0.02262 0.02262 0.02262 5.16% FillPatchIterator::Initialize 20 0.02056 0.02056 0.02056 4.69% FillPatchIterator::FillFromLevel0() 20 0.01977 0.01977 0.01977 4.51% FillPatchSingleLevel 20 0.01975 0.01975 0.01975 4.50% StateDataPhysBCFunct::() 20 0.01771 0.01771 0.01771 4.04% MLCellLinOp::apply() 500 0.01631 0.01631 0.01631 3.72% MLMG::mgVcycle_down::0 36 0.01467 0.01467 0.01467 3.35% MLPoisson::Fsmooth() 1440 0.01436 0.01436 0.01436 3.28% FabArray::FillBoundary() 1766 0.01362 0.01362 0.01362 3.11% FillBoundary_nowait() 1766 0.01327 0.01327 0.01327 3.03% StateData::FillBoundary(geom) 160 0.01301 0.01301 0.01301 2.97% Castro::initialize_do_advance() 5 0.01147 0.01147 0.01147 2.62% Castro::computeTemp() 30 0.01105 0.01105 0.01105 2.52% MLMG::mgVcycle_up::0 36 0.01088 0.01088 0.01088 2.48% MLCellLinOp::correctionResidual() 216 0.009791 0.009791 0.009791 2.23% Castro::normalize_species() 30 0.00934 0.00934 0.00934 2.13% MLPoisson::define() 6 0.009327 0.009327 0.009327 2.13% amrex::Dot() 484 0.009184 0.009184 0.009184 2.09% MLMG:computeResOfCorrection() 180 0.008608 0.008608 0.008608 1.96% Castro::do_old_sources() 5 0.008576 0.008576 0.008576 1.96% Gravity::get_new_grav_vector() 5 0.007929 0.007929 0.007929 1.81% Castro::construct_old_gravity() 5 0.007502 0.007502 0.007502 1.71% Gravity::get_old_grav_vector() 5 0.007496 0.007496 0.007496 1.71% amrex::Copy() 463 0.007285 0.007285 0.007285 1.66% MLMG::mgVcycle_down::1 36 0.007134 0.007134 0.007134 1.63% Castro::initialize_advance() 5 0.006906 0.006906 0.006906 1.58% Castro::do_new_sources() 5 0.00687 0.00687 0.00687 1.57% FabArray::ParallelCopy() 380 0.006689 0.006689 0.006689 1.53% FabArray::setVal() 537 0.006593 0.006593 0.006593 1.50% FabArray::ParallelCopy_nowait() 380 0.006573 0.006573 0.006573 1.50% MLMG::mgVcycle_down::2 36 0.006508 0.006508 0.006508 1.48% FabArray::norminf() 326 0.006485 0.006485 0.006485 1.48% MLCellLinOp::defineAuxData() 6 0.006364 0.006364 0.006364 1.45% MLMG::mgVcycle_down::3 36 0.006338 0.006338 0.006338 1.45% MLMG::mgVcycle_down::4 36 0.006304 0.006304 0.006304 1.44% Castro::post_restart() 1 0.006155 0.006155 0.006155 1.40% Castro::enforce_min_density() 30 0.005842 0.005842 0.005842 1.33% Castro::expand_state() 5 0.005807 0.005807 0.005807 1.32% Gravity::multilevel_solve_for_new_phi() 1 0.005788 0.005788 0.005788 1.32% Gravity::actual_multilevel_solve() 1 0.005772 0.005772 0.005772 1.32% Gravity::fill_multipole_BCs() 6 0.005743 0.005743 0.005743 1.31% Castro::post_timestep() 5 0.005547 0.005547 0.005547 1.27% MLCGSolver::ParallelAllReduce 659 0.00553 0.00553 0.00553 1.26% MLMG::addInterpCorrection() 180 0.005417 0.005417 0.005417 1.24% MLMG::mgVcycle_up::4 36 0.005141 0.005141 0.005141 1.17% MLMG::mgVcycle_up::1 36 0.005092 0.005092 0.005092 1.16% amrex::average_down 180 0.005088 0.005088 0.005088 1.16% MLMG::mgVcycle_up::2 36 0.005011 0.005011 0.005011 1.14% MLMG::mgVcycle_up::3 36 0.004889 0.004889 0.004889 1.12% MLPoisson::Fapply() 500 0.004656 0.004656 0.004656 1.06% Castro::reset_internal_energy(MultiFab) 30 0.003801 0.003801 0.003801 0.87% FabArray::Saxpy() 355 0.003788 0.003788 0.003788 0.86% FabArray::Xpay() 361 0.003643 0.003643 0.003643 0.83% MLCellLinOp::solutionResidual() 42 0.003497 0.003497 0.003497 0.80% Castro::estTimeStep() 10 0.003163 0.003163 0.003163 0.72% MLCellLinOp::defineBC() 6 0.002824 0.002824 0.002824 0.64% MLMG::prepareForSolve() 6 0.002755 0.002755 0.002755 0.63% MLMG::computeResidual() 36 0.002735 0.002735 0.002735 0.62% BndryData::define() 6 0.002704 0.002704 0.002704 0.62% FabArray::LinComb() 242 0.002275 0.002275 0.002275 0.52% Castro::computeNewDt() 5 0.001993 0.001993 0.001993 0.45% amrex::Add() 72 0.001896 0.001896 0.001896 0.43% Castro::reset_internal_energy(Fab) 240 0.001646 0.001646 0.001646 0.38% Castro::construct_new_source() 25 0.001622 0.001622 0.001622 0.37% Castro::construct_new_gravity_source() 5 0.001547 0.001547 0.001547 0.35% Castro::finalize_do_advance() 5 0.001257 0.001257 0.001257 0.29% Castro::construct_old_source() 25 0.001238 0.001238 0.001238 0.28% Castro::construct_old_gravity_source() 5 0.001211 0.001211 0.001211 0.28% Castro::enforce_speed_limit() 30 0.001118 0.001118 0.001118 0.25% MLMG::ResNormInf() 42 0.000996 0.000996 0.000996 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009705 0.0009705 0.0009705 0.22% Castro::apply_source_to_state() 10 0.000954 0.000954 0.000954 0.22% check_for_negative_density() 5 0.0009401 0.0009401 0.0009401 0.21% MLMG::getGradSolution() 6 0.0008581 0.0008581 0.0008581 0.20% MLCellLinOp::compGrad() 6 0.0008551 0.0008551 0.0008551 0.20% MLCellLinOp::setLevelBC() 6 0.0008049 0.0008049 0.0008049 0.18% MLMG::computeMLResidual() 6 0.0007819 0.0007819 0.0007819 0.18% FabArrayBase::getCPC() 632 0.00076 0.00076 0.00076 0.17% MLPoisson::prepareForSolve() 6 0.0007205 0.0007205 0.0007205 0.16% MLCellLinOp::prepareForSolve() 6 0.0007114 0.0007114 0.0007114 0.16% Gravity::update_max_rhs() 6 0.0007083 0.0007083 0.0007083 0.16% FabArray::setDomainBndry() 20 0.0006906 0.0006906 0.0006906 0.16% FabArray::mult() 22 0.0006894 0.0006894 0.0006894 0.16% Castro::check_for_nan() 10 0.0006604 0.0006604 0.0006604 0.15% MultiFab::contains_nan() 10 0.000653 0.000653 0.000653 0.15% Amr::InitAmr() 1 0.0004162 0.0004162 0.0004162 0.09% FabArrayBase::CPC::define() 244 0.0004108 0.0004108 0.0004108 0.09% FabArrayBase::getFB() 1766 0.0003532 0.0003532 0.0003532 0.08% Castro::finalize_advance() 5 0.0002966 0.0002966 0.0002966 0.07% Gravity::swapTimeLevels() 5 0.0002325 0.0002325 0.0002325 0.05% MultiFab::max() 6 0.0002059 0.0002059 0.0002059 0.05% MLMG::MLResNormInf() 6 0.0001883 0.0001883 0.0001883 0.04% Castro::buildMetrics() 1 0.0001505 0.0001505 0.0001505 0.03% MLLinOp::define() 6 0.0001243 0.0001243 0.0001243 0.03% MLMG::MLRhsNormInf() 6 0.0001193 0.0001193 0.0001193 0.03% MLLinOp::defineGrids() 6 0.0001187 0.0001187 0.0001187 0.03% Castro::create_source_corrector() 5 7.565e-05 7.565e-05 7.565e-05 0.02% FabArrayBase::FB::FB() 26 5.684e-05 5.684e-05 5.684e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.992e-05 2.992e-05 2.992e-05 0.01% Amr::writeSmallPlotFile() 1 2.905e-05 2.905e-05 2.905e-05 0.01% Castro::swap_state_time_levels() 5 2.54e-05 2.54e-05 2.54e-05 0.01% Castro::initMFs() 1 2.454e-05 2.454e-05 2.454e-05 0.01% makeSFC 30 2.297e-05 2.297e-05 2.297e-05 0.01% DistributionMapping::Distribute() 31 1.041e-05 1.041e-05 1.041e-05 0.00% Amr::initSubcycle() 1 9.14e-06 9.14e-06 9.14e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.102e-06 4.102e-06 4.102e-06 0.00% Castro::retry_advance_ctu() 5 2.191e-06 2.191e-06 2.191e-06 0.00% Gravity::set_mass_offset() 6 1.993e-06 1.993e-06 1.993e-06 0.00% Castro::FluxRegCrseInit 5 1.481e-06 1.481e-06 1.481e-06 0.00% Castro::FluxRegFineAdd() 5 1.363e-06 1.363e-06 1.363e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.203e-06 1.203e-06 1.203e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 191 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 155 MiB 692 MiB Castro::initMFs() 48 48 61 MiB 68 MiB Castro::swap_state_time_levels() 32 32 49 MiB 55 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB FillPatchIterator::Initialize 160 160 1022 KiB 39 MiB Castro::initialize_do_advance() 40 40 29 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1886 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6836 KiB 14 MiB MLMG::prepareForSolve() 361 361 3198 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 177 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 185 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6823 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 22 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3210 B 2048 KiB Gravity::solve_for_phi() 40 40 589 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 26 KiB 2048 KiB BndryData::define() 576 576 294 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 189 KiB 671 KiB Castro::estTimeStep() 10 10 3468 B 480 KiB VisMF::Write(FabArray) 112 112 1271 B 320 KiB Castro::normalize_species() 30 30 6934 B 320 KiB amrex::average_down 469 469 1168 B 257 KiB MLMG::addInterpCorrection() 468 468 1043 B 257 KiB amrex::Dot() 592 592 3069 B 160 KiB FabArray::norminf() 398 398 2173 B 160 KiB check_for_negative_density() 5 5 343 B 160 KiB MultiFab::max() 6 6 73 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3101 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 44 B 10 KiB FillBoundary_nowait() 336 336 255 B 9648 B MLCellLinOp::applyBC() 3892 3892 202 B 9344 B amrex::Copy() 56 56 5961 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 37 B 2464 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 324 324 109 B 1472 B MLCellLinOp::defineBC() 36 36 331 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1106 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 78 KiB 8192 KiB VisMF::Write(FabArray) 120 120 184 KiB 3584 KiB VisMF::Read() 24 24 169 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3101 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 44 B 10 KiB FillBoundary_nowait() 336 336 255 B 9648 B MLCellLinOp::applyBC() 1946 1946 201 B 9328 B amrex::Copy() 56 56 5961 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2687 B 3072 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B StateData::FillBoundary(geom) 960 960 37 B 2464 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 508 B 1296 B amrex::average_down 37 37 227 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 22 B 400 B FabArray::norminf() 398 398 15 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-30-g3396b1df1175) finalized