Initializing CUDA... CUDA initialized with 1 device. AMReX (23.07-52-g98d22d21af45) initialized Starting run at 07:36:50 UTC on 2023-07-31. Successfully read inputs file ... Castro git describe: 23.07-20-g4174f17eb AMReX git describe: 23.07-52-g98d22d21a Microphysics git describe: 23.07-48-g588beb9c reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058054242 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.03312481 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047438274 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048825012 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.071449125 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.073886804 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.067396701 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057188748 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.06651719 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.070323456 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072106232 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066127007 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067811143 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057379648 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033143245 seconds Ending run at 07:36:51 UTC on 2023-07-31. Run time = 0.939428401 Run time without initialization = 0.800319444 Average number of zones advanced per microsecond: 3.275 Average number of zones advanced per microsecond per rank: 3.275 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9395 ... 0.9395 ... 0.9395 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2945 0.2945 0.2945 31.35% VisMF::Write(FabArray) 11 0.232 0.232 0.232 24.69% MLCellLinOp::applyBC() 4433 0.07456 0.07456 0.07456 7.94% MLPoisson::Fsmooth() 3280 0.03138 0.03138 0.03138 3.34% FillBoundary_nowait() 4023 0.03059 0.03059 0.03059 3.26% StateData::FillBoundary(geom) 328 0.02413 0.02413 0.02413 2.57% amrex::Dot() 1114 0.01979 0.01979 0.01979 2.11% Castro::normalize_species() 62 0.01888 0.01888 0.01888 2.01% amrex::Copy() 1029 0.0145 0.0145 0.0145 1.54% FabArray::norminf() 743 0.01386 0.01386 0.01386 1.48% Castro::computeTemp() 63 0.0138 0.0138 0.0138 1.47% FabArray::ParallelCopy_nowait() 861 0.01305 0.01305 0.01305 1.39% FabArray::setVal() 1144 0.01274 0.01274 0.01274 1.36% StateDataPhysBCFunct::() 41 0.01234 0.01234 0.01234 1.31% Castro::enforce_min_density() 62 0.01123 0.01123 0.01123 1.20% MLPoisson::Fapply() 1142 0.009973 0.009973 0.009973 1.06% MLCellLinOp::defineAuxData() 11 0.009312 0.009312 0.009312 0.99% FabArray::Saxpy() 813 0.007861 0.007861 0.007861 0.84% FabArray::Xpay() 821 0.007826 0.007826 0.007826 0.83% MLMG::addInterpCorrection() 410 0.006606 0.006606 0.006606 0.70% Gravity::fill_multipole_BCs() 11 0.006474 0.006474 0.006474 0.69% amrex::average_down 410 0.005846 0.005846 0.005846 0.62% Castro::reset_internal_energy(MultiFab) 63 0.004978 0.004978 0.004978 0.53% Castro::estTimeStep() 21 0.004744 0.004744 0.004744 0.50% FabArray::LinComb() 557 0.004386 0.004386 0.004386 0.47% amrex::Add() 164 0.004306 0.004306 0.004306 0.46% Amr::checkPoint() 3 0.004068 0.004068 0.004068 0.43% Castro::enforce_speed_limit() 62 0.003591 0.003591 0.003591 0.38% BndryData::define() 11 0.003534 0.003534 0.003534 0.38% Castro::construct_new_gravity_source() 10 0.003128 0.003128 0.003128 0.33% Castro::construct_old_gravity_source() 10 0.002762 0.002762 0.002762 0.29% Amr::writePlotFile() 2 0.002387 0.002387 0.002387 0.25% MLCGSolver::bicgstab 82 0.002153 0.002153 0.002153 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001633 0.001633 0.001633 0.17% Castro::reset_internal_energy(Fab) 504 0.001581 0.001581 0.001581 0.17% Gravity::actual_solve_with_mlmg() 11 0.001485 0.001485 0.001485 0.16% check_for_negative_density() 10 0.001431 0.001431 0.001431 0.15% MLCellLinOp::setLevelBC() 11 0.001374 0.001374 0.001374 0.15% FabArray::mult() 43 0.001295 0.001295 0.001295 0.14% FabArray::setDomainBndry() 41 0.001283 0.001283 0.001283 0.14% Castro::initData() 1 0.001238 0.001238 0.001238 0.13% MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.13% MLCellLinOp::smooth() 1640 0.001146 0.001146 0.001146 0.12% MLCellLinOp::prepareForSolve() 11 0.00109 0.00109 0.00109 0.12% MLCellLinOp::compGrad() 11 0.0009045 0.0009045 0.0009045 0.10% FabArray::FillBoundary() 4023 0.0008818 0.0008818 0.0008818 0.09% MLMG::prepareForSolve() 11 0.0008574 0.0008574 0.0008574 0.09% FabArrayBase::getCPC() 1323 0.000745 0.000745 0.000745 0.08% FabArrayBase::CPC::define() 454 0.0006657 0.0006657 0.0006657 0.07% FabArrayBase::getFB() 4023 0.0006513 0.0006513 0.0006513 0.07% Gravity::get_new_grav_vector() 11 0.0005374 0.0005374 0.0005374 0.06% MLCellLinOp::apply() 1142 0.000478 0.000478 0.000478 0.05% Gravity::get_old_grav_vector() 10 0.0004692 0.0004692 0.0004692 0.05% Amr::InitAmr() 1 0.0004589 0.0004589 0.0004589 0.05% Amr::coarseTimeStep() 10 0.000439 0.000439 0.000439 0.05% MLLinOp::defineGrids() 11 0.0003566 0.0003566 0.0003566 0.04% MLMG::mgVcycle() 82 0.0003522 0.0003522 0.0003522 0.04% AmrLevel::FillPatch() 41 0.0003269 0.0003269 0.0003269 0.03% MLCGSolver::ParallelAllReduce 1514 0.0003012 0.0003012 0.0003012 0.03% main() 1 0.000297 0.000297 0.000297 0.03% MultiFab::max() 11 0.0002564 0.0002564 0.0002564 0.03% FabArray::ParallelCopy() 861 0.0002358 0.0002358 0.0002358 0.03% MLCellLinOp::correctionResidual() 492 0.000225 0.000225 0.000225 0.02% MLCellLinOp::defineBC() 11 0.0002087 0.0002087 0.0002087 0.02% FillPatchIterator::Initialize 41 0.0002041 0.0002041 0.0002041 0.02% Castro::create_source_corrector() 10 0.0001859 0.0001859 0.0001859 0.02% Castro::subcycle_advance_ctu() 10 0.0001763 0.0001763 0.0001763 0.02% Amr::timeStep() 10 0.0001662 0.0001662 0.0001662 0.02% StateData::checkPoint() 12 0.0001376 0.0001376 0.0001376 0.01% Gravity::update_max_rhs() 11 0.0001295 0.0001295 0.0001295 0.01% MLMG:computeResOfCorrection() 410 0.0001288 0.0001288 0.0001288 0.01% Gravity::solve_for_phi() 10 0.0001107 0.0001107 0.0001107 0.01% Castro::post_timestep() 10 0.0001014 0.0001014 0.0001014 0.01% MLMG::actualBottomSolve() 82 8.895e-05 8.895e-05 8.895e-05 0.01% FabArrayBase::FB::FB() 56 8.873e-05 8.873e-05 8.873e-05 0.01% MLMG::mgVcycle_down::0 82 8.762e-05 8.762e-05 8.762e-05 0.01% Castro::Castro() 1 8.586e-05 8.586e-05 8.586e-05 0.01% MLMG::mgVcycle_down::2 82 8.422e-05 8.422e-05 8.422e-05 0.01% MLMG::mgVcycle_down::1 82 8.418e-05 8.418e-05 8.418e-05 0.01% MLMG::mgVcycle_down::4 82 8.312e-05 8.312e-05 8.312e-05 0.01% MLMG::mgVcycle_down::3 82 8.036e-05 8.036e-05 8.036e-05 0.01% MLMG::solve() 11 7.669e-05 7.669e-05 7.669e-05 0.01% AmrLevel::checkPoint() 3 7.525e-05 7.525e-05 7.525e-05 0.01% Castro::initialize_advance() 10 7.178e-05 7.178e-05 7.178e-05 0.01% Castro::clean_state() 62 6.607e-05 6.607e-05 6.607e-05 0.01% MLMG::mgVcycle_up::4 82 6.262e-05 6.262e-05 6.262e-05 0.01% MLMG::oneIter() 82 5.955e-05 5.955e-05 5.955e-05 0.01% Castro::finalize_advance() 10 5.812e-05 5.812e-05 5.812e-05 0.01% Castro::initialize_do_advance() 10 5.68e-05 5.68e-05 5.68e-05 0.01% MLMG::mgVcycle_up::0 82 5.399e-05 5.399e-05 5.399e-05 0.01% MLMG::mgVcycle_up::1 82 5.082e-05 5.082e-05 5.082e-05 0.01% MLMG::mgVcycle_up::3 82 5.039e-05 5.039e-05 5.039e-05 0.01% Castro::finalize_do_advance() 10 5.037e-05 5.037e-05 5.037e-05 0.01% MLMG::mgVcycle_up::2 82 4.956e-05 4.956e-05 4.956e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.932e-05 4.932e-05 4.932e-05 0.01% Castro::do_advance_ctu() 10 4.914e-05 4.914e-05 4.914e-05 0.01% Castro::advance() 10 4.654e-05 4.654e-05 4.654e-05 0.00% MLCellLinOp::solutionResidual() 93 4.644e-05 4.644e-05 4.644e-05 0.00% Castro::swap_state_time_levels() 10 4.432e-05 4.432e-05 4.432e-05 0.00% Castro::construct_new_source() 50 4.33e-05 4.33e-05 4.33e-05 0.00% StateData::define() 4 3.52e-05 3.52e-05 3.52e-05 0.00% MLMG::mgVcycle_bottom 82 3.311e-05 3.311e-05 3.311e-05 0.00% Castro::enforce_consistent_e() 1 3.303e-05 3.303e-05 3.303e-05 0.00% MLMG::computeResidual() 82 3.29e-05 3.29e-05 3.29e-05 0.00% MLMG::ResNormInf() 93 3.207e-05 3.207e-05 3.207e-05 0.00% FillPatchSingleLevel 41 3.075e-05 3.075e-05 3.075e-05 0.00% MLPoisson::define() 11 2.777e-05 2.777e-05 2.777e-05 0.00% Amr::writeSmallPlotFile() 1 2.563e-05 2.563e-05 2.563e-05 0.00% makeSFC 55 2.476e-05 2.476e-05 2.476e-05 0.00% Castro::construct_new_gravity() 10 2.392e-05 2.392e-05 2.392e-05 0.00% Castro::initMFs() 1 2.38e-05 2.38e-05 2.38e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.104e-05 2.104e-05 2.104e-05 0.00% Amr::FinalizeInit() 1 2.049e-05 2.049e-05 2.049e-05 0.00% Castro::do_old_sources() 10 2.017e-05 2.017e-05 2.017e-05 0.00% Castro::do_new_sources() 10 1.849e-05 1.849e-05 1.849e-05 0.00% Castro::construct_old_source() 50 1.801e-05 1.801e-05 1.801e-05 0.00% Castro::buildMetrics() 1 1.733e-05 1.733e-05 1.733e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.716e-05 1.716e-05 1.716e-05 0.00% MLLinOp::define() 11 1.675e-05 1.675e-05 1.675e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.645e-05 1.645e-05 1.645e-05 0.00% DistributionMapping::Distribute() 56 1.488e-05 1.488e-05 1.488e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.346e-05 1.346e-05 1.346e-05 0.00% Amr::defBaseLevel() 1 1.262e-05 1.262e-05 1.262e-05 0.00% Castro::check_for_nan() 20 1.081e-05 1.081e-05 1.081e-05 0.00% Castro::apply_source_to_state() 20 1.045e-05 1.045e-05 1.045e-05 0.00% Castro::construct_old_gravity() 10 1.011e-05 1.011e-05 1.011e-05 0.00% Amr::initSubcycle() 1 8.479e-06 8.479e-06 8.479e-06 0.00% Gravity::swapTimeLevels() 10 8.447e-06 8.447e-06 8.447e-06 0.00% MLPoisson::prepareForSolve() 11 8.377e-06 8.377e-06 8.377e-06 0.00% Gravity::actual_multilevel_solve() 1 8.106e-06 8.106e-06 8.106e-06 0.00% MLMG::computeMLResidual() 11 7.479e-06 7.479e-06 7.479e-06 0.00% MLMG::getGradSolution() 11 6.144e-06 6.144e-06 6.144e-06 0.00% Castro::computeNewDt() 9 6.087e-06 6.087e-06 6.087e-06 0.00% Castro::expand_state() 10 5.524e-06 5.524e-06 5.524e-06 0.00% AmrLevel::checkPointPost() 3 5.522e-06 5.522e-06 5.522e-06 0.00% Gravity::set_mass_offset() 11 4.786e-06 4.786e-06 4.786e-06 0.00% Amr::InitializeInit() 1 4.643e-06 4.643e-06 4.643e-06 0.00% Castro::retry_advance_ctu() 10 4.061e-06 4.061e-06 4.061e-06 0.00% Castro::post_init() 1 3.753e-06 3.753e-06 3.753e-06 0.00% MLMG::MLRhsNormInf() 11 3.719e-06 3.719e-06 3.719e-06 0.00% MLMG::MLResNormInf() 11 3.322e-06 3.322e-06 3.322e-06 0.00% Castro::computeInitialDt() 2 3.055e-06 3.055e-06 3.055e-06 0.00% Amr::init() 1 2.848e-06 2.848e-06 2.848e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.71e-06 2.71e-06 2.71e-06 0.00% Castro::FluxRegCrseInit 10 2.518e-06 2.518e-06 2.518e-06 0.00% Castro::FluxRegFineAdd() 10 2.113e-06 2.113e-06 2.113e-06 0.00% AmrLevel::checkPointPre() 3 1.947e-06 1.947e-06 1.947e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.842e-06 1.842e-06 1.842e-06 0.00% Castro::post_regrid() 1 1.355e-06 1.355e-06 1.355e-06 0.00% Amr::initialInit() 1 1.204e-06 1.204e-06 1.204e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9394 0.9394 0.9394 100.00% Amr::coarseTimeStep() 10 0.767 0.767 0.767 81.64% Amr::timeStep() 10 0.649 0.649 0.649 69.08% Castro::advance() 10 0.6383 0.6383 0.6383 67.94% Castro::subcycle_advance_ctu() 10 0.6251 0.6251 0.6251 66.54% Castro::do_advance_ctu() 10 0.6249 0.6249 0.6249 66.52% Castro::construct_ctu_hydro_source() 10 0.3037 0.3037 0.3037 32.33% Gravity::solve_phi_with_mlmg() 11 0.2756 0.2756 0.2756 29.34% Gravity::actual_solve_with_mlmg() 11 0.2687 0.2687 0.2687 28.60% Castro::construct_new_gravity() 10 0.2514 0.2514 0.2514 26.76% MLMG::solve() 11 0.2487 0.2487 0.2487 26.47% Gravity::solve_for_phi() 10 0.2362 0.2362 0.2362 25.14% MLMG::oneIter() 82 0.2348 0.2348 0.2348 24.99% VisMF::Write(FabArray) 11 0.232 0.232 0.232 24.69% MLMG::mgVcycle() 82 0.2312 0.2312 0.2312 24.60% Amr::checkPoint() 3 0.1727 0.1727 0.1727 18.39% AmrLevel::checkPoint() 3 0.1687 0.1687 0.1687 17.95% StateData::checkPoint() 12 0.1686 0.1686 0.1686 17.95% Amr::init() 1 0.1385 0.1385 0.1385 14.74% MLCellLinOp::smooth() 1640 0.114 0.114 0.114 12.13% MLCellLinOp::applyBC() 4433 0.1068 0.1068 0.1068 11.37% MLMG::mgVcycle_bottom 82 0.07102 0.07102 0.07102 7.56% MLMG::actualBottomSolve() 82 0.07099 0.07099 0.07099 7.56% MLCGSolver::bicgstab 82 0.07031 0.07031 0.07031 7.48% Amr::writePlotFile() 2 0.06641 0.06641 0.06641 7.07% Castro::clean_state() 62 0.05339 0.05339 0.05339 5.68% Amr::initialInit() 1 0.04719 0.04719 0.04719 5.02% AmrLevel::FillPatch() 41 0.04616 0.04616 0.04616 4.91% Amr::FinalizeInit() 1 0.04307 0.04307 0.04307 4.58% FillPatchIterator::Initialize 41 0.04204 0.04204 0.04204 4.47% Castro::post_init() 1 0.04178 0.04178 0.04178 4.45% FillPatchIterator::FillFromLevel0() 41 0.04055 0.04055 0.04055 4.32% FillPatchSingleLevel 41 0.0405 0.0405 0.0405 4.31% Gravity::multilevel_solve_for_new_phi() 1 0.03989 0.03989 0.03989 4.25% Gravity::actual_multilevel_solve() 1 0.03988 0.03988 0.03988 4.24% StateDataPhysBCFunct::() 41 0.03647 0.03647 0.03647 3.88% MLCellLinOp::apply() 1142 0.03529 0.03529 0.03529 3.76% MLMG::mgVcycle_down::0 82 0.03317 0.03317 0.03317 3.53% FabArray::FillBoundary() 4023 0.03221 0.03221 0.03221 3.43% MLPoisson::Fsmooth() 3280 0.03138 0.03138 0.03138 3.34% FillBoundary_nowait() 4023 0.03133 0.03133 0.03133 3.33% MLMG::mgVcycle_up::0 82 0.02516 0.02516 0.02516 2.68% StateData::FillBoundary(geom) 328 0.02413 0.02413 0.02413 2.57% MLCellLinOp::correctionResidual() 492 0.02163 0.02163 0.02163 2.30% Castro::initialize_do_advance() 10 0.02096 0.02096 0.02096 2.23% Castro::computeTemp() 63 0.02036 0.02036 0.02036 2.17% amrex::Dot() 1114 0.01979 0.01979 0.01979 2.11% MLMG:computeResOfCorrection() 410 0.0191 0.0191 0.0191 2.03% Castro::normalize_species() 62 0.01888 0.01888 0.01888 2.01% Castro::do_old_sources() 10 0.01884 0.01884 0.01884 2.01% Gravity::get_new_grav_vector() 11 0.01688 0.01688 0.01688 1.80% MLPoisson::define() 11 0.01575 0.01575 0.01575 1.68% MLMG::mgVcycle_down::1 82 0.01531 0.01531 0.01531 1.63% Castro::construct_old_gravity() 10 0.01467 0.01467 0.01467 1.56% Gravity::get_old_grav_vector() 10 0.01466 0.01466 0.01466 1.56% amrex::Copy() 1029 0.0145 0.0145 0.0145 1.54% MLMG::mgVcycle_down::2 82 0.01437 0.01437 0.01437 1.53% FabArray::ParallelCopy() 861 0.01409 0.01409 0.01409 1.50% MLMG::mgVcycle_down::3 82 0.014 0.014 0.014 1.49% FabArray::norminf() 743 0.01386 0.01386 0.01386 1.48% FabArray::ParallelCopy_nowait() 861 0.01386 0.01386 0.01386 1.47% MLMG::mgVcycle_down::4 82 0.01381 0.01381 0.01381 1.47% FabArray::setVal() 1144 0.01274 0.01274 0.01274 1.36% Castro::initialize_advance() 10 0.01256 0.01256 0.01256 1.34% Castro::do_new_sources() 10 0.0125 0.0125 0.0125 1.33% MLCGSolver::ParallelAllReduce 1514 0.0119 0.0119 0.0119 1.27% Castro::expand_state() 10 0.01181 0.01181 0.01181 1.26% MLMG::addInterpCorrection() 410 0.01163 0.01163 0.01163 1.24% Castro::enforce_min_density() 62 0.01123 0.01123 0.01123 1.20% MLMG::mgVcycle_up::4 82 0.01119 0.01119 0.01119 1.19% MLMG::mgVcycle_up::1 82 0.01117 0.01117 0.01117 1.19% amrex::average_down 410 0.01092 0.01092 0.01092 1.16% MLMG::mgVcycle_up::2 82 0.0109 0.0109 0.0109 1.16% MLMG::mgVcycle_up::3 82 0.0107 0.0107 0.0107 1.14% MLCellLinOp::defineAuxData() 11 0.0106 0.0106 0.0106 1.13% Castro::post_timestep() 10 0.01058 0.01058 0.01058 1.13% MLPoisson::Fapply() 1142 0.009973 0.009973 0.009973 1.06% FabArray::Saxpy() 813 0.007861 0.007861 0.007861 0.84% FabArray::Xpay() 821 0.007826 0.007826 0.007826 0.83% MLCellLinOp::solutionResidual() 93 0.006987 0.006987 0.006987 0.74% Gravity::fill_multipole_BCs() 11 0.006735 0.006735 0.006735 0.72% Castro::reset_internal_energy(MultiFab) 63 0.006559 0.006559 0.006559 0.70% MLMG::computeResidual() 82 0.00603 0.00603 0.00603 0.64% Castro::estTimeStep() 21 0.004744 0.004744 0.004744 0.50% MLCellLinOp::defineBC() 11 0.004693 0.004693 0.004693 0.50% MLMG::prepareForSolve() 11 0.004511 0.004511 0.004511 0.48% BndryData::define() 11 0.004485 0.004485 0.004485 0.48% FabArray::LinComb() 557 0.004386 0.004386 0.004386 0.47% amrex::Add() 164 0.004306 0.004306 0.004306 0.46% Amr::InitializeInit() 1 0.004112 0.004112 0.004112 0.44% Amr::defBaseLevel() 1 0.004107 0.004107 0.004107 0.44% Castro::initData() 1 0.003615 0.003615 0.003615 0.38% Castro::enforce_speed_limit() 62 0.003591 0.003591 0.003591 0.38% Castro::construct_new_source() 50 0.003171 0.003171 0.003171 0.34% Castro::construct_new_gravity_source() 10 0.003128 0.003128 0.003128 0.33% Castro::construct_old_source() 50 0.00278 0.00278 0.00278 0.30% Castro::construct_old_gravity_source() 10 0.002762 0.002762 0.002762 0.29% Castro::computeNewDt() 9 0.002152 0.002152 0.002152 0.23% MLMG::ResNormInf() 93 0.002089 0.002089 0.002089 0.22% Castro::apply_source_to_state() 20 0.001802 0.001802 0.001802 0.19% Castro::finalize_do_advance() 10 0.001794 0.001794 0.001794 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001633 0.001633 0.001633 0.17% Castro::reset_internal_energy(Fab) 504 0.001581 0.001581 0.001581 0.17% check_for_negative_density() 10 0.001431 0.001431 0.001431 0.15% FabArrayBase::getCPC() 1323 0.001411 0.001411 0.001411 0.15% MLCellLinOp::setLevelBC() 11 0.001374 0.001374 0.001374 0.15% MLMG::getGradSolution() 11 0.001373 0.001373 0.001373 0.15% MLCellLinOp::compGrad() 11 0.001366 0.001366 0.001366 0.15% FabArray::mult() 43 0.001295 0.001295 0.001295 0.14% FabArray::setDomainBndry() 41 0.001283 0.001283 0.001283 0.14% Castro::check_for_nan() 20 0.001209 0.001209 0.001209 0.13% MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.13% Castro::post_regrid() 1 0.001122 0.001122 0.001122 0.12% MLPoisson::prepareForSolve() 11 0.001098 0.001098 0.001098 0.12% MLCellLinOp::prepareForSolve() 11 0.00109 0.00109 0.00109 0.12% MLMG::computeMLResidual() 11 0.0009978 0.0009978 0.0009978 0.11% Castro::computeInitialDt() 2 0.0008565 0.0008565 0.0008565 0.09% Gravity::update_max_rhs() 11 0.0008143 0.0008143 0.0008143 0.09% FabArrayBase::getFB() 4023 0.0007401 0.0007401 0.0007401 0.08% FabArrayBase::CPC::define() 454 0.0006657 0.0006657 0.0006657 0.07% Castro::finalize_advance() 10 0.0005592 0.0005592 0.0005592 0.06% Amr::InitAmr() 1 0.0004674 0.0004674 0.0004674 0.05% Gravity::swapTimeLevels() 10 0.0004274 0.0004274 0.0004274 0.05% MLLinOp::define() 11 0.0004269 0.0004269 0.0004269 0.05% Castro::Castro() 1 0.000424 0.000424 0.000424 0.05% MLLinOp::defineGrids() 11 0.0004102 0.0004102 0.0004102 0.04% MLMG::MLResNormInf() 11 0.0002831 0.0002831 0.0002831 0.03% MultiFab::max() 11 0.0002564 0.0002564 0.0002564 0.03% MLMG::MLRhsNormInf() 11 0.0002154 0.0002154 0.0002154 0.02% Castro::create_source_corrector() 10 0.0001859 0.0001859 0.0001859 0.02% Castro::buildMetrics() 1 0.0001542 0.0001542 0.0001542 0.02% FabArrayBase::FB::FB() 56 8.873e-05 8.873e-05 8.873e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.173e-05 5.173e-05 5.173e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.164e-05 5.164e-05 5.164e-05 0.01% Castro::swap_state_time_levels() 10 4.432e-05 4.432e-05 4.432e-05 0.00% makeSFC 55 3.827e-05 3.827e-05 3.827e-05 0.00% StateData::define() 4 3.52e-05 3.52e-05 3.52e-05 0.00% Castro::enforce_consistent_e() 1 3.303e-05 3.303e-05 3.303e-05 0.00% Amr::writeSmallPlotFile() 1 2.563e-05 2.563e-05 2.563e-05 0.00% Castro::initMFs() 1 2.38e-05 2.38e-05 2.38e-05 0.00% DistributionMapping::Distribute() 56 1.488e-05 1.488e-05 1.488e-05 0.00% Amr::initSubcycle() 1 8.479e-06 8.479e-06 8.479e-06 0.00% AmrLevel::checkPointPost() 3 5.522e-06 5.522e-06 5.522e-06 0.00% Gravity::set_mass_offset() 11 4.786e-06 4.786e-06 4.786e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.08e-06 4.08e-06 4.08e-06 0.00% Castro::retry_advance_ctu() 10 4.061e-06 4.061e-06 4.061e-06 0.00% Castro::FluxRegCrseInit 10 2.518e-06 2.518e-06 2.518e-06 0.00% Castro::FluxRegFineAdd() 10 2.113e-06 2.113e-06 2.113e-06 0.00% AmrLevel::checkPointPre() 3 1.947e-06 1.947e-06 1.947e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.842e-06 1.842e-06 1.842e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 85 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 148 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 970 KiB 39 MiB Castro::initialize_do_advance() 80 80 25 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1986 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7610 KiB 14 MiB MLMG::prepareForSolve() 660 660 3257 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 186 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 159 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7530 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 11 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1656 B 2048 KiB Gravity::solve_for_phi() 80 80 514 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 86 KiB 2048 KiB BndryData::define() 1056 1056 297 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 189 KiB 671 KiB Castro::estTimeStep() 21 21 2461 B 480 KiB VisMF::Write(FabArray) 656 656 3138 B 320 KiB Castro::normalize_species() 62 62 6546 B 320 KiB amrex::average_down 1067 1067 1182 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1065 B 257 KiB amrex::Dot() 1360 1360 3100 B 160 KiB FabArray::norminf() 907 907 2183 B 160 KiB check_for_negative_density() 10 10 241 B 160 KiB Castro::initData() 1 1 25 B 160 KiB MultiFab::max() 11 11 42 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 25 B 20 KiB MLPoisson::Fsmooth() 132 132 3185 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 41 B 10 KiB FillBoundary_nowait() 760 760 285 B 9648 B MLCellLinOp::applyBC() 8866 8866 205 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3859 B 6144 B StateData::FillBoundary(geom) 1992 1992 42 B 2880 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 108 B 1472 B MLCellLinOp::defineBC() 66 66 335 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 590 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 47 KiB 8192 KiB VisMF::Write(FabArray) 744 744 569 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3185 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 42 B 10 KiB FillBoundary_nowait() 760 760 285 B 9648 B MLCellLinOp::applyBC() 4433 4433 204 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3859 B 6144 B Gravity::get_new_grav_vector() 3 3 2918 B 3072 B StateData::FillBoundary(geom) 1992 1992 43 B 2880 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 271 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 22 B 400 B FabArray::norminf() 907 907 16 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2422 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.07-52-g98d22d21af45) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.07-52-g98d22d21af45) initialized Starting run at 07:36:52 UTC on 2023-07-31. Successfully read inputs file ... Castro git describe: 23.07-20-g4174f17eb AMReX git describe: 23.07-52-g98d22d21a Microphysics git describe: 23.07-48-g588beb9c reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.505326192 Restart time = 0.045900333 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048865934 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048461575 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.078396147 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.079103571 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.05326007 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032883833 seconds Ending run at 07:36:52 UTC on 2023-07-31. Run time = 0.387917147 Run time without initialization = 0.341434504 Average number of zones advanced per microsecond: 3.839 Average number of zones advanced per microsecond per rank: 3.839 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3879 ... 0.3879 ... 0.3879 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1278 0.1278 0.1278 32.94% VisMF::Read() 3 0.03961 0.03961 0.03961 10.21% MLCellLinOp::applyBC() 1946 0.03242 0.03242 0.03242 8.36% VisMF::Write(FabArray) 1 0.03125 0.03125 0.03125 8.06% MLPoisson::Fsmooth() 1440 0.01351 0.01351 0.01351 3.48% FillBoundary_nowait() 1766 0.01276 0.01276 0.01276 3.29% StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 2.99% Castro::normalize_species() 30 0.01028 0.01028 0.01028 2.65% amrex::Dot() 484 0.008468 0.008468 0.008468 2.18% amrex::Copy() 463 0.006871 0.006871 0.006871 1.77% Castro::computeTemp() 30 0.006288 0.006288 0.006288 1.62% FabArray::setVal() 537 0.006055 0.006055 0.006055 1.56% FabArray::norminf() 326 0.006035 0.006035 0.006035 1.56% FabArray::ParallelCopy_nowait() 380 0.005916 0.005916 0.005916 1.52% MLCellLinOp::defineAuxData() 6 0.005051 0.005051 0.005051 1.30% StateDataPhysBCFunct::() 20 0.004769 0.004769 0.004769 1.23% Castro::enforce_min_density() 30 0.004385 0.004385 0.004385 1.13% MLPoisson::Fapply() 500 0.00429 0.00429 0.00429 1.11% FabArray::Saxpy() 355 0.003508 0.003508 0.003508 0.90% Gravity::fill_multipole_BCs() 6 0.003449 0.003449 0.003449 0.89% FabArray::Xpay() 361 0.003394 0.003394 0.003394 0.87% MLMG::addInterpCorrection() 180 0.002866 0.002866 0.002866 0.74% Castro::estTimeStep() 10 0.002816 0.002816 0.002816 0.73% amrex::average_down 180 0.002546 0.002546 0.002546 0.66% Amr::restart() 1 0.002496 0.002496 0.002496 0.64% Castro::reset_internal_energy(MultiFab) 30 0.00216 0.00216 0.00216 0.56% BndryData::define() 6 0.001925 0.001925 0.001925 0.50% FabArray::LinComb() 242 0.001857 0.001857 0.001857 0.48% amrex::Add() 72 0.001815 0.001815 0.001815 0.47% Castro::enforce_speed_limit() 30 0.001655 0.001655 0.001655 0.43% Castro::construct_new_gravity_source() 5 0.001574 0.001574 0.001574 0.41% Amr::writePlotFile() 1 0.00145 0.00145 0.00145 0.37% Castro::construct_old_gravity_source() 5 0.001445 0.001445 0.001445 0.37% MLCGSolver::bicgstab 36 0.0009509 0.0009509 0.0009509 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008743 0.0008743 0.0008743 0.23% Castro::reset_internal_energy(Fab) 240 0.0008228 0.0008228 0.0008228 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007873 0.0007873 0.0007873 0.20% MLCellLinOp::setLevelBC() 6 0.0007305 0.0007305 0.0007305 0.19% check_for_negative_density() 5 0.0007165 0.0007165 0.0007165 0.18% Castro::subcycle_advance_ctu() 5 0.0006901 0.0006901 0.0006901 0.18% FabArray::mult() 22 0.0006396 0.0006396 0.0006396 0.16% FabArray::setDomainBndry() 20 0.0006242 0.0006242 0.0006242 0.16% MLCellLinOp::prepareForSolve() 6 0.0005945 0.0005945 0.0005945 0.15% MultiFab::contains_nan() 10 0.0005891 0.0005891 0.0005891 0.15% MLCellLinOp::smooth() 720 0.000502 0.000502 0.000502 0.13% MLCellLinOp::compGrad() 6 0.0004768 0.0004768 0.0004768 0.12% MLMG::prepareForSolve() 6 0.0004601 0.0004601 0.0004601 0.12% FabArrayBase::CPC::define() 244 0.0004196 0.0004196 0.0004196 0.11% Amr::InitAmr() 1 0.0003919 0.0003919 0.0003919 0.10% FabArray::FillBoundary() 1766 0.0003713 0.0003713 0.0003713 0.10% FabArrayBase::getCPC() 632 0.0003663 0.0003663 0.0003663 0.09% main() 1 0.0002864 0.0002864 0.0002864 0.07% FabArrayBase::getFB() 1766 0.0002671 0.0002671 0.0002671 0.07% Gravity::get_old_grav_vector() 5 0.0002636 0.0002636 0.0002636 0.07% Gravity::get_new_grav_vector() 5 0.0002316 0.0002316 0.0002316 0.06% Amr::coarseTimeStep() 5 0.0002197 0.0002197 0.0002197 0.06% MLCellLinOp::apply() 500 0.0002043 0.0002043 0.0002043 0.05% MLLinOp::defineGrids() 6 0.000169 0.000169 0.000169 0.04% Castro::construct_new_source() 25 0.0001689 0.0001689 0.0001689 0.04% AmrLevel::FillPatch() 20 0.0001635 0.0001635 0.0001635 0.04% MLMG::mgVcycle() 36 0.0001488 0.0001488 0.0001488 0.04% MultiFab::max() 6 0.0001371 0.0001371 0.0001371 0.04% MLCGSolver::ParallelAllReduce 659 0.000129 0.000129 0.000129 0.03% FillPatchIterator::Initialize 20 0.0001101 0.0001101 0.0001101 0.03% MLCellLinOp::defineBC() 6 0.0001098 0.0001098 0.0001098 0.03% FabArray::ParallelCopy() 380 0.0001036 0.0001036 0.0001036 0.03% Castro::finalize_do_advance() 5 9.806e-05 9.806e-05 9.806e-05 0.03% MLCellLinOp::correctionResidual() 216 9.208e-05 9.208e-05 9.208e-05 0.02% Castro::do_new_sources() 5 9.085e-05 9.085e-05 9.085e-05 0.02% Castro::create_source_corrector() 5 8.123e-05 8.123e-05 8.123e-05 0.02% Amr::timeStep() 5 8.059e-05 8.059e-05 8.059e-05 0.02% AmrLevel::restart() 1 7.635e-05 7.635e-05 7.635e-05 0.02% Gravity::update_max_rhs() 6 6.605e-05 6.605e-05 6.605e-05 0.02% FabArrayBase::FB::FB() 26 6.073e-05 6.073e-05 6.073e-05 0.02% StateData::restartDoit() 4 6.059e-05 6.059e-05 6.059e-05 0.02% MLMG:computeResOfCorrection() 180 5.555e-05 5.555e-05 5.555e-05 0.01% Gravity::solve_for_phi() 5 5.426e-05 5.426e-05 5.426e-05 0.01% MLMG::actualBottomSolve() 36 3.978e-05 3.978e-05 3.978e-05 0.01% MLMG::mgVcycle_down::0 36 3.939e-05 3.939e-05 3.939e-05 0.01% MLMG::mgVcycle_down::1 36 3.881e-05 3.881e-05 3.881e-05 0.01% Castro::advance() 5 3.702e-05 3.702e-05 3.702e-05 0.01% MLMG::mgVcycle_down::2 36 3.642e-05 3.642e-05 3.642e-05 0.01% MLMG::mgVcycle_down::4 36 3.621e-05 3.621e-05 3.621e-05 0.01% MLMG::mgVcycle_down::3 36 3.593e-05 3.593e-05 3.593e-05 0.01% Castro::do_old_sources() 5 3.547e-05 3.547e-05 3.547e-05 0.01% MLMG::solve() 6 3.464e-05 3.464e-05 3.464e-05 0.01% Castro::initialize_advance() 5 3.441e-05 3.441e-05 3.441e-05 0.01% Castro::clean_state() 30 3.342e-05 3.342e-05 3.342e-05 0.01% Castro::construct_old_source() 25 3.196e-05 3.196e-05 3.196e-05 0.01% Castro::initialize_do_advance() 5 2.899e-05 2.899e-05 2.899e-05 0.01% MLMG::mgVcycle_up::4 36 2.866e-05 2.866e-05 2.866e-05 0.01% Castro::finalize_advance() 5 2.864e-05 2.864e-05 2.864e-05 0.01% Amr::writeSmallPlotFile() 1 2.767e-05 2.767e-05 2.767e-05 0.01% Castro::post_timestep() 5 2.74e-05 2.74e-05 2.74e-05 0.01% Castro::buildMetrics() 1 2.683e-05 2.683e-05 2.683e-05 0.01% Castro::initMFs() 1 2.568e-05 2.568e-05 2.568e-05 0.01% MLMG::oneIter() 36 2.418e-05 2.418e-05 2.418e-05 0.01% Castro::do_advance_ctu() 5 2.406e-05 2.406e-05 2.406e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.374e-05 2.374e-05 2.374e-05 0.01% Castro::swap_state_time_levels() 5 2.348e-05 2.348e-05 2.348e-05 0.01% MLMG::mgVcycle_up::0 36 2.323e-05 2.323e-05 2.323e-05 0.01% MLMG::mgVcycle_up::3 36 2.241e-05 2.241e-05 2.241e-05 0.01% MLMG::mgVcycle_up::2 36 2.167e-05 2.167e-05 2.167e-05 0.01% MLMG::mgVcycle_up::1 36 2.129e-05 2.129e-05 2.129e-05 0.01% MLCellLinOp::solutionResidual() 42 2.102e-05 2.102e-05 2.102e-05 0.01% MLPoisson::define() 6 2.079e-05 2.079e-05 2.079e-05 0.01% Castro::post_restart() 1 1.951e-05 1.951e-05 1.951e-05 0.01% MLMG::ResNormInf() 42 1.598e-05 1.598e-05 1.598e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.572e-05 1.572e-05 1.572e-05 0.00% MLMG::mgVcycle_bottom 36 1.496e-05 1.496e-05 1.496e-05 0.00% FillPatchSingleLevel 20 1.46e-05 1.46e-05 1.46e-05 0.00% MLMG::computeResidual() 36 1.385e-05 1.385e-05 1.385e-05 0.00% makeSFC 30 1.24e-05 1.24e-05 1.24e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.229e-05 1.229e-05 1.229e-05 0.00% MLLinOp::define() 6 1.171e-05 1.171e-05 1.171e-05 0.00% Castro::construct_new_gravity() 5 1.162e-05 1.162e-05 1.162e-05 0.00% Amr::initSubcycle() 1 8.403e-06 8.403e-06 8.403e-06 0.00% DistributionMapping::Distribute() 31 7.941e-06 7.941e-06 7.941e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.589e-06 7.589e-06 7.589e-06 0.00% Gravity::actual_multilevel_solve() 1 7.482e-06 7.482e-06 7.482e-06 0.00% Castro::construct_old_gravity() 5 5.792e-06 5.792e-06 5.792e-06 0.00% Castro::apply_source_to_state() 10 5.688e-06 5.688e-06 5.688e-06 0.00% Castro::check_for_nan() 10 5.457e-06 5.457e-06 5.457e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.36e-06 4.36e-06 4.36e-06 0.00% MLPoisson::prepareForSolve() 6 4.358e-06 4.358e-06 4.358e-06 0.00% Gravity::swapTimeLevels() 5 4.305e-06 4.305e-06 4.305e-06 0.00% MLMG::computeMLResidual() 6 3.622e-06 3.622e-06 3.622e-06 0.00% MLMG::getGradSolution() 6 3.206e-06 3.206e-06 3.206e-06 0.00% Castro::FluxRegCrseInit 5 3.012e-06 3.012e-06 3.012e-06 0.00% Gravity::set_mass_offset() 6 2.984e-06 2.984e-06 2.984e-06 0.00% Castro::computeNewDt() 5 2.938e-06 2.938e-06 2.938e-06 0.00% Castro::expand_state() 5 2.766e-06 2.766e-06 2.766e-06 0.00% Castro::retry_advance_ctu() 5 2.389e-06 2.389e-06 2.389e-06 0.00% MLMG::MLResNormInf() 6 2.282e-06 2.282e-06 2.282e-06 0.00% MLMG::MLRhsNormInf() 6 2.131e-06 2.131e-06 2.131e-06 0.00% Castro::FluxRegFineAdd() 5 1.614e-06 1.614e-06 1.614e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.022e-06 1.022e-06 1.022e-06 0.00% Amr::init() 1 9.38e-07 9.38e-07 9.38e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3879 0.3879 0.3879 100.00% Amr::coarseTimeStep() 5 0.3083 0.3083 0.3083 79.47% Amr::timeStep() 5 0.3068 0.3068 0.3068 79.08% Castro::advance() 5 0.3021 0.3021 0.3021 77.88% Castro::subcycle_advance_ctu() 5 0.2954 0.2954 0.2954 76.15% Castro::do_advance_ctu() 5 0.2947 0.2947 0.2947 75.97% Castro::construct_ctu_hydro_source() 5 0.1324 0.1324 0.1324 34.14% Castro::construct_new_gravity() 5 0.1264 0.1264 0.1264 32.59% Gravity::solve_phi_with_mlmg() 6 0.1221 0.1221 0.1221 31.47% Gravity::solve_for_phi() 5 0.119 0.119 0.119 30.66% Gravity::actual_solve_with_mlmg() 6 0.1184 0.1184 0.1184 30.51% MLMG::solve() 6 0.1075 0.1075 0.1075 27.72% MLMG::oneIter() 36 0.1008 0.1008 0.1008 25.99% MLMG::mgVcycle() 36 0.09932 0.09932 0.09932 25.60% MLCellLinOp::smooth() 720 0.04889 0.04889 0.04889 12.60% Amr::init() 1 0.04594 0.04594 0.04594 11.84% Amr::restart() 1 0.04594 0.04594 0.04594 11.84% MLCellLinOp::applyBC() 1946 0.04589 0.04589 0.04589 11.83% AmrLevel::restart() 1 0.03981 0.03981 0.03981 10.26% StateData::restartDoit() 4 0.03973 0.03973 0.03973 10.24% VisMF::Read() 3 0.03961 0.03961 0.03961 10.21% Amr::writePlotFile() 1 0.03297 0.03297 0.03297 8.50% VisMF::Write(FabArray) 1 0.03125 0.03125 0.03125 8.06% MLMG::mgVcycle_bottom 36 0.03054 0.03054 0.03054 7.87% MLMG::actualBottomSolve() 36 0.03053 0.03053 0.03053 7.87% MLCGSolver::bicgstab 36 0.03023 0.03023 0.03023 7.79% Castro::clean_state() 30 0.02562 0.02562 0.02562 6.60% AmrLevel::FillPatch() 20 0.0212 0.0212 0.0212 5.46% FillPatchIterator::Initialize 20 0.01916 0.01916 0.01916 4.94% FillPatchIterator::FillFromLevel0() 20 0.01842 0.01842 0.01842 4.75% FillPatchSingleLevel 20 0.0184 0.0184 0.0184 4.74% StateDataPhysBCFunct::() 20 0.01636 0.01636 0.01636 4.22% MLCellLinOp::apply() 500 0.01525 0.01525 0.01525 3.93% MLMG::mgVcycle_down::0 36 0.01416 0.01416 0.01416 3.65% MLPoisson::Fsmooth() 1440 0.01351 0.01351 0.01351 3.48% FabArray::FillBoundary() 1766 0.01346 0.01346 0.01346 3.47% FillBoundary_nowait() 1766 0.01309 0.01309 0.01309 3.37% StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 2.99% MLMG::mgVcycle_up::0 36 0.01074 0.01074 0.01074 2.77% Castro::normalize_species() 30 0.01028 0.01028 0.01028 2.65% Castro::initialize_do_advance() 5 0.01005 0.01005 0.01005 2.59% Castro::do_old_sources() 5 0.009927 0.009927 0.009927 2.56% Castro::computeTemp() 30 0.009271 0.009271 0.009271 2.39% MLCellLinOp::correctionResidual() 216 0.009246 0.009246 0.009246 2.38% MLPoisson::define() 6 0.008583 0.008583 0.008583 2.21% amrex::Dot() 484 0.008468 0.008468 0.008468 2.18% MLMG:computeResOfCorrection() 180 0.008134 0.008134 0.008134 2.10% Gravity::get_new_grav_vector() 5 0.007363 0.007363 0.007363 1.90% Castro::construct_old_gravity() 5 0.007114 0.007114 0.007114 1.83% Gravity::get_old_grav_vector() 5 0.007109 0.007109 0.007109 1.83% amrex::Copy() 463 0.006871 0.006871 0.006871 1.77% Castro::do_new_sources() 5 0.006647 0.006647 0.006647 1.71% MLMG::mgVcycle_down::1 36 0.006604 0.006604 0.006604 1.70% FabArray::ParallelCopy() 380 0.006412 0.006412 0.006412 1.65% Castro::initialize_advance() 5 0.006368 0.006368 0.006368 1.64% FabArray::ParallelCopy_nowait() 380 0.006308 0.006308 0.006308 1.63% MLMG::mgVcycle_down::2 36 0.006171 0.006171 0.006171 1.59% FabArray::setVal() 537 0.006055 0.006055 0.006055 1.56% MLMG::mgVcycle_down::3 36 0.006037 0.006037 0.006037 1.56% FabArray::norminf() 326 0.006035 0.006035 0.006035 1.56% MLMG::mgVcycle_down::4 36 0.00594 0.00594 0.00594 1.53% MLCellLinOp::defineAuxData() 6 0.005773 0.005773 0.005773 1.49% Castro::expand_state() 5 0.005224 0.005224 0.005224 1.35% MLCGSolver::ParallelAllReduce 659 0.005119 0.005119 0.005119 1.32% MLMG::addInterpCorrection() 180 0.005046 0.005046 0.005046 1.30% MLMG::mgVcycle_up::4 36 0.00481 0.00481 0.00481 1.24% MLMG::mgVcycle_up::1 36 0.004808 0.004808 0.004808 1.24% amrex::average_down 180 0.004757 0.004757 0.004757 1.23% MLMG::mgVcycle_up::2 36 0.004723 0.004723 0.004723 1.22% MLMG::mgVcycle_up::3 36 0.00464 0.00464 0.00464 1.20% Castro::post_timestep() 5 0.004599 0.004599 0.004599 1.19% Castro::enforce_min_density() 30 0.004385 0.004385 0.004385 1.13% MLPoisson::Fapply() 500 0.00429 0.00429 0.00429 1.11% Gravity::fill_multipole_BCs() 6 0.003601 0.003601 0.003601 0.93% FabArray::Saxpy() 355 0.003508 0.003508 0.003508 0.90% Castro::post_restart() 1 0.003464 0.003464 0.003464 0.89% FabArray::Xpay() 361 0.003394 0.003394 0.003394 0.87% Gravity::multilevel_solve_for_new_phi() 1 0.003351 0.003351 0.003351 0.86% Gravity::actual_multilevel_solve() 1 0.003335 0.003335 0.003335 0.86% MLCellLinOp::solutionResidual() 42 0.003158 0.003158 0.003158 0.81% Castro::reset_internal_energy(MultiFab) 30 0.002982 0.002982 0.002982 0.77% Castro::estTimeStep() 10 0.002816 0.002816 0.002816 0.73% MLMG::computeResidual() 36 0.002614 0.002614 0.002614 0.67% MLCellLinOp::defineBC() 6 0.00258 0.00258 0.00258 0.67% BndryData::define() 6 0.00247 0.00247 0.00247 0.64% MLMG::prepareForSolve() 6 0.002431 0.002431 0.002431 0.63% FabArray::LinComb() 242 0.001857 0.001857 0.001857 0.48% amrex::Add() 72 0.001815 0.001815 0.001815 0.47% Castro::construct_new_source() 25 0.001743 0.001743 0.001743 0.45% Castro::enforce_speed_limit() 30 0.001655 0.001655 0.001655 0.43% Castro::finalize_do_advance() 5 0.001641 0.001641 0.001641 0.42% Castro::construct_new_gravity_source() 5 0.001574 0.001574 0.001574 0.41% Castro::construct_old_source() 25 0.001477 0.001477 0.001477 0.38% Castro::construct_old_gravity_source() 5 0.001445 0.001445 0.001445 0.37% Castro::computeNewDt() 5 0.001276 0.001276 0.001276 0.33% MLMG::ResNormInf() 42 0.0009237 0.0009237 0.0009237 0.24% Castro::apply_source_to_state() 10 0.0009193 0.0009193 0.0009193 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008743 0.0008743 0.0008743 0.23% Castro::reset_internal_energy(Fab) 240 0.0008228 0.0008228 0.0008228 0.21% FabArrayBase::getCPC() 632 0.0007859 0.0007859 0.0007859 0.20% MLMG::getGradSolution() 6 0.000738 0.000738 0.000738 0.19% MLCellLinOp::compGrad() 6 0.0007348 0.0007348 0.0007348 0.19% MLCellLinOp::setLevelBC() 6 0.0007305 0.0007305 0.0007305 0.19% check_for_negative_density() 5 0.0007165 0.0007165 0.0007165 0.18% FabArray::mult() 22 0.0006396 0.0006396 0.0006396 0.16% FabArray::setDomainBndry() 20 0.0006242 0.0006242 0.0006242 0.16% MLPoisson::prepareForSolve() 6 0.0005989 0.0005989 0.0005989 0.15% Castro::check_for_nan() 10 0.0005946 0.0005946 0.0005946 0.15% MLCellLinOp::prepareForSolve() 6 0.0005945 0.0005945 0.0005945 0.15% MultiFab::contains_nan() 10 0.0005891 0.0005891 0.0005891 0.15% MLMG::computeMLResidual() 6 0.0005624 0.0005624 0.0005624 0.14% Gravity::update_max_rhs() 6 0.0004434 0.0004434 0.0004434 0.11% FabArrayBase::CPC::define() 244 0.0004196 0.0004196 0.0004196 0.11% Amr::InitAmr() 1 0.0004003 0.0004003 0.0004003 0.10% FabArrayBase::getFB() 1766 0.0003278 0.0003278 0.0003278 0.08% Castro::finalize_advance() 5 0.0002968 0.0002968 0.0002968 0.08% Gravity::swapTimeLevels() 5 0.0002212 0.0002212 0.0002212 0.06% MLLinOp::define() 6 0.0002084 0.0002084 0.0002084 0.05% MLLinOp::defineGrids() 6 0.0001967 0.0001967 0.0001967 0.05% MLMG::MLResNormInf() 6 0.0001502 0.0001502 0.0001502 0.04% Castro::buildMetrics() 1 0.0001499 0.0001499 0.0001499 0.04% MultiFab::max() 6 0.0001371 0.0001371 0.0001371 0.04% MLMG::MLRhsNormInf() 6 0.0001151 0.0001151 0.0001151 0.03% Castro::create_source_corrector() 5 8.123e-05 8.123e-05 8.123e-05 0.02% FabArrayBase::FB::FB() 26 6.073e-05 6.073e-05 6.073e-05 0.02% Amr::writeSmallPlotFile() 1 2.767e-05 2.767e-05 2.767e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.669e-05 2.669e-05 2.669e-05 0.01% Castro::initMFs() 1 2.568e-05 2.568e-05 2.568e-05 0.01% Castro::swap_state_time_levels() 5 2.348e-05 2.348e-05 2.348e-05 0.01% makeSFC 30 1.911e-05 1.911e-05 1.911e-05 0.00% Amr::initSubcycle() 1 8.403e-06 8.403e-06 8.403e-06 0.00% DistributionMapping::Distribute() 31 7.941e-06 7.941e-06 7.941e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.591e-06 5.591e-06 5.591e-06 0.00% Castro::FluxRegCrseInit 5 3.012e-06 3.012e-06 3.012e-06 0.00% Gravity::set_mass_offset() 6 2.984e-06 2.984e-06 2.984e-06 0.00% Castro::retry_advance_ctu() 5 2.389e-06 2.389e-06 2.389e-06 0.00% Castro::FluxRegFineAdd() 5 1.614e-06 1.614e-06 1.614e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.022e-06 1.022e-06 1.022e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 204 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 153 MiB 692 MiB Castro::initMFs() 48 48 60 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB FillPatchIterator::Initialize 160 160 1072 KiB 39 MiB Castro::initialize_do_advance() 40 40 29 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2377 KiB 28 MiB Castro::initialize_advance() 40 40 18 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6733 KiB 14 MiB MLMG::prepareForSolve() 361 361 3404 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 190 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 194 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6729 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 14 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2211 B 2048 KiB Gravity::solve_for_phi() 40 40 626 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 17 KiB 2048 KiB BndryData::define() 576 576 313 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 200 KiB 671 KiB Castro::estTimeStep() 10 10 3446 B 480 KiB VisMF::Write(FabArray) 112 112 1263 B 320 KiB Castro::normalize_species() 30 30 8625 B 320 KiB amrex::average_down 469 469 1198 B 257 KiB MLMG::addInterpCorrection() 468 468 1123 B 257 KiB amrex::Dot() 592 592 3205 B 160 KiB FabArray::norminf() 398 398 2293 B 160 KiB check_for_negative_density() 5 5 292 B 160 KiB MultiFab::max() 6 6 54 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 30 B 20 KiB MLPoisson::Fsmooth() 60 60 3311 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 286 B 9648 B MLCellLinOp::applyBC() 3892 3892 217 B 9344 B amrex::Copy() 56 56 5969 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 45 B 2880 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCGSolver::bicgstab 324 324 113 B 1472 B MLCellLinOp::defineBC() 36 36 351 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1433 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 113 KiB 8192 KiB VisMF::Write(FabArray) 120 120 248 KiB 3584 KiB VisMF::Read() 24 24 198 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3311 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 286 B 9648 B MLCellLinOp::applyBC() 1946 1946 216 B 9328 B amrex::Copy() 56 56 5969 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2659 B 3072 B StateData::FillBoundary(geom) 960 960 45 B 2880 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 523 B 1296 B amrex::average_down 37 37 225 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 398 398 16 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2422 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.07-52-g98d22d21af45) finalized