Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-28-g831d1487fa19) initialized Starting run at 09:09:49 UTC on 2023-03-29. Successfully read inputs file ... Castro git describe: 23.03-17-gf84bee6a2 AMReX git describe: 23.03-28-g831d1487f Microphysics git describe: 23.03-23-g27bca043 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058432521 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.033467872 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.046586296 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048648817 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.05660124 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059040732 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.060287176 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.058159049 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051895422 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049453477 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06686918 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.074066444 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063663498 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.058044866 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033667596 seconds Ending run at 09:09:50 UTC on 2023-03-29. Run time = 0.868615753 Run time without initialization = 0.727649261 Average number of zones advanced per microsecond: 3.603 Average number of zones advanced per microsecond per rank: 3.603 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8686 ... 0.8686 ... 0.8686 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2350 0.2350 0.2350 27.05% Castro::construct_ctu_hydro_source() 10 0.2114 0.2114 0.2114 24.33% MLCellLinOp::applyBC() 4433 0.07725 0.07725 0.07725 8.89% MLPoisson::Fsmooth() 3280 0.03255 0.03255 0.03255 3.75% FillBoundary_nowait() 4023 0.03069 0.03069 0.03069 3.53% StateData::FillBoundary(geom) 328 0.02505 0.02505 0.02505 2.88% amrex::Dot() 1114 0.02083 0.02083 0.02083 2.40% Castro::normalize_species() 62 0.01605 0.01605 0.01605 1.85% Castro::computeTemp() 63 0.0151 0.0151 0.0151 1.74% amrex::Copy() 1029 0.01492 0.01492 0.01492 1.72% FabArray::norminf() 743 0.0145 0.0145 0.0145 1.67% FabArray::ParallelCopy_nowait() 861 0.01356 0.01356 0.01356 1.56% FabArray::setVal() 1144 0.01331 0.01331 0.01331 1.53% StateDataPhysBCFunct::() 41 0.01281 0.01281 0.01281 1.47% Castro::enforce_min_density() 62 0.01134 0.01134 0.01134 1.31% MLPoisson::Fapply() 1142 0.01045 0.01045 0.01045 1.20% MLCellLinOp::defineAuxData() 11 0.009785 0.009785 0.009785 1.13% FabArray::Saxpy() 813 0.008249 0.008249 0.008249 0.95% FabArray::Xpay() 821 0.008142 0.008142 0.008142 0.94% MLMG::addInterpCorrection() 410 0.006844 0.006844 0.006844 0.79% Gravity::fill_multipole_BCs() 11 0.0067 0.0067 0.0067 0.77% Castro::estTimeStep() 21 0.00631 0.00631 0.00631 0.73% amrex::average_down 410 0.006028 0.006028 0.006028 0.69% Castro::reset_internal_energy(MultiFab) 63 0.004603 0.004603 0.004603 0.53% FabArray::LinComb() 557 0.004588 0.004588 0.004588 0.53% amrex::Add() 164 0.004392 0.004392 0.004392 0.51% Amr::checkPoint() 3 0.003947 0.003947 0.003947 0.45% BndryData::define() 11 0.003792 0.003792 0.003792 0.44% Castro::do_advance_ctu() 10 0.003242 0.003242 0.003242 0.37% Castro::construct_new_gravity_source() 10 0.002578 0.002578 0.002578 0.30% Amr::writePlotFile() 2 0.002347 0.002347 0.002347 0.27% Castro::construct_old_gravity_source() 10 0.002137 0.002137 0.002137 0.25% MLCGSolver::bicgstab 82 0.002134 0.002134 0.002134 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001714 0.001714 0.001714 0.20% Castro::reset_internal_energy(Fab) 504 0.001626 0.001626 0.001626 0.19% Gravity::actual_solve_with_mlmg() 11 0.001463 0.001463 0.001463 0.17% MLCellLinOp::setLevelBC() 11 0.001404 0.001404 0.001404 0.16% FabArray::mult() 43 0.001323 0.001323 0.001323 0.15% FabArray::setDomainBndry() 41 0.001296 0.001296 0.001296 0.15% Castro::initData() 1 0.001253 0.001253 0.001253 0.14% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% MLCellLinOp::prepareForSolve() 11 0.001121 0.001121 0.001121 0.13% MLCellLinOp::smooth() 1640 0.001032 0.001032 0.001032 0.12% Castro::subcycle_advance_ctu() 10 0.0009376 0.0009376 0.0009376 0.11% MLCellLinOp::compGrad() 11 0.0009128 0.0009128 0.0009128 0.11% MLMG::prepareForSolve() 11 0.0008812 0.0008812 0.0008812 0.10% FabArray::FillBoundary() 4023 0.0008255 0.0008255 0.0008255 0.10% Castro::enforce_speed_limit() 62 0.0007959 0.0007959 0.0007959 0.09% FabArrayBase::getCPC() 1323 0.0007369 0.0007369 0.0007369 0.08% FabArrayBase::CPC::define() 454 0.0006954 0.0006954 0.0006954 0.08% Gravity::get_new_grav_vector() 11 0.0006431 0.0006431 0.0006431 0.07% FabArrayBase::getFB() 4023 0.0006119 0.0006119 0.0006119 0.07% Gravity::get_old_grav_vector() 10 0.00056 0.00056 0.00056 0.06% Amr::InitAmr() 1 0.0005237 0.0005237 0.0005237 0.06% MLCellLinOp::apply() 1142 0.0004342 0.0004342 0.0004342 0.05% MLLinOp::defineGrids() 11 0.0003802 0.0003802 0.0003802 0.04% Amr::coarseTimeStep() 10 0.0003762 0.0003762 0.0003762 0.04% MLMG::mgVcycle() 82 0.0003544 0.0003544 0.0003544 0.04% MLCGSolver::ParallelAllReduce 1514 0.000289 0.000289 0.000289 0.03% MultiFab::max() 11 0.000285 0.000285 0.000285 0.03% main() 1 0.0002824 0.0002824 0.0002824 0.03% FabArray::ParallelCopy() 861 0.0002543 0.0002543 0.0002543 0.03% FillPatchIterator::Initialize 41 0.0002264 0.0002264 0.0002264 0.03% MLCellLinOp::defineBC() 11 0.0002203 0.0002203 0.0002203 0.03% MLCellLinOp::correctionResidual() 492 0.0002143 0.0002143 0.0002143 0.02% Amr::timeStep() 10 0.0001743 0.0001743 0.0001743 0.02% Gravity::update_max_rhs() 11 0.00014 0.00014 0.00014 0.02% StateData::checkPoint() 12 0.0001363 0.0001363 0.0001363 0.02% MLMG:computeResOfCorrection() 410 0.0001228 0.0001228 0.0001228 0.01% Gravity::solve_for_phi() 10 0.0001071 0.0001071 0.0001071 0.01% Castro::advance() 10 0.0001058 0.0001058 0.0001058 0.01% Castro::finalize_advance() 10 0.0001019 0.0001019 0.0001019 0.01% MLMG::mgVcycle_down::0 82 9.579e-05 9.579e-05 9.579e-05 0.01% Castro::Castro() 1 9.237e-05 9.237e-05 9.237e-05 0.01% FabArrayBase::FB::FB() 56 9.111e-05 9.111e-05 9.111e-05 0.01% MLMG::actualBottomSolve() 82 8.997e-05 8.997e-05 8.997e-05 0.01% Castro::initialize_advance() 10 8.535e-05 8.535e-05 8.535e-05 0.01% Castro::expand_state() 10 7.977e-05 7.977e-05 7.977e-05 0.01% Castro::create_source_corrector() 10 7.856e-05 7.856e-05 7.856e-05 0.01% MLMG::solve() 11 7.699e-05 7.699e-05 7.699e-05 0.01% AmrLevel::checkPoint() 3 7.636e-05 7.636e-05 7.636e-05 0.01% Castro::clean_state() 62 7.424e-05 7.424e-05 7.424e-05 0.01% MLMG::mgVcycle_down::1 82 7.264e-05 7.264e-05 7.264e-05 0.01% MLMG::mgVcycle_down::2 82 7.105e-05 7.105e-05 7.105e-05 0.01% MLMG::mgVcycle_down::3 82 6.662e-05 6.662e-05 6.662e-05 0.01% MLMG::mgVcycle_down::4 82 6.635e-05 6.635e-05 6.635e-05 0.01% MLMG::mgVcycle_up::4 82 5.702e-05 5.702e-05 5.702e-05 0.01% Castro::initialize_do_advance() 10 5.389e-05 5.389e-05 5.389e-05 0.01% MLMG::oneIter() 82 5.225e-05 5.225e-05 5.225e-05 0.01% MLMG::mgVcycle_up::0 82 5.146e-05 5.146e-05 5.146e-05 0.01% MLMG::mgVcycle_up::1 82 4.901e-05 4.901e-05 4.901e-05 0.01% MLCellLinOp::solutionResidual() 93 4.888e-05 4.888e-05 4.888e-05 0.01% MLMG::mgVcycle_up::3 82 4.782e-05 4.782e-05 4.782e-05 0.01% MLMG::mgVcycle_up::2 82 4.707e-05 4.707e-05 4.707e-05 0.01% Castro::swap_state_time_levels() 10 4.273e-05 4.273e-05 4.273e-05 0.00% MLMG::ResNormInf() 93 3.516e-05 3.516e-05 3.516e-05 0.00% Castro::finalize_do_advance() 10 3.484e-05 3.484e-05 3.484e-05 0.00% StateData::define() 4 3.472e-05 3.472e-05 3.472e-05 0.00% MLMG::mgVcycle_bottom 82 3.391e-05 3.391e-05 3.391e-05 0.00% Castro::enforce_consistent_e() 1 3.325e-05 3.325e-05 3.325e-05 0.00% MLMG::computeResidual() 82 2.959e-05 2.959e-05 2.959e-05 0.00% FillPatchSingleLevel 41 2.913e-05 2.913e-05 2.913e-05 0.00% MLPoisson::define() 11 2.851e-05 2.851e-05 2.851e-05 0.00% Castro::construct_new_gravity() 10 2.775e-05 2.775e-05 2.775e-05 0.00% Amr::writeSmallPlotFile() 1 2.618e-05 2.618e-05 2.618e-05 0.00% Castro::initMFs() 1 2.546e-05 2.546e-05 2.546e-05 0.00% makeSFC 55 2.294e-05 2.294e-05 2.294e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.23e-05 2.23e-05 2.23e-05 0.00% Amr::FinalizeInit() 1 1.985e-05 1.985e-05 1.985e-05 0.00% Castro::construct_old_source() 50 1.97e-05 1.97e-05 1.97e-05 0.00% Amr::defBaseLevel() 1 1.953e-05 1.953e-05 1.953e-05 0.00% Castro::construct_new_source() 50 1.898e-05 1.898e-05 1.898e-05 0.00% Castro::buildMetrics() 1 1.874e-05 1.874e-05 1.874e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.724e-05 1.724e-05 1.724e-05 0.00% Castro::do_new_sources() 10 1.688e-05 1.688e-05 1.688e-05 0.00% Castro::do_old_sources() 10 1.666e-05 1.666e-05 1.666e-05 0.00% MLPoisson::prepareForSolve() 11 1.481e-05 1.481e-05 1.481e-05 0.00% DistributionMapping::Distribute() 56 1.42e-05 1.42e-05 1.42e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.41e-05 1.41e-05 1.41e-05 0.00% MLMG::computeMLResidual() 11 1.391e-05 1.391e-05 1.391e-05 0.00% MLLinOp::define() 11 1.07e-05 1.07e-05 1.07e-05 0.00% Castro::check_for_nan() 20 1.052e-05 1.052e-05 1.052e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.051e-05 1.051e-05 1.051e-05 0.00% Castro::construct_old_gravity() 10 1.047e-05 1.047e-05 1.047e-05 0.00% Gravity::swapTimeLevels() 10 1.034e-05 1.034e-05 1.034e-05 0.00% Castro::post_timestep() 10 9.538e-06 9.538e-06 9.538e-06 0.00% Castro::apply_source_to_state() 20 9.206e-06 9.206e-06 9.206e-06 0.00% Amr::initSubcycle() 1 8.809e-06 8.809e-06 8.809e-06 0.00% Gravity::actual_multilevel_solve() 1 7.875e-06 7.875e-06 7.875e-06 0.00% Castro::computeNewDt() 9 7.213e-06 7.213e-06 7.213e-06 0.00% MLMG::getGradSolution() 11 5.588e-06 5.588e-06 5.588e-06 0.00% Gravity::set_mass_offset() 11 5.251e-06 5.251e-06 5.251e-06 0.00% Castro::retry_advance_ctu() 10 4.916e-06 4.916e-06 4.916e-06 0.00% Amr::InitializeInit() 1 4.591e-06 4.591e-06 4.591e-06 0.00% Castro::FluxRegFineAdd() 10 4.359e-06 4.359e-06 4.359e-06 0.00% Castro::post_init() 1 4.078e-06 4.078e-06 4.078e-06 0.00% AmrLevel::checkPointPost() 3 3.981e-06 3.981e-06 3.981e-06 0.00% MLMG::MLRhsNormInf() 11 3.763e-06 3.763e-06 3.763e-06 0.00% Castro::FluxRegCrseInit 10 3.603e-06 3.603e-06 3.603e-06 0.00% MLMG::MLResNormInf() 11 3.538e-06 3.538e-06 3.538e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.47e-06 3.47e-06 3.47e-06 0.00% Amr::init() 1 2.921e-06 2.921e-06 2.921e-06 0.00% Castro::computeInitialDt() 2 2.645e-06 2.645e-06 2.645e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.895e-06 1.895e-06 1.895e-06 0.00% AmrLevel::checkPointPre() 3 1.829e-06 1.829e-06 1.829e-06 0.00% Amr::initialInit() 1 1.399e-06 1.399e-06 1.399e-06 0.00% Castro::post_regrid() 1 1.083e-06 1.083e-06 1.083e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8686 0.8686 0.8686 100.00% Amr::coarseTimeStep() 10 0.6938 0.6938 0.6938 79.87% Amr::timeStep() 10 0.5725 0.5725 0.5725 65.91% Castro::advance() 10 0.5641 0.5641 0.5641 64.94% Castro::subcycle_advance_ctu() 10 0.5526 0.5526 0.5526 63.61% Castro::do_advance_ctu() 10 0.5516 0.5516 0.5516 63.51% Gravity::solve_phi_with_mlmg() 11 0.2851 0.2851 0.2851 32.83% Gravity::actual_solve_with_mlmg() 11 0.278 0.278 0.278 32.00% Castro::construct_new_gravity() 10 0.2604 0.2604 0.2604 29.98% MLMG::solve() 11 0.2571 0.2571 0.2571 29.60% Gravity::solve_for_phi() 10 0.2444 0.2444 0.2444 28.14% MLMG::oneIter() 82 0.2427 0.2427 0.2427 27.94% MLMG::mgVcycle() 82 0.239 0.239 0.239 27.52% VisMF::Write(FabArray) 11 0.235 0.235 0.235 27.05% Castro::construct_ctu_hydro_source() 10 0.2114 0.2114 0.2114 24.33% Amr::checkPoint() 3 0.1748 0.1748 0.1748 20.12% AmrLevel::checkPoint() 3 0.1708 0.1708 0.1708 19.66% StateData::checkPoint() 12 0.1707 0.1707 0.1707 19.65% Amr::init() 1 0.1403 0.1403 0.1403 16.15% MLCellLinOp::smooth() 1640 0.117 0.117 0.117 13.47% MLCellLinOp::applyBC() 4433 0.1095 0.1095 0.1095 12.60% MLMG::mgVcycle_bottom 82 0.07434 0.07434 0.07434 8.56% MLMG::actualBottomSolve() 82 0.07431 0.07431 0.07431 8.55% MLCGSolver::bicgstab 82 0.0736 0.0736 0.0736 8.47% Amr::writePlotFile() 2 0.06726 0.06726 0.06726 7.74% Castro::clean_state() 62 0.04886 0.04886 0.04886 5.63% Amr::initialInit() 1 0.04829 0.04829 0.04829 5.56% Amr::FinalizeInit() 1 0.04425 0.04425 0.04425 5.09% FillPatchIterator::Initialize 41 0.04349 0.04349 0.04349 5.01% Castro::post_init() 1 0.04295 0.04295 0.04295 4.94% FillPatchSingleLevel 41 0.04197 0.04197 0.04197 4.83% Gravity::multilevel_solve_for_new_phi() 1 0.04113 0.04113 0.04113 4.73% Gravity::actual_multilevel_solve() 1 0.04111 0.04111 0.04111 4.73% StateDataPhysBCFunct::() 41 0.03786 0.03786 0.03786 4.36% MLCellLinOp::apply() 1142 0.03648 0.03648 0.03648 4.20% MLMG::mgVcycle_down::0 82 0.03372 0.03372 0.03372 3.88% MLPoisson::Fsmooth() 3280 0.03255 0.03255 0.03255 3.75% FabArray::FillBoundary() 4023 0.03221 0.03221 0.03221 3.71% FillBoundary_nowait() 4023 0.03139 0.03139 0.03139 3.61% MLMG::mgVcycle_up::0 82 0.02551 0.02551 0.02551 2.94% StateData::FillBoundary(geom) 328 0.02505 0.02505 0.02505 2.88% MLCellLinOp::correctionResidual() 492 0.02232 0.02232 0.02232 2.57% Castro::computeTemp() 63 0.02133 0.02133 0.02133 2.46% amrex::Dot() 1114 0.02083 0.02083 0.02083 2.40% MLMG:computeResOfCorrection() 410 0.01966 0.01966 0.01966 2.26% Castro::initialize_do_advance() 10 0.01943 0.01943 0.01943 2.24% Gravity::get_new_grav_vector() 11 0.01752 0.01752 0.01752 2.02% MLPoisson::define() 11 0.01662 0.01662 0.01662 1.91% Castro::normalize_species() 62 0.01605 0.01605 0.01605 1.85% MLMG::mgVcycle_down::1 82 0.01585 0.01585 0.01585 1.82% Castro::construct_old_gravity() 10 0.01579 0.01579 0.01579 1.82% Gravity::get_old_grav_vector() 10 0.01578 0.01578 0.01578 1.82% amrex::Copy() 1029 0.01492 0.01492 0.01492 1.72% MLMG::mgVcycle_down::2 82 0.01489 0.01489 0.01489 1.71% FabArray::ParallelCopy() 861 0.01461 0.01461 0.01461 1.68% MLMG::mgVcycle_down::3 82 0.01455 0.01455 0.01455 1.68% FabArray::norminf() 743 0.0145 0.0145 0.0145 1.67% FabArray::ParallelCopy_nowait() 861 0.01435 0.01435 0.01435 1.65% MLMG::mgVcycle_down::4 82 0.01431 0.01431 0.01431 1.65% FabArray::setVal() 1144 0.01331 0.01331 0.01331 1.53% Castro::do_new_sources() 10 0.0125 0.0125 0.0125 1.44% MLCGSolver::ParallelAllReduce 1514 0.01249 0.01249 0.01249 1.44% MLMG::addInterpCorrection() 410 0.01208 0.01208 0.01208 1.39% MLMG::mgVcycle_up::4 82 0.01163 0.01163 0.01163 1.34% Castro::expand_state() 10 0.01155 0.01155 0.01155 1.33% MLMG::mgVcycle_up::1 82 0.01153 0.01153 0.01153 1.33% Castro::enforce_min_density() 62 0.01134 0.01134 0.01134 1.31% amrex::average_down 410 0.01132 0.01132 0.01132 1.30% MLMG::mgVcycle_up::2 82 0.01128 0.01128 0.01128 1.30% MLCellLinOp::defineAuxData() 11 0.01114 0.01114 0.01114 1.28% MLMG::mgVcycle_up::3 82 0.01106 0.01106 0.01106 1.27% Castro::initialize_advance() 10 0.01077 0.01077 0.01077 1.24% MLPoisson::Fapply() 1142 0.01045 0.01045 0.01045 1.20% Castro::do_old_sources() 10 0.0102 0.0102 0.0102 1.17% Castro::post_timestep() 10 0.008257 0.008257 0.008257 0.95% FabArray::Saxpy() 813 0.008249 0.008249 0.008249 0.95% FabArray::Xpay() 821 0.008142 0.008142 0.008142 0.94% MLCellLinOp::solutionResidual() 93 0.007227 0.007227 0.007227 0.83% Gravity::fill_multipole_BCs() 11 0.00695 0.00695 0.00695 0.80% Castro::estTimeStep() 21 0.00631 0.00631 0.00631 0.73% Castro::reset_internal_energy(MultiFab) 63 0.006229 0.006229 0.006229 0.72% MLMG::computeResidual() 82 0.006213 0.006213 0.006213 0.72% MLCellLinOp::defineBC() 11 0.005011 0.005011 0.005011 0.58% BndryData::define() 11 0.004791 0.004791 0.004791 0.55% MLMG::prepareForSolve() 11 0.0047 0.0047 0.0047 0.54% FabArray::LinComb() 557 0.004588 0.004588 0.004588 0.53% amrex::Add() 164 0.004392 0.004392 0.004392 0.51% Amr::InitializeInit() 1 0.004038 0.004038 0.004038 0.46% Amr::defBaseLevel() 1 0.004034 0.004034 0.004034 0.46% Castro::computeNewDt() 9 0.003836 0.003836 0.003836 0.44% Castro::initData() 1 0.003512 0.003512 0.003512 0.40% Castro::construct_new_source() 50 0.002597 0.002597 0.002597 0.30% Castro::construct_new_gravity_source() 10 0.002578 0.002578 0.002578 0.30% Castro::construct_old_source() 50 0.002156 0.002156 0.002156 0.25% Castro::construct_old_gravity_source() 10 0.002137 0.002137 0.002137 0.25% MLMG::ResNormInf() 93 0.002099 0.002099 0.002099 0.24% Castro::apply_source_to_state() 20 0.001827 0.001827 0.001827 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001714 0.001714 0.001714 0.20% Castro::reset_internal_energy(Fab) 504 0.001626 0.001626 0.001626 0.19% FabArrayBase::getCPC() 1323 0.001432 0.001432 0.001432 0.16% MLCellLinOp::setLevelBC() 11 0.001404 0.001404 0.001404 0.16% MLMG::getGradSolution() 11 0.001399 0.001399 0.001399 0.16% MLCellLinOp::compGrad() 11 0.001393 0.001393 0.001393 0.16% FabArray::mult() 43 0.001323 0.001323 0.001323 0.15% FabArray::setDomainBndry() 41 0.001296 0.001296 0.001296 0.15% Castro::check_for_nan() 20 0.001204 0.001204 0.001204 0.14% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% MLPoisson::prepareForSolve() 11 0.001136 0.001136 0.001136 0.13% Castro::post_regrid() 1 0.001132 0.001132 0.001132 0.13% MLCellLinOp::prepareForSolve() 11 0.001121 0.001121 0.001121 0.13% MLMG::computeMLResidual() 11 0.001058 0.001058 0.001058 0.12% Castro::computeInitialDt() 2 0.0008927 0.0008927 0.0008927 0.10% Gravity::update_max_rhs() 11 0.000885 0.000885 0.000885 0.10% Castro::enforce_speed_limit() 62 0.0007959 0.0007959 0.0007959 0.09% FabArrayBase::getFB() 4023 0.000703 0.000703 0.000703 0.08% FabArrayBase::CPC::define() 454 0.0006954 0.0006954 0.0006954 0.08% Castro::finalize_advance() 10 0.0006402 0.0006402 0.0006402 0.07% Amr::InitAmr() 1 0.0005325 0.0005325 0.0005325 0.06% Gravity::swapTimeLevels() 10 0.0004853 0.0004853 0.0004853 0.06% Castro::Castro() 1 0.0004523 0.0004523 0.0004523 0.05% MLLinOp::define() 11 0.0004429 0.0004429 0.0004429 0.05% MLLinOp::defineGrids() 11 0.0004322 0.0004322 0.0004322 0.05% MLMG::MLResNormInf() 11 0.0002873 0.0002873 0.0002873 0.03% MultiFab::max() 11 0.000285 0.000285 0.000285 0.03% MLMG::MLRhsNormInf() 11 0.0002219 0.0002219 0.0002219 0.03% Castro::buildMetrics() 1 0.0001677 0.0001677 0.0001677 0.02% FabArrayBase::FB::FB() 56 9.111e-05 9.111e-05 9.111e-05 0.01% Castro::create_source_corrector() 10 7.856e-05 7.856e-05 7.856e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.003e-05 5.003e-05 5.003e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.523e-05 4.523e-05 4.523e-05 0.01% Castro::swap_state_time_levels() 10 4.273e-05 4.273e-05 4.273e-05 0.00% makeSFC 55 3.593e-05 3.593e-05 3.593e-05 0.00% Castro::finalize_do_advance() 10 3.484e-05 3.484e-05 3.484e-05 0.00% StateData::define() 4 3.472e-05 3.472e-05 3.472e-05 0.00% Castro::enforce_consistent_e() 1 3.325e-05 3.325e-05 3.325e-05 0.00% Amr::writeSmallPlotFile() 1 2.618e-05 2.618e-05 2.618e-05 0.00% Castro::initMFs() 1 2.546e-05 2.546e-05 2.546e-05 0.00% DistributionMapping::Distribute() 56 1.42e-05 1.42e-05 1.42e-05 0.00% Amr::initSubcycle() 1 8.809e-06 8.809e-06 8.809e-06 0.00% Gravity::set_mass_offset() 11 5.251e-06 5.251e-06 5.251e-06 0.00% Castro::retry_advance_ctu() 10 4.916e-06 4.916e-06 4.916e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.684e-06 4.684e-06 4.684e-06 0.00% Castro::FluxRegFineAdd() 10 4.359e-06 4.359e-06 4.359e-06 0.00% AmrLevel::checkPointPost() 3 3.981e-06 3.981e-06 3.981e-06 0.00% Castro::FluxRegCrseInit 10 3.603e-06 3.603e-06 3.603e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.895e-06 1.895e-06 1.895e-06 0.00% AmrLevel::checkPointPre() 3 1.829e-06 1.829e-06 1.829e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 94 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 110 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1059 KiB 39 MiB Castro::initialize_do_advance() 80 80 24 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2176 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7610 KiB 14 MiB MLMG::prepareForSolve() 660 660 3641 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 209 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 186 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7525 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1935 B 2048 KiB Gravity::solve_for_phi() 80 80 575 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 96 KiB 2048 KiB BndryData::define() 1056 1056 332 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 211 KiB 671 KiB Castro::estTimeStep() 21 21 3068 B 480 KiB VisMF::Write(FabArray) 656 656 3494 B 320 KiB Castro::normalize_species() 62 62 6015 B 320 KiB amrex::average_down 1067 1067 1288 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1190 B 257 KiB amrex::Dot() 1360 1360 3532 B 160 KiB FabArray::norminf() 907 907 2464 B 160 KiB Castro::do_advance_ctu() 10 10 583 B 160 KiB MultiFab::max() 11 11 50 B 160 KiB Castro::initData() 1 1 27 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3559 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 309 B 9648 B MLCellLinOp::applyBC() 8866 8866 230 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3858 B 6144 B StateData::FillBoundary(geom) 1992 1992 50 B 2976 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 123 B 1472 B MLCellLinOp::defineBC() 66 66 374 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 600 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 50 KiB 8192 KiB VisMF::Write(FabArray) 744 744 631 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3559 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 309 B 9648 B MLCellLinOp::applyBC() 4433 4433 228 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3858 B 6144 B Gravity::get_new_grav_vector() 3 3 2901 B 3072 B StateData::FillBoundary(geom) 1992 1992 50 B 2976 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 303 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 907 907 18 B 272 B Castro::estTimeStep() 21 21 0 B 32 B MultiFab::contains_nan() 20 20 0 B 16 B MultiFab::max() 11 11 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B Castro::do_advance_ctu() 10 10 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-28-g831d1487fa19) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-28-g831d1487fa19) initialized Starting run at 09:09:50 UTC on 2023-03-29. Successfully read inputs file ... Castro git describe: 23.03-17-gf84bee6a2 AMReX git describe: 23.03-28-g831d1487f Microphysics git describe: 23.03-23-g27bca043 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.470291663 Restart time = 0.045944025 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051487515 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048450594 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057883167 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059762023 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.075309034 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033072929 seconds Ending run at 09:09:51 UTC on 2023-03-29. Run time = 0.373071564 Run time without initialization = 0.3265381 Average number of zones advanced per microsecond: 4.014 Average number of zones advanced per microsecond per rank: 4.014 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3731 ... 0.3731 ... 0.3731 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1070 0.1070 0.1070 28.67% VisMF::Read() 3 0.03953 0.03953 0.03953 10.60% MLCellLinOp::applyBC() 1946 0.03325 0.03325 0.03325 8.91% VisMF::Write(FabArray) 1 0.03152 0.03152 0.03152 8.45% MLPoisson::Fsmooth() 1440 0.01405 0.01405 0.01405 3.77% FillBoundary_nowait() 1766 0.01282 0.01282 0.01282 3.43% StateData::FillBoundary(geom) 160 0.01207 0.01207 0.01207 3.23% Castro::normalize_species() 30 0.009539 0.009539 0.009539 2.56% amrex::Dot() 484 0.008883 0.008883 0.008883 2.38% amrex::Copy() 463 0.006982 0.006982 0.006982 1.87% Castro::computeTemp() 30 0.006954 0.006954 0.006954 1.86% FabArray::setVal() 537 0.006335 0.006335 0.006335 1.70% FabArray::norminf() 326 0.006222 0.006222 0.006222 1.67% FabArray::ParallelCopy_nowait() 380 0.006149 0.006149 0.006149 1.65% StateDataPhysBCFunct::() 20 0.005909 0.005909 0.005909 1.58% MLCellLinOp::defineAuxData() 6 0.005602 0.005602 0.005602 1.50% Castro::enforce_min_density() 30 0.005538 0.005538 0.005538 1.48% MLPoisson::Fapply() 500 0.004457 0.004457 0.004457 1.19% FabArray::Saxpy() 355 0.003642 0.003642 0.003642 0.98% FabArray::Xpay() 361 0.003528 0.003528 0.003528 0.95% MLMG::addInterpCorrection() 180 0.003008 0.003008 0.003008 0.81% Gravity::fill_multipole_BCs() 6 0.002754 0.002754 0.002754 0.74% amrex::average_down 180 0.002679 0.002679 0.002679 0.72% Castro::estTimeStep() 10 0.002516 0.002516 0.002516 0.67% Amr::restart() 1 0.002509 0.002509 0.002509 0.67% BndryData::define() 6 0.002138 0.002138 0.002138 0.57% FabArray::LinComb() 242 0.001932 0.001932 0.001932 0.52% amrex::Add() 72 0.001853 0.001853 0.001853 0.50% Castro::reset_internal_energy(MultiFab) 30 0.001853 0.001853 0.001853 0.50% Castro::construct_new_gravity_source() 5 0.001739 0.001739 0.001739 0.47% Castro::do_advance_ctu() 5 0.001738 0.001738 0.001738 0.47% Amr::writePlotFile() 1 0.001397 0.001397 0.001397 0.37% Castro::construct_old_gravity_source() 5 0.001339 0.001339 0.001339 0.36% MLCGSolver::bicgstab 36 0.0009544 0.0009544 0.0009544 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009284 0.0009284 0.0009284 0.25% Castro::post_timestep() 5 0.000851 0.000851 0.000851 0.23% Castro::reset_internal_energy(Fab) 240 0.00085 0.00085 0.00085 0.23% Gravity::actual_solve_with_mlmg() 6 0.0007908 0.0007908 0.0007908 0.21% MLCellLinOp::setLevelBC() 6 0.0007437 0.0007437 0.0007437 0.20% FabArray::setDomainBndry() 20 0.0006696 0.0006696 0.0006696 0.18% FabArray::mult() 22 0.0006472 0.0006472 0.0006472 0.17% MLCellLinOp::prepareForSolve() 6 0.0006167 0.0006167 0.0006167 0.17% MultiFab::contains_nan() 10 0.0005935 0.0005935 0.0005935 0.16% MLMG::prepareForSolve() 6 0.0004884 0.0004884 0.0004884 0.13% MLCellLinOp::compGrad() 6 0.0004847 0.0004847 0.0004847 0.13% Castro::enforce_speed_limit() 30 0.0004843 0.0004843 0.0004843 0.13% MLCellLinOp::smooth() 720 0.000462 0.000462 0.000462 0.12% FabArrayBase::CPC::define() 244 0.0004251 0.0004251 0.0004251 0.11% Amr::InitAmr() 1 0.0003989 0.0003989 0.0003989 0.11% FabArrayBase::getCPC() 632 0.0003675 0.0003675 0.0003675 0.10% FabArray::FillBoundary() 1766 0.0003624 0.0003624 0.0003624 0.10% Gravity::get_old_grav_vector() 5 0.0003496 0.0003496 0.0003496 0.09% main() 1 0.0003029 0.0003029 0.0003029 0.08% Amr::coarseTimeStep() 5 0.0002918 0.0002918 0.0002918 0.08% Gravity::get_new_grav_vector() 5 0.0002741 0.0002741 0.0002741 0.07% FabArrayBase::getFB() 1766 0.0002545 0.0002545 0.0002545 0.07% MLCellLinOp::apply() 500 0.0001875 0.0001875 0.0001875 0.05% MLMG::mgVcycle() 36 0.000159 0.000159 0.000159 0.04% MLLinOp::defineGrids() 6 0.0001368 0.0001368 0.0001368 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.04% MLCGSolver::ParallelAllReduce 659 0.0001263 0.0001263 0.0001263 0.03% Castro::construct_new_source() 25 0.0001227 0.0001227 0.0001227 0.03% MLCellLinOp::defineBC() 6 0.0001204 0.0001204 0.0001204 0.03% FillPatchIterator::Initialize 20 0.0001194 0.0001194 0.0001194 0.03% FabArray::ParallelCopy() 380 0.0001162 0.0001162 0.0001162 0.03% Castro::subcycle_advance_ctu() 5 0.0001134 0.0001134 0.0001134 0.03% MLCellLinOp::correctionResidual() 216 9.467e-05 9.467e-05 9.467e-05 0.03% Amr::timeStep() 5 9.462e-05 9.462e-05 9.462e-05 0.03% Castro::finalize_advance() 5 7.703e-05 7.703e-05 7.703e-05 0.02% AmrLevel::restart() 1 7.424e-05 7.424e-05 7.424e-05 0.02% StateData::restartDoit() 4 6.684e-05 6.684e-05 6.684e-05 0.02% Gravity::solve_for_phi() 5 6.466e-05 6.466e-05 6.466e-05 0.02% FabArrayBase::FB::FB() 26 6.173e-05 6.173e-05 6.173e-05 0.02% Gravity::update_max_rhs() 6 6.153e-05 6.153e-05 6.153e-05 0.02% Castro::construct_new_gravity() 5 5.212e-05 5.212e-05 5.212e-05 0.01% MLMG:computeResOfCorrection() 180 5.164e-05 5.164e-05 5.164e-05 0.01% Castro::expand_state() 5 4.899e-05 4.899e-05 4.899e-05 0.01% Castro::clean_state() 30 4.164e-05 4.164e-05 4.164e-05 0.01% MLMG::actualBottomSolve() 36 4.16e-05 4.16e-05 4.16e-05 0.01% MLMG::mgVcycle_down::0 36 3.935e-05 3.935e-05 3.935e-05 0.01% Castro::advance() 5 3.482e-05 3.482e-05 3.482e-05 0.01% Castro::initialize_advance() 5 3.481e-05 3.481e-05 3.481e-05 0.01% MLMG::solve() 6 3.476e-05 3.476e-05 3.476e-05 0.01% MLMG::mgVcycle_down::3 36 3.395e-05 3.395e-05 3.395e-05 0.01% MLMG::mgVcycle_down::1 36 3.319e-05 3.319e-05 3.319e-05 0.01% MLMG::mgVcycle_down::2 36 3.164e-05 3.164e-05 3.164e-05 0.01% MLMG::mgVcycle_down::4 36 3.142e-05 3.142e-05 3.142e-05 0.01% Castro::initialize_do_advance() 5 2.762e-05 2.762e-05 2.762e-05 0.01% MLMG::mgVcycle_up::4 36 2.731e-05 2.731e-05 2.731e-05 0.01% Castro::buildMetrics() 1 2.724e-05 2.724e-05 2.724e-05 0.01% Amr::writeSmallPlotFile() 1 2.666e-05 2.666e-05 2.666e-05 0.01% Castro::initMFs() 1 2.653e-05 2.653e-05 2.653e-05 0.01% MLPoisson::prepareForSolve() 6 2.594e-05 2.594e-05 2.594e-05 0.01% MLMG::mgVcycle_up::1 36 2.578e-05 2.578e-05 2.578e-05 0.01% MLMG::oneIter() 36 2.358e-05 2.358e-05 2.358e-05 0.01% Castro::swap_state_time_levels() 5 2.342e-05 2.342e-05 2.342e-05 0.01% MLMG::mgVcycle_up::0 36 2.284e-05 2.284e-05 2.284e-05 0.01% Castro::create_source_corrector() 5 2.276e-05 2.276e-05 2.276e-05 0.01% MLMG::mgVcycle_up::3 36 2.239e-05 2.239e-05 2.239e-05 0.01% MLMG::mgVcycle_up::2 36 2.14e-05 2.14e-05 2.14e-05 0.01% Castro::computeNewDt() 5 2.12e-05 2.12e-05 2.12e-05 0.01% MLCellLinOp::solutionResidual() 42 2.072e-05 2.072e-05 2.072e-05 0.01% Castro::post_restart() 1 1.959e-05 1.959e-05 1.959e-05 0.01% Castro::finalize_do_advance() 5 1.952e-05 1.952e-05 1.952e-05 0.01% MLMG::ResNormInf() 42 1.736e-05 1.736e-05 1.736e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.667e-05 1.667e-05 1.667e-05 0.00% MLPoisson::define() 6 1.57e-05 1.57e-05 1.57e-05 0.00% MLMG::mgVcycle_bottom 36 1.526e-05 1.526e-05 1.526e-05 0.00% FillPatchSingleLevel 20 1.427e-05 1.427e-05 1.427e-05 0.00% MLMG::computeResidual() 36 1.326e-05 1.326e-05 1.326e-05 0.00% makeSFC 30 1.29e-05 1.29e-05 1.29e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.236e-05 1.236e-05 1.236e-05 0.00% Castro::construct_old_source() 25 1.017e-05 1.017e-05 1.017e-05 0.00% MLMG::computeMLResidual() 6 9.766e-06 9.766e-06 9.766e-06 0.00% Castro::do_new_sources() 5 9.121e-06 9.121e-06 9.121e-06 0.00% DistributionMapping::Distribute() 31 9.072e-06 9.072e-06 9.072e-06 0.00% Gravity::actual_multilevel_solve() 1 8.313e-06 8.313e-06 8.313e-06 0.00% Amr::initSubcycle() 1 8.311e-06 8.311e-06 8.311e-06 0.00% Castro::do_old_sources() 5 8.189e-06 8.189e-06 8.189e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.165e-06 7.165e-06 7.165e-06 0.00% Castro::apply_source_to_state() 10 5.676e-06 5.676e-06 5.676e-06 0.00% Castro::construct_old_gravity() 5 5.519e-06 5.519e-06 5.519e-06 0.00% Castro::check_for_nan() 10 5.363e-06 5.363e-06 5.363e-06 0.00% MLLinOp::define() 6 5.266e-06 5.266e-06 5.266e-06 0.00% Gravity::swapTimeLevels() 5 4.368e-06 4.368e-06 4.368e-06 0.00% Gravity::set_mass_offset() 6 3.841e-06 3.841e-06 3.841e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.966e-06 2.966e-06 2.966e-06 0.00% MLMG::getGradSolution() 6 2.94e-06 2.94e-06 2.94e-06 0.00% MLMG::MLResNormInf() 6 2.295e-06 2.295e-06 2.295e-06 0.00% MLMG::MLRhsNormInf() 6 2.205e-06 2.205e-06 2.205e-06 0.00% Castro::retry_advance_ctu() 5 1.788e-06 1.788e-06 1.788e-06 0.00% Castro::FluxRegCrseInit 5 1.446e-06 1.446e-06 1.446e-06 0.00% Castro::FluxRegFineAdd() 5 1.305e-06 1.305e-06 1.305e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.058e-06 1.058e-06 1.058e-06 0.00% Amr::init() 1 9.74e-07 9.74e-07 9.74e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3731 0.3731 0.3731 100.00% Amr::coarseTimeStep() 5 0.2932 0.2932 0.2932 78.58% Amr::timeStep() 5 0.2916 0.2916 0.2916 78.14% Castro::advance() 5 0.2852 0.2852 0.2852 76.45% Castro::subcycle_advance_ctu() 5 0.2793 0.2793 0.2793 74.85% Castro::do_advance_ctu() 5 0.2791 0.2791 0.2791 74.82% Castro::construct_new_gravity() 5 0.1301 0.1301 0.1301 34.88% Gravity::solve_phi_with_mlmg() 6 0.1255 0.1255 0.1255 33.64% Gravity::actual_solve_with_mlmg() 6 0.1225 0.1225 0.1225 32.84% Gravity::solve_for_phi() 5 0.1223 0.1223 0.1223 32.78% MLMG::solve() 6 0.1109 0.1109 0.1109 29.72% Castro::construct_ctu_hydro_source() 5 0.107 0.107 0.107 28.67% MLMG::oneIter() 36 0.104 0.104 0.104 27.86% MLMG::mgVcycle() 36 0.1024 0.1024 0.1024 27.45% MLCellLinOp::smooth() 720 0.04998 0.04998 0.04998 13.39% MLCellLinOp::applyBC() 1946 0.04674 0.04674 0.04674 12.53% Amr::init() 1 0.046 0.046 0.046 12.33% Amr::restart() 1 0.046 0.046 0.046 12.33% AmrLevel::restart() 1 0.03973 0.03973 0.03973 10.65% StateData::restartDoit() 4 0.03965 0.03965 0.03965 10.63% VisMF::Read() 3 0.03953 0.03953 0.03953 10.60% Amr::writePlotFile() 1 0.03317 0.03317 0.03317 8.89% MLMG::mgVcycle_bottom 36 0.03171 0.03171 0.03171 8.50% MLMG::actualBottomSolve() 36 0.0317 0.0317 0.0317 8.50% VisMF::Write(FabArray) 1 0.03152 0.03152 0.03152 8.45% MLCGSolver::bicgstab 36 0.03139 0.03139 0.03139 8.41% Castro::clean_state() 30 0.02526 0.02526 0.02526 6.77% FillPatchIterator::Initialize 20 0.02081 0.02081 0.02081 5.58% FillPatchSingleLevel 20 0.02002 0.02002 0.02002 5.37% StateDataPhysBCFunct::() 20 0.01797 0.01797 0.01797 4.82% MLCellLinOp::apply() 500 0.01566 0.01566 0.01566 4.20% MLMG::mgVcycle_down::0 36 0.01434 0.01434 0.01434 3.84% MLPoisson::Fsmooth() 1440 0.01405 0.01405 0.01405 3.77% FabArray::FillBoundary() 1766 0.01349 0.01349 0.01349 3.62% FillBoundary_nowait() 1766 0.01313 0.01313 0.01313 3.52% StateData::FillBoundary(geom) 160 0.01207 0.01207 0.01207 3.23% MLMG::mgVcycle_up::0 36 0.01084 0.01084 0.01084 2.91% Castro::initialize_do_advance() 5 0.009838 0.009838 0.009838 2.64% Castro::computeTemp() 30 0.009658 0.009658 0.009658 2.59% MLCellLinOp::correctionResidual() 216 0.009554 0.009554 0.009554 2.56% Castro::normalize_species() 30 0.009539 0.009539 0.009539 2.56% MLPoisson::define() 6 0.009377 0.009377 0.009377 2.51% amrex::Dot() 484 0.008883 0.008883 0.008883 2.38% MLMG:computeResOfCorrection() 180 0.008393 0.008393 0.008393 2.25% Castro::construct_old_gravity() 5 0.007998 0.007998 0.007998 2.14% Gravity::get_old_grav_vector() 5 0.007992 0.007992 0.007992 2.14% Gravity::get_new_grav_vector() 5 0.007663 0.007663 0.007663 2.05% Castro::do_new_sources() 5 0.007097 0.007097 0.007097 1.90% amrex::Copy() 463 0.006982 0.006982 0.006982 1.87% MLMG::mgVcycle_down::1 36 0.006842 0.006842 0.006842 1.83% FabArray::ParallelCopy() 380 0.006656 0.006656 0.006656 1.78% FabArray::ParallelCopy_nowait() 380 0.006539 0.006539 0.006539 1.75% MLMG::mgVcycle_down::2 36 0.006398 0.006398 0.006398 1.71% MLCellLinOp::defineAuxData() 6 0.006361 0.006361 0.006361 1.70% Castro::do_old_sources() 5 0.006337 0.006337 0.006337 1.70% FabArray::setVal() 537 0.006335 0.006335 0.006335 1.70% MLMG::mgVcycle_down::3 36 0.006251 0.006251 0.006251 1.68% FabArray::norminf() 326 0.006222 0.006222 0.006222 1.67% Castro::post_timestep() 5 0.006211 0.006211 0.006211 1.66% MLMG::mgVcycle_down::4 36 0.0062 0.0062 0.0062 1.66% Castro::expand_state() 5 0.00588 0.00588 0.00588 1.58% Castro::initialize_advance() 5 0.005606 0.005606 0.005606 1.50% Castro::enforce_min_density() 30 0.005538 0.005538 0.005538 1.48% MLCGSolver::ParallelAllReduce 659 0.005355 0.005355 0.005355 1.44% MLMG::addInterpCorrection() 180 0.005293 0.005293 0.005293 1.42% amrex::average_down 180 0.005013 0.005013 0.005013 1.34% MLMG::mgVcycle_up::4 36 0.005005 0.005005 0.005005 1.34% MLMG::mgVcycle_up::1 36 0.004964 0.004964 0.004964 1.33% MLMG::mgVcycle_up::2 36 0.004899 0.004899 0.004899 1.31% MLMG::mgVcycle_up::3 36 0.00479 0.00479 0.00479 1.28% MLPoisson::Fapply() 500 0.004457 0.004457 0.004457 1.19% FabArray::Saxpy() 355 0.003642 0.003642 0.003642 0.98% Castro::post_restart() 1 0.003592 0.003592 0.003592 0.96% FabArray::Xpay() 361 0.003528 0.003528 0.003528 0.95% Gravity::multilevel_solve_for_new_phi() 1 0.003469 0.003469 0.003469 0.93% Gravity::actual_multilevel_solve() 1 0.003453 0.003453 0.003453 0.93% MLCellLinOp::solutionResidual() 42 0.003232 0.003232 0.003232 0.87% Gravity::fill_multipole_BCs() 6 0.002875 0.002875 0.002875 0.77% MLCellLinOp::defineBC() 6 0.00283 0.00283 0.00283 0.76% BndryData::define() 6 0.002709 0.002709 0.002709 0.73% Castro::reset_internal_energy(MultiFab) 30 0.002703 0.002703 0.002703 0.72% MLMG::computeResidual() 36 0.002675 0.002675 0.002675 0.72% MLMG::prepareForSolve() 6 0.002592 0.002592 0.002592 0.69% Castro::estTimeStep() 10 0.002516 0.002516 0.002516 0.67% FabArray::LinComb() 242 0.001932 0.001932 0.001932 0.52% Castro::construct_new_source() 25 0.001861 0.001861 0.001861 0.50% amrex::Add() 72 0.001853 0.001853 0.001853 0.50% Castro::construct_new_gravity_source() 5 0.001739 0.001739 0.001739 0.47% Castro::construct_old_source() 25 0.001349 0.001349 0.001349 0.36% Castro::construct_old_gravity_source() 5 0.001339 0.001339 0.001339 0.36% Castro::computeNewDt() 5 0.001327 0.001327 0.001327 0.36% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009284 0.0009284 0.0009284 0.25% Castro::apply_source_to_state() 10 0.0009214 0.0009214 0.0009214 0.25% MLMG::ResNormInf() 42 0.0009189 0.0009189 0.0009189 0.25% Castro::reset_internal_energy(Fab) 240 0.00085 0.00085 0.00085 0.23% FabArrayBase::getCPC() 632 0.0007926 0.0007926 0.0007926 0.21% MLMG::getGradSolution() 6 0.0007499 0.0007499 0.0007499 0.20% MLCellLinOp::compGrad() 6 0.000747 0.000747 0.000747 0.20% MLCellLinOp::setLevelBC() 6 0.0007437 0.0007437 0.0007437 0.20% FabArray::setDomainBndry() 20 0.0006696 0.0006696 0.0006696 0.18% FabArray::mult() 22 0.0006472 0.0006472 0.0006472 0.17% MLPoisson::prepareForSolve() 6 0.0006427 0.0006427 0.0006427 0.17% MLCellLinOp::prepareForSolve() 6 0.0006167 0.0006167 0.0006167 0.17% Castro::check_for_nan() 10 0.0005989 0.0005989 0.0005989 0.16% MultiFab::contains_nan() 10 0.0005935 0.0005935 0.0005935 0.16% MLMG::computeMLResidual() 6 0.00058 0.00058 0.00058 0.16% Castro::enforce_speed_limit() 30 0.0004843 0.0004843 0.0004843 0.13% Gravity::update_max_rhs() 6 0.0004618 0.0004618 0.0004618 0.12% FabArrayBase::CPC::define() 244 0.0004251 0.0004251 0.0004251 0.11% Amr::InitAmr() 1 0.0004072 0.0004072 0.0004072 0.11% Castro::finalize_advance() 5 0.0003409 0.0003409 0.0003409 0.09% FabArrayBase::getFB() 1766 0.0003162 0.0003162 0.0003162 0.08% Gravity::swapTimeLevels() 5 0.0002376 0.0002376 0.0002376 0.06% MLLinOp::define() 6 0.0001708 0.0001708 0.0001708 0.05% MLLinOp::defineGrids() 6 0.0001655 0.0001655 0.0001655 0.04% MLMG::MLResNormInf() 6 0.0001504 0.0001504 0.0001504 0.04% Castro::buildMetrics() 1 0.0001469 0.0001469 0.0001469 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.04% MLMG::MLRhsNormInf() 6 0.000118 0.000118 0.000118 0.03% FabArrayBase::FB::FB() 26 6.173e-05 6.173e-05 6.173e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.771e-05 2.771e-05 2.771e-05 0.01% Amr::writeSmallPlotFile() 1 2.666e-05 2.666e-05 2.666e-05 0.01% Castro::initMFs() 1 2.653e-05 2.653e-05 2.653e-05 0.01% Castro::swap_state_time_levels() 5 2.342e-05 2.342e-05 2.342e-05 0.01% Castro::create_source_corrector() 5 2.276e-05 2.276e-05 2.276e-05 0.01% makeSFC 30 2.055e-05 2.055e-05 2.055e-05 0.01% Castro::finalize_do_advance() 5 1.952e-05 1.952e-05 1.952e-05 0.01% DistributionMapping::Distribute() 31 9.072e-06 9.072e-06 9.072e-06 0.00% Amr::initSubcycle() 1 8.311e-06 8.311e-06 8.311e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.395e-06 4.395e-06 4.395e-06 0.00% Gravity::set_mass_offset() 6 3.841e-06 3.841e-06 3.841e-06 0.00% Castro::retry_advance_ctu() 5 1.788e-06 1.788e-06 1.788e-06 0.00% Castro::FluxRegCrseInit 5 1.446e-06 1.446e-06 1.446e-06 0.00% Castro::FluxRegFineAdd() 5 1.305e-06 1.305e-06 1.305e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.058e-06 1.058e-06 1.058e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 220 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 122 MiB 692 MiB Castro::initMFs() 48 48 60 MiB 68 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB Castro::initialize_do_advance() 40 40 29 MiB 39 MiB FillPatchIterator::Initialize 160 160 1209 KiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2487 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6700 KiB 14 MiB MLMG::prepareForSolve() 361 361 3647 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 220 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 210 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6695 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 11 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2395 B 2048 KiB Gravity::solve_for_phi() 40 40 669 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 18 KiB 2048 KiB BndryData::define() 576 576 335 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 216 KiB 671 KiB Castro::estTimeStep() 10 10 3104 B 480 KiB VisMF::Write(FabArray) 112 112 1321 B 320 KiB Castro::normalize_species() 30 30 8321 B 320 KiB amrex::average_down 469 469 1271 B 257 KiB MLMG::addInterpCorrection() 468 468 1197 B 257 KiB amrex::Dot() 592 592 3498 B 160 KiB FabArray::norminf() 398 398 2456 B 160 KiB Castro::do_advance_ctu() 5 5 686 B 160 KiB MultiFab::max() 6 6 56 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 31 B 20 KiB MLPoisson::Fsmooth() 60 60 3546 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 52 B 10 KiB FillBoundary_nowait() 336 336 298 B 9648 B MLCellLinOp::applyBC() 3892 3892 230 B 9344 B amrex::Copy() 56 56 5953 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 52 B 2880 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCGSolver::bicgstab 324 324 122 B 1472 B MLCellLinOp::defineBC() 36 36 377 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1335 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 122 KiB 8192 KiB VisMF::Write(FabArray) 120 120 260 KiB 3584 KiB VisMF::Read() 24 24 207 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3546 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 52 B 10 KiB FillBoundary_nowait() 336 336 298 B 9648 B MLCellLinOp::applyBC() 1946 1946 229 B 9328 B amrex::Copy() 56 56 5954 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2638 B 3072 B StateData::FillBoundary(geom) 960 960 53 B 2880 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 542 B 1296 B amrex::average_down 37 37 222 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 25 B 400 B FabArray::norminf() 398 398 17 B 272 B Castro::estTimeStep() 10 10 0 B 32 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::do_advance_ctu() 5 5 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-28-g831d1487fa19) finalized