Initializing AMReX (23.10-22-gbe6c6415467d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (23.10-22-gbe6c6415467d) initialized Starting run at 08:23:04 UTC on 2023-10-30. Successfully read inputs file ... Castro git describe: 23.10-46-g13f63f26e AMReX git describe: 23.10-22-gbe6c64154 Microphysics git describe: 23.10-27-g04ba99af reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.047844148 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.026642832 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.051394976 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051628078 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072892394 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.079891653 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.056899045 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047123927 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.075556143 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.076652615 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058140069 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057376721 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060619468 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.04595424 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026681265 seconds Ending run at 08:23:05 UTC on 2023-10-30. Run time = 0.89073378 Run time without initialization = 0.761439005 Average number of zones advanced per microsecond: 3.443 Average number of zones advanced per microsecond per rank: 3.443 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8908 ... 0.8908 ... 0.8908 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2677 0.2677 0.2677 30.06% VisMF::Write(FabArray) 11 0.1857 0.1857 0.1857 20.85% MLCellLinOp::applyBC() 4433 0.07777 0.07777 0.07777 8.73% MLPoisson::Fsmooth() 3280 0.03228 0.03228 0.03228 3.62% FillBoundary_nowait() 4023 0.03194 0.03194 0.03194 3.59% StateData::FillBoundary(geom) 328 0.02533 0.02533 0.02533 2.84% amrex::Dot() 1114 0.02032 0.02032 0.02032 2.28% Castro::normalize_species() 62 0.01903 0.01903 0.01903 2.14% Castro::computeTemp() 63 0.01516 0.01516 0.01516 1.70% amrex::Copy() 1029 0.01479 0.01479 0.01479 1.66% FabArray::norminf() 743 0.01428 0.01428 0.01428 1.60% FabArray::ParallelCopy_nowait() 861 0.01315 0.01315 0.01315 1.48% FabArray::setVal() 1144 0.01314 0.01314 0.01314 1.47% amrex::average_down 410 0.01236 0.01236 0.01236 1.39% StateDataPhysBCFunct::() 41 0.01226 0.01226 0.01226 1.38% MLPoisson::Fapply() 1142 0.01036 0.01036 0.01036 1.16% Castro::enforce_min_density() 62 0.01009 0.01009 0.01009 1.13% MLCellLinOp::defineAuxData() 11 0.00974 0.00974 0.00974 1.09% Gravity::fill_multipole_BCs() 11 0.009194 0.009194 0.009194 1.03% FabArray::Saxpy() 813 0.008168 0.008168 0.008168 0.92% FabArray::Xpay() 821 0.008046 0.008046 0.008046 0.90% MLMG::addInterpCorrection() 410 0.006743 0.006743 0.006743 0.76% Castro::reset_internal_energy(MultiFab) 63 0.006631 0.006631 0.006631 0.74% Castro::estTimeStep() 21 0.006119 0.006119 0.006119 0.69% Amr::checkPoint() 3 0.005818 0.005818 0.005818 0.65% FabArray::LinComb() 557 0.004477 0.004477 0.004477 0.50% amrex::Add() 164 0.004405 0.004405 0.004405 0.49% BndryData::define() 11 0.003578 0.003578 0.003578 0.40% Castro::construct_new_gravity_source() 10 0.003346 0.003346 0.003346 0.38% Castro::construct_old_gravity_source() 10 0.002996 0.002996 0.002996 0.34% MLCGSolver::bicgstab 82 0.002341 0.002341 0.002341 0.26% Amr::writePlotFile() 2 0.002208 0.002208 0.002208 0.25% Castro::enforce_speed_limit() 62 0.001951 0.001951 0.001951 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00171 0.00171 0.00171 0.19% Gravity::actual_solve_with_mlmg() 11 0.001608 0.001608 0.001608 0.18% Castro::reset_internal_energy(Fab) 504 0.00152 0.00152 0.00152 0.17% Castro::initData() 1 0.001508 0.001508 0.001508 0.17% MLCellLinOp::setLevelBC() 11 0.001483 0.001483 0.001483 0.17% check_for_negative_density() 10 0.001433 0.001433 0.001433 0.16% FabArray::mult() 43 0.001334 0.001334 0.001334 0.15% FabArray::setDomainBndry() 41 0.00129 0.00129 0.00129 0.14% MLCellLinOp::prepareForSolve() 11 0.001267 0.001267 0.001267 0.14% MultiFab::contains_nan() 20 0.001259 0.001259 0.001259 0.14% MLCellLinOp::smooth() 1640 0.001236 0.001236 0.001236 0.14% MLCellLinOp::compGrad() 11 0.001053 0.001053 0.001053 0.12% MLMG::prepareForSolve() 11 0.0009637 0.0009637 0.0009637 0.11% FabArray::FillBoundary() 4023 0.0008237 0.0008237 0.0008237 0.09% FabArrayBase::getCPC() 1323 0.0007991 0.0007991 0.0007991 0.09% FabArrayBase::CPC::define() 454 0.0006801 0.0006801 0.0006801 0.08% FabArrayBase::getFB() 4023 0.0006502 0.0006502 0.0006502 0.07% Gravity::get_new_grav_vector() 11 0.000601 0.000601 0.000601 0.07% MLCellLinOp::apply() 1142 0.0004928 0.0004928 0.0004928 0.06% Amr::InitAmr() 1 0.0004795 0.0004795 0.0004795 0.05% Gravity::get_old_grav_vector() 10 0.0004711 0.0004711 0.0004711 0.05% AmrLevel::FillPatch() 41 0.0003731 0.0003731 0.0003731 0.04% Amr::coarseTimeStep() 10 0.0003359 0.0003359 0.0003359 0.04% MultiFab::max() 11 0.0003165 0.0003165 0.0003165 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002989 0.0002989 0.0002989 0.03% main() 1 0.0002926 0.0002926 0.0002926 0.03% FabArray::ParallelCopy() 861 0.0002589 0.0002589 0.0002589 0.03% MLCellLinOp::defineBC() 11 0.0002552 0.0002552 0.0002552 0.03% MLCellLinOp::correctionResidual() 492 0.0002433 0.0002433 0.0002433 0.03% FillPatchIterator::Initialize 41 0.0002142 0.0002142 0.0002142 0.02% MLMG::mgVcycle() 82 0.0001996 0.0001996 0.0001996 0.02% Castro::subcycle_advance_ctu() 10 0.0001846 0.0001846 0.0001846 0.02% Castro::create_source_corrector() 10 0.0001771 0.0001771 0.0001771 0.02% Amr::timeStep() 10 0.0001599 0.0001599 0.0001599 0.02% MLLinOp::defineGrids() 11 0.0001548 0.0001548 0.0001548 0.02% Gravity::solve_for_phi() 10 0.0001486 0.0001486 0.0001486 0.02% StateData::checkPoint() 12 0.0001379 0.0001379 0.0001379 0.02% Gravity::update_max_rhs() 11 0.00013 0.00013 0.00013 0.01% Castro::advance() 10 0.0001154 0.0001154 0.0001154 0.01% MLMG:computeResOfCorrection() 410 0.0001117 0.0001117 0.0001117 0.01% FabArrayBase::FB::FB() 56 0.0001035 0.0001035 0.0001035 0.01% MLMG::actualBottomSolve() 82 9.339e-05 9.339e-05 9.339e-05 0.01% MLMG::mgVcycle_down::0 82 8.806e-05 8.806e-05 8.806e-05 0.01% Castro::Castro() 1 8.527e-05 8.527e-05 8.527e-05 0.01% Castro::do_new_sources() 10 8.488e-05 8.488e-05 8.488e-05 0.01% MLMG::mgVcycle_down::1 82 7.887e-05 7.887e-05 7.887e-05 0.01% AmrLevel::checkPoint() 3 7.762e-05 7.762e-05 7.762e-05 0.01% Castro::initialize_advance() 10 7.744e-05 7.744e-05 7.744e-05 0.01% MLMG::mgVcycle_down::2 82 7.732e-05 7.732e-05 7.732e-05 0.01% MLMG::solve() 11 7.402e-05 7.402e-05 7.402e-05 0.01% MLMG::mgVcycle_down::4 82 7.342e-05 7.342e-05 7.342e-05 0.01% Castro::enforce_consistent_e() 1 7.31e-05 7.31e-05 7.31e-05 0.01% MLMG::mgVcycle_down::3 82 7.134e-05 7.134e-05 7.134e-05 0.01% Castro::clean_state() 62 6.946e-05 6.946e-05 6.946e-05 0.01% Castro::finalize_advance() 10 6.732e-05 6.732e-05 6.732e-05 0.01% Castro::initialize_do_advance() 10 5.751e-05 5.751e-05 5.751e-05 0.01% MLMG::mgVcycle_up::4 82 5.743e-05 5.743e-05 5.743e-05 0.01% MLMG::oneIter() 82 5.49e-05 5.49e-05 5.49e-05 0.01% MLCellLinOp::solutionResidual() 93 5.159e-05 5.159e-05 5.159e-05 0.01% MLMG::mgVcycle_up::0 82 4.867e-05 4.867e-05 4.867e-05 0.01% MLMG::mgVcycle_up::3 82 4.786e-05 4.786e-05 4.786e-05 0.01% MLMG::mgVcycle_up::1 82 4.652e-05 4.652e-05 4.652e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.574e-05 4.574e-05 4.574e-05 0.01% Castro::construct_new_source() 50 4.568e-05 4.568e-05 4.568e-05 0.01% MLMG::mgVcycle_up::2 82 4.531e-05 4.531e-05 4.531e-05 0.01% Castro::do_advance_ctu() 10 4.508e-05 4.508e-05 4.508e-05 0.01% Gravity::solve_phi_with_mlmg() 11 4.391e-05 4.391e-05 4.391e-05 0.00% Castro::finalize_do_advance() 10 4.377e-05 4.377e-05 4.377e-05 0.00% StateData::define() 4 4.009e-05 4.009e-05 4.009e-05 0.00% Castro::swap_state_time_levels() 10 3.79e-05 3.79e-05 3.79e-05 0.00% MLMG::ResNormInf() 93 3.676e-05 3.676e-05 3.676e-05 0.00% FillPatchSingleLevel 41 3.316e-05 3.316e-05 3.316e-05 0.00% Amr::writeSmallPlotFile() 1 3.254e-05 3.254e-05 3.254e-05 0.00% MLMG::mgVcycle_bottom 82 3.221e-05 3.221e-05 3.221e-05 0.00% MLMG::computeResidual() 82 3.199e-05 3.199e-05 3.199e-05 0.00% Castro::initMFs() 1 2.998e-05 2.998e-05 2.998e-05 0.00% Amr::defBaseLevel() 1 2.929e-05 2.929e-05 2.929e-05 0.00% Castro::construct_new_gravity() 10 2.837e-05 2.837e-05 2.837e-05 0.00% makeSFC 55 2.427e-05 2.427e-05 2.427e-05 0.00% Castro::buildMetrics() 1 2.382e-05 2.382e-05 2.382e-05 0.00% MLPoisson::define() 11 2.284e-05 2.284e-05 2.284e-05 0.00% Castro::do_old_sources() 10 2.163e-05 2.163e-05 2.163e-05 0.00% Amr::FinalizeInit() 1 2.083e-05 2.083e-05 2.083e-05 0.00% Castro::construct_old_source() 50 1.906e-05 1.906e-05 1.906e-05 0.00% DistributionMapping::Distribute() 56 1.837e-05 1.837e-05 1.837e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.808e-05 1.808e-05 1.808e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.488e-05 1.488e-05 1.488e-05 0.00% MLMG::computeMLResidual() 11 1.409e-05 1.409e-05 1.409e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.378e-05 1.378e-05 1.378e-05 0.00% MLPoisson::prepareForSolve() 11 1.308e-05 1.308e-05 1.308e-05 0.00% Castro::apply_source_to_state() 20 1.149e-05 1.149e-05 1.149e-05 0.00% Castro::construct_old_gravity() 10 1.11e-05 1.11e-05 1.11e-05 0.00% Castro::check_for_nan() 20 1.064e-05 1.064e-05 1.064e-05 0.00% MLLinOp::define() 11 1.044e-05 1.044e-05 1.044e-05 0.00% Gravity::swapTimeLevels() 10 1.002e-05 1.002e-05 1.002e-05 0.00% Castro::post_timestep() 10 9.749e-06 9.749e-06 9.749e-06 0.00% Amr::initSubcycle() 1 9.476e-06 9.476e-06 9.476e-06 0.00% Castro::post_init() 1 9.476e-06 9.476e-06 9.476e-06 0.00% Gravity::actual_multilevel_solve() 1 7.862e-06 7.862e-06 7.862e-06 0.00% Castro::computeNewDt() 9 6.326e-06 6.326e-06 6.326e-06 0.00% MLMG::getGradSolution() 11 6.024e-06 6.024e-06 6.024e-06 0.00% Castro::expand_state() 10 5.44e-06 5.44e-06 5.44e-06 0.00% AmrLevel::checkPointPost() 3 4.954e-06 4.954e-06 4.954e-06 0.00% Amr::InitializeInit() 1 4.523e-06 4.523e-06 4.523e-06 0.00% Castro::retry_advance_ctu() 10 4.37e-06 4.37e-06 4.37e-06 0.00% MLMG::MLRhsNormInf() 11 4.164e-06 4.164e-06 4.164e-06 0.00% Gravity::set_mass_offset() 11 3.957e-06 3.957e-06 3.957e-06 0.00% MLMG::MLResNormInf() 11 3.709e-06 3.709e-06 3.709e-06 0.00% Castro::FluxRegCrseInit 10 2.865e-06 2.865e-06 2.865e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.771e-06 2.771e-06 2.771e-06 0.00% Castro::computeInitialDt() 2 2.539e-06 2.539e-06 2.539e-06 0.00% Castro::FluxRegFineAdd() 10 2.416e-06 2.416e-06 2.416e-06 0.00% Amr::init() 1 2.186e-06 2.186e-06 2.186e-06 0.00% AmrLevel::checkPointPre() 3 1.908e-06 1.908e-06 1.908e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.873e-06 1.873e-06 1.873e-06 0.00% Castro::post_regrid() 1 1.242e-06 1.242e-06 1.242e-06 0.00% Amr::initialInit() 1 9.51e-07 9.51e-07 9.51e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8907 0.8907 0.8907 100.00% Amr::coarseTimeStep() 10 0.7345 0.7345 0.7345 82.46% Amr::timeStep() 10 0.6375 0.6375 0.6375 71.57% Castro::advance() 10 0.6274 0.6274 0.6274 70.44% Castro::subcycle_advance_ctu() 10 0.6136 0.6136 0.6136 68.88% Castro::do_advance_ctu() 10 0.6134 0.6134 0.6134 68.86% Gravity::solve_phi_with_mlmg() 11 0.2943 0.2943 0.2943 33.03% Gravity::actual_solve_with_mlmg() 11 0.2846 0.2846 0.2846 31.95% Castro::construct_ctu_hydro_source() 10 0.2779 0.2779 0.2779 31.20% Castro::construct_new_gravity() 10 0.2653 0.2653 0.2653 29.78% MLMG::solve() 11 0.2638 0.2638 0.2638 29.62% Gravity::solve_for_phi() 10 0.2498 0.2498 0.2498 28.04% MLMG::oneIter() 82 0.249 0.249 0.249 27.96% MLMG::mgVcycle() 82 0.2454 0.2454 0.2454 27.55% VisMF::Write(FabArray) 11 0.1857 0.1857 0.1857 20.85% Amr::checkPoint() 3 0.141 0.141 0.141 15.83% AmrLevel::checkPoint() 3 0.1352 0.1352 0.1352 15.18% StateData::checkPoint() 12 0.1351 0.1351 0.1351 15.17% Amr::init() 1 0.1287 0.1287 0.1287 14.45% MLCellLinOp::smooth() 1640 0.1184 0.1184 0.1184 13.30% MLCellLinOp::applyBC() 4433 0.1113 0.1113 0.1113 12.49% MLMG::mgVcycle_bottom 82 0.07349 0.07349 0.07349 8.25% MLMG::actualBottomSolve() 82 0.07345 0.07345 0.07345 8.25% MLCGSolver::bicgstab 82 0.07275 0.07275 0.07275 8.17% Amr::initialInit() 1 0.05409 0.05409 0.05409 6.07% Castro::clean_state() 62 0.05345 0.05345 0.05345 6.00% Amr::writePlotFile() 2 0.05343 0.05343 0.05343 6.00% Amr::FinalizeInit() 1 0.04888 0.04888 0.04888 5.49% Castro::post_init() 1 0.0474 0.0474 0.0474 5.32% AmrLevel::FillPatch() 41 0.04739 0.04739 0.04739 5.32% Gravity::multilevel_solve_for_new_phi() 1 0.04495 0.04495 0.04495 5.05% Gravity::actual_multilevel_solve() 1 0.04493 0.04493 0.04493 5.04% FillPatchIterator::Initialize 41 0.04318 0.04318 0.04318 4.85% FillPatchIterator::FillFromLevel0() 41 0.04168 0.04168 0.04168 4.68% FillPatchSingleLevel 41 0.04163 0.04163 0.04163 4.67% MLMG::mgVcycle_down::0 82 0.04078 0.04078 0.04078 4.58% StateDataPhysBCFunct::() 41 0.03759 0.03759 0.03759 4.22% MLCellLinOp::apply() 1142 0.03674 0.03674 0.03674 4.12% FabArray::FillBoundary() 4023 0.03351 0.03351 0.03351 3.76% FillBoundary_nowait() 4023 0.03269 0.03269 0.03269 3.67% MLPoisson::Fsmooth() 3280 0.03228 0.03228 0.03228 3.62% MLMG::mgVcycle_up::0 82 0.02618 0.02618 0.02618 2.94% StateData::FillBoundary(geom) 328 0.02533 0.02533 0.02533 2.84% Castro::computeTemp() 63 0.02331 0.02331 0.02331 2.62% MLCellLinOp::correctionResidual() 492 0.02235 0.02235 0.02235 2.51% Castro::initialize_do_advance() 10 0.02178 0.02178 0.02178 2.45% amrex::Dot() 1114 0.02032 0.02032 0.02032 2.28% MLMG:computeResOfCorrection() 410 0.0197 0.0197 0.0197 2.21% Castro::normalize_species() 62 0.01903 0.01903 0.01903 2.14% Castro::do_old_sources() 10 0.01865 0.01865 0.01865 2.09% amrex::average_down 410 0.01759 0.01759 0.01759 1.97% Gravity::get_new_grav_vector() 11 0.01753 0.01753 0.01753 1.97% MLPoisson::define() 11 0.01614 0.01614 0.01614 1.81% MLMG::mgVcycle_down::1 82 0.01608 0.01608 0.01608 1.81% amrex::Copy() 1029 0.01479 0.01479 0.01479 1.66% MLMG::mgVcycle_down::2 82 0.01478 0.01478 0.01478 1.66% Castro::construct_old_gravity() 10 0.0146 0.0146 0.0146 1.64% Gravity::get_old_grav_vector() 10 0.01458 0.01458 0.01458 1.64% MLMG::mgVcycle_down::3 82 0.01441 0.01441 0.01441 1.62% FabArray::norminf() 743 0.01428 0.01428 0.01428 1.60% FabArray::ParallelCopy() 861 0.01427 0.01427 0.01427 1.60% MLMG::mgVcycle_down::4 82 0.01426 0.01426 0.01426 1.60% FabArray::ParallelCopy_nowait() 861 0.01401 0.01401 0.01401 1.57% Castro::initialize_advance() 10 0.01319 0.01319 0.01319 1.48% FabArray::setVal() 1144 0.01314 0.01314 0.01314 1.47% Castro::expand_state() 10 0.01232 0.01232 0.01232 1.38% MLCGSolver::ParallelAllReduce 1514 0.01218 0.01218 0.01218 1.37% Castro::do_new_sources() 10 0.01183 0.01183 0.01183 1.33% MLMG::addInterpCorrection() 410 0.01177 0.01177 0.01177 1.32% MLMG::mgVcycle_up::1 82 0.01153 0.01153 0.01153 1.29% MLMG::mgVcycle_up::4 82 0.01142 0.01142 0.01142 1.28% MLMG::mgVcycle_up::2 82 0.01127 0.01127 0.01127 1.27% MLCellLinOp::defineAuxData() 11 0.01109 0.01109 0.01109 1.24% MLMG::mgVcycle_up::3 82 0.01098 0.01098 0.01098 1.23% MLPoisson::Fapply() 1142 0.01036 0.01036 0.01036 1.16% Castro::enforce_min_density() 62 0.01009 0.01009 0.01009 1.13% Castro::post_timestep() 10 0.009918 0.009918 0.009918 1.11% Gravity::fill_multipole_BCs() 11 0.009421 0.009421 0.009421 1.06% FabArray::Saxpy() 813 0.008168 0.008168 0.008168 0.92% Castro::reset_internal_energy(MultiFab) 63 0.008152 0.008152 0.008152 0.92% FabArray::Xpay() 821 0.008046 0.008046 0.008046 0.90% MLCellLinOp::solutionResidual() 93 0.007437 0.007437 0.007437 0.83% MLMG::computeResidual() 82 0.006187 0.006187 0.006187 0.69% Castro::estTimeStep() 21 0.006119 0.006119 0.006119 0.69% Amr::InitializeInit() 1 0.005205 0.005205 0.005205 0.58% Amr::defBaseLevel() 1 0.0052 0.0052 0.0052 0.58% MLMG::prepareForSolve() 11 0.004881 0.004881 0.004881 0.55% MLCellLinOp::defineBC() 11 0.004813 0.004813 0.004813 0.54% BndryData::define() 11 0.004557 0.004557 0.004557 0.51% Castro::initData() 1 0.004549 0.004549 0.004549 0.51% FabArray::LinComb() 557 0.004477 0.004477 0.004477 0.50% amrex::Add() 164 0.004405 0.004405 0.004405 0.49% Castro::construct_new_source() 50 0.003392 0.003392 0.003392 0.38% Castro::construct_new_gravity_source() 10 0.003346 0.003346 0.003346 0.38% Castro::construct_old_source() 50 0.003015 0.003015 0.003015 0.34% Castro::construct_old_gravity_source() 10 0.002996 0.002996 0.002996 0.34% Castro::computeNewDt() 9 0.002702 0.002702 0.002702 0.30% Castro::finalize_do_advance() 10 0.002372 0.002372 0.002372 0.27% MLMG::ResNormInf() 93 0.002139 0.002139 0.002139 0.24% Castro::enforce_speed_limit() 62 0.001951 0.001951 0.001951 0.22% Castro::apply_source_to_state() 20 0.001849 0.001849 0.001849 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00171 0.00171 0.00171 0.19% MLMG::getGradSolution() 11 0.001541 0.001541 0.001541 0.17% MLCellLinOp::compGrad() 11 0.001535 0.001535 0.001535 0.17% Castro::reset_internal_energy(Fab) 504 0.00152 0.00152 0.00152 0.17% MLCellLinOp::setLevelBC() 11 0.001483 0.001483 0.001483 0.17% FabArrayBase::getCPC() 1323 0.001479 0.001479 0.001479 0.17% check_for_negative_density() 10 0.001433 0.001433 0.001433 0.16% FabArray::mult() 43 0.001334 0.001334 0.001334 0.15% MLMG::computeMLResidual() 11 0.001296 0.001296 0.001296 0.15% FabArray::setDomainBndry() 41 0.00129 0.00129 0.00129 0.14% MLPoisson::prepareForSolve() 11 0.00128 0.00128 0.00128 0.14% Castro::check_for_nan() 20 0.001269 0.001269 0.001269 0.14% MLCellLinOp::prepareForSolve() 11 0.001267 0.001267 0.001267 0.14% MultiFab::contains_nan() 20 0.001259 0.001259 0.001259 0.14% Castro::post_regrid() 1 0.001181 0.001181 0.001181 0.13% Castro::computeInitialDt() 2 0.001098 0.001098 0.001098 0.12% Gravity::update_max_rhs() 11 0.0009516 0.0009516 0.0009516 0.11% FabArrayBase::getFB() 4023 0.0007537 0.0007537 0.0007537 0.08% FabArrayBase::CPC::define() 454 0.0006801 0.0006801 0.0006801 0.08% Castro::finalize_advance() 10 0.0005836 0.0005836 0.0005836 0.07% Castro::Castro() 1 0.0005605 0.0005605 0.0005605 0.06% Amr::InitAmr() 1 0.0004889 0.0004889 0.0004889 0.05% Gravity::swapTimeLevels() 10 0.0004341 0.0004341 0.0004341 0.05% MLMG::MLResNormInf() 11 0.000323 0.000323 0.000323 0.04% MultiFab::max() 11 0.0003165 0.0003165 0.0003165 0.04% Castro::buildMetrics() 1 0.0002767 0.0002767 0.0002767 0.03% MLLinOp::define() 11 0.00022 0.00022 0.00022 0.02% MLMG::MLRhsNormInf() 11 0.0002182 0.0002182 0.0002182 0.02% MLLinOp::defineGrids() 11 0.0002096 0.0002096 0.0002096 0.02% Castro::create_source_corrector() 10 0.0001771 0.0001771 0.0001771 0.02% FabArrayBase::FB::FB() 56 0.0001035 0.0001035 0.0001035 0.01% Castro::enforce_consistent_e() 1 7.31e-05 7.31e-05 7.31e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.497e-05 5.497e-05 5.497e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.286e-05 5.286e-05 5.286e-05 0.01% StateData::define() 4 4.009e-05 4.009e-05 4.009e-05 0.00% makeSFC 55 3.908e-05 3.908e-05 3.908e-05 0.00% Castro::swap_state_time_levels() 10 3.79e-05 3.79e-05 3.79e-05 0.00% Amr::writeSmallPlotFile() 1 3.254e-05 3.254e-05 3.254e-05 0.00% Castro::initMFs() 1 2.998e-05 2.998e-05 2.998e-05 0.00% DistributionMapping::Distribute() 56 1.837e-05 1.837e-05 1.837e-05 0.00% Amr::initSubcycle() 1 9.476e-06 9.476e-06 9.476e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 6.322e-06 6.322e-06 6.322e-06 0.00% AmrLevel::checkPointPost() 3 4.954e-06 4.954e-06 4.954e-06 0.00% Castro::retry_advance_ctu() 10 4.37e-06 4.37e-06 4.37e-06 0.00% Gravity::set_mass_offset() 11 3.957e-06 3.957e-06 3.957e-06 0.00% Castro::FluxRegCrseInit 10 2.865e-06 2.865e-06 2.865e-06 0.00% Castro::FluxRegFineAdd() 10 2.416e-06 2.416e-06 2.416e-06 0.00% AmrLevel::checkPointPre() 3 1.908e-06 1.908e-06 1.908e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.873e-06 1.873e-06 1.873e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 93 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 141 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1056 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1679 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7621 KiB 14 MiB MLMG::prepareForSolve() 660 660 3644 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 204 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 167 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7514 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 18 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2057 B 2048 KiB Gravity::solve_for_phi() 80 80 573 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 103 KiB 2048 KiB BndryData::define() 1056 1056 332 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 211 KiB 671 KiB Castro::estTimeStep() 21 21 3352 B 480 KiB VisMF::Write(FabArray) 656 656 3381 B 320 KiB Castro::normalize_species() 62 62 6956 B 320 KiB amrex::average_down 1067 1067 3139 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1132 B 257 KiB amrex::Dot() 1360 1360 3360 B 160 KiB FabArray::norminf() 907 907 2370 B 160 KiB check_for_negative_density() 10 10 238 B 160 KiB Castro::initData() 1 1 53 B 160 KiB MultiFab::max() 11 11 55 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3561 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 315 B 9648 B MLCellLinOp::applyBC() 8866 8866 219 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3941 B 6144 B StateData::FillBoundary(geom) 1992 1992 47 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 118 B 1472 B MLCellLinOp::defineBC() 66 66 375 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 589 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 39 KiB 8192 KiB VisMF::Write(FabArray) 744 744 466 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3561 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 314 B 9648 B MLCellLinOp::applyBC() 4433 4433 217 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3941 B 6144 B Gravity::get_new_grav_vector() 3 3 2888 B 3072 B StateData::FillBoundary(geom) 1992 1992 47 B 3024 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 278 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 303 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 24 B 400 B FabArray::norminf() 907 907 17 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.10-22-gbe6c6415467d) finalized Initializing AMReX (23.10-22-gbe6c6415467d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (23.10-22-gbe6c6415467d) initialized Starting run at 08:23:06 UTC on 2023-10-30. Successfully read inputs file ... Castro git describe: 23.10-46-g13f63f26e AMReX git describe: 23.10-22-gbe6c64154 Microphysics git describe: 23.10-27-g04ba99af reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.489149773 Restart time = 0.069292119 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.055966406 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047949909 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.071487089 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.071023 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.053029382 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027265453 seconds Ending run at 08:23:06 UTC on 2023-10-30. Run time = 0.396989031 Run time without initialization = 0.327124299 Average number of zones advanced per microsecond: 4.007 Average number of zones advanced per microsecond per rank: 4.007 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.397 ... 0.397 ... 0.397 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1099 0.1099 0.1099 27.69% VisMF::Read() 3 0.05804 0.05804 0.05804 14.62% MLCellLinOp::applyBC() 1946 0.03361 0.03361 0.03361 8.47% VisMF::Write(FabArray) 1 0.02576 0.02576 0.02576 6.49% MLPoisson::Fsmooth() 1440 0.01391 0.01391 0.01391 3.50% FillBoundary_nowait() 1766 0.01301 0.01301 0.01301 3.28% StateData::FillBoundary(geom) 160 0.01212 0.01212 0.01212 3.05% Castro::normalize_species() 30 0.009084 0.009084 0.009084 2.29% amrex::Dot() 484 0.008727 0.008727 0.008727 2.20% Castro::computeTemp() 30 0.007307 0.007307 0.007307 1.84% amrex::Copy() 463 0.00706 0.00706 0.00706 1.78% Castro::enforce_min_density() 30 0.006807 0.006807 0.006807 1.71% FabArray::norminf() 326 0.006484 0.006484 0.006484 1.63% FabArray::setVal() 537 0.006408 0.006408 0.006408 1.61% FabArray::ParallelCopy_nowait() 380 0.005898 0.005898 0.005898 1.49% Gravity::fill_multipole_BCs() 6 0.005479 0.005479 0.005479 1.38% StateDataPhysBCFunct::() 20 0.00525 0.00525 0.00525 1.32% MLCellLinOp::defineAuxData() 6 0.005246 0.005246 0.005246 1.32% Amr::restart() 1 0.004603 0.004603 0.004603 1.16% MLPoisson::Fapply() 500 0.004497 0.004497 0.004497 1.13% Castro::estTimeStep() 10 0.003735 0.003735 0.003735 0.94% FabArray::Saxpy() 355 0.003665 0.003665 0.003665 0.92% FabArray::Xpay() 361 0.003515 0.003515 0.003515 0.89% MLMG::addInterpCorrection() 180 0.003024 0.003024 0.003024 0.76% amrex::average_down 180 0.002644 0.002644 0.002644 0.67% Castro::enforce_speed_limit() 30 0.002633 0.002633 0.002633 0.66% Castro::reset_internal_energy(MultiFab) 30 0.002087 0.002087 0.002087 0.53% FabArray::LinComb() 242 0.001964 0.001964 0.001964 0.49% BndryData::define() 6 0.001955 0.001955 0.001955 0.49% amrex::Add() 72 0.001876 0.001876 0.001876 0.47% Castro::construct_new_gravity_source() 5 0.00178 0.00178 0.00178 0.45% Castro::construct_old_gravity_source() 5 0.001519 0.001519 0.001519 0.38% Amr::writePlotFile() 1 0.001331 0.001331 0.001331 0.34% MLCGSolver::bicgstab 36 0.00102 0.00102 0.00102 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008998 0.0008998 0.0008998 0.23% check_for_negative_density() 5 0.0008873 0.0008873 0.0008873 0.22% Gravity::actual_solve_with_mlmg() 6 0.0008637 0.0008637 0.0008637 0.22% MLCellLinOp::setLevelBC() 6 0.0008094 0.0008094 0.0008094 0.20% Castro::reset_internal_energy(Fab) 240 0.0007998 0.0007998 0.0007998 0.20% MLCellLinOp::prepareForSolve() 6 0.0007212 0.0007212 0.0007212 0.18% FabArray::mult() 22 0.0006766 0.0006766 0.0006766 0.17% FabArray::setDomainBndry() 20 0.0006625 0.0006625 0.0006625 0.17% MultiFab::contains_nan() 10 0.0006501 0.0006501 0.0006501 0.16% MLCellLinOp::compGrad() 6 0.0005868 0.0005868 0.0005868 0.15% MLMG::prepareForSolve() 6 0.0005454 0.0005454 0.0005454 0.14% MLCellLinOp::smooth() 720 0.0005327 0.0005327 0.0005327 0.13% FabArrayBase::CPC::define() 244 0.0004114 0.0004114 0.0004114 0.10% Amr::InitAmr() 1 0.0003993 0.0003993 0.0003993 0.10% FabArray::FillBoundary() 1766 0.0003681 0.0003681 0.0003681 0.09% FabArrayBase::getCPC() 632 0.0003625 0.0003625 0.0003625 0.09% Gravity::get_old_grav_vector() 5 0.000345 0.000345 0.000345 0.09% main() 1 0.0002682 0.0002682 0.0002682 0.07% FabArrayBase::getFB() 1766 0.0002671 0.0002671 0.0002671 0.07% Gravity::get_new_grav_vector() 5 0.0002614 0.0002614 0.0002614 0.07% MLCellLinOp::apply() 500 0.0002057 0.0002057 0.0002057 0.05% MultiFab::max() 6 0.0002007 0.0002007 0.0002007 0.05% AmrLevel::FillPatch() 20 0.000181 0.000181 0.000181 0.05% Amr::coarseTimeStep() 5 0.0001648 0.0001648 0.0001648 0.04% Castro::subcycle_advance_ctu() 5 0.0001618 0.0001618 0.0001618 0.04% MLCellLinOp::defineBC() 6 0.0001298 0.0001298 0.0001298 0.03% MLCGSolver::ParallelAllReduce 659 0.0001288 0.0001288 0.0001288 0.03% Castro::advance() 5 0.0001189 0.0001189 0.0001189 0.03% FabArray::ParallelCopy() 380 0.0001107 0.0001107 0.0001107 0.03% MLCellLinOp::correctionResidual() 216 0.0001016 0.0001016 0.0001016 0.03% FillPatchIterator::Initialize 20 9.902e-05 9.902e-05 9.902e-05 0.02% Castro::initialize_do_advance() 5 8.914e-05 8.914e-05 8.914e-05 0.02% MLMG::mgVcycle() 36 8.796e-05 8.796e-05 8.796e-05 0.02% Castro::do_advance_ctu() 5 8.568e-05 8.568e-05 8.568e-05 0.02% Amr::timeStep() 5 8.513e-05 8.513e-05 8.513e-05 0.02% MLLinOp::defineGrids() 6 8.481e-05 8.481e-05 8.481e-05 0.02% Castro::create_source_corrector() 5 8.395e-05 8.395e-05 8.395e-05 0.02% Gravity::solve_for_phi() 5 7.775e-05 7.775e-05 7.775e-05 0.02% AmrLevel::restart() 1 7.747e-05 7.747e-05 7.747e-05 0.02% Castro::initialize_advance() 5 7.59e-05 7.59e-05 7.59e-05 0.02% Castro::finalize_do_advance() 5 7.075e-05 7.075e-05 7.075e-05 0.02% StateData::restartDoit() 4 6.885e-05 6.885e-05 6.885e-05 0.02% Gravity::update_max_rhs() 6 6.406e-05 6.406e-05 6.406e-05 0.02% FabArrayBase::FB::FB() 26 6.01e-05 6.01e-05 6.01e-05 0.02% Castro::construct_new_source() 25 5.204e-05 5.204e-05 5.204e-05 0.01% MLMG:computeResOfCorrection() 180 4.854e-05 4.854e-05 4.854e-05 0.01% Castro::construct_old_source() 25 4.697e-05 4.697e-05 4.697e-05 0.01% MLMG::actualBottomSolve() 36 4.488e-05 4.488e-05 4.488e-05 0.01% MLMG::mgVcycle_down::0 36 3.874e-05 3.874e-05 3.874e-05 0.01% MLMG::solve() 6 3.594e-05 3.594e-05 3.594e-05 0.01% MLMG::mgVcycle_down::1 36 3.469e-05 3.469e-05 3.469e-05 0.01% MLMG::mgVcycle_down::2 36 3.318e-05 3.318e-05 3.318e-05 0.01% Castro::clean_state() 30 3.244e-05 3.244e-05 3.244e-05 0.01% MLMG::mgVcycle_down::4 36 3.238e-05 3.238e-05 3.238e-05 0.01% MLMG::mgVcycle_down::3 36 3.156e-05 3.156e-05 3.156e-05 0.01% Castro::initMFs() 1 3.105e-05 3.105e-05 3.105e-05 0.01% Castro::post_restart() 1 2.982e-05 2.982e-05 2.982e-05 0.01% Castro::finalize_advance() 5 2.975e-05 2.975e-05 2.975e-05 0.01% Castro::buildMetrics() 1 2.857e-05 2.857e-05 2.857e-05 0.01% Amr::writeSmallPlotFile() 1 2.853e-05 2.853e-05 2.853e-05 0.01% MLMG::mgVcycle_up::4 36 2.722e-05 2.722e-05 2.722e-05 0.01% Castro::computeNewDt() 5 2.679e-05 2.679e-05 2.679e-05 0.01% Castro::swap_state_time_levels() 5 2.678e-05 2.678e-05 2.678e-05 0.01% MLMG::oneIter() 36 2.4e-05 2.4e-05 2.4e-05 0.01% MLCellLinOp::solutionResidual() 42 2.305e-05 2.305e-05 2.305e-05 0.01% MLMG::mgVcycle_up::3 36 2.09e-05 2.09e-05 2.09e-05 0.01% MLMG::mgVcycle_up::0 36 2.073e-05 2.073e-05 2.073e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.071e-05 2.071e-05 2.071e-05 0.01% MLMG::mgVcycle_up::2 36 2.009e-05 2.009e-05 2.009e-05 0.01% Gravity::solve_phi_with_mlmg() 6 1.912e-05 1.912e-05 1.912e-05 0.00% MLMG::mgVcycle_up::1 36 1.87e-05 1.87e-05 1.87e-05 0.00% MLMG::ResNormInf() 42 1.865e-05 1.865e-05 1.865e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.708e-05 1.708e-05 1.708e-05 0.00% DistributionMapping::Distribute() 31 1.514e-05 1.514e-05 1.514e-05 0.00% MLMG::mgVcycle_bottom 36 1.465e-05 1.465e-05 1.465e-05 0.00% FillPatchSingleLevel 20 1.462e-05 1.462e-05 1.462e-05 0.00% makeSFC 30 1.433e-05 1.433e-05 1.433e-05 0.00% MLMG::computeResidual() 36 1.384e-05 1.384e-05 1.384e-05 0.00% Castro::construct_new_gravity() 5 1.37e-05 1.37e-05 1.37e-05 0.00% MLPoisson::define() 6 1.318e-05 1.318e-05 1.318e-05 0.00% Castro::do_old_sources() 5 1.189e-05 1.189e-05 1.189e-05 0.00% Castro::do_new_sources() 5 9.645e-06 9.645e-06 9.645e-06 0.00% Amr::initSubcycle() 1 9.241e-06 9.241e-06 9.241e-06 0.00% Gravity::actual_multilevel_solve() 1 8.967e-06 8.967e-06 8.967e-06 0.00% MLMG::computeMLResidual() 6 8.854e-06 8.854e-06 8.854e-06 0.00% MLPoisson::prepareForSolve() 6 8.73e-06 8.73e-06 8.73e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.776e-06 7.776e-06 7.776e-06 0.00% Castro::construct_old_gravity() 5 6.794e-06 6.794e-06 6.794e-06 0.00% Castro::check_for_nan() 10 6.782e-06 6.782e-06 6.782e-06 0.00% MLLinOp::define() 6 6.189e-06 6.189e-06 6.189e-06 0.00% Castro::apply_source_to_state() 10 5.925e-06 5.925e-06 5.925e-06 0.00% Castro::post_timestep() 5 4.934e-06 4.934e-06 4.934e-06 0.00% Gravity::swapTimeLevels() 5 4.508e-06 4.508e-06 4.508e-06 0.00% MLMG::getGradSolution() 6 3.21e-06 3.21e-06 3.21e-06 0.00% Castro::expand_state() 5 3.087e-06 3.087e-06 3.087e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.98e-06 2.98e-06 2.98e-06 0.00% MLMG::MLResNormInf() 6 2.446e-06 2.446e-06 2.446e-06 0.00% Castro::retry_advance_ctu() 5 2.24e-06 2.24e-06 2.24e-06 0.00% MLMG::MLRhsNormInf() 6 2.189e-06 2.189e-06 2.189e-06 0.00% Gravity::set_mass_offset() 6 2.136e-06 2.136e-06 2.136e-06 0.00% Castro::FluxRegCrseInit 5 1.514e-06 1.514e-06 1.514e-06 0.00% Castro::FluxRegFineAdd() 5 1.285e-06 1.285e-06 1.285e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.093e-06 1.093e-06 1.093e-06 0.00% Amr::init() 1 9.46e-07 9.46e-07 9.46e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.397 0.397 0.397 100.00% Amr::coarseTimeStep() 5 0.2996 0.2996 0.2996 75.47% Amr::timeStep() 5 0.2969 0.2969 0.2969 74.78% Castro::advance() 5 0.2908 0.2908 0.2908 73.26% Castro::subcycle_advance_ctu() 5 0.2829 0.2829 0.2829 71.26% Castro::do_advance_ctu() 5 0.2828 0.2828 0.2828 71.22% Castro::construct_new_gravity() 5 0.1306 0.1306 0.1306 32.89% Gravity::solve_phi_with_mlmg() 6 0.1285 0.1285 0.1285 32.35% Gravity::solve_for_phi() 5 0.1229 0.1229 0.1229 30.96% Gravity::actual_solve_with_mlmg() 6 0.1227 0.1227 0.1227 30.91% Castro::construct_ctu_hydro_source() 5 0.1152 0.1152 0.1152 29.01% MLMG::solve() 6 0.1115 0.1115 0.1115 28.07% MLMG::oneIter() 36 0.1042 0.1042 0.1042 26.24% MLMG::mgVcycle() 36 0.1026 0.1026 0.1026 25.85% Amr::init() 1 0.06934 0.06934 0.06934 17.46% Amr::restart() 1 0.06934 0.06934 0.06934 17.46% AmrLevel::restart() 1 0.05841 0.05841 0.05841 14.71% StateData::restartDoit() 4 0.05832 0.05832 0.05832 14.69% VisMF::Read() 3 0.05804 0.05804 0.05804 14.62% MLCellLinOp::smooth() 720 0.05031 0.05031 0.05031 12.67% MLCellLinOp::applyBC() 1946 0.04731 0.04731 0.04731 11.92% MLMG::mgVcycle_bottom 36 0.03194 0.03194 0.03194 8.05% MLMG::actualBottomSolve() 36 0.03193 0.03193 0.03193 8.04% MLCGSolver::bicgstab 36 0.03162 0.03162 0.03162 7.96% Castro::clean_state() 30 0.02875 0.02875 0.02875 7.24% Amr::writePlotFile() 1 0.02734 0.02734 0.02734 6.89% VisMF::Write(FabArray) 1 0.02576 0.02576 0.02576 6.49% AmrLevel::FillPatch() 20 0.02223 0.02223 0.02223 5.60% FillPatchIterator::Initialize 20 0.02017 0.02017 0.02017 5.08% FillPatchIterator::FillFromLevel0() 20 0.01941 0.01941 0.01941 4.89% FillPatchSingleLevel 20 0.01939 0.01939 0.01939 4.88% StateDataPhysBCFunct::() 20 0.01737 0.01737 0.01737 4.37% MLCellLinOp::apply() 500 0.01587 0.01587 0.01587 4.00% MLMG::mgVcycle_down::0 36 0.01458 0.01458 0.01458 3.67% MLPoisson::Fsmooth() 1440 0.01391 0.01391 0.01391 3.50% FabArray::FillBoundary() 1766 0.0137 0.0137 0.0137 3.45% FillBoundary_nowait() 1766 0.01333 0.01333 0.01333 3.36% StateData::FillBoundary(geom) 160 0.01212 0.01212 0.01212 3.05% Castro::initialize_do_advance() 5 0.01155 0.01155 0.01155 2.91% MLMG::mgVcycle_up::0 36 0.01092 0.01092 0.01092 2.75% Castro::computeTemp() 30 0.01019 0.01019 0.01019 2.57% Castro::do_old_sources() 5 0.009742 0.009742 0.009742 2.45% MLCellLinOp::correctionResidual() 216 0.009545 0.009545 0.009545 2.40% Castro::normalize_species() 30 0.009084 0.009084 0.009084 2.29% MLPoisson::define() 6 0.008747 0.008747 0.008747 2.20% amrex::Dot() 484 0.008727 0.008727 0.008727 2.20% MLMG:computeResOfCorrection() 180 0.00839 0.00839 0.00839 2.11% Gravity::get_new_grav_vector() 5 0.007515 0.007515 0.007515 1.89% Castro::initialize_advance() 5 0.00751 0.00751 0.00751 1.89% Castro::construct_old_gravity() 5 0.007332 0.007332 0.007332 1.85% Gravity::get_old_grav_vector() 5 0.007326 0.007326 0.007326 1.85% amrex::Copy() 463 0.00706 0.00706 0.00706 1.78% MLMG::mgVcycle_down::1 36 0.006925 0.006925 0.006925 1.74% Castro::enforce_min_density() 30 0.006807 0.006807 0.006807 1.71% Castro::do_new_sources() 5 0.006564 0.006564 0.006564 1.65% FabArray::norminf() 326 0.006484 0.006484 0.006484 1.63% FabArray::setVal() 537 0.006408 0.006408 0.006408 1.61% FabArray::ParallelCopy() 380 0.006408 0.006408 0.006408 1.61% MLMG::mgVcycle_down::2 36 0.006333 0.006333 0.006333 1.60% FabArray::ParallelCopy_nowait() 380 0.006297 0.006297 0.006297 1.59% MLMG::mgVcycle_down::3 36 0.006181 0.006181 0.006181 1.56% Castro::post_restart() 1 0.006146 0.006146 0.006146 1.55% MLMG::mgVcycle_down::4 36 0.00612 0.00612 0.00612 1.54% Castro::expand_state() 5 0.005992 0.005992 0.005992 1.51% MLCellLinOp::defineAuxData() 6 0.005977 0.005977 0.005977 1.51% Castro::post_timestep() 5 0.005966 0.005966 0.005966 1.50% Gravity::multilevel_solve_for_new_phi() 1 0.005779 0.005779 0.005779 1.46% Gravity::actual_multilevel_solve() 1 0.005762 0.005762 0.005762 1.45% Gravity::fill_multipole_BCs() 6 0.005595 0.005595 0.005595 1.41% MLCGSolver::ParallelAllReduce 659 0.005309 0.005309 0.005309 1.34% MLMG::addInterpCorrection() 180 0.005212 0.005212 0.005212 1.31% MLMG::mgVcycle_up::4 36 0.004992 0.004992 0.004992 1.26% MLMG::mgVcycle_up::1 36 0.004954 0.004954 0.004954 1.25% amrex::average_down 180 0.004853 0.004853 0.004853 1.22% MLMG::mgVcycle_up::2 36 0.004838 0.004838 0.004838 1.22% MLMG::mgVcycle_up::3 36 0.004753 0.004753 0.004753 1.20% MLPoisson::Fapply() 500 0.004497 0.004497 0.004497 1.13% Castro::estTimeStep() 10 0.003735 0.003735 0.003735 0.94% FabArray::Saxpy() 355 0.003665 0.003665 0.003665 0.92% FabArray::Xpay() 361 0.003515 0.003515 0.003515 0.89% MLCellLinOp::solutionResidual() 42 0.003418 0.003418 0.003418 0.86% Castro::reset_internal_energy(MultiFab) 30 0.002887 0.002887 0.002887 0.73% MLMG::prepareForSolve() 6 0.00274 0.00274 0.00274 0.69% MLMG::computeResidual() 36 0.002677 0.002677 0.002677 0.67% Castro::enforce_speed_limit() 30 0.002633 0.002633 0.002633 0.66% MLCellLinOp::defineBC() 6 0.002629 0.002629 0.002629 0.66% Castro::computeNewDt() 5 0.002548 0.002548 0.002548 0.64% BndryData::define() 6 0.002499 0.002499 0.002499 0.63% FabArray::LinComb() 242 0.001964 0.001964 0.001964 0.49% amrex::Add() 72 0.001876 0.001876 0.001876 0.47% Castro::construct_new_source() 25 0.001832 0.001832 0.001832 0.46% Castro::construct_new_gravity_source() 5 0.00178 0.00178 0.00178 0.45% Castro::construct_old_source() 25 0.001565 0.001565 0.001565 0.39% Castro::construct_old_gravity_source() 5 0.001519 0.001519 0.001519 0.38% Castro::finalize_do_advance() 5 0.001285 0.001285 0.001285 0.32% Castro::apply_source_to_state() 10 0.0009481 0.0009481 0.0009481 0.24% MLMG::ResNormInf() 42 0.000943 0.000943 0.000943 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008998 0.0008998 0.0008998 0.23% check_for_negative_density() 5 0.0008873 0.0008873 0.0008873 0.22% MLMG::getGradSolution() 6 0.0008613 0.0008613 0.0008613 0.22% MLCellLinOp::compGrad() 6 0.000858 0.000858 0.000858 0.22% MLCellLinOp::setLevelBC() 6 0.0008094 0.0008094 0.0008094 0.20% Castro::reset_internal_energy(Fab) 240 0.0007998 0.0007998 0.0007998 0.20% FabArrayBase::getCPC() 632 0.0007739 0.0007739 0.0007739 0.19% MLMG::computeMLResidual() 6 0.0007639 0.0007639 0.0007639 0.19% MLPoisson::prepareForSolve() 6 0.00073 0.00073 0.00073 0.18% MLCellLinOp::prepareForSolve() 6 0.0007212 0.0007212 0.0007212 0.18% Gravity::update_max_rhs() 6 0.000697 0.000697 0.000697 0.18% FabArray::mult() 22 0.0006766 0.0006766 0.0006766 0.17% FabArray::setDomainBndry() 20 0.0006625 0.0006625 0.0006625 0.17% Castro::check_for_nan() 10 0.0006569 0.0006569 0.0006569 0.17% MultiFab::contains_nan() 10 0.0006501 0.0006501 0.0006501 0.16% FabArrayBase::CPC::define() 244 0.0004114 0.0004114 0.0004114 0.10% Amr::InitAmr() 1 0.0004086 0.0004086 0.0004086 0.10% FabArrayBase::getFB() 1766 0.0003272 0.0003272 0.0003272 0.08% Castro::finalize_advance() 5 0.0002915 0.0002915 0.0002915 0.07% Gravity::swapTimeLevels() 5 0.0002237 0.0002237 0.0002237 0.06% MultiFab::max() 6 0.0002007 0.0002007 0.0002007 0.05% MLMG::MLResNormInf() 6 0.0001857 0.0001857 0.0001857 0.05% Castro::buildMetrics() 1 0.000149 0.000149 0.000149 0.04% MLLinOp::define() 6 0.000128 0.000128 0.000128 0.03% MLLinOp::defineGrids() 6 0.0001218 0.0001218 0.0001218 0.03% MLMG::MLRhsNormInf() 6 0.0001215 0.0001215 0.0001215 0.03% Castro::create_source_corrector() 5 8.395e-05 8.395e-05 8.395e-05 0.02% FabArrayBase::FB::FB() 26 6.01e-05 6.01e-05 6.01e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.593e-05 3.593e-05 3.593e-05 0.01% Castro::initMFs() 1 3.105e-05 3.105e-05 3.105e-05 0.01% Amr::writeSmallPlotFile() 1 2.853e-05 2.853e-05 2.853e-05 0.01% makeSFC 30 2.815e-05 2.815e-05 2.815e-05 0.01% Castro::swap_state_time_levels() 5 2.678e-05 2.678e-05 2.678e-05 0.01% DistributionMapping::Distribute() 31 1.514e-05 1.514e-05 1.514e-05 0.00% Amr::initSubcycle() 1 9.241e-06 9.241e-06 9.241e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.301e-06 4.301e-06 4.301e-06 0.00% Castro::retry_advance_ctu() 5 2.24e-06 2.24e-06 2.24e-06 0.00% Gravity::set_mass_offset() 6 2.136e-06 2.136e-06 2.136e-06 0.00% Castro::FluxRegCrseInit 5 1.514e-06 1.514e-06 1.514e-06 0.00% Castro::FluxRegFineAdd() 5 1.285e-06 1.285e-06 1.285e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.093e-06 1.093e-06 1.093e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 217 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 126 MiB 692 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 45 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1131 KiB 39 MiB Castro::initialize_do_advance() 40 40 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1919 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6363 KiB 14 MiB MLMG::prepareForSolve() 361 361 3445 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 191 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 193 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6348 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 24 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3481 B 2048 KiB Gravity::solve_for_phi() 40 40 632 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 29 KiB 2048 KiB BndryData::define() 576 576 317 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 203 KiB 671 KiB Castro::estTimeStep() 10 10 4494 B 480 KiB VisMF::Write(FabArray) 112 112 1363 B 320 KiB Castro::normalize_species() 30 30 7444 B 320 KiB amrex::average_down 469 469 1171 B 257 KiB MLMG::addInterpCorrection() 468 468 1115 B 257 KiB amrex::Dot() 592 592 3234 B 160 KiB FabArray::norminf() 398 398 2416 B 160 KiB check_for_negative_density() 5 5 327 B 160 KiB MultiFab::max() 6 6 78 B 160 KiB FabArray::setVal() 66 66 17 KiB 23 KiB MultiFab::contains_nan() 10 10 32 B 20 KiB MLPoisson::Fsmooth() 60 60 3340 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 48 B 10 KiB FillBoundary_nowait() 336 336 285 B 9648 B MLCellLinOp::applyBC() 3892 3892 215 B 9344 B amrex::Copy() 56 56 5587 B 8464 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 47 B 2976 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 324 324 115 B 1472 B MLCellLinOp::defineBC() 36 36 357 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1236 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 87 KiB 8192 KiB VisMF::Write(FabArray) 120 120 186 KiB 3584 KiB VisMF::Read() 24 24 220 KiB 3000 KiB FabArray::setVal() 66 66 17 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3340 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 48 B 10 KiB FillBoundary_nowait() 336 336 285 B 9648 B MLCellLinOp::applyBC() 1946 1946 214 B 9328 B amrex::Copy() 56 56 5587 B 8464 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B Gravity::get_old_grav_vector() 3 3 2485 B 3072 B StateData::FillBoundary(geom) 960 960 47 B 2976 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 512 B 1296 B amrex::average_down 37 37 206 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 398 398 16 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.10-22-gbe6c6415467d) finalized