Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-11-g3129db344622) initialized Starting run at 10:11:36 UTC on 2022-12-13. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-11-g3129db344 Microphysics git describe: 22.12-16-g3e8f3c22 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056625493 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032273137 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.044920825 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.047086259 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048395629 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.053477559 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.069103998 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.055501877 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.058757599 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051664241 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057153067 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057335618 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.058844673 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.05557658 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032333471 seconds Ending run at 10:11:37 UTC on 2022-12-13. Run time = 0.827597774 Run time without initialization = 0.690798569 Average number of zones advanced per microsecond: 3.795 Average number of zones advanced per microsecond per rank: 3.795 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8276 ... 0.8276 ... 0.8276 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2256 0.2256 0.2256 27.26% Castro::construct_ctu_hydro_source() 10 0.1932 0.1932 0.1932 23.35% MLCellLinOp::applyBC() 4433 0.07345 0.07345 0.07345 8.87% MLPoisson::Fsmooth() 3280 0.05915 0.05915 0.05915 7.15% StateData::FillBoundary(geom) 328 0.02327 0.02327 0.02327 2.81% amrex::Dot() 1114 0.01979 0.01979 0.01979 2.39% Castro::normalize_species() 62 0.01485 0.01485 0.01485 1.79% amrex::Copy() 1029 0.01453 0.01453 0.01453 1.76% FabArray::norminf() 743 0.01396 0.01396 0.01396 1.69% FillBoundary_nowait() 4023 0.01388 0.01388 0.01388 1.68% Castro::computeTemp() 63 0.01308 0.01308 0.01308 1.58% FabArray::setVal() 1144 0.01283 0.01283 0.01283 1.55% StateDataPhysBCFunct::() 41 0.01281 0.01281 0.01281 1.55% FabArray::ParallelCopy_nowait() 861 0.01279 0.01279 0.01279 1.55% MLPoisson::Fapply() 1142 0.01023 0.01023 0.01023 1.24% MLCellLinOp::defineAuxData() 11 0.009415 0.009415 0.009415 1.14% Castro::enforce_min_density() 62 0.008584 0.008584 0.008584 1.04% FabArray::Saxpy() 813 0.008041 0.008041 0.008041 0.97% FabArray::Xpay() 821 0.007982 0.007982 0.007982 0.96% MLMG::addInterpCorrection() 410 0.006409 0.006409 0.006409 0.77% Gravity::fill_multipole_BCs() 11 0.006121 0.006121 0.006121 0.74% amrex::average_down 410 0.005722 0.005722 0.005722 0.69% FabArray::LinComb() 557 0.004367 0.004367 0.004367 0.53% Castro::reset_internal_energy(MultiFab) 63 0.004078 0.004078 0.004078 0.49% Amr::checkPoint() 3 0.003958 0.003958 0.003958 0.48% Castro::estTimeStep() 21 0.003652 0.003652 0.003652 0.44% BndryData::define() 11 0.003447 0.003447 0.003447 0.42% Castro::do_advance_ctu() 10 0.003241 0.003241 0.003241 0.39% Castro::construct_new_gravity_source() 10 0.003115 0.003115 0.003115 0.38% amrex::Add() 164 0.002337 0.002337 0.002337 0.28% Castro::construct_old_gravity_source() 10 0.002313 0.002313 0.002313 0.28% Amr::writePlotFile() 2 0.002301 0.002301 0.002301 0.28% MLCGSolver::bicgstab 82 0.002031 0.002031 0.002031 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001604 0.001604 0.001604 0.19% Castro::reset_internal_energy(Fab) 504 0.001506 0.001506 0.001506 0.18% Castro::enforce_speed_limit() 62 0.00141 0.00141 0.00141 0.17% MLCellLinOp::setLevelBC() 11 0.001343 0.001343 0.001343 0.16% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% FabArray::setDomainBndry() 41 0.001312 0.001312 0.001312 0.16% Gravity::actual_solve_with_mlmg() 11 0.001292 0.001292 0.001292 0.16% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% MLCellLinOp::smooth() 1640 0.001151 0.001151 0.001151 0.14% Castro::initData() 1 0.001111 0.001111 0.001111 0.13% MLCellLinOp::prepareForSolve() 11 0.001076 0.001076 0.001076 0.13% MLCellLinOp::compGrad() 11 0.0008966 0.0008966 0.0008966 0.11% FabArray::FillBoundary() 4023 0.0008892 0.0008892 0.0008892 0.11% MLMG::prepareForSolve() 11 0.0008129 0.0008129 0.0008129 0.10% FabArrayBase::getCPC() 1323 0.0007548 0.0007548 0.0007548 0.09% FabArrayBase::CPC::define() 454 0.0006898 0.0006898 0.0006898 0.08% FabArrayBase::getFB() 4023 0.0006229 0.0006229 0.0006229 0.08% Gravity::get_new_grav_vector() 11 0.0006079 0.0006079 0.0006079 0.07% Gravity::get_old_grav_vector() 10 0.0005337 0.0005337 0.0005337 0.06% Amr::InitAmr() 1 0.0004767 0.0004767 0.0004767 0.06% MLCellLinOp::apply() 1142 0.0004743 0.0004743 0.0004743 0.06% MLMG::mgVcycle() 82 0.0003933 0.0003933 0.0003933 0.05% Amr::coarseTimeStep() 10 0.0003627 0.0003627 0.0003627 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002881 0.0002881 0.0002881 0.03% MLLinOp::defineGrids() 11 0.0002828 0.0002828 0.0002828 0.03% main() 1 0.0002757 0.0002757 0.0002757 0.03% MultiFab::max() 11 0.0002531 0.0002531 0.0002531 0.03% FabArray::ParallelCopy() 861 0.000252 0.000252 0.000252 0.03% MLCellLinOp::correctionResidual() 492 0.0002256 0.0002256 0.0002256 0.03% FillPatchIterator::Initialize 41 0.0002116 0.0002116 0.0002116 0.03% MLCellLinOp::defineBC() 11 0.0001927 0.0001927 0.0001927 0.02% Amr::timeStep() 10 0.0001535 0.0001535 0.0001535 0.02% StateData::checkPoint() 12 0.0001361 0.0001361 0.0001361 0.02% Castro::subcycle_advance_ctu() 10 0.000133 0.000133 0.000133 0.02% MLMG:computeResOfCorrection() 410 0.0001218 0.0001218 0.0001218 0.01% Gravity::solve_for_phi() 10 0.0001174 0.0001174 0.0001174 0.01% Castro::advance() 10 0.0001168 0.0001168 0.0001168 0.01% Gravity::update_max_rhs() 11 0.0001155 0.0001155 0.0001155 0.01% Castro::finalize_advance() 10 0.0001133 0.0001133 0.0001133 0.01% MLMG::actualBottomSolve() 82 9.58e-05 9.58e-05 9.58e-05 0.01% MLMG::mgVcycle_down::0 82 9.439e-05 9.439e-05 9.439e-05 0.01% MLMG::mgVcycle_down::1 82 8.658e-05 8.658e-05 8.658e-05 0.01% FabArrayBase::FB::FB() 56 8.36e-05 8.36e-05 8.36e-05 0.01% MLMG::mgVcycle_down::2 82 8.235e-05 8.235e-05 8.235e-05 0.01% Castro::clean_state() 62 8.117e-05 8.117e-05 8.117e-05 0.01% MLMG::mgVcycle_down::3 82 7.827e-05 7.827e-05 7.827e-05 0.01% MLMG::mgVcycle_down::4 82 7.742e-05 7.742e-05 7.742e-05 0.01% Castro::expand_state() 10 7.601e-05 7.601e-05 7.601e-05 0.01% Castro::initialize_advance() 10 7.479e-05 7.479e-05 7.479e-05 0.01% AmrLevel::checkPoint() 3 7.33e-05 7.33e-05 7.33e-05 0.01% Castro::Castro() 1 7.174e-05 7.174e-05 7.174e-05 0.01% MLMG::solve() 11 7.041e-05 7.041e-05 7.041e-05 0.01% MLMG::mgVcycle_up::4 82 6.179e-05 6.179e-05 6.179e-05 0.01% MLMG::mgVcycle_up::0 82 5.281e-05 5.281e-05 5.281e-05 0.01% MLMG::oneIter() 82 5.254e-05 5.254e-05 5.254e-05 0.01% MLCellLinOp::solutionResidual() 93 5.211e-05 5.211e-05 5.211e-05 0.01% MLMG::mgVcycle_up::1 82 4.953e-05 4.953e-05 4.953e-05 0.01% MLMG::mgVcycle_up::3 82 4.841e-05 4.841e-05 4.841e-05 0.01% MLMG::mgVcycle_up::2 82 4.784e-05 4.784e-05 4.784e-05 0.01% Castro::finalize_do_advance() 10 4.432e-05 4.432e-05 4.432e-05 0.01% Castro::initialize_do_advance() 10 4.427e-05 4.427e-05 4.427e-05 0.01% Castro::swap_state_time_levels() 10 3.558e-05 3.558e-05 3.558e-05 0.00% MLMG::ResNormInf() 93 3.406e-05 3.406e-05 3.406e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.342e-05 3.342e-05 3.342e-05 0.00% Castro::enforce_consistent_e() 1 3.269e-05 3.269e-05 3.269e-05 0.00% MLMG::mgVcycle_bottom 82 3.259e-05 3.259e-05 3.259e-05 0.00% StateData::define() 4 3.199e-05 3.199e-05 3.199e-05 0.00% MLMG::computeResidual() 82 3.156e-05 3.156e-05 3.156e-05 0.00% Castro::construct_new_gravity() 10 3.035e-05 3.035e-05 3.035e-05 0.00% FillPatchSingleLevel 41 2.987e-05 2.987e-05 2.987e-05 0.00% Castro::post_timestep() 10 2.958e-05 2.958e-05 2.958e-05 0.00% makeSFC 55 2.745e-05 2.745e-05 2.745e-05 0.00% MLPoisson::define() 11 2.546e-05 2.546e-05 2.546e-05 0.00% Amr::writeSmallPlotFile() 1 2.455e-05 2.455e-05 2.455e-05 0.00% Castro::do_old_sources() 10 2.275e-05 2.275e-05 2.275e-05 0.00% Amr::FinalizeInit() 1 2.255e-05 2.255e-05 2.255e-05 0.00% Castro::initMFs() 1 1.99e-05 1.99e-05 1.99e-05 0.00% Castro::construct_old_source() 50 1.799e-05 1.799e-05 1.799e-05 0.00% DistributionMapping::Distribute() 56 1.765e-05 1.765e-05 1.765e-05 0.00% Castro::do_new_sources() 10 1.678e-05 1.678e-05 1.678e-05 0.00% Castro::construct_new_source() 50 1.669e-05 1.669e-05 1.669e-05 0.00% Castro::buildMetrics() 1 1.57e-05 1.57e-05 1.57e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.546e-05 1.546e-05 1.546e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.488e-05 1.488e-05 1.488e-05 0.00% Amr::defBaseLevel() 1 1.304e-05 1.304e-05 1.304e-05 0.00% Castro::apply_source_to_state() 20 1.11e-05 1.11e-05 1.11e-05 0.00% Castro::check_for_nan() 20 1.036e-05 1.036e-05 1.036e-05 0.00% Castro::construct_old_gravity() 10 1.012e-05 1.012e-05 1.012e-05 0.00% MLLinOp::define() 11 1.007e-05 1.007e-05 1.007e-05 0.00% Gravity::swapTimeLevels() 10 9.382e-06 9.382e-06 9.382e-06 0.00% Castro::post_init() 1 8.918e-06 8.918e-06 8.918e-06 0.00% Amr::initSubcycle() 1 8.875e-06 8.875e-06 8.875e-06 0.00% AmrLevel::AmrLevel(dm) 1 8.833e-06 8.833e-06 8.833e-06 0.00% MLPoisson::prepareForSolve() 11 8.277e-06 8.277e-06 8.277e-06 0.00% Gravity::actual_multilevel_solve() 1 7.209e-06 7.209e-06 7.209e-06 0.00% MLMG::computeMLResidual() 11 7.155e-06 7.155e-06 7.155e-06 0.00% MLMG::getGradSolution() 11 5.93e-06 5.93e-06 5.93e-06 0.00% Castro::computeNewDt() 9 5.805e-06 5.805e-06 5.805e-06 0.00% Amr::InitializeInit() 1 4.984e-06 4.984e-06 4.984e-06 0.00% Gravity::set_mass_offset() 11 4.809e-06 4.809e-06 4.809e-06 0.00% AmrLevel::checkPointPost() 3 4.497e-06 4.497e-06 4.497e-06 0.00% Castro::retry_advance_ctu() 10 4.194e-06 4.194e-06 4.194e-06 0.00% MLMG::MLRhsNormInf() 11 3.82e-06 3.82e-06 3.82e-06 0.00% Castro::create_source_corrector() 10 3.506e-06 3.506e-06 3.506e-06 0.00% MLMG::MLResNormInf() 11 3.494e-06 3.494e-06 3.494e-06 0.00% Castro::FluxRegCrseInit 10 3.07e-06 3.07e-06 3.07e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.948e-06 2.948e-06 2.948e-06 0.00% Amr::init() 1 2.824e-06 2.824e-06 2.824e-06 0.00% Castro::computeInitialDt() 2 2.481e-06 2.481e-06 2.481e-06 0.00% Castro::FluxRegFineAdd() 10 2.394e-06 2.394e-06 2.394e-06 0.00% AmrLevel::checkPointPre() 3 2.138e-06 2.138e-06 2.138e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.045e-06 2.045e-06 2.045e-06 0.00% Castro::post_regrid() 1 9.95e-07 9.95e-07 9.95e-07 0.00% Amr::initialInit() 1 9.06e-07 9.06e-07 9.06e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8276 0.8276 0.8276 100.00% Amr::coarseTimeStep() 10 0.6583 0.6583 0.6583 79.54% Amr::timeStep() 10 0.5448 0.5448 0.5448 65.82% Castro::advance() 10 0.538 0.538 0.538 65.00% Castro::subcycle_advance_ctu() 10 0.5251 0.5251 0.5251 63.45% Castro::do_advance_ctu() 10 0.525 0.525 0.525 63.43% Gravity::solve_phi_with_mlmg() 11 0.283 0.283 0.283 34.20% Gravity::actual_solve_with_mlmg() 11 0.2764 0.2764 0.2764 33.40% Castro::construct_new_gravity() 10 0.2585 0.2585 0.2585 31.24% MLMG::solve() 11 0.2568 0.2568 0.2568 31.02% Gravity::solve_for_phi() 10 0.2432 0.2432 0.2432 29.39% MLMG::oneIter() 82 0.2428 0.2428 0.2428 29.34% MLMG::mgVcycle() 82 0.2412 0.2412 0.2412 29.14% VisMF::Write(FabArray) 11 0.2256 0.2256 0.2256 27.26% Castro::construct_ctu_hydro_source() 10 0.1932 0.1932 0.1932 23.35% Amr::checkPoint() 3 0.1678 0.1678 0.1678 20.28% AmrLevel::checkPoint() 3 0.1639 0.1639 0.1639 19.80% StateData::checkPoint() 12 0.1638 0.1638 0.1638 19.79% Amr::init() 1 0.1362 0.1362 0.1362 16.45% MLCellLinOp::smooth() 1640 0.1265 0.1265 0.1265 15.29% MLCellLinOp::applyBC() 4433 0.08893 0.08893 0.08893 10.74% MLMG::mgVcycle_bottom 82 0.07124 0.07124 0.07124 8.61% MLMG::actualBottomSolve() 82 0.07121 0.07121 0.07121 8.60% MLCGSolver::bicgstab 82 0.07052 0.07052 0.07052 8.52% Amr::writePlotFile() 2 0.06472 0.06472 0.06472 7.82% MLPoisson::Fsmooth() 3280 0.05915 0.05915 0.05915 7.15% Amr::initialInit() 1 0.04717 0.04717 0.04717 5.70% Amr::FinalizeInit() 1 0.04333 0.04333 0.04333 5.24% Castro::clean_state() 62 0.04287 0.04287 0.04287 5.18% Castro::post_init() 1 0.04206 0.04206 0.04206 5.08% FillPatchIterator::Initialize 41 0.04167 0.04167 0.04167 5.03% Gravity::multilevel_solve_for_new_phi() 1 0.04023 0.04023 0.04023 4.86% Gravity::actual_multilevel_solve() 1 0.04022 0.04022 0.04022 4.86% FillPatchSingleLevel 41 0.04015 0.04015 0.04015 4.85% StateDataPhysBCFunct::() 41 0.03608 0.03608 0.03608 4.36% MLMG::mgVcycle_down::0 82 0.03448 0.03448 0.03448 4.17% MLCellLinOp::apply() 1142 0.03293 0.03293 0.03293 3.98% MLMG::mgVcycle_up::0 82 0.02937 0.02937 0.02937 3.55% StateData::FillBoundary(geom) 328 0.02327 0.02327 0.02327 2.81% amrex::Dot() 1114 0.01979 0.01979 0.01979 2.39% MLCellLinOp::correctionResidual() 492 0.01937 0.01937 0.01937 2.34% Castro::initialize_do_advance() 10 0.0191 0.0191 0.0191 2.31% Castro::computeTemp() 63 0.01867 0.01867 0.01867 2.26% MLMG:computeResOfCorrection() 410 0.01683 0.01683 0.01683 2.03% Gravity::get_new_grav_vector() 11 0.01682 0.01682 0.01682 2.03% MLMG::mgVcycle_down::1 82 0.0159 0.0159 0.0159 1.92% MLPoisson::define() 11 0.01565 0.01565 0.01565 1.89% FabArray::FillBoundary() 4023 0.01548 0.01548 0.01548 1.87% MLMG::mgVcycle_down::2 82 0.0154 0.0154 0.0154 1.86% Castro::normalize_species() 62 0.01485 0.01485 0.01485 1.79% Castro::construct_old_gravity() 10 0.01473 0.01473 0.01473 1.78% Gravity::get_old_grav_vector() 10 0.01472 0.01472 0.01472 1.78% FillBoundary_nowait() 4023 0.01459 0.01459 0.01459 1.76% MLMG::mgVcycle_down::3 82 0.01454 0.01454 0.01454 1.76% amrex::Copy() 1029 0.01453 0.01453 0.01453 1.76% FabArray::norminf() 743 0.01396 0.01396 0.01396 1.69% FabArray::ParallelCopy() 861 0.01388 0.01388 0.01388 1.68% MLMG::mgVcycle_down::4 82 0.01376 0.01376 0.01376 1.66% FabArray::ParallelCopy_nowait() 861 0.01363 0.01363 0.01363 1.65% FabArray::setVal() 1144 0.01283 0.01283 0.01283 1.55% Castro::initialize_advance() 10 0.01211 0.01211 0.01211 1.46% MLMG::mgVcycle_up::2 82 0.01193 0.01193 0.01193 1.44% MLCGSolver::ParallelAllReduce 1514 0.01189 0.01189 0.01189 1.44% MLMG::mgVcycle_up::1 82 0.01172 0.01172 0.01172 1.42% MLMG::addInterpCorrection() 410 0.01135 0.01135 0.01135 1.37% Castro::expand_state() 10 0.01134 0.01134 0.01134 1.37% MLMG::mgVcycle_up::3 82 0.01127 0.01127 0.01127 1.36% MLMG::mgVcycle_up::4 82 0.01118 0.01118 0.01118 1.35% Castro::do_new_sources() 10 0.01116 0.01116 0.01116 1.35% MLCellLinOp::defineAuxData() 11 0.01073 0.01073 0.01073 1.30% amrex::average_down 410 0.01062 0.01062 0.01062 1.28% MLPoisson::Fapply() 1142 0.01023 0.01023 0.01023 1.24% Castro::do_old_sources() 10 0.009496 0.009496 0.009496 1.15% Castro::enforce_min_density() 62 0.008584 0.008584 0.008584 1.04% FabArray::Saxpy() 813 0.008041 0.008041 0.008041 0.97% FabArray::Xpay() 821 0.007982 0.007982 0.007982 0.96% MLCellLinOp::solutionResidual() 93 0.007036 0.007036 0.007036 0.85% Castro::post_timestep() 10 0.006634 0.006634 0.006634 0.80% Gravity::fill_multipole_BCs() 11 0.006372 0.006372 0.006372 0.77% MLMG::computeResidual() 82 0.006091 0.006091 0.006091 0.74% Castro::reset_internal_energy(MultiFab) 63 0.005584 0.005584 0.005584 0.67% MLCellLinOp::defineBC() 11 0.004543 0.004543 0.004543 0.55% MLMG::prepareForSolve() 11 0.004483 0.004483 0.004483 0.54% FabArray::LinComb() 557 0.004367 0.004367 0.004367 0.53% BndryData::define() 11 0.004351 0.004351 0.004351 0.53% Amr::InitializeInit() 1 0.003831 0.003831 0.003831 0.46% Amr::defBaseLevel() 1 0.003826 0.003826 0.003826 0.46% Castro::estTimeStep() 21 0.003652 0.003652 0.003652 0.44% Castro::initData() 1 0.003365 0.003365 0.003365 0.41% Castro::construct_new_source() 50 0.003132 0.003132 0.003132 0.38% Castro::construct_new_gravity_source() 10 0.003115 0.003115 0.003115 0.38% amrex::Add() 164 0.002337 0.002337 0.002337 0.28% Castro::construct_old_source() 50 0.002331 0.002331 0.002331 0.28% Castro::construct_old_gravity_source() 10 0.002313 0.002313 0.002313 0.28% MLMG::ResNormInf() 93 0.002112 0.002112 0.002112 0.26% Castro::apply_source_to_state() 20 0.001813 0.001813 0.001813 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001604 0.001604 0.001604 0.19% Castro::computeNewDt() 9 0.00153 0.00153 0.00153 0.18% Castro::reset_internal_energy(Fab) 504 0.001506 0.001506 0.001506 0.18% FabArrayBase::getCPC() 1323 0.001445 0.001445 0.001445 0.17% Castro::enforce_speed_limit() 62 0.00141 0.00141 0.00141 0.17% MLMG::getGradSolution() 11 0.001359 0.001359 0.001359 0.16% MLCellLinOp::compGrad() 11 0.001353 0.001353 0.001353 0.16% MLCellLinOp::setLevelBC() 11 0.001343 0.001343 0.001343 0.16% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% FabArray::setDomainBndry() 41 0.001312 0.001312 0.001312 0.16% Castro::check_for_nan() 20 0.001193 0.001193 0.001193 0.14% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% Castro::post_regrid() 1 0.001105 0.001105 0.001105 0.13% MLPoisson::prepareForSolve() 11 0.001085 0.001085 0.001085 0.13% MLCellLinOp::prepareForSolve() 11 0.001076 0.001076 0.001076 0.13% MLMG::computeMLResidual() 11 0.0009834 0.0009834 0.0009834 0.12% Gravity::update_max_rhs() 11 0.0008022 0.0008022 0.0008022 0.10% FabArrayBase::getFB() 4023 0.0007065 0.0007065 0.0007065 0.09% FabArrayBase::CPC::define() 454 0.0006898 0.0006898 0.0006898 0.08% Castro::finalize_advance() 10 0.0006268 0.0006268 0.0006268 0.08% Castro::computeInitialDt() 2 0.0005748 0.0005748 0.0005748 0.07% Amr::InitAmr() 1 0.0004855 0.0004855 0.0004855 0.06% Gravity::swapTimeLevels() 10 0.0004286 0.0004286 0.0004286 0.05% Castro::Castro() 1 0.0004029 0.0004029 0.0004029 0.05% MLLinOp::define() 11 0.0003541 0.0003541 0.0003541 0.04% MLLinOp::defineGrids() 11 0.000344 0.000344 0.000344 0.04% MLMG::MLResNormInf() 11 0.0002815 0.0002815 0.0002815 0.03% MultiFab::max() 11 0.0002531 0.0002531 0.0002531 0.03% MLMG::MLRhsNormInf() 11 0.0002188 0.0002188 0.0002188 0.03% Castro::buildMetrics() 1 0.0001537 0.0001537 0.0001537 0.02% FabArrayBase::FB::FB() 56 8.36e-05 8.36e-05 8.36e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.917e-05 5.917e-05 5.917e-05 0.01% Castro::finalize_do_advance() 10 4.432e-05 4.432e-05 4.432e-05 0.01% makeSFC 55 4.371e-05 4.371e-05 4.371e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.082e-05 4.082e-05 4.082e-05 0.00% Castro::swap_state_time_levels() 10 3.558e-05 3.558e-05 3.558e-05 0.00% Castro::enforce_consistent_e() 1 3.269e-05 3.269e-05 3.269e-05 0.00% StateData::define() 4 3.199e-05 3.199e-05 3.199e-05 0.00% Amr::writeSmallPlotFile() 1 2.455e-05 2.455e-05 2.455e-05 0.00% Castro::initMFs() 1 1.99e-05 1.99e-05 1.99e-05 0.00% DistributionMapping::Distribute() 56 1.765e-05 1.765e-05 1.765e-05 0.00% Amr::initSubcycle() 1 8.875e-06 8.875e-06 8.875e-06 0.00% Gravity::set_mass_offset() 11 4.809e-06 4.809e-06 4.809e-06 0.00% AmrLevel::checkPointPost() 3 4.497e-06 4.497e-06 4.497e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.342e-06 4.342e-06 4.342e-06 0.00% Castro::retry_advance_ctu() 10 4.194e-06 4.194e-06 4.194e-06 0.00% Castro::create_source_corrector() 10 3.506e-06 3.506e-06 3.506e-06 0.00% Castro::FluxRegCrseInit 10 3.07e-06 3.07e-06 3.07e-06 0.00% Castro::FluxRegFineAdd() 10 2.394e-06 2.394e-06 2.394e-06 0.00% AmrLevel::checkPointPre() 3 2.138e-06 2.138e-06 2.138e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.045e-06 2.045e-06 2.045e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-11-g3129db344622) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-11-g3129db344622) initialized Starting run at 10:11:38 UTC on 2022-12-13. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-11-g3129db344 Microphysics git describe: 22.12-16-g3e8f3c22 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.45528865 Restart time = 0.072627732 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047441413 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.045794064 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047239123 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054410305 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.058118124 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032310327 seconds Ending run at 10:11:38 UTC on 2022-12-13. Run time = 0.358881145 Run time without initialization = 0.285689645 Average number of zones advanced per microsecond: 4.588 Average number of zones advanced per microsecond per rank: 4.588 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3589 ... 0.3589 ... 0.3589 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0782 0.0782 0.0782 21.80% VisMF::Read() 3 0.04022 0.04022 0.04022 11.21% MLCellLinOp::applyBC() 1946 0.03168 0.03168 0.03168 8.83% VisMF::Write(FabArray) 1 0.03081 0.03081 0.03081 8.58% Amr::restart() 1 0.02874 0.02874 0.02874 8.01% MLPoisson::Fsmooth() 1440 0.02508 0.02508 0.02508 6.99% StateData::FillBoundary(geom) 160 0.01143 0.01143 0.01143 3.19% amrex::Dot() 484 0.008399 0.008399 0.008399 2.34% amrex::Copy() 463 0.006765 0.006765 0.006765 1.88% FabArray::setVal() 537 0.006058 0.006058 0.006058 1.69% FillBoundary_nowait() 1766 0.006047 0.006047 0.006047 1.68% FabArray::norminf() 326 0.005992 0.005992 0.005992 1.67% Castro::computeTemp() 30 0.005945 0.005945 0.005945 1.66% FabArray::ParallelCopy_nowait() 380 0.005787 0.005787 0.005787 1.61% Castro::normalize_species() 30 0.005782 0.005782 0.005782 1.61% StateDataPhysBCFunct::() 20 0.005603 0.005603 0.005603 1.56% MLCellLinOp::defineAuxData() 6 0.005059 0.005059 0.005059 1.41% MLPoisson::Fapply() 500 0.004394 0.004394 0.004394 1.22% Castro::estTimeStep() 10 0.003682 0.003682 0.003682 1.03% FabArray::Saxpy() 355 0.003559 0.003559 0.003559 0.99% FabArray::Xpay() 361 0.003455 0.003455 0.003455 0.96% Gravity::fill_multipole_BCs() 6 0.002909 0.002909 0.002909 0.81% MLMG::addInterpCorrection() 180 0.00275 0.00275 0.00275 0.77% amrex::average_down 180 0.002472 0.002472 0.002472 0.69% Castro::enforce_min_density() 30 0.002107 0.002107 0.002107 0.59% BndryData::define() 6 0.001882 0.001882 0.001882 0.52% FabArray::LinComb() 242 0.001859 0.001859 0.001859 0.52% Castro::reset_internal_energy(MultiFab) 30 0.001854 0.001854 0.001854 0.52% Castro::construct_new_gravity_source() 5 0.001648 0.001648 0.001648 0.46% Amr::writePlotFile() 1 0.001345 0.001345 0.001345 0.37% Castro::construct_old_gravity_source() 5 0.001238 0.001238 0.001238 0.34% Castro::do_advance_ctu() 5 0.001081 0.001081 0.001081 0.30% amrex::Add() 72 0.001013 0.001013 0.001013 0.28% MLCGSolver::bicgstab 36 0.0009153 0.0009153 0.0009153 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008683 0.0008683 0.0008683 0.24% MLCellLinOp::setLevelBC() 6 0.0007097 0.0007097 0.0007097 0.20% Castro::reset_internal_energy(Fab) 240 0.0006995 0.0006995 0.0006995 0.19% Gravity::actual_solve_with_mlmg() 6 0.0006928 0.0006928 0.0006928 0.19% FabArray::mult() 22 0.0006443 0.0006443 0.0006443 0.18% FabArray::setDomainBndry() 20 0.0006163 0.0006163 0.0006163 0.17% MultiFab::contains_nan() 10 0.0005829 0.0005829 0.0005829 0.16% MLCellLinOp::prepareForSolve() 6 0.0005809 0.0005809 0.0005809 0.16% MLCellLinOp::smooth() 720 0.0005515 0.0005515 0.0005515 0.15% Castro::enforce_speed_limit() 30 0.0005451 0.0005451 0.0005451 0.15% MLCellLinOp::compGrad() 6 0.0004764 0.0004764 0.0004764 0.13% MLMG::prepareForSolve() 6 0.0004469 0.0004469 0.0004469 0.12% FabArrayBase::CPC::define() 244 0.000408 0.000408 0.000408 0.11% FabArray::FillBoundary() 1766 0.0003862 0.0003862 0.0003862 0.11% Amr::InitAmr() 1 0.0003836 0.0003836 0.0003836 0.11% FabArrayBase::getCPC() 632 0.0003639 0.0003639 0.0003639 0.10% Gravity::get_old_grav_vector() 5 0.0002868 0.0002868 0.0002868 0.08% Gravity::get_new_grav_vector() 5 0.0002667 0.0002667 0.0002667 0.07% main() 1 0.0002623 0.0002623 0.0002623 0.07% FabArrayBase::getFB() 1766 0.0002584 0.0002584 0.0002584 0.07% MLCellLinOp::apply() 500 0.0002324 0.0002324 0.0002324 0.06% MLMG::mgVcycle() 36 0.0001764 0.0001764 0.0001764 0.05% Amr::coarseTimeStep() 5 0.0001531 0.0001531 0.0001531 0.04% MultiFab::max() 6 0.0001343 0.0001343 0.0001343 0.04% MLCGSolver::ParallelAllReduce 659 0.0001295 0.0001295 0.0001295 0.04% AmrLevel::restart() 1 0.0001247 0.0001247 0.0001247 0.03% FabArray::ParallelCopy() 380 0.0001156 0.0001156 0.0001156 0.03% MLCellLinOp::defineBC() 6 0.0001032 0.0001032 0.0001032 0.03% MLCellLinOp::correctionResidual() 216 0.0001018 0.0001018 0.0001018 0.03% FillPatchIterator::Initialize 20 9.999e-05 9.999e-05 9.999e-05 0.03% StateData::restartDoit() 4 9.156e-05 9.156e-05 9.156e-05 0.03% MLLinOp::defineGrids() 6 8.804e-05 8.804e-05 8.804e-05 0.02% Amr::timeStep() 5 8.409e-05 8.409e-05 8.409e-05 0.02% Gravity::solve_for_phi() 5 8.261e-05 8.261e-05 8.261e-05 0.02% Castro::subcycle_advance_ctu() 5 6.814e-05 6.814e-05 6.814e-05 0.02% Castro::advance() 5 6.707e-05 6.707e-05 6.707e-05 0.02% Gravity::update_max_rhs() 6 5.952e-05 5.952e-05 5.952e-05 0.02% FabArrayBase::FB::FB() 26 5.626e-05 5.626e-05 5.626e-05 0.02% MLMG:computeResOfCorrection() 180 5.432e-05 5.432e-05 5.432e-05 0.02% Castro::expand_state() 5 4.552e-05 4.552e-05 4.552e-05 0.01% MLMG::actualBottomSolve() 36 4.332e-05 4.332e-05 4.332e-05 0.01% MLMG::mgVcycle_down::0 36 4.271e-05 4.271e-05 4.271e-05 0.01% MLMG::mgVcycle_down::2 36 4.026e-05 4.026e-05 4.026e-05 0.01% Castro::clean_state() 30 3.966e-05 3.966e-05 3.966e-05 0.01% MLMG::mgVcycle_down::1 36 3.957e-05 3.957e-05 3.957e-05 0.01% Castro::create_source_corrector() 5 3.747e-05 3.747e-05 3.747e-05 0.01% MLMG::mgVcycle_down::4 36 3.496e-05 3.496e-05 3.496e-05 0.01% MLMG::mgVcycle_down::3 36 3.415e-05 3.415e-05 3.415e-05 0.01% MLMG::solve() 6 3.291e-05 3.291e-05 3.291e-05 0.01% Castro::initialize_advance() 5 3.113e-05 3.113e-05 3.113e-05 0.01% MLMG::mgVcycle_up::4 36 2.884e-05 2.884e-05 2.884e-05 0.01% Castro::finalize_advance() 5 2.661e-05 2.661e-05 2.661e-05 0.01% MLMG::oneIter() 36 2.552e-05 2.552e-05 2.552e-05 0.01% MLMG::mgVcycle_up::0 36 2.509e-05 2.509e-05 2.509e-05 0.01% Castro::buildMetrics() 1 2.483e-05 2.483e-05 2.483e-05 0.01% Amr::writeSmallPlotFile() 1 2.345e-05 2.345e-05 2.345e-05 0.01% Castro::initialize_do_advance() 5 2.277e-05 2.277e-05 2.277e-05 0.01% MLMG::mgVcycle_up::3 36 2.261e-05 2.261e-05 2.261e-05 0.01% MLCellLinOp::solutionResidual() 42 2.204e-05 2.204e-05 2.204e-05 0.01% Castro::swap_state_time_levels() 5 2.201e-05 2.201e-05 2.201e-05 0.01% MLMG::mgVcycle_up::2 36 2.198e-05 2.198e-05 2.198e-05 0.01% Castro::post_restart() 1 2.128e-05 2.128e-05 2.128e-05 0.01% MLMG::mgVcycle_up::1 36 2.109e-05 2.109e-05 2.109e-05 0.01% Castro::initMFs() 1 1.982e-05 1.982e-05 1.982e-05 0.01% Castro::finalize_do_advance() 5 1.959e-05 1.959e-05 1.959e-05 0.01% MLPoisson::define() 6 1.954e-05 1.954e-05 1.954e-05 0.01% MLMG::ResNormInf() 42 1.776e-05 1.776e-05 1.776e-05 0.00% MLMG::mgVcycle_bottom 36 1.518e-05 1.518e-05 1.518e-05 0.00% makeSFC 30 1.453e-05 1.453e-05 1.453e-05 0.00% MLMG::computeResidual() 36 1.409e-05 1.409e-05 1.409e-05 0.00% FillPatchSingleLevel 20 1.402e-05 1.402e-05 1.402e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.319e-05 1.319e-05 1.319e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.282e-05 1.282e-05 1.282e-05 0.00% DistributionMapping::Distribute() 31 1.21e-05 1.21e-05 1.21e-05 0.00% Castro::construct_new_gravity() 5 1.178e-05 1.178e-05 1.178e-05 0.00% Castro::construct_old_source() 25 9.812e-06 9.812e-06 9.812e-06 0.00% Castro::construct_new_source() 25 9.262e-06 9.262e-06 9.262e-06 0.00% Castro::do_new_sources() 5 8.746e-06 8.746e-06 8.746e-06 0.00% Amr::initSubcycle() 1 8.655e-06 8.655e-06 8.655e-06 0.00% Castro::do_old_sources() 5 8.322e-06 8.322e-06 8.322e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.838e-06 7.838e-06 7.838e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 6.829e-06 6.829e-06 6.829e-06 0.00% Gravity::actual_multilevel_solve() 1 6.545e-06 6.545e-06 6.545e-06 0.00% Castro::check_for_nan() 10 5.771e-06 5.771e-06 5.771e-06 0.00% MLLinOp::define() 6 5.653e-06 5.653e-06 5.653e-06 0.00% Castro::apply_source_to_state() 10 5.512e-06 5.512e-06 5.512e-06 0.00% Castro::construct_old_gravity() 5 5.043e-06 5.043e-06 5.043e-06 0.00% Gravity::swapTimeLevels() 5 5.003e-06 5.003e-06 5.003e-06 0.00% Castro::post_timestep() 5 4.945e-06 4.945e-06 4.945e-06 0.00% MLPoisson::prepareForSolve() 6 4.229e-06 4.229e-06 4.229e-06 0.00% MLMG::computeMLResidual() 6 3.63e-06 3.63e-06 3.63e-06 0.00% AmrLevel::AmrLevel() 1 3.619e-06 3.619e-06 3.619e-06 0.00% MLMG::getGradSolution() 6 3.158e-06 3.158e-06 3.158e-06 0.00% Castro::computeNewDt() 5 3.014e-06 3.014e-06 3.014e-06 0.00% Gravity::set_mass_offset() 6 2.299e-06 2.299e-06 2.299e-06 0.00% MLMG::MLResNormInf() 6 2.252e-06 2.252e-06 2.252e-06 0.00% MLMG::MLRhsNormInf() 6 2.187e-06 2.187e-06 2.187e-06 0.00% Castro::retry_advance_ctu() 5 1.797e-06 1.797e-06 1.797e-06 0.00% Castro::FluxRegCrseInit 5 1.502e-06 1.502e-06 1.502e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.101e-06 1.101e-06 1.101e-06 0.00% Castro::FluxRegFineAdd() 5 1.069e-06 1.069e-06 1.069e-06 0.00% Amr::init() 1 1.006e-06 1.006e-06 1.006e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3589 0.3589 0.3589 100.00% Amr::coarseTimeStep() 5 0.2531 0.2531 0.2531 70.53% Amr::timeStep() 5 0.25 0.25 0.25 69.67% Castro::advance() 5 0.2476 0.2476 0.2476 69.00% Castro::subcycle_advance_ctu() 5 0.2422 0.2422 0.2422 67.50% Castro::do_advance_ctu() 5 0.2422 0.2422 0.2422 67.48% Castro::construct_new_gravity() 5 0.1292 0.1292 0.1292 35.99% Gravity::solve_phi_with_mlmg() 6 0.1245 0.1245 0.1245 34.68% Gravity::solve_for_phi() 5 0.1216 0.1216 0.1216 33.87% Gravity::actual_solve_with_mlmg() 6 0.1213 0.1213 0.1213 33.80% MLMG::solve() 6 0.1107 0.1107 0.1107 30.85% MLMG::oneIter() 36 0.1041 0.1041 0.1041 29.00% MLMG::mgVcycle() 36 0.1034 0.1034 0.1034 28.80% Castro::construct_ctu_hydro_source() 5 0.07823 0.07823 0.07823 21.80% Amr::init() 1 0.07267 0.07267 0.07267 20.25% Amr::restart() 1 0.07267 0.07267 0.07267 20.25% MLCellLinOp::smooth() 720 0.05412 0.05412 0.05412 15.08% AmrLevel::restart() 1 0.0405 0.0405 0.0405 11.28% StateData::restartDoit() 4 0.04036 0.04036 0.04036 11.25% VisMF::Read() 3 0.04022 0.04022 0.04022 11.21% MLCellLinOp::applyBC() 1946 0.03843 0.03843 0.03843 10.71% Amr::writePlotFile() 1 0.03239 0.03239 0.03239 9.03% VisMF::Write(FabArray) 1 0.03081 0.03081 0.03081 8.58% MLMG::mgVcycle_bottom 36 0.03046 0.03046 0.03046 8.49% MLMG::actualBottomSolve() 36 0.03045 0.03045 0.03045 8.48% MLCGSolver::bicgstab 36 0.03015 0.03015 0.03015 8.40% MLPoisson::Fsmooth() 1440 0.02508 0.02508 0.02508 6.99% FillPatchIterator::Initialize 20 0.01976 0.01976 0.01976 5.51% FillPatchSingleLevel 20 0.01905 0.01905 0.01905 5.31% StateDataPhysBCFunct::() 20 0.01704 0.01704 0.01704 4.75% Castro::clean_state() 30 0.01697 0.01697 0.01697 4.73% MLMG::mgVcycle_down::0 36 0.01475 0.01475 0.01475 4.11% MLCellLinOp::apply() 500 0.01431 0.01431 0.01431 3.99% MLMG::mgVcycle_up::0 36 0.01263 0.01263 0.01263 3.52% StateData::FillBoundary(geom) 160 0.01143 0.01143 0.01143 3.19% Castro::initialize_do_advance() 5 0.008935 0.008935 0.008935 2.49% Castro::computeTemp() 30 0.008498 0.008498 0.008498 2.37% MLPoisson::define() 6 0.008448 0.008448 0.008448 2.35% amrex::Dot() 484 0.008399 0.008399 0.008399 2.34% MLCellLinOp::correctionResidual() 216 0.008376 0.008376 0.008376 2.33% Gravity::get_new_grav_vector() 5 0.007487 0.007487 0.007487 2.09% MLMG:computeResOfCorrection() 180 0.007273 0.007273 0.007273 2.03% Castro::construct_old_gravity() 5 0.007237 0.007237 0.007237 2.02% Gravity::get_old_grav_vector() 5 0.007232 0.007232 0.007232 2.01% MLMG::mgVcycle_down::1 36 0.006835 0.006835 0.006835 1.90% amrex::Copy() 463 0.006765 0.006765 0.006765 1.88% FabArray::FillBoundary() 1766 0.006747 0.006747 0.006747 1.88% Castro::do_new_sources() 5 0.006653 0.006653 0.006653 1.85% MLMG::mgVcycle_down::2 36 0.00663 0.00663 0.00663 1.85% FillBoundary_nowait() 1766 0.006361 0.006361 0.006361 1.77% FabArray::ParallelCopy() 380 0.006284 0.006284 0.006284 1.75% MLMG::mgVcycle_down::3 36 0.006273 0.006273 0.006273 1.75% FabArray::ParallelCopy_nowait() 380 0.006169 0.006169 0.006169 1.72% FabArray::setVal() 537 0.006058 0.006058 0.006058 1.69% FabArray::norminf() 326 0.005992 0.005992 0.005992 1.67% MLMG::mgVcycle_down::4 36 0.005901 0.005901 0.005901 1.64% MLCellLinOp::defineAuxData() 6 0.005801 0.005801 0.005801 1.62% Castro::normalize_species() 30 0.005782 0.005782 0.005782 1.61% Castro::expand_state() 5 0.005683 0.005683 0.005683 1.58% MLMG::mgVcycle_up::2 36 0.005097 0.005097 0.005097 1.42% MLCGSolver::ParallelAllReduce 659 0.00508 0.00508 0.00508 1.42% Castro::initialize_advance() 5 0.005042 0.005042 0.005042 1.40% MLMG::mgVcycle_up::1 36 0.005006 0.005006 0.005006 1.39% MLMG::addInterpCorrection() 180 0.004889 0.004889 0.004889 1.36% MLMG::mgVcycle_up::4 36 0.004816 0.004816 0.004816 1.34% MLMG::mgVcycle_up::3 36 0.004805 0.004805 0.004805 1.34% amrex::average_down 180 0.004619 0.004619 0.004619 1.29% MLPoisson::Fapply() 500 0.004394 0.004394 0.004394 1.22% Castro::do_old_sources() 5 0.003891 0.003891 0.003891 1.08% Castro::estTimeStep() 10 0.003682 0.003682 0.003682 1.03% FabArray::Saxpy() 355 0.003559 0.003559 0.003559 0.99% FabArray::Xpay() 361 0.003455 0.003455 0.003455 0.96% Castro::post_restart() 1 0.003264 0.003264 0.003264 0.91% Gravity::multilevel_solve_for_new_phi() 1 0.003153 0.003153 0.003153 0.88% MLCellLinOp::solutionResidual() 42 0.003147 0.003147 0.003147 0.88% Gravity::actual_multilevel_solve() 1 0.00314 0.00314 0.00314 0.87% Gravity::fill_multipole_BCs() 6 0.003035 0.003035 0.003035 0.85% Castro::computeNewDt() 5 0.002954 0.002954 0.002954 0.82% MLMG::computeResidual() 36 0.002618 0.002618 0.002618 0.73% Castro::reset_internal_energy(MultiFab) 30 0.002553 0.002553 0.002553 0.71% MLCellLinOp::defineBC() 6 0.002502 0.002502 0.002502 0.70% BndryData::define() 6 0.002399 0.002399 0.002399 0.67% MLMG::prepareForSolve() 6 0.002397 0.002397 0.002397 0.67% Castro::post_timestep() 5 0.002317 0.002317 0.002317 0.65% Castro::enforce_min_density() 30 0.002107 0.002107 0.002107 0.59% FabArray::LinComb() 242 0.001859 0.001859 0.001859 0.52% Castro::construct_new_source() 25 0.001657 0.001657 0.001657 0.46% Castro::construct_new_gravity_source() 5 0.001648 0.001648 0.001648 0.46% Castro::construct_old_source() 25 0.001248 0.001248 0.001248 0.35% Castro::construct_old_gravity_source() 5 0.001238 0.001238 0.001238 0.34% amrex::Add() 72 0.001013 0.001013 0.001013 0.28% MLMG::ResNormInf() 42 0.0009262 0.0009262 0.0009262 0.26% Castro::apply_source_to_state() 10 0.0009221 0.0009221 0.0009221 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008683 0.0008683 0.0008683 0.24% FabArrayBase::getCPC() 632 0.0007719 0.0007719 0.0007719 0.22% MLMG::getGradSolution() 6 0.0007358 0.0007358 0.0007358 0.21% MLCellLinOp::compGrad() 6 0.0007327 0.0007327 0.0007327 0.20% MLCellLinOp::setLevelBC() 6 0.0007097 0.0007097 0.0007097 0.20% Castro::reset_internal_energy(Fab) 240 0.0006995 0.0006995 0.0006995 0.19% FabArray::mult() 22 0.0006443 0.0006443 0.0006443 0.18% FabArray::setDomainBndry() 20 0.0006163 0.0006163 0.0006163 0.17% Castro::check_for_nan() 10 0.0005886 0.0005886 0.0005886 0.16% MLPoisson::prepareForSolve() 6 0.0005851 0.0005851 0.0005851 0.16% MultiFab::contains_nan() 10 0.0005829 0.0005829 0.0005829 0.16% MLCellLinOp::prepareForSolve() 6 0.0005809 0.0005809 0.0005809 0.16% MLMG::computeMLResidual() 6 0.0005466 0.0005466 0.0005466 0.15% Castro::enforce_speed_limit() 30 0.0005451 0.0005451 0.0005451 0.15% Gravity::update_max_rhs() 6 0.0004291 0.0004291 0.0004291 0.12% FabArrayBase::CPC::define() 244 0.000408 0.000408 0.000408 0.11% Amr::InitAmr() 1 0.0003923 0.0003923 0.0003923 0.11% FabArrayBase::getFB() 1766 0.0003147 0.0003147 0.0003147 0.09% Castro::finalize_advance() 5 0.0002831 0.0002831 0.0002831 0.08% Gravity::swapTimeLevels() 5 0.0002184 0.0002184 0.0002184 0.06% MLMG::MLResNormInf() 6 0.0001476 0.0001476 0.0001476 0.04% Castro::buildMetrics() 1 0.0001401 0.0001401 0.0001401 0.04% MultiFab::max() 6 0.0001343 0.0001343 0.0001343 0.04% MLLinOp::define() 6 0.0001256 0.0001256 0.0001256 0.03% MLLinOp::defineGrids() 6 0.0001199 0.0001199 0.0001199 0.03% MLMG::MLRhsNormInf() 6 0.0001131 0.0001131 0.0001131 0.03% FabArrayBase::FB::FB() 26 5.626e-05 5.626e-05 5.626e-05 0.02% Castro::create_source_corrector() 5 3.747e-05 3.747e-05 3.747e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.076e-05 3.076e-05 3.076e-05 0.01% Amr::writeSmallPlotFile() 1 2.345e-05 2.345e-05 2.345e-05 0.01% makeSFC 30 2.292e-05 2.292e-05 2.292e-05 0.01% Castro::swap_state_time_levels() 5 2.201e-05 2.201e-05 2.201e-05 0.01% Castro::initMFs() 1 1.982e-05 1.982e-05 1.982e-05 0.01% Castro::finalize_do_advance() 5 1.959e-05 1.959e-05 1.959e-05 0.01% DistributionMapping::Distribute() 31 1.21e-05 1.21e-05 1.21e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.054e-05 1.054e-05 1.054e-05 0.00% Amr::initSubcycle() 1 8.655e-06 8.655e-06 8.655e-06 0.00% AmrLevel::AmrLevel() 1 3.619e-06 3.619e-06 3.619e-06 0.00% Gravity::set_mass_offset() 6 2.299e-06 2.299e-06 2.299e-06 0.00% Castro::retry_advance_ctu() 5 1.797e-06 1.797e-06 1.797e-06 0.00% Castro::FluxRegCrseInit 5 1.502e-06 1.502e-06 1.502e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.101e-06 1.101e-06 1.101e-06 0.00% Castro::FluxRegFineAdd() 5 1.069e-06 1.069e-06 1.069e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-11-g3129db344622) finalized