Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-1-g4a53367b19e1) initialized Starting run at 09:52:31 UTC on 2022-12-06. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-1-g4a53367b1 Microphysics git describe: 22.12-10-g85f387df reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.054364873 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.031261585 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049408242 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.05176471 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.05226723 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.051985477 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.079922201 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.050648406 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.067300272 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.058455425 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051294115 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.055331211 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066517497 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.050948398 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031489278 seconds Ending run at 09:52:32 UTC on 2022-12-06. Run time = 0.857490208 Run time without initialization = 0.71794696 Average number of zones advanced per microsecond: 3.651 Average number of zones advanced per microsecond per rank: 3.651 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8575 ... 0.8575 ... 0.8575 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2107 0.2107 0.2107 24.57% Castro::construct_ctu_hydro_source() 10 0.1896 0.1896 0.1896 22.11% MLCellLinOp::applyBC() 4433 0.08335 0.08335 0.08335 9.72% MLPoisson::Fsmooth() 3280 0.06574 0.06574 0.06574 7.67% StateData::FillBoundary(geom) 328 0.02503 0.02503 0.02503 2.92% amrex::Dot() 1114 0.02348 0.02348 0.02348 2.74% amrex::Copy() 1029 0.01627 0.01627 0.01627 1.90% FabArray::LinComb() 1586 0.01506 0.01506 0.01506 1.76% FabArray::setVal() 1144 0.01474 0.01474 0.01474 1.72% FillBoundary_nowait() 4023 0.01453 0.01453 0.01453 1.69% Castro::normalize_species() 62 0.01406 0.01406 0.01406 1.64% FabArray::ParallelCopy_nowait() 861 0.01365 0.01365 0.01365 1.59% FabArray::norminf() 639 0.01357 0.01357 0.01357 1.58% Castro::computeTemp() 63 0.01321 0.01321 0.01321 1.54% StateDataPhysBCFunct::() 41 0.01227 0.01227 0.01227 1.43% MLPoisson::Fapply() 1142 0.01223 0.01223 0.01223 1.43% MLCellLinOp::defineAuxData() 11 0.01186 0.01186 0.01186 1.38% Castro::enforce_min_density() 62 0.009073 0.009073 0.009073 1.06% Gravity::fill_multipole_BCs() 11 0.008769 0.008769 0.008769 1.02% MLMG::addInterpCorrection() 410 0.007908 0.007908 0.007908 0.92% amrex::average_down 410 0.007142 0.007142 0.007142 0.83% FabArray::Xpay() 585 0.006845 0.006845 0.006845 0.80% Amr::checkPoint() 3 0.005182 0.005182 0.005182 0.60% Castro::estTimeStep() 21 0.004369 0.004369 0.004369 0.51% Castro::reset_internal_energy(MultiFab) 63 0.004259 0.004259 0.004259 0.50% BndryData::define() 11 0.004082 0.004082 0.004082 0.48% Castro::construct_new_gravity_source() 10 0.003259 0.003259 0.003259 0.38% MLCGSolver::bicgstab 82 0.003236 0.003236 0.003236 0.38% Castro::do_advance_ctu() 10 0.002893 0.002893 0.002893 0.34% amrex::Add() 164 0.002607 0.002607 0.002607 0.30% Castro::construct_old_gravity_source() 10 0.002591 0.002591 0.002591 0.30% Amr::writePlotFile() 2 0.002365 0.002365 0.002365 0.28% MLMG::ResNormInf() 93 0.002134 0.002134 0.002134 0.25% FabArray::Saxpy() 20 0.001812 0.001812 0.001812 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00172 0.00172 0.00172 0.20% MLCellLinOp::setLevelBC() 11 0.001605 0.001605 0.001605 0.19% Castro::reset_internal_energy(Fab) 504 0.001592 0.001592 0.001592 0.19% Castro::enforce_speed_limit() 62 0.001575 0.001575 0.001575 0.18% Gravity::actual_solve_with_mlmg() 11 0.00139 0.00139 0.00139 0.16% FabArray::setDomainBndry() 41 0.00135 0.00135 0.00135 0.16% FabArray::mult() 43 0.001336 0.001336 0.001336 0.16% Castro::initData() 1 0.001326 0.001326 0.001326 0.15% MLMG::prepareForSolve() 11 0.001246 0.001246 0.001246 0.15% MLCellLinOp::prepareForSolve() 11 0.001211 0.001211 0.001211 0.14% MultiFab::contains_nan() 20 0.0012 0.0012 0.0012 0.14% MLCellLinOp::smooth() 1640 0.001018 0.001018 0.001018 0.12% MLCellLinOp::compGrad() 11 0.0009515 0.0009515 0.0009515 0.11% FabArray::FillBoundary() 4023 0.0007901 0.0007901 0.0007901 0.09% FabArrayBase::getCPC() 1323 0.0007506 0.0007506 0.0007506 0.09% FabArrayBase::CPC::define() 454 0.0006808 0.0006808 0.0006808 0.08% Gravity::get_new_grav_vector() 11 0.0006415 0.0006415 0.0006415 0.07% FabArrayBase::getFB() 4023 0.000621 0.000621 0.000621 0.07% Gravity::get_old_grav_vector() 10 0.0005559 0.0005559 0.0005559 0.06% MLLinOp::defineGrids() 11 0.0004884 0.0004884 0.0004884 0.06% Amr::InitAmr() 1 0.0004702 0.0004702 0.0004702 0.05% MLCellLinOp::apply() 1142 0.0004561 0.0004561 0.0004561 0.05% MLMG::mgVcycle() 82 0.0003888 0.0003888 0.0003888 0.05% Amr::coarseTimeStep() 10 0.0003439 0.0003439 0.0003439 0.04% CGSolver::sxay() 1586 0.0003353 0.0003353 0.0003353 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002987 0.0002987 0.0002987 0.03% FillPatchIterator::Initialize 41 0.0002941 0.0002941 0.0002941 0.03% MLCellLinOp::defineBC() 11 0.0002777 0.0002777 0.0002777 0.03% main() 1 0.0002772 0.0002772 0.0002772 0.03% FabArray::ParallelCopy() 861 0.0002667 0.0002667 0.0002667 0.03% MultiFab::max() 11 0.0002639 0.0002639 0.0002639 0.03% Castro::subcycle_advance_ctu() 10 0.0002499 0.0002499 0.0002499 0.03% MLMG::MLRhsNormInf() 11 0.0002262 0.0002262 0.0002262 0.03% MLCellLinOp::correctionResidual() 492 0.0002004 0.0002004 0.0002004 0.02% Amr::timeStep() 10 0.0001481 0.0001481 0.0001481 0.02% MLMG:computeResOfCorrection() 410 0.0001455 0.0001455 0.0001455 0.02% Gravity::update_max_rhs() 11 0.0001413 0.0001413 0.0001413 0.02% StateData::checkPoint() 12 0.0001311 0.0001311 0.0001311 0.02% Gravity::solve_for_phi() 10 0.0001242 0.0001242 0.0001242 0.01% MLMG::mgVcycle_down::0 82 0.0001051 0.0001051 0.0001051 0.01% MLMG::mgVcycle_down::1 82 9.468e-05 9.468e-05 9.468e-05 0.01% Castro::finalize_advance() 10 8.926e-05 8.926e-05 8.926e-05 0.01% MLMG::mgVcycle_down::2 82 8.78e-05 8.78e-05 8.78e-05 0.01% FabArrayBase::FB::FB() 56 8.727e-05 8.727e-05 8.727e-05 0.01% Castro::initialize_advance() 10 8.563e-05 8.563e-05 8.563e-05 0.01% MLMG::mgVcycle_down::3 82 8.476e-05 8.476e-05 8.476e-05 0.01% MLMG::mgVcycle_down::4 82 8.458e-05 8.458e-05 8.458e-05 0.01% Castro::Castro() 1 8.26e-05 8.26e-05 8.26e-05 0.01% Castro::clean_state() 62 7.818e-05 7.818e-05 7.818e-05 0.01% MLMG::actualBottomSolve() 82 7.755e-05 7.755e-05 7.755e-05 0.01% AmrLevel::checkPoint() 3 7.735e-05 7.735e-05 7.735e-05 0.01% Castro::expand_state() 10 7.646e-05 7.646e-05 7.646e-05 0.01% MLMG::solve() 11 7.49e-05 7.49e-05 7.49e-05 0.01% StateData::define() 4 7.09e-05 7.09e-05 7.09e-05 0.01% MLMG::mgVcycle_up::4 82 6.864e-05 6.864e-05 6.864e-05 0.01% MLMG::oneIter() 82 6.83e-05 6.83e-05 6.83e-05 0.01% Castro::initialize_do_advance() 10 6.601e-05 6.601e-05 6.601e-05 0.01% Castro::construct_new_gravity() 10 5.945e-05 5.945e-05 5.945e-05 0.01% MLMG::mgVcycle_up::0 82 5.419e-05 5.419e-05 5.419e-05 0.01% MLMG::mgVcycle_up::1 82 5.231e-05 5.231e-05 5.231e-05 0.01% MLMG::mgVcycle_up::2 82 5.138e-05 5.138e-05 5.138e-05 0.01% MLMG::mgVcycle_up::3 82 5.043e-05 5.043e-05 5.043e-05 0.01% Castro::advance() 10 4.736e-05 4.736e-05 4.736e-05 0.01% MLCellLinOp::solutionResidual() 93 4.521e-05 4.521e-05 4.521e-05 0.01% MLMG::computeResidual() 82 4.44e-05 4.44e-05 4.44e-05 0.01% Castro::swap_state_time_levels() 10 4.246e-05 4.246e-05 4.246e-05 0.00% MLMG::mgVcycle_bottom 82 3.584e-05 3.584e-05 3.584e-05 0.00% Castro::construct_new_source() 50 3.582e-05 3.582e-05 3.582e-05 0.00% Castro::enforce_consistent_e() 1 3.539e-05 3.539e-05 3.539e-05 0.00% MLPoisson::define() 11 3.515e-05 3.515e-05 3.515e-05 0.00% Castro::finalize_do_advance() 10 3.429e-05 3.429e-05 3.429e-05 0.00% FillPatchSingleLevel 41 3.114e-05 3.114e-05 3.114e-05 0.00% Castro::initMFs() 1 2.998e-05 2.998e-05 2.998e-05 0.00% MultiFab::Add() 82 2.977e-05 2.977e-05 2.977e-05 0.00% MLLinOp::define() 11 2.535e-05 2.535e-05 2.535e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.488e-05 2.488e-05 2.488e-05 0.00% makeSFC 55 2.452e-05 2.452e-05 2.452e-05 0.00% Amr::writeSmallPlotFile() 1 2.401e-05 2.401e-05 2.401e-05 0.00% Castro::buildMetrics() 1 2.312e-05 2.312e-05 2.312e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.008e-05 2.008e-05 2.008e-05 0.00% Amr::defBaseLevel() 1 2.001e-05 2.001e-05 2.001e-05 0.00% Amr::FinalizeInit() 1 1.928e-05 1.928e-05 1.928e-05 0.00% Castro::construct_old_source() 50 1.781e-05 1.781e-05 1.781e-05 0.00% Castro::do_new_sources() 10 1.716e-05 1.716e-05 1.716e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.615e-05 1.615e-05 1.615e-05 0.00% Castro::do_old_sources() 10 1.53e-05 1.53e-05 1.53e-05 0.00% DistributionMapping::Distribute() 56 1.444e-05 1.444e-05 1.444e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.37e-05 1.37e-05 1.37e-05 0.00% Castro::apply_source_to_state() 20 1.097e-05 1.097e-05 1.097e-05 0.00% Castro::check_for_nan() 20 1.085e-05 1.085e-05 1.085e-05 0.00% Castro::construct_old_gravity() 10 1.042e-05 1.042e-05 1.042e-05 0.00% MLPoisson::prepareForSolve() 11 9.544e-06 9.544e-06 9.544e-06 0.00% Castro::post_init() 1 9.305e-06 9.305e-06 9.305e-06 0.00% Castro::post_timestep() 10 9.203e-06 9.203e-06 9.203e-06 0.00% MLMG::computeMLResidual() 11 9.147e-06 9.147e-06 9.147e-06 0.00% Gravity::swapTimeLevels() 10 8.901e-06 8.901e-06 8.901e-06 0.00% Gravity::actual_multilevel_solve() 1 8.77e-06 8.77e-06 8.77e-06 0.00% Amr::initSubcycle() 1 8.24e-06 8.24e-06 8.24e-06 0.00% Castro::computeNewDt() 9 7.099e-06 7.099e-06 7.099e-06 0.00% MLMG::getGradSolution() 11 6.638e-06 6.638e-06 6.638e-06 0.00% MultiFab::Copy() 11 5.665e-06 5.665e-06 5.665e-06 0.00% Amr::InitializeInit() 1 5.143e-06 5.143e-06 5.143e-06 0.00% AmrLevel::checkPointPost() 3 4.871e-06 4.871e-06 4.871e-06 0.00% Castro::create_source_corrector() 10 4.831e-06 4.831e-06 4.831e-06 0.00% Gravity::set_mass_offset() 11 4.409e-06 4.409e-06 4.409e-06 0.00% Castro::retry_advance_ctu() 10 4.122e-06 4.122e-06 4.122e-06 0.00% MLMG::MLResNormInf() 11 3.652e-06 3.652e-06 3.652e-06 0.00% Castro::computeInitialDt() 2 3.122e-06 3.122e-06 3.122e-06 0.00% Castro::FluxRegCrseInit 10 3.085e-06 3.085e-06 3.085e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.743e-06 2.743e-06 2.743e-06 0.00% Amr::init() 1 2.393e-06 2.393e-06 2.393e-06 0.00% Castro::FluxRegFineAdd() 10 1.983e-06 1.983e-06 1.983e-06 0.00% AmrLevel::checkPointPre() 3 1.902e-06 1.902e-06 1.902e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.9e-06 1.9e-06 1.9e-06 0.00% Castro::post_regrid() 1 1.334e-06 1.334e-06 1.334e-06 0.00% Amr::initialInit() 1 1.048e-06 1.048e-06 1.048e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8575 0.8575 0.8575 100.00% Amr::coarseTimeStep() 10 0.6863 0.6863 0.6863 80.03% Amr::timeStep() 10 0.5819 0.5819 0.5819 67.86% Castro::advance() 10 0.5749 0.5749 0.5749 67.04% Castro::subcycle_advance_ctu() 10 0.5615 0.5615 0.5615 65.48% Castro::do_advance_ctu() 10 0.5612 0.5612 0.5612 65.45% Gravity::solve_phi_with_mlmg() 11 0.327 0.327 0.327 38.13% Gravity::actual_solve_with_mlmg() 11 0.3177 0.3177 0.3177 37.05% Castro::construct_new_gravity() 10 0.2978 0.2978 0.2978 34.72% MLMG::solve() 11 0.2941 0.2941 0.2941 34.30% Gravity::solve_for_phi() 10 0.2818 0.2818 0.2818 32.86% MLMG::oneIter() 82 0.2786 0.2786 0.2786 32.49% MLMG::mgVcycle() 82 0.2768 0.2768 0.2768 32.28% VisMF::Write(FabArray) 11 0.2107 0.2107 0.2107 24.57% Castro::construct_ctu_hydro_source() 10 0.1896 0.1896 0.1896 22.11% Amr::checkPoint() 3 0.1561 0.1561 0.1561 18.20% AmrLevel::checkPoint() 3 0.1509 0.1509 0.1509 17.60% StateData::checkPoint() 12 0.1508 0.1508 0.1508 17.59% MLCellLinOp::smooth() 1640 0.1405 0.1405 0.1405 16.38% Amr::init() 1 0.1389 0.1389 0.1389 16.20% MLCellLinOp::applyBC() 4433 0.09938 0.09938 0.09938 11.59% MLMG::mgVcycle_bottom 82 0.08614 0.08614 0.08614 10.05% MLMG::actualBottomSolve() 82 0.08611 0.08611 0.08611 10.04% MLCGSolver::bicgstab 82 0.08527 0.08527 0.08527 9.94% MLPoisson::Fsmooth() 3280 0.06574 0.06574 0.06574 7.67% Amr::writePlotFile() 2 0.06287 0.06287 0.06287 7.33% Amr::initialInit() 1 0.0532 0.0532 0.0532 6.20% Amr::FinalizeInit() 1 0.04888 0.04888 0.04888 5.70% Castro::post_init() 1 0.04759 0.04759 0.04759 5.55% Gravity::multilevel_solve_for_new_phi() 1 0.04567 0.04567 0.04567 5.33% Gravity::actual_multilevel_solve() 1 0.04565 0.04565 0.04565 5.32% FillPatchIterator::Initialize 41 0.04301 0.04301 0.04301 5.02% Castro::clean_state() 62 0.04294 0.04294 0.04294 5.01% FillPatchSingleLevel 41 0.04137 0.04137 0.04137 4.82% MLCellLinOp::apply() 1142 0.03784 0.03784 0.03784 4.41% StateDataPhysBCFunct::() 41 0.0373 0.0373 0.0373 4.35% MLMG::mgVcycle_down::0 82 0.03628 0.03628 0.03628 4.23% MLMG::mgVcycle_up::0 82 0.03111 0.03111 0.03111 3.63% StateData::FillBoundary(geom) 328 0.02503 0.02503 0.02503 2.92% amrex::Dot() 1114 0.02348 0.02348 0.02348 2.74% MLCellLinOp::correctionResidual() 492 0.02207 0.02207 0.02207 2.57% MLPoisson::define() 11 0.01915 0.01915 0.01915 2.23% Castro::computeTemp() 63 0.01906 0.01906 0.01906 2.22% Castro::initialize_do_advance() 10 0.01905 0.01905 0.01905 2.22% MLMG:computeResOfCorrection() 410 0.01905 0.01905 0.01905 2.22% MLMG::mgVcycle_down::1 82 0.01837 0.01837 0.01837 2.14% MLMG::mgVcycle_down::2 82 0.0179 0.0179 0.0179 2.09% Gravity::get_new_grav_vector() 11 0.01757 0.01757 0.01757 2.05% MLMG::mgVcycle_down::3 82 0.01702 0.01702 0.01702 1.98% MLMG::mgVcycle_down::4 82 0.01627 0.01627 0.01627 1.90% amrex::Copy() 1029 0.01627 0.01627 0.01627 1.90% FabArray::FillBoundary() 4023 0.01603 0.01603 0.01603 1.87% CGSolver::sxay() 1586 0.0154 0.0154 0.0154 1.80% Castro::construct_old_gravity() 10 0.01527 0.01527 0.01527 1.78% Gravity::get_old_grav_vector() 10 0.01526 0.01526 0.01526 1.78% FillBoundary_nowait() 4023 0.01524 0.01524 0.01524 1.78% FabArray::LinComb() 1586 0.01506 0.01506 0.01506 1.76% FabArray::ParallelCopy() 861 0.01474 0.01474 0.01474 1.72% FabArray::setVal() 1144 0.01474 0.01474 0.01474 1.72% FabArray::ParallelCopy_nowait() 861 0.01448 0.01448 0.01448 1.69% Castro::normalize_species() 62 0.01406 0.01406 0.01406 1.64% MLCGSolver::ParallelAllReduce 1514 0.014 0.014 0.014 1.63% MLMG::mgVcycle_up::2 82 0.01374 0.01374 0.01374 1.60% MLMG::mgVcycle_up::1 82 0.01357 0.01357 0.01357 1.58% FabArray::norminf() 639 0.01357 0.01357 0.01357 1.58% MLMG::addInterpCorrection() 410 0.01328 0.01328 0.01328 1.55% MLCellLinOp::defineAuxData() 11 0.01321 0.01321 0.01321 1.54% MLMG::mgVcycle_up::3 82 0.01304 0.01304 0.01304 1.52% MLMG::mgVcycle_up::4 82 0.01296 0.01296 0.01296 1.51% Castro::initialize_advance() 10 0.01279 0.01279 0.01279 1.49% amrex::average_down 410 0.01248 0.01248 0.01248 1.46% MLPoisson::Fapply() 1142 0.01223 0.01223 0.01223 1.43% Castro::expand_state() 10 0.01162 0.01162 0.01162 1.36% Castro::do_old_sources() 10 0.01034 0.01034 0.01034 1.21% Castro::do_new_sources() 10 0.01016 0.01016 0.01016 1.19% Castro::enforce_min_density() 62 0.009073 0.009073 0.009073 1.06% Gravity::fill_multipole_BCs() 11 0.00901 0.00901 0.00901 1.05% MLCellLinOp::solutionResidual() 93 0.007407 0.007407 0.007407 0.86% FabArray::Xpay() 585 0.006845 0.006845 0.006845 0.80% Castro::post_timestep() 10 0.006802 0.006802 0.006802 0.79% MLMG::computeResidual() 82 0.006395 0.006395 0.006395 0.75% Castro::reset_internal_energy(MultiFab) 63 0.005851 0.005851 0.005851 0.68% MLMG::prepareForSolve() 11 0.005613 0.005613 0.005613 0.65% MLCellLinOp::defineBC() 11 0.00534 0.00534 0.00534 0.62% BndryData::define() 11 0.005062 0.005062 0.005062 0.59% Castro::estTimeStep() 21 0.004369 0.004369 0.004369 0.51% Amr::InitializeInit() 1 0.00431 0.00431 0.00431 0.50% Amr::defBaseLevel() 1 0.004305 0.004305 0.004305 0.50% Castro::initData() 1 0.003746 0.003746 0.003746 0.44% Castro::construct_new_source() 50 0.003295 0.003295 0.003295 0.38% Castro::construct_new_gravity_source() 10 0.003259 0.003259 0.003259 0.38% Castro::construct_old_source() 50 0.002609 0.002609 0.002609 0.30% amrex::Add() 164 0.002607 0.002607 0.002607 0.30% Castro::construct_old_gravity_source() 10 0.002591 0.002591 0.002591 0.30% MLMG::ResNormInf() 93 0.002134 0.002134 0.002134 0.25% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.21% FabArray::Saxpy() 20 0.001812 0.001812 0.001812 0.21% MultiFab::Add() 82 0.001754 0.001754 0.001754 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00172 0.00172 0.00172 0.20% Castro::computeNewDt() 9 0.001663 0.001663 0.001663 0.19% MLCellLinOp::setLevelBC() 11 0.001605 0.001605 0.001605 0.19% Castro::reset_internal_energy(Fab) 504 0.001592 0.001592 0.001592 0.19% Castro::enforce_speed_limit() 62 0.001575 0.001575 0.001575 0.18% MLMG::getGradSolution() 11 0.00147 0.00147 0.00147 0.17% MLCellLinOp::compGrad() 11 0.001463 0.001463 0.001463 0.17% FabArrayBase::getCPC() 1323 0.001431 0.001431 0.001431 0.17% FabArray::setDomainBndry() 41 0.00135 0.00135 0.00135 0.16% FabArray::mult() 43 0.001336 0.001336 0.001336 0.16% MLPoisson::prepareForSolve() 11 0.00122 0.00122 0.00122 0.14% Castro::check_for_nan() 20 0.001211 0.001211 0.001211 0.14% MLCellLinOp::prepareForSolve() 11 0.001211 0.001211 0.001211 0.14% MultiFab::contains_nan() 20 0.0012 0.0012 0.0012 0.14% Castro::post_regrid() 1 0.001109 0.001109 0.001109 0.13% MLMG::computeMLResidual() 11 0.001066 0.001066 0.001066 0.12% Gravity::update_max_rhs() 11 0.0008625 0.0008625 0.0008625 0.10% Castro::computeInitialDt() 2 0.0008586 0.0008586 0.0008586 0.10% FabArrayBase::getFB() 4023 0.0007083 0.0007083 0.0007083 0.08% FabArrayBase::CPC::define() 454 0.0006808 0.0006808 0.0006808 0.08% Castro::finalize_advance() 10 0.0006175 0.0006175 0.0006175 0.07% MLLinOp::define() 11 0.0005673 0.0005673 0.0005673 0.07% MLLinOp::defineGrids() 11 0.000542 0.000542 0.000542 0.06% Amr::InitAmr() 1 0.0004784 0.0004784 0.0004784 0.06% Gravity::swapTimeLevels() 10 0.0004538 0.0004538 0.0004538 0.05% Castro::Castro() 1 0.0004478 0.0004478 0.0004478 0.05% MultiFab::Copy() 11 0.0003497 0.0003497 0.0003497 0.04% MLMG::MLResNormInf() 11 0.0002889 0.0002889 0.0002889 0.03% MultiFab::max() 11 0.0002639 0.0002639 0.0002639 0.03% MLMG::MLRhsNormInf() 11 0.0002262 0.0002262 0.0002262 0.03% Castro::buildMetrics() 1 0.0001693 0.0001693 0.0001693 0.02% FabArrayBase::FB::FB() 56 8.727e-05 8.727e-05 8.727e-05 0.01% AmrLevel::AmrLevel(dm) 1 8.705e-05 8.705e-05 8.705e-05 0.01% StateData::define() 4 7.09e-05 7.09e-05 7.09e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.162e-05 5.162e-05 5.162e-05 0.01% Castro::swap_state_time_levels() 10 4.246e-05 4.246e-05 4.246e-05 0.00% makeSFC 55 3.792e-05 3.792e-05 3.792e-05 0.00% Castro::enforce_consistent_e() 1 3.539e-05 3.539e-05 3.539e-05 0.00% Castro::finalize_do_advance() 10 3.429e-05 3.429e-05 3.429e-05 0.00% Castro::initMFs() 1 2.998e-05 2.998e-05 2.998e-05 0.00% Amr::writeSmallPlotFile() 1 2.401e-05 2.401e-05 2.401e-05 0.00% DistributionMapping::Distribute() 56 1.444e-05 1.444e-05 1.444e-05 0.00% Amr::initSubcycle() 1 8.24e-06 8.24e-06 8.24e-06 0.00% AmrLevel::checkPointPost() 3 4.871e-06 4.871e-06 4.871e-06 0.00% Castro::create_source_corrector() 10 4.831e-06 4.831e-06 4.831e-06 0.00% Gravity::set_mass_offset() 11 4.409e-06 4.409e-06 4.409e-06 0.00% Castro::retry_advance_ctu() 10 4.122e-06 4.122e-06 4.122e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.787e-06 3.787e-06 3.787e-06 0.00% Castro::FluxRegCrseInit 10 3.085e-06 3.085e-06 3.085e-06 0.00% Castro::FluxRegFineAdd() 10 1.983e-06 1.983e-06 1.983e-06 0.00% AmrLevel::checkPointPre() 3 1.902e-06 1.902e-06 1.902e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.9e-06 1.9e-06 1.9e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-1-g4a53367b19e1) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-1-g4a53367b19e1) initialized Starting run at 09:52:32 UTC on 2022-12-06. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-1-g4a53367b1 Microphysics git describe: 22.12-10-g85f387df reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.475542467 Restart time = 0.049744267 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.055696105 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.053121345 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062315281 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063184369 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068474951 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033201448 seconds Ending run at 09:52:33 UTC on 2022-12-06. Run time = 0.386802807 Run time without initialization = 0.336463578 Average number of zones advanced per microsecond: 3.896 Average number of zones advanced per microsecond per rank: 3.896 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3868 ... 0.3868 ... 0.3868 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0942 0.0942 0.0942 24.34% VisMF::Read() 3 0.04166 0.04166 0.04166 10.77% MLCellLinOp::applyBC() 1946 0.03554 0.03554 0.03554 9.19% VisMF::Write(FabArray) 1 0.03159 0.03159 0.03159 8.17% MLPoisson::Fsmooth() 1440 0.02769 0.02769 0.02769 7.16% StateData::FillBoundary(geom) 160 0.01205 0.01205 0.01205 3.12% amrex::Dot() 484 0.009904 0.009904 0.009904 2.56% Castro::normalize_species() 30 0.009226 0.009226 0.009226 2.38% amrex::Copy() 463 0.007526 0.007526 0.007526 1.95% Castro::computeTemp() 30 0.007432 0.007432 0.007432 1.92% Castro::enforce_min_density() 30 0.007247 0.007247 0.007247 1.87% FabArray::setVal() 537 0.006932 0.006932 0.006932 1.79% FillBoundary_nowait() 1766 0.006392 0.006392 0.006392 1.65% MLCellLinOp::defineAuxData() 6 0.00632 0.00632 0.00632 1.63% FabArray::LinComb() 690 0.006248 0.006248 0.006248 1.62% FabArray::ParallelCopy_nowait() 380 0.006155 0.006155 0.006155 1.59% FabArray::norminf() 278 0.005719 0.005719 0.005719 1.48% StateDataPhysBCFunct::() 20 0.005357 0.005357 0.005357 1.38% MLPoisson::Fapply() 500 0.005233 0.005233 0.005233 1.35% Gravity::fill_multipole_BCs() 6 0.004699 0.004699 0.004699 1.21% Amr::restart() 1 0.003768 0.003768 0.003768 0.97% MLMG::addInterpCorrection() 180 0.003425 0.003425 0.003425 0.89% amrex::average_down 180 0.003051 0.003051 0.003051 0.79% FabArray::Xpay() 258 0.00295 0.00295 0.00295 0.76% Castro::estTimeStep() 10 0.002684 0.002684 0.002684 0.69% BndryData::define() 6 0.002223 0.002223 0.002223 0.57% Castro::reset_internal_energy(MultiFab) 30 0.001832 0.001832 0.001832 0.47% Castro::construct_new_gravity_source() 5 0.001671 0.001671 0.001671 0.43% Castro::reset_internal_energy(Fab) 240 0.001621 0.001621 0.001621 0.42% Castro::do_advance_ctu() 5 0.001605 0.001605 0.001605 0.41% Castro::subcycle_advance_ctu() 5 0.001491 0.001491 0.001491 0.39% Amr::writePlotFile() 1 0.001439 0.001439 0.001439 0.37% Castro::construct_old_gravity_source() 5 0.001437 0.001437 0.001437 0.37% MLCGSolver::bicgstab 36 0.001406 0.001406 0.001406 0.36% amrex::Add() 72 0.001108 0.001108 0.001108 0.29% MLMG::ResNormInf() 42 0.0009322 0.0009322 0.0009322 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009185 0.0009185 0.0009185 0.24% FabArray::Saxpy() 10 0.0009181 0.0009181 0.0009181 0.24% Castro::create_source_corrector() 5 0.0008841 0.0008841 0.0008841 0.23% Castro::initialize_advance() 5 0.0008552 0.0008552 0.0008552 0.22% MLCellLinOp::setLevelBC() 6 0.0008275 0.0008275 0.0008275 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007416 0.0007416 0.0007416 0.19% Castro::post_timestep() 5 0.0006996 0.0006996 0.0006996 0.18% FabArray::setDomainBndry() 20 0.0006616 0.0006616 0.0006616 0.17% FabArray::mult() 22 0.0006593 0.0006593 0.0006593 0.17% MLMG::prepareForSolve() 6 0.0006489 0.0006489 0.0006489 0.17% MLCellLinOp::prepareForSolve() 6 0.000641 0.000641 0.000641 0.17% MultiFab::contains_nan() 10 0.0006199 0.0006199 0.0006199 0.16% Castro::enforce_speed_limit() 30 0.0005021 0.0005021 0.0005021 0.13% MLCellLinOp::smooth() 720 0.0004983 0.0004983 0.0004983 0.13% MLCellLinOp::compGrad() 6 0.0004916 0.0004916 0.0004916 0.13% Amr::InitAmr() 1 0.0004175 0.0004175 0.0004175 0.11% FabArrayBase::CPC::define() 244 0.0003997 0.0003997 0.0003997 0.10% FabArrayBase::getCPC() 632 0.0003667 0.0003667 0.0003667 0.09% Gravity::get_old_grav_vector() 5 0.0003586 0.0003586 0.0003586 0.09% FabArray::FillBoundary() 1766 0.0003507 0.0003507 0.0003507 0.09% main() 1 0.0002872 0.0002872 0.0002872 0.07% Gravity::get_new_grav_vector() 5 0.0002767 0.0002767 0.0002767 0.07% FabArrayBase::getFB() 1766 0.0002705 0.0002705 0.0002705 0.07% Amr::coarseTimeStep() 5 0.0002239 0.0002239 0.0002239 0.06% MLCellLinOp::apply() 500 0.0001993 0.0001993 0.0001993 0.05% MLMG::mgVcycle() 36 0.0001661 0.0001661 0.0001661 0.04% CGSolver::sxay() 690 0.0001637 0.0001637 0.0001637 0.04% MLLinOp::defineGrids() 6 0.0001557 0.0001557 0.0001557 0.04% MLCellLinOp::defineBC() 6 0.0001458 0.0001458 0.0001458 0.04% MultiFab::max() 6 0.0001435 0.0001435 0.0001435 0.04% Amr::timeStep() 5 0.0001404 0.0001404 0.0001404 0.04% FillPatchIterator::Initialize 20 0.0001355 0.0001355 0.0001355 0.04% MLCGSolver::ParallelAllReduce 659 0.0001317 0.0001317 0.0001317 0.03% FabArray::ParallelCopy() 380 0.0001174 0.0001174 0.0001174 0.03% MLMG::MLRhsNormInf() 6 0.0001159 0.0001159 0.0001159 0.03% Gravity::update_max_rhs() 6 0.0001006 0.0001006 0.0001006 0.03% MLCellLinOp::correctionResidual() 216 9.053e-05 9.053e-05 9.053e-05 0.02% Castro::computeNewDt() 5 8.337e-05 8.337e-05 8.337e-05 0.02% AmrLevel::restart() 1 8.335e-05 8.335e-05 8.335e-05 0.02% Castro::advance() 5 7.997e-05 7.997e-05 7.997e-05 0.02% StateData::restartDoit() 4 7.657e-05 7.657e-05 7.657e-05 0.02% Castro::construct_new_gravity() 5 7.286e-05 7.286e-05 7.286e-05 0.02% MLMG:computeResOfCorrection() 180 6.428e-05 6.428e-05 6.428e-05 0.02% FabArrayBase::FB::FB() 26 5.841e-05 5.841e-05 5.841e-05 0.02% Gravity::solve_for_phi() 5 5.645e-05 5.645e-05 5.645e-05 0.01% MLMG::mgVcycle_down::0 36 4.75e-05 4.75e-05 4.75e-05 0.01% Castro::finalize_advance() 5 4.44e-05 4.44e-05 4.44e-05 0.01% Castro::clean_state() 30 4.325e-05 4.325e-05 4.325e-05 0.01% Castro::construct_old_gravity() 5 4.215e-05 4.215e-05 4.215e-05 0.01% MLMG::mgVcycle_down::1 36 4.189e-05 4.189e-05 4.189e-05 0.01% Castro::do_new_sources() 5 4.09e-05 4.09e-05 4.09e-05 0.01% Castro::expand_state() 5 3.923e-05 3.923e-05 3.923e-05 0.01% MLMG::actualBottomSolve() 36 3.751e-05 3.751e-05 3.751e-05 0.01% MLMG::mgVcycle_down::2 36 3.73e-05 3.73e-05 3.73e-05 0.01% MLMG::mgVcycle_down::4 36 3.641e-05 3.641e-05 3.641e-05 0.01% Castro::buildMetrics() 1 3.627e-05 3.627e-05 3.627e-05 0.01% Castro::construct_old_source() 25 3.595e-05 3.595e-05 3.595e-05 0.01% MLMG::solve() 6 3.544e-05 3.544e-05 3.544e-05 0.01% MLMG::mgVcycle_down::3 36 3.525e-05 3.525e-05 3.525e-05 0.01% Castro::post_restart() 1 3.306e-05 3.306e-05 3.306e-05 0.01% Castro::initialize_do_advance() 5 3.269e-05 3.269e-05 3.269e-05 0.01% MLMG::mgVcycle_up::4 36 3.255e-05 3.255e-05 3.255e-05 0.01% Castro::swap_state_time_levels() 5 3.155e-05 3.155e-05 3.155e-05 0.01% Castro::construct_new_source() 25 3.053e-05 3.053e-05 3.053e-05 0.01% Castro::initMFs() 1 2.992e-05 2.992e-05 2.992e-05 0.01% MLPoisson::define() 6 2.735e-05 2.735e-05 2.735e-05 0.01% Amr::writeSmallPlotFile() 1 2.628e-05 2.628e-05 2.628e-05 0.01% MLMG::mgVcycle_up::0 36 2.598e-05 2.598e-05 2.598e-05 0.01% MLMG::oneIter() 36 2.591e-05 2.591e-05 2.591e-05 0.01% MLMG::mgVcycle_up::3 36 2.234e-05 2.234e-05 2.234e-05 0.01% MLMG::mgVcycle_up::2 36 2.151e-05 2.151e-05 2.151e-05 0.01% MLLinOp::define() 6 2.091e-05 2.091e-05 2.091e-05 0.01% MLMG::mgVcycle_up::1 36 2.06e-05 2.06e-05 2.06e-05 0.01% MLCellLinOp::solutionResidual() 42 2.032e-05 2.032e-05 2.032e-05 0.01% MLMG::computeResidual() 36 1.943e-05 1.943e-05 1.943e-05 0.01% Castro::finalize_do_advance() 5 1.828e-05 1.828e-05 1.828e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.769e-05 1.769e-05 1.769e-05 0.00% MLMG::mgVcycle_bottom 36 1.685e-05 1.685e-05 1.685e-05 0.00% FillPatchSingleLevel 20 1.512e-05 1.512e-05 1.512e-05 0.00% makeSFC 30 1.354e-05 1.354e-05 1.354e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.302e-05 1.302e-05 1.302e-05 0.00% MultiFab::Add() 36 1.254e-05 1.254e-05 1.254e-05 0.00% Amr::initSubcycle() 1 8.961e-06 8.961e-06 8.961e-06 0.00% Gravity::actual_multilevel_solve() 1 8.254e-06 8.254e-06 8.254e-06 0.00% DistributionMapping::Distribute() 31 8.104e-06 8.104e-06 8.104e-06 0.00% Castro::do_old_sources() 5 8.011e-06 8.011e-06 8.011e-06 0.00% Castro::check_for_nan() 10 7.942e-06 7.942e-06 7.942e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.234e-06 7.234e-06 7.234e-06 0.00% Castro::apply_source_to_state() 10 5.805e-06 5.805e-06 5.805e-06 0.00% Gravity::swapTimeLevels() 5 5.801e-06 5.801e-06 5.801e-06 0.00% MLPoisson::prepareForSolve() 6 4.894e-06 4.894e-06 4.894e-06 0.00% Castro::retry_advance_ctu() 5 4.022e-06 4.022e-06 4.022e-06 0.00% MLMG::computeMLResidual() 6 4.009e-06 4.009e-06 4.009e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.155e-06 3.155e-06 3.155e-06 0.00% MLMG::getGradSolution() 6 3.068e-06 3.068e-06 3.068e-06 0.00% MultiFab::Copy() 6 2.98e-06 2.98e-06 2.98e-06 0.00% Gravity::set_mass_offset() 6 2.85e-06 2.85e-06 2.85e-06 0.00% Castro::FluxRegCrseInit 5 2.702e-06 2.702e-06 2.702e-06 0.00% Castro::FluxRegFineAdd() 5 2.156e-06 2.156e-06 2.156e-06 0.00% MLMG::MLResNormInf() 6 2.144e-06 2.144e-06 2.144e-06 0.00% Amr::init() 1 1.171e-06 1.171e-06 1.171e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.123e-06 1.123e-06 1.123e-06 0.00% AmrLevel::AmrLevel() 1 1.001e-06 1.001e-06 1.001e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3868 0.3868 0.3868 99.99% Amr::coarseTimeStep() 5 0.303 0.303 0.303 78.33% Amr::timeStep() 5 0.3011 0.3011 0.3011 77.82% Castro::advance() 5 0.2951 0.2951 0.2951 76.29% Castro::subcycle_advance_ctu() 5 0.287 0.287 0.287 74.20% Castro::do_advance_ctu() 5 0.2856 0.2856 0.2856 73.82% Castro::construct_new_gravity() 5 0.1476 0.1476 0.1476 38.15% Gravity::solve_phi_with_mlmg() 6 0.1433 0.1433 0.1433 37.04% Gravity::solve_for_phi() 5 0.1397 0.1397 0.1397 36.11% Gravity::actual_solve_with_mlmg() 6 0.1383 0.1383 0.1383 35.76% MLMG::solve() 6 0.1258 0.1258 0.1258 32.51% MLMG::oneIter() 36 0.1184 0.1184 0.1184 30.62% MLMG::mgVcycle() 36 0.1177 0.1177 0.1177 30.42% Castro::construct_ctu_hydro_source() 5 0.09416 0.09416 0.09416 24.34% MLCellLinOp::smooth() 720 0.05973 0.05973 0.05973 15.44% Amr::init() 1 0.04979 0.04979 0.04979 12.87% Amr::restart() 1 0.04978 0.04978 0.04978 12.87% MLCellLinOp::applyBC() 1946 0.04261 0.04261 0.04261 11.02% AmrLevel::restart() 1 0.04188 0.04188 0.04188 10.83% StateData::restartDoit() 4 0.0418 0.0418 0.0418 10.80% VisMF::Read() 3 0.04166 0.04166 0.04166 10.77% MLMG::mgVcycle_bottom 36 0.03633 0.03633 0.03633 9.39% MLMG::actualBottomSolve() 36 0.03632 0.03632 0.03632 9.39% MLCGSolver::bicgstab 36 0.03596 0.03596 0.03596 9.29% Amr::writePlotFile() 1 0.03328 0.03328 0.03328 8.60% VisMF::Write(FabArray) 1 0.03159 0.03159 0.03159 8.17% Castro::clean_state() 30 0.0279 0.0279 0.0279 7.21% MLPoisson::Fsmooth() 1440 0.02769 0.02769 0.02769 7.16% FillPatchIterator::Initialize 20 0.02022 0.02022 0.02022 5.23% FillPatchSingleLevel 20 0.01943 0.01943 0.01943 5.02% StateDataPhysBCFunct::() 20 0.01741 0.01741 0.01741 4.50% MLCellLinOp::apply() 500 0.01624 0.01624 0.01624 4.20% MLMG::mgVcycle_down::0 36 0.01567 0.01567 0.01567 4.05% MLMG::mgVcycle_up::0 36 0.01341 0.01341 0.01341 3.47% StateData::FillBoundary(geom) 160 0.01205 0.01205 0.01205 3.12% Castro::computeTemp() 30 0.01089 0.01089 0.01089 2.81% MLPoisson::define() 6 0.01022 0.01022 0.01022 2.64% amrex::Dot() 484 0.009904 0.009904 0.009904 2.56% Castro::initialize_do_advance() 5 0.009728 0.009728 0.009728 2.51% MLCellLinOp::correctionResidual() 216 0.009461 0.009461 0.009461 2.45% Castro::normalize_species() 30 0.009226 0.009226 0.009226 2.38% Castro::do_new_sources() 5 0.00891 0.00891 0.00891 2.30% MLMG:computeResOfCorrection() 180 0.008156 0.008156 0.008156 2.11% MLMG::mgVcycle_down::1 36 0.007806 0.007806 0.007806 2.02% Castro::construct_old_gravity() 5 0.007687 0.007687 0.007687 1.99% Gravity::get_new_grav_vector() 5 0.007679 0.007679 0.007679 1.99% Castro::initialize_advance() 5 0.007666 0.007666 0.007666 1.98% Gravity::get_old_grav_vector() 5 0.007645 0.007645 0.007645 1.98% MLMG::mgVcycle_down::2 36 0.007598 0.007598 0.007598 1.96% amrex::Copy() 463 0.007526 0.007526 0.007526 1.95% Castro::enforce_min_density() 30 0.007247 0.007247 0.007247 1.87% MLMG::mgVcycle_down::3 36 0.00719 0.00719 0.00719 1.86% FabArray::FillBoundary() 1766 0.007071 0.007071 0.007071 1.83% MLCellLinOp::defineAuxData() 6 0.007061 0.007061 0.007061 1.83% FabArray::setVal() 537 0.006932 0.006932 0.006932 1.79% MLMG::mgVcycle_down::4 36 0.006862 0.006862 0.006862 1.77% FillBoundary_nowait() 1766 0.006721 0.006721 0.006721 1.74% FabArray::ParallelCopy() 380 0.006662 0.006662 0.006662 1.72% FabArray::ParallelCopy_nowait() 380 0.006544 0.006544 0.006544 1.69% CGSolver::sxay() 690 0.006412 0.006412 0.006412 1.66% FabArray::LinComb() 690 0.006248 0.006248 0.006248 1.62% MLCGSolver::ParallelAllReduce 659 0.005928 0.005928 0.005928 1.53% MLMG::mgVcycle_up::2 36 0.005803 0.005803 0.005803 1.50% MLMG::mgVcycle_up::1 36 0.005789 0.005789 0.005789 1.50% Castro::post_timestep() 5 0.005777 0.005777 0.005777 1.49% MLMG::addInterpCorrection() 180 0.005751 0.005751 0.005751 1.49% FabArray::norminf() 278 0.005719 0.005719 0.005719 1.48% Castro::expand_state() 5 0.005606 0.005606 0.005606 1.45% Castro::do_old_sources() 5 0.005589 0.005589 0.005589 1.44% MLMG::mgVcycle_up::4 36 0.00553 0.00553 0.00553 1.43% MLMG::mgVcycle_up::3 36 0.00551 0.00551 0.00551 1.42% amrex::average_down 180 0.005382 0.005382 0.005382 1.39% MLPoisson::Fapply() 500 0.005233 0.005233 0.005233 1.35% Gravity::fill_multipole_BCs() 6 0.004821 0.004821 0.004821 1.25% Castro::post_restart() 1 0.00394 0.00394 0.00394 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003809 0.003809 0.003809 0.98% Gravity::actual_multilevel_solve() 1 0.003791 0.003791 0.003791 0.98% Castro::reset_internal_energy(MultiFab) 30 0.003453 0.003453 0.003453 0.89% MLCellLinOp::solutionResidual() 42 0.0033 0.0033 0.0033 0.85% FabArray::Xpay() 258 0.00295 0.00295 0.00295 0.76% MLMG::prepareForSolve() 6 0.002936 0.002936 0.002936 0.76% MLCellLinOp::defineBC() 6 0.002924 0.002924 0.002924 0.76% BndryData::define() 6 0.002778 0.002778 0.002778 0.72% MLMG::computeResidual() 36 0.002742 0.002742 0.002742 0.71% Castro::estTimeStep() 10 0.002684 0.002684 0.002684 0.69% Castro::computeNewDt() 5 0.001719 0.001719 0.001719 0.44% Castro::construct_new_source() 25 0.001701 0.001701 0.001701 0.44% Castro::construct_new_gravity_source() 5 0.001671 0.001671 0.001671 0.43% Castro::reset_internal_energy(Fab) 240 0.001621 0.001621 0.001621 0.42% Castro::construct_old_source() 25 0.001473 0.001473 0.001473 0.38% Castro::construct_old_gravity_source() 5 0.001437 0.001437 0.001437 0.37% amrex::Add() 72 0.001108 0.001108 0.001108 0.29% MLMG::ResNormInf() 42 0.0009322 0.0009322 0.0009322 0.24% Castro::apply_source_to_state() 10 0.0009239 0.0009239 0.0009239 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009185 0.0009185 0.0009185 0.24% FabArray::Saxpy() 10 0.0009181 0.0009181 0.0009181 0.24% Castro::create_source_corrector() 5 0.0008841 0.0008841 0.0008841 0.23% MLCellLinOp::setLevelBC() 6 0.0008275 0.0008275 0.0008275 0.21% MLMG::getGradSolution() 6 0.0007698 0.0007698 0.0007698 0.20% MLCellLinOp::compGrad() 6 0.0007667 0.0007667 0.0007667 0.20% FabArrayBase::getCPC() 632 0.0007664 0.0007664 0.0007664 0.20% MultiFab::Add() 36 0.0007374 0.0007374 0.0007374 0.19% FabArray::setDomainBndry() 20 0.0006616 0.0006616 0.0006616 0.17% FabArray::mult() 22 0.0006593 0.0006593 0.0006593 0.17% MLPoisson::prepareForSolve() 6 0.0006459 0.0006459 0.0006459 0.17% MLCellLinOp::prepareForSolve() 6 0.000641 0.000641 0.000641 0.17% Castro::check_for_nan() 10 0.0006278 0.0006278 0.0006278 0.16% MultiFab::contains_nan() 10 0.0006199 0.0006199 0.0006199 0.16% MLMG::computeMLResidual() 6 0.0005811 0.0005811 0.0005811 0.15% Gravity::update_max_rhs() 6 0.0005186 0.0005186 0.0005186 0.13% Castro::enforce_speed_limit() 30 0.0005021 0.0005021 0.0005021 0.13% Amr::InitAmr() 1 0.0004264 0.0004264 0.0004264 0.11% FabArrayBase::CPC::define() 244 0.0003997 0.0003997 0.0003997 0.10% Castro::finalize_advance() 5 0.0003501 0.0003501 0.0003501 0.09% FabArrayBase::getFB() 1766 0.0003289 0.0003289 0.0003289 0.09% Gravity::swapTimeLevels() 5 0.0002454 0.0002454 0.0002454 0.06% MLLinOp::define() 6 0.0002054 0.0002054 0.0002054 0.05% MLLinOp::defineGrids() 6 0.0001845 0.0001845 0.0001845 0.05% MultiFab::Copy() 6 0.0001806 0.0001806 0.0001806 0.05% Castro::buildMetrics() 1 0.000162 0.000162 0.000162 0.04% MLMG::MLResNormInf() 6 0.0001512 0.0001512 0.0001512 0.04% MultiFab::max() 6 0.0001435 0.0001435 0.0001435 0.04% MLMG::MLRhsNormInf() 6 0.0001159 0.0001159 0.0001159 0.03% FabArrayBase::FB::FB() 26 5.841e-05 5.841e-05 5.841e-05 0.02% Castro::swap_state_time_levels() 5 3.155e-05 3.155e-05 3.155e-05 0.01% Castro::initMFs() 1 2.992e-05 2.992e-05 2.992e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.765e-05 2.765e-05 2.765e-05 0.01% Amr::writeSmallPlotFile() 1 2.628e-05 2.628e-05 2.628e-05 0.01% makeSFC 30 2.042e-05 2.042e-05 2.042e-05 0.01% Castro::finalize_do_advance() 5 1.828e-05 1.828e-05 1.828e-05 0.00% Amr::initSubcycle() 1 8.961e-06 8.961e-06 8.961e-06 0.00% DistributionMapping::Distribute() 31 8.104e-06 8.104e-06 8.104e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.38e-06 4.38e-06 4.38e-06 0.00% Castro::retry_advance_ctu() 5 4.022e-06 4.022e-06 4.022e-06 0.00% Gravity::set_mass_offset() 6 2.85e-06 2.85e-06 2.85e-06 0.00% Castro::FluxRegCrseInit 5 2.702e-06 2.702e-06 2.702e-06 0.00% Castro::FluxRegFineAdd() 5 2.156e-06 2.156e-06 2.156e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.123e-06 1.123e-06 1.123e-06 0.00% AmrLevel::AmrLevel() 1 1.001e-06 1.001e-06 1.001e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-1-g4a53367b19e1) finalized