Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-18-ga537fb0516d5) initialized Starting run at 08:26:55 UTC on 2022-04-14. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-18-ga537fb051 Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038482186 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.023253556 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.111591723 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048759263 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049065128 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.056865012 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.064391157 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035292714 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.061789931 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051234933 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.055761339 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056866741 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061789602 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035362878 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022128151 seconds Ending run at 08:26:56 UTC on 2022-04-14. Run time = 0.831354074 Run time without initialization = 0.711389403 Average number of zones advanced per microsecond: 3.685 Average number of zones advanced per microsecond per rank: 3.685 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8314 ... 0.8314 ... 0.8314 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2169 0.2169 0.2169 26.09% VisMF::Write(FabArray) 11 0.1474 0.1474 0.1474 17.73% MLCellLinOp::applyBC() 4379 0.09166 0.09166 0.09166 11.03% MLPoisson::Fsmooth() 3240 0.05818 0.05818 0.05818 7.00% FabArray::setVal() 1135 0.02598 0.02598 0.02598 3.13% StateDataPhysBCFunct::() 41 0.0257 0.0257 0.0257 3.09% StateData::FillBoundary(geom) 328 0.02265 0.02265 0.02265 2.72% MLCGSolver::bicgstab 81 0.02127 0.02127 0.02127 2.56% MultiFab::Dot() 1100 0.02082 0.02082 0.02082 2.50% FillBoundary_nowait() 3974 0.01771 0.01771 0.01771 2.13% FabArray::ParallelCopy_nowait() 851 0.01572 0.01572 0.01572 1.89% Castro::computeTemp() 63 0.01312 0.01312 0.01312 1.58% MultiFab::LinComb() 1566 0.0119 0.0119 0.0119 1.43% Gravity::fill_multipole_BCs() 11 0.01155 0.01155 0.01155 1.39% MLPoisson::Fapply() 1128 0.01027 0.01027 0.01027 1.24% MLCellLinOp::defineAuxData() 11 0.0101 0.0101 0.0101 1.21% Castro::enforce_min_density() 62 0.0085 0.0085 0.0085 1.02% MLMG::addInterpCorrection() 405 0.006849 0.006849 0.006849 0.82% amrex::average_down 405 0.006313 0.006313 0.006313 0.76% MultiFab::Xpay() 578 0.005907 0.005907 0.005907 0.71% FabArray::setDomainBndry() 41 0.005398 0.005398 0.005398 0.65% Castro::reset_internal_energy() 63 0.005316 0.005316 0.005316 0.64% Castro::estTimeStep() 21 0.005309 0.005309 0.005309 0.64% Castro::expand_state() 10 0.004792 0.004792 0.004792 0.58% Castro::normalize_species() 62 0.004356 0.004356 0.004356 0.52% Amr::checkPoint() 3 0.003996 0.003996 0.003996 0.48% BndryData::define() 11 0.003614 0.003614 0.003614 0.43% Castro::do_advance_ctu() 10 0.003284 0.003284 0.003284 0.40% Amr::writePlotFile() 2 0.003179 0.003179 0.003179 0.38% Castro::construct_new_gravity_source() 10 0.003053 0.003053 0.003053 0.37% Castro::enforce_speed_limit() 62 0.002661 0.002661 0.002661 0.32% Gravity::get_new_grav_vector() 11 0.002661 0.002661 0.002661 0.32% Castro::construct_old_gravity_source() 10 0.002552 0.002552 0.002552 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002246 0.002246 0.002246 0.27% MLCellLinOp::compGrad() 11 0.001988 0.001988 0.001988 0.24% MLMG::ResNormInf() 92 0.001907 0.001907 0.001907 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% Gravity::get_old_grav_vector() 10 0.001748 0.001748 0.001748 0.21% MLMG::oneIter() 81 0.001668 0.001668 0.001668 0.20% MLCellLinOp::setLevelBC() 11 0.00151 0.00151 0.00151 0.18% Gravity::actual_solve_with_mlmg() 11 0.001387 0.001387 0.001387 0.17% MLCellLinOp::prepareForSolve() 11 0.001315 0.001315 0.001315 0.16% FabArray::mult() 43 0.001314 0.001314 0.001314 0.16% MLCellLinOp::smooth() 1620 0.001276 0.001276 0.001276 0.15% Castro::initData() 1 0.00119 0.00119 0.00119 0.14% FabArray::FillBoundary() 3974 0.0008554 0.0008554 0.0008554 0.10% FabArrayBase::getCPC() 1313 0.0008182 0.0008182 0.0008182 0.10% MLMG::prepareForSolve() 11 0.000764 0.000764 0.000764 0.09% FabArrayBase::CPC::define() 454 0.0007028 0.0007028 0.0007028 0.08% FabArrayBase::getFB() 3974 0.0006807 0.0006807 0.0006807 0.08% Gravity::update_max_rhs() 11 0.0005486 0.0005486 0.0005486 0.07% MLCellLinOp::apply() 1128 0.00054 0.00054 0.00054 0.06% MultiFab::Copy() 11 0.0005127 0.0005127 0.0005127 0.06% Amr::InitAmr() 1 0.0004724 0.0004724 0.0004724 0.06% CGSolver::sxay() 1566 0.0004645 0.0004645 0.0004645 0.06% Gravity::solve_for_phi() 10 0.0004427 0.0004427 0.0004427 0.05% MLMG::mgVcycle() 81 0.0004034 0.0004034 0.0004034 0.05% MultiFab::min() 10 0.0003212 0.0003212 0.0003212 0.04% MLCGSolver::ParallelAllReduce 1495 0.0003108 0.0003108 0.0003108 0.04% FabArray::ParallelCopy() 851 0.0003029 0.0003029 0.0003029 0.04% main() 1 0.0002989 0.0002989 0.0002989 0.04% Castro::construct_new_gravity() 10 0.000259 0.000259 0.000259 0.03% FillPatchIterator::Initialize 41 0.0002514 0.0002514 0.0002514 0.03% MultiFab::max() 11 0.000251 0.000251 0.000251 0.03% MLCellLinOp::correctionResidual() 486 0.0002359 0.0002359 0.0002359 0.03% Gravity::actual_multilevel_solve() 1 0.0002254 0.0002254 0.0002254 0.03% Amr::coarseTimeStep() 10 0.0002107 0.0002107 0.0002107 0.03% Amr::timeStep() 10 0.0001957 0.0001957 0.0001957 0.02% MLMG::MLRhsNormInf() 11 0.0001945 0.0001945 0.0001945 0.02% MLCellLinOp::defineBC() 11 0.0001831 0.0001831 0.0001831 0.02% MLLinOp::defineGrids() 11 0.0001754 0.0001754 0.0001754 0.02% Castro::subcycle_advance_ctu() 10 0.0001509 0.0001509 0.0001509 0.02% Amr::defBaseLevel() 1 0.0001487 0.0001487 0.0001487 0.02% StateData::checkPoint() 12 0.0001311 0.0001311 0.0001311 0.02% MLMG:computeResOfCorrection() 405 0.0001289 0.0001289 0.0001289 0.02% MLMG::actualBottomSolve() 81 0.0001095 0.0001095 0.0001095 0.01% MLMG::mgVcycle_down::0 81 9.629e-05 9.629e-05 9.629e-05 0.01% Castro::advance() 10 9.47e-05 9.47e-05 9.47e-05 0.01% FabArrayBase::FB::FB() 56 8.845e-05 8.845e-05 8.845e-05 0.01% MLMG::mgVcycle_down::1 81 8.528e-05 8.528e-05 8.528e-05 0.01% MLMG::mgVcycle_down::2 81 8.012e-05 8.012e-05 8.012e-05 0.01% MLMG::solve() 11 7.892e-05 7.892e-05 7.892e-05 0.01% Castro::initialize_advance() 10 7.834e-05 7.834e-05 7.834e-05 0.01% MLMG::mgVcycle_down::4 81 7.596e-05 7.596e-05 7.596e-05 0.01% MLMG::mgVcycle_down::3 81 7.561e-05 7.561e-05 7.561e-05 0.01% Castro::clean_state() 62 7.486e-05 7.486e-05 7.486e-05 0.01% AmrLevel::checkPoint() 3 7.207e-05 7.207e-05 7.207e-05 0.01% MLMG::mgVcycle_up::4 81 6.77e-05 6.77e-05 6.77e-05 0.01% Castro::finalize_advance() 10 5.83e-05 5.83e-05 5.83e-05 0.01% Castro::initialize_do_advance() 10 5.696e-05 5.696e-05 5.696e-05 0.01% MLMG::mgVcycle_up::0 81 5.645e-05 5.645e-05 5.645e-05 0.01% MLMG::mgVcycle_up::3 81 5.379e-05 5.379e-05 5.379e-05 0.01% MLMG::mgVcycle_up::1 81 5.311e-05 5.311e-05 5.311e-05 0.01% MLMG::mgVcycle_up::2 81 5.119e-05 5.119e-05 5.119e-05 0.01% MLCellLinOp::solutionResidual() 92 5.115e-05 5.115e-05 5.115e-05 0.01% StateData::define() 4 4.47e-05 4.47e-05 4.47e-05 0.01% Gravity::solve_phi_with_mlmg() 11 4.08e-05 4.08e-05 4.08e-05 0.00% Castro::construct_new_source() 50 3.893e-05 3.893e-05 3.893e-05 0.00% Castro::finalize_do_advance() 10 3.721e-05 3.721e-05 3.721e-05 0.00% Castro::swap_state_time_levels() 10 3.71e-05 3.71e-05 3.71e-05 0.00% MLMG::mgVcycle_bottom 81 3.401e-05 3.401e-05 3.401e-05 0.00% Castro::enforce_consistent_e() 1 3.399e-05 3.399e-05 3.399e-05 0.00% MLMG::computeResidual() 81 3.28e-05 3.28e-05 3.28e-05 0.00% Castro::post_timestep() 10 2.947e-05 2.947e-05 2.947e-05 0.00% FillPatchSingleLevel 41 2.88e-05 2.88e-05 2.88e-05 0.00% MLLinOp::define() 11 2.635e-05 2.635e-05 2.635e-05 0.00% makeSFC 55 2.545e-05 2.545e-05 2.545e-05 0.00% Amr::writeSmallPlotFile() 1 2.524e-05 2.524e-05 2.524e-05 0.00% MLPoisson::define() 11 2.329e-05 2.329e-05 2.329e-05 0.00% Amr::FinalizeInit() 1 2.173e-05 2.173e-05 2.173e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 2.123e-05 2.123e-05 2.123e-05 0.00% Castro::do_new_sources() 10 1.694e-05 1.694e-05 1.694e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.693e-05 1.693e-05 1.693e-05 0.00% Castro::construct_old_source() 50 1.628e-05 1.628e-05 1.628e-05 0.00% Castro::do_old_sources() 10 1.541e-05 1.541e-05 1.541e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.514e-05 1.514e-05 1.514e-05 0.00% DistributionMapping::Distribute() 56 1.485e-05 1.485e-05 1.485e-05 0.00% Castro::apply_source_to_state() 20 1.145e-05 1.145e-05 1.145e-05 0.00% Castro::construct_old_gravity() 10 1.067e-05 1.067e-05 1.067e-05 0.00% Castro::create_source_corrector() 10 1.022e-05 1.022e-05 1.022e-05 0.00% Amr::initSubcycle() 1 1.009e-05 1.009e-05 1.009e-05 0.00% Gravity::swapTimeLevels() 10 8.9e-06 8.9e-06 8.9e-06 0.00% MLPoisson::prepareForSolve() 11 8.563e-06 8.563e-06 8.563e-06 0.00% Castro::retry_advance_ctu() 10 7.506e-06 7.506e-06 7.506e-06 0.00% Amr::InitializeInit() 1 6.752e-06 6.752e-06 6.752e-06 0.00% MLMG::computeMLResidual() 11 6.496e-06 6.496e-06 6.496e-06 0.00% MLMG::buildFineMask() 11 6.142e-06 6.142e-06 6.142e-06 0.00% Castro::computeNewDt() 9 6.128e-06 6.128e-06 6.128e-06 0.00% MLMG::getGradSolution() 11 6.001e-06 6.001e-06 6.001e-06 0.00% MLMG::MLResNormInf() 11 4.556e-06 4.556e-06 4.556e-06 0.00% Gravity::set_mass_offset() 11 4.424e-06 4.424e-06 4.424e-06 0.00% AmrLevel::checkPointPost() 3 4.385e-06 4.385e-06 4.385e-06 0.00% Castro::post_init() 1 4.224e-06 4.224e-06 4.224e-06 0.00% Castro::FluxRegFineAdd() 10 3.248e-06 3.248e-06 3.248e-06 0.00% Castro::computeInitialDt() 2 2.906e-06 2.906e-06 2.906e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.795e-06 2.795e-06 2.795e-06 0.00% Amr::init() 1 2.393e-06 2.393e-06 2.393e-06 0.00% AmrLevel::checkPointPre() 3 2.348e-06 2.348e-06 2.348e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.041e-06 2.041e-06 2.041e-06 0.00% Amr::initialInit() 1 1.146e-06 1.146e-06 1.146e-06 0.00% Castro::post_regrid() 1 1.02e-06 1.02e-06 1.02e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8314 0.8314 0.8314 100.00% Amr::coarseTimeStep() 10 0.689 0.689 0.689 82.88% Amr::timeStep() 10 0.6153 0.6153 0.6153 74.01% Castro::advance() 10 0.6095 0.6095 0.6095 73.32% Castro::subcycle_advance_ctu() 10 0.5949 0.5949 0.5949 71.55% Castro::do_advance_ctu() 10 0.5947 0.5947 0.5947 71.53% Gravity::solve_phi_with_mlmg() 11 0.3187 0.3187 0.3187 38.33% Gravity::actual_solve_with_mlmg() 11 0.3069 0.3069 0.3069 36.91% Castro::construct_new_gravity() 10 0.2945 0.2945 0.2945 35.43% MLMG::solve() 11 0.2843 0.2843 0.2843 34.20% Gravity::solve_for_phi() 10 0.2746 0.2746 0.2746 33.02% MLMG::oneIter() 81 0.2686 0.2686 0.2686 32.30% MLMG::mgVcycle() 81 0.2669 0.2669 0.2669 32.10% Castro::construct_ctu_hydro_source() 10 0.2169 0.2169 0.2169 26.09% VisMF::Write(FabArray) 11 0.1474 0.1474 0.1474 17.73% MLCellLinOp::smooth() 1620 0.1419 0.1419 0.1419 17.07% Amr::init() 1 0.1193 0.1193 0.1193 14.36% MLCellLinOp::applyBC() 4379 0.111 0.111 0.111 13.35% Amr::checkPoint() 3 0.1093 0.1093 0.1093 13.14% AmrLevel::checkPoint() 3 0.1053 0.1053 0.1053 12.66% StateData::checkPoint() 12 0.1052 0.1052 0.1052 12.65% MLMG::mgVcycle_bottom 81 0.07557 0.07557 0.07557 9.09% MLMG::actualBottomSolve() 81 0.07554 0.07554 0.07554 9.09% MLCGSolver::bicgstab 81 0.07482 0.07482 0.07482 9.00% FillPatchIterator::Initialize 41 0.05825 0.05825 0.05825 7.01% MLPoisson::Fsmooth() 3240 0.05818 0.05818 0.05818 7.00% Amr::initialInit() 1 0.0575 0.0575 0.0575 6.92% FillPatchSingleLevel 41 0.0526 0.0526 0.0526 6.33% StateDataPhysBCFunct::() 41 0.04835 0.04835 0.04835 5.82% Amr::FinalizeInit() 1 0.04827 0.04827 0.04827 5.81% Castro::post_init() 1 0.04752 0.04752 0.04752 5.72% Amr::writePlotFile() 2 0.04549 0.04549 0.04549 5.47% Gravity::multilevel_solve_for_new_phi() 1 0.0448 0.0448 0.0448 5.39% Gravity::actual_multilevel_solve() 1 0.04478 0.04478 0.04478 5.39% MLCellLinOp::apply() 1128 0.03879 0.03879 0.03879 4.67% MLMG::mgVcycle_down::0 81 0.03786 0.03786 0.03786 4.55% Castro::clean_state() 62 0.03328 0.03328 0.03328 4.00% MLMG::mgVcycle_up::0 81 0.03241 0.03241 0.03241 3.90% Castro::initialize_do_advance() 10 0.03004 0.03004 0.03004 3.61% FabArray::setVal() 1135 0.02598 0.02598 0.02598 3.13% Castro::expand_state() 10 0.02483 0.02483 0.02483 2.99% StateData::FillBoundary(geom) 328 0.02265 0.02265 0.02265 2.72% Gravity::get_new_grav_vector() 11 0.02228 0.02228 0.02228 2.68% MLCellLinOp::correctionResidual() 486 0.02183 0.02183 0.02183 2.63% MultiFab::Dot() 1100 0.02082 0.02082 0.02082 2.50% FabArray::FillBoundary() 3974 0.01934 0.01934 0.01934 2.33% MLMG:computeResOfCorrection() 405 0.01892 0.01892 0.01892 2.28% FillBoundary_nowait() 3974 0.01848 0.01848 0.01848 2.22% Castro::computeTemp() 63 0.01844 0.01844 0.01844 2.22% MLMG::mgVcycle_down::1 81 0.01807 0.01807 0.01807 2.17% MLMG::mgVcycle_down::2 81 0.01743 0.01743 0.01743 2.10% MLPoisson::define() 11 0.01708 0.01708 0.01708 2.05% FabArray::ParallelCopy() 851 0.01689 0.01689 0.01689 2.03% MLMG::mgVcycle_down::3 81 0.01666 0.01666 0.01666 2.00% FabArray::ParallelCopy_nowait() 851 0.01659 0.01659 0.01659 2.00% Castro::construct_old_gravity() 10 0.016 0.016 0.016 1.92% Gravity::get_old_grav_vector() 10 0.01599 0.01599 0.01599 1.92% MLMG::mgVcycle_down::4 81 0.01584 0.01584 0.01584 1.91% Castro::initialize_advance() 10 0.0145 0.0145 0.0145 1.74% MLMG::mgVcycle_up::2 81 0.01357 0.01357 0.01357 1.63% MLMG::mgVcycle_up::1 81 0.01335 0.01335 0.01335 1.61% MLMG::addInterpCorrection() 405 0.01318 0.01318 0.01318 1.59% MLMG::mgVcycle_up::3 81 0.01294 0.01294 0.01294 1.56% MLMG::mgVcycle_up::4 81 0.01279 0.01279 0.01279 1.54% amrex::average_down 405 0.01265 0.01265 0.01265 1.52% MLCGSolver::ParallelAllReduce 1495 0.01256 0.01256 0.01256 1.51% CGSolver::sxay() 1566 0.01237 0.01237 0.01237 1.49% MultiFab::LinComb() 1566 0.0119 0.0119 0.0119 1.43% MLCellLinOp::defineAuxData() 11 0.01175 0.01175 0.01175 1.41% Gravity::fill_multipole_BCs() 11 0.01155 0.01155 0.01155 1.39% Castro::do_new_sources() 10 0.0115 0.0115 0.0115 1.38% MLPoisson::Fapply() 1128 0.01027 0.01027 0.01027 1.24% Amr::InitializeInit() 1 0.009224 0.009224 0.009224 1.11% Amr::defBaseLevel() 1 0.009217 0.009217 0.009217 1.11% Castro::enforce_min_density() 62 0.0085 0.0085 0.0085 1.02% MLCellLinOp::solutionResidual() 92 0.007789 0.007789 0.007789 0.94% Castro::do_old_sources() 10 0.007701 0.007701 0.007701 0.93% MLMG::computeResidual() 81 0.006713 0.006713 0.006713 0.81% MultiFab::Xpay() 578 0.005907 0.005907 0.005907 0.71% MLMG::prepareForSolve() 11 0.005749 0.005749 0.005749 0.69% Castro::post_timestep() 10 0.005598 0.005598 0.005598 0.67% FabArray::setDomainBndry() 41 0.005398 0.005398 0.005398 0.65% Castro::reset_internal_energy() 63 0.005316 0.005316 0.005316 0.64% Castro::estTimeStep() 21 0.005309 0.005309 0.005309 0.64% MLCellLinOp::defineBC() 11 0.005045 0.005045 0.005045 0.61% BndryData::define() 11 0.004862 0.004862 0.004862 0.58% Castro::normalize_species() 62 0.004356 0.004356 0.004356 0.52% Castro::initData() 1 0.003454 0.003454 0.003454 0.42% Castro::construct_new_source() 50 0.003091 0.003091 0.003091 0.37% Castro::construct_new_gravity_source() 10 0.003053 0.003053 0.003053 0.37% Castro::enforce_speed_limit() 62 0.002661 0.002661 0.002661 0.32% Castro::construct_old_source() 50 0.002569 0.002569 0.002569 0.31% MLMG::getGradSolution() 11 0.002565 0.002565 0.002565 0.31% MLCellLinOp::compGrad() 11 0.002559 0.002559 0.002559 0.31% Castro::construct_old_gravity_source() 10 0.002552 0.002552 0.002552 0.31% Castro::computeNewDt() 9 0.002282 0.002282 0.002282 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002246 0.002246 0.002246 0.27% MLMG::ResNormInf() 92 0.001907 0.001907 0.001907 0.23% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.22% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% FabArrayBase::getCPC() 1313 0.001521 0.001521 0.001521 0.18% MLCellLinOp::setLevelBC() 11 0.00151 0.00151 0.00151 0.18% MLPoisson::prepareForSolve() 11 0.001323 0.001323 0.001323 0.16% MLCellLinOp::prepareForSolve() 11 0.001315 0.001315 0.001315 0.16% FabArray::mult() 43 0.001314 0.001314 0.001314 0.16% MLMG::computeMLResidual() 11 0.001116 0.001116 0.001116 0.13% Gravity::swapTimeLevels() 10 0.00108 0.00108 0.00108 0.13% Gravity::update_max_rhs() 11 0.0009398 0.0009398 0.0009398 0.11% FabArrayBase::getFB() 3974 0.0007692 0.0007692 0.0007692 0.09% FabArrayBase::CPC::define() 454 0.0007028 0.0007028 0.0007028 0.08% Castro::computeInitialDt() 2 0.0006909 0.0006909 0.0006909 0.08% Castro::post_regrid() 1 0.0005173 0.0005173 0.0005173 0.06% MultiFab::Copy() 11 0.0005127 0.0005127 0.0005127 0.06% Amr::InitAmr() 1 0.0004825 0.0004825 0.0004825 0.06% MultiFab::min() 10 0.0003212 0.0003212 0.0003212 0.04% MLLinOp::define() 11 0.000264 0.000264 0.000264 0.03% MultiFab::max() 11 0.000251 0.000251 0.000251 0.03% MLMG::MLResNormInf() 11 0.0002491 0.0002491 0.0002491 0.03% MLLinOp::defineGrids() 11 0.0002376 0.0002376 0.0002376 0.03% MLMG::MLRhsNormInf() 11 0.0001945 0.0001945 0.0001945 0.02% FabArrayBase::FB::FB() 56 8.845e-05 8.845e-05 8.845e-05 0.01% Castro::finalize_advance() 10 6.155e-05 6.155e-05 6.155e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.015e-05 6.015e-05 6.015e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.984e-05 5.984e-05 5.984e-05 0.01% StateData::define() 4 4.47e-05 4.47e-05 4.47e-05 0.01% makeSFC 55 3.892e-05 3.892e-05 3.892e-05 0.00% Castro::finalize_do_advance() 10 3.721e-05 3.721e-05 3.721e-05 0.00% Castro::swap_state_time_levels() 10 3.71e-05 3.71e-05 3.71e-05 0.00% Castro::enforce_consistent_e() 1 3.399e-05 3.399e-05 3.399e-05 0.00% Amr::writeSmallPlotFile() 1 2.524e-05 2.524e-05 2.524e-05 0.00% DistributionMapping::Distribute() 56 1.485e-05 1.485e-05 1.485e-05 0.00% Castro::create_source_corrector() 10 1.022e-05 1.022e-05 1.022e-05 0.00% Amr::initSubcycle() 1 1.009e-05 1.009e-05 1.009e-05 0.00% Castro::retry_advance_ctu() 10 7.506e-06 7.506e-06 7.506e-06 0.00% MLMG::buildFineMask() 11 6.142e-06 6.142e-06 6.142e-06 0.00% Gravity::set_mass_offset() 11 4.424e-06 4.424e-06 4.424e-06 0.00% AmrLevel::checkPointPost() 3 4.385e-06 4.385e-06 4.385e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.175e-06 4.175e-06 4.175e-06 0.00% Castro::FluxRegFineAdd() 10 3.248e-06 3.248e-06 3.248e-06 0.00% AmrLevel::checkPointPre() 3 2.348e-06 2.348e-06 2.348e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.041e-06 2.041e-06 2.041e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-18-ga537fb0516d5) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-18-ga537fb0516d5) initialized Starting run at 08:26:56 UTC on 2022-04-14. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-18-ga537fb051 Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.485864989 Restart time = 0.055392291 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.113178793 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051717346 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056532285 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056941745 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061133363 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023796964 seconds Ending run at 08:26:57 UTC on 2022-04-14. Run time = 0.419630265 Run time without initialization = 0.363673058 Average number of zones advanced per microsecond: 3.604 Average number of zones advanced per microsecond per rank: 3.604 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4197 ... 0.4197 ... 0.4197 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1349 0.1349 0.1349 32.14% VisMF::Read() 3 0.04378 0.04378 0.04378 10.43% MLCellLinOp::applyBC() 1946 0.03779 0.03779 0.03779 9.01% MLPoisson::Fsmooth() 1440 0.02566 0.02566 0.02566 6.11% VisMF::Write(FabArray) 1 0.02245 0.02245 0.02245 5.35% FabArray::setVal() 537 0.01627 0.01627 0.01627 3.88% StateData::FillBoundary(geom) 160 0.01161 0.01161 0.01161 2.77% MLCGSolver::bicgstab 36 0.009437 0.009437 0.009437 2.25% MultiFab::Dot() 484 0.009178 0.009178 0.009178 2.19% FillBoundary_nowait() 1766 0.007217 0.007217 0.007217 1.72% FabArray::ParallelCopy_nowait() 380 0.007085 0.007085 0.007085 1.69% StateDataPhysBCFunct::() 20 0.006942 0.006942 0.006942 1.65% Gravity::fill_multipole_BCs() 6 0.00606 0.00606 0.00606 1.44% Castro::computeTemp() 30 0.005553 0.005553 0.005553 1.32% MLCellLinOp::defineAuxData() 6 0.005532 0.005532 0.005532 1.32% MultiFab::LinComb() 690 0.005348 0.005348 0.005348 1.27% MLPoisson::Fapply() 500 0.004636 0.004636 0.004636 1.10% Castro::expand_state() 5 0.004064 0.004064 0.004064 0.97% FabArray::setDomainBndry() 20 0.003802 0.003802 0.003802 0.91% Castro::enforce_min_density() 30 0.003659 0.003659 0.003659 0.87% Amr::restart() 1 0.003037 0.003037 0.003037 0.72% Castro::estTimeStep() 10 0.003013 0.003013 0.003013 0.72% MLMG::addInterpCorrection() 180 0.003002 0.003002 0.003002 0.72% amrex::average_down 180 0.002796 0.002796 0.002796 0.67% Castro::do_advance_ctu() 5 0.002722 0.002722 0.002722 0.65% MultiFab::Xpay() 258 0.002661 0.002661 0.002661 0.63% Castro::reset_internal_energy() 30 0.002467 0.002467 0.002467 0.59% Castro::normalize_species() 30 0.002215 0.002215 0.002215 0.53% BndryData::define() 6 0.002006 0.002006 0.002006 0.48% Gravity::get_new_grav_vector() 5 0.001765 0.001765 0.001765 0.42% Castro::construct_new_gravity_source() 5 0.001545 0.001545 0.001545 0.37% MLCellLinOp::compGrad() 6 0.001438 0.001438 0.001438 0.34% Amr::writePlotFile() 1 0.001437 0.001437 0.001437 0.34% Castro::construct_old_gravity_source() 5 0.001317 0.001317 0.001317 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001207 0.001207 0.001207 0.29% MultiFab::Saxpy() 10 0.0009194 0.0009194 0.0009194 0.22% Gravity::get_old_grav_vector() 5 0.0009177 0.0009177 0.0009177 0.22% MLMG::ResNormInf() 42 0.0008602 0.0008602 0.0008602 0.20% MLCellLinOp::setLevelBC() 6 0.0008384 0.0008384 0.0008384 0.20% Castro::post_timestep() 5 0.0008315 0.0008315 0.0008315 0.20% MLMG::oneIter() 36 0.0007513 0.0007513 0.0007513 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007342 0.0007342 0.0007342 0.17% MLCellLinOp::prepareForSolve() 6 0.0006951 0.0006951 0.0006951 0.17% FabArray::mult() 22 0.0006528 0.0006528 0.0006528 0.16% Castro::enforce_speed_limit() 30 0.0005257 0.0005257 0.0005257 0.13% Gravity::update_max_rhs() 6 0.0005177 0.0005177 0.0005177 0.12% MLCellLinOp::smooth() 720 0.0005091 0.0005091 0.0005091 0.12% FabArrayBase::getCPC() 632 0.0004322 0.0004322 0.0004322 0.10% MLMG::prepareForSolve() 6 0.0004317 0.0004317 0.0004317 0.10% FabArrayBase::CPC::define() 244 0.0004221 0.0004221 0.0004221 0.10% FabArray::FillBoundary() 1766 0.0004094 0.0004094 0.0004094 0.10% MultiFab::Copy() 6 0.0003778 0.0003778 0.0003778 0.09% Amr::InitAmr() 1 0.0003676 0.0003676 0.0003676 0.09% FabArrayBase::getFB() 1766 0.0003103 0.0003103 0.0003103 0.07% main() 1 0.0002846 0.0002846 0.0002846 0.07% CGSolver::sxay() 690 0.0002226 0.0002226 0.0002226 0.05% Gravity::solve_for_phi() 5 0.0002223 0.0002223 0.0002223 0.05% MLCellLinOp::apply() 500 0.0002197 0.0002197 0.0002197 0.05% Gravity::actual_multilevel_solve() 1 0.0002189 0.0002189 0.0002189 0.05% Castro::construct_new_gravity() 5 0.0002103 0.0002103 0.0002103 0.05% MLMG::mgVcycle() 36 0.0001853 0.0001853 0.0001853 0.04% MultiFab::min() 5 0.0001688 0.0001688 0.0001688 0.04% Castro::construct_new_source() 25 0.000166 0.000166 0.000166 0.04% FabArray::ParallelCopy() 380 0.0001484 0.0001484 0.0001484 0.04% MLCGSolver::ParallelAllReduce 659 0.0001441 0.0001441 0.0001441 0.03% MultiFab::max() 6 0.0001339 0.0001339 0.0001339 0.03% Amr::coarseTimeStep() 5 0.0001333 0.0001333 0.0001333 0.03% FillPatchIterator::Initialize 20 0.000125 0.000125 0.000125 0.03% Amr::timeStep() 5 0.0001175 0.0001175 0.0001175 0.03% MLMG::MLRhsNormInf() 6 0.0001046 0.0001046 0.0001046 0.02% MLCellLinOp::defineBC() 6 0.0001043 0.0001043 0.0001043 0.02% MLLinOp::defineGrids() 6 0.0001041 0.0001041 0.0001041 0.02% MLCellLinOp::correctionResidual() 216 9.581e-05 9.581e-05 9.581e-05 0.02% AmrLevel::restart() 1 7.566e-05 7.566e-05 7.566e-05 0.02% StateData::restartDoit() 4 7.045e-05 7.045e-05 7.045e-05 0.02% Castro::subcycle_advance_ctu() 5 6.844e-05 6.844e-05 6.844e-05 0.02% Castro::initialize_do_advance() 5 6.705e-05 6.705e-05 6.705e-05 0.02% MLMG:computeResOfCorrection() 180 6.361e-05 6.361e-05 6.361e-05 0.02% FabArrayBase::FB::FB() 26 6.003e-05 6.003e-05 6.003e-05 0.01% Castro::create_source_corrector() 5 5.893e-05 5.893e-05 5.893e-05 0.01% Castro::advance() 5 5.165e-05 5.165e-05 5.165e-05 0.01% MLMG::actualBottomSolve() 36 4.844e-05 4.844e-05 4.844e-05 0.01% MLMG::mgVcycle_down::0 36 4.147e-05 4.147e-05 4.147e-05 0.01% MLMG::mgVcycle_down::1 36 4.126e-05 4.126e-05 4.126e-05 0.01% Castro::do_new_sources() 5 3.795e-05 3.795e-05 3.795e-05 0.01% MLMG::mgVcycle_down::2 36 3.776e-05 3.776e-05 3.776e-05 0.01% Castro::initialize_advance() 5 3.729e-05 3.729e-05 3.729e-05 0.01% MLMG::solve() 6 3.683e-05 3.683e-05 3.683e-05 0.01% MLMG::mgVcycle_down::4 36 3.634e-05 3.634e-05 3.634e-05 0.01% MLMG::mgVcycle_down::3 36 3.605e-05 3.605e-05 3.605e-05 0.01% Castro::clean_state() 30 3.59e-05 3.59e-05 3.59e-05 0.01% Castro::construct_old_source() 25 2.987e-05 2.987e-05 2.987e-05 0.01% MLMG::mgVcycle_up::4 36 2.892e-05 2.892e-05 2.892e-05 0.01% Castro::finalize_advance() 5 2.869e-05 2.869e-05 2.869e-05 0.01% MLLinOp::define() 6 2.815e-05 2.815e-05 2.815e-05 0.01% Amr::writeSmallPlotFile() 1 2.666e-05 2.666e-05 2.666e-05 0.01% Castro::swap_state_time_levels() 5 2.577e-05 2.577e-05 2.577e-05 0.01% Castro::post_restart() 1 2.516e-05 2.516e-05 2.516e-05 0.01% MLMG::mgVcycle_up::0 36 2.411e-05 2.411e-05 2.411e-05 0.01% MLMG::mgVcycle_up::3 36 2.363e-05 2.363e-05 2.363e-05 0.01% MLMG::mgVcycle_up::2 36 2.355e-05 2.355e-05 2.355e-05 0.01% MLCellLinOp::solutionResidual() 42 2.318e-05 2.318e-05 2.318e-05 0.01% Castro::construct_old_gravity() 5 2.307e-05 2.307e-05 2.307e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.271e-05 2.271e-05 2.271e-05 0.01% MLMG::mgVcycle_up::1 36 2.214e-05 2.214e-05 2.214e-05 0.01% Castro::finalize_do_advance() 5 2.194e-05 2.194e-05 2.194e-05 0.01% MLMG::mgVcycle_bottom 36 1.804e-05 1.804e-05 1.804e-05 0.00% MLMG::computeResidual() 36 1.623e-05 1.623e-05 1.623e-05 0.00% MLPoisson::define() 6 1.584e-05 1.584e-05 1.584e-05 0.00% makeSFC 30 1.539e-05 1.539e-05 1.539e-05 0.00% FillPatchSingleLevel 20 1.523e-05 1.523e-05 1.523e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.522e-05 1.522e-05 1.522e-05 0.00% DistributionMapping::Distribute() 31 1.017e-05 1.017e-05 1.017e-05 0.00% Amr::initSubcycle() 1 9.866e-06 9.866e-06 9.866e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.111e-06 8.111e-06 8.111e-06 0.00% Castro::do_old_sources() 5 8.103e-06 8.103e-06 8.103e-06 0.00% Castro::apply_source_to_state() 10 5.287e-06 5.287e-06 5.287e-06 0.00% MLPoisson::prepareForSolve() 6 4.76e-06 4.76e-06 4.76e-06 0.00% Gravity::swapTimeLevels() 5 4.519e-06 4.519e-06 4.519e-06 0.00% MLMG::buildFineMask() 6 3.707e-06 3.707e-06 3.707e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.686e-06 3.686e-06 3.686e-06 0.00% Castro::computeNewDt() 5 3.342e-06 3.342e-06 3.342e-06 0.00% MLMG::computeMLResidual() 6 3.198e-06 3.198e-06 3.198e-06 0.00% MLMG::getGradSolution() 6 3.163e-06 3.163e-06 3.163e-06 0.00% Gravity::set_mass_offset() 6 3.002e-06 3.002e-06 3.002e-06 0.00% MLMG::MLResNormInf() 6 2.637e-06 2.637e-06 2.637e-06 0.00% Castro::retry_advance_ctu() 5 1.817e-06 1.817e-06 1.817e-06 0.00% Castro::FluxRegFineAdd() 5 1.726e-06 1.726e-06 1.726e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.253e-06 1.253e-06 1.253e-06 0.00% Amr::init() 1 9.76e-07 9.76e-07 9.76e-07 0.00% AmrLevel::AmrLevel() 1 7.95e-07 7.95e-07 7.95e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4196 0.4196 0.4196 100.00% Amr::coarseTimeStep() 5 0.3396 0.3396 0.3396 80.93% Amr::timeStep() 5 0.3378 0.3378 0.3378 80.48% Castro::advance() 5 0.3341 0.3341 0.3341 79.60% Castro::subcycle_advance_ctu() 5 0.3239 0.3239 0.3239 77.18% Castro::do_advance_ctu() 5 0.3238 0.3238 0.3238 77.16% Castro::construct_new_gravity() 5 0.1463 0.1463 0.1463 34.85% Gravity::solve_phi_with_mlmg() 6 0.1425 0.1425 0.1425 33.95% Gravity::solve_for_phi() 5 0.1368 0.1368 0.1368 32.60% Gravity::actual_solve_with_mlmg() 6 0.1363 0.1363 0.1363 32.47% Castro::construct_ctu_hydro_source() 5 0.1349 0.1349 0.1349 32.14% MLMG::solve() 6 0.1235 0.1235 0.1235 29.43% MLMG::oneIter() 36 0.1156 0.1156 0.1156 27.55% MLMG::mgVcycle() 36 0.1149 0.1149 0.1149 27.37% MLCellLinOp::smooth() 720 0.05999 0.05999 0.05999 14.29% Amr::init() 1 0.05544 0.05544 0.05544 13.21% Amr::restart() 1 0.05544 0.05544 0.05544 13.21% MLCellLinOp::applyBC() 1946 0.04579 0.04579 0.04579 10.91% AmrLevel::restart() 1 0.04471 0.04471 0.04471 10.65% StateData::restartDoit() 4 0.04463 0.04463 0.04463 10.64% VisMF::Read() 3 0.04378 0.04378 0.04378 10.43% MLMG::mgVcycle_bottom 36 0.0333 0.0333 0.0333 7.94% MLMG::actualBottomSolve() 36 0.03328 0.03328 0.03328 7.93% MLCGSolver::bicgstab 36 0.03297 0.03297 0.03297 7.86% MLPoisson::Fsmooth() 1440 0.02566 0.02566 0.02566 6.11% FillPatchIterator::Initialize 20 0.02458 0.02458 0.02458 5.86% Amr::writePlotFile() 1 0.02388 0.02388 0.02388 5.69% VisMF::Write(FabArray) 1 0.02245 0.02245 0.02245 5.35% FillPatchSingleLevel 20 0.02065 0.02065 0.02065 4.92% StateDataPhysBCFunct::() 20 0.01855 0.01855 0.01855 4.42% MLCellLinOp::apply() 500 0.01654 0.01654 0.01654 3.94% FabArray::setVal() 537 0.01627 0.01627 0.01627 3.88% MLMG::mgVcycle_down::0 36 0.01596 0.01596 0.01596 3.80% Castro::initialize_do_advance() 5 0.01451 0.01451 0.01451 3.46% Castro::clean_state() 30 0.01446 0.01446 0.01446 3.44% MLMG::mgVcycle_up::0 36 0.01361 0.01361 0.01361 3.24% Castro::expand_state() 5 0.01204 0.01204 0.01204 2.87% StateData::FillBoundary(geom) 160 0.01161 0.01161 0.01161 2.77% Castro::initialize_advance() 5 0.01008 0.01008 0.01008 2.40% MLPoisson::define() 6 0.009463 0.009463 0.009463 2.25% MLCellLinOp::correctionResidual() 216 0.009345 0.009345 0.009345 2.23% Gravity::get_new_grav_vector() 5 0.009225 0.009225 0.009225 2.20% MultiFab::Dot() 484 0.009178 0.009178 0.009178 2.19% MLMG:computeResOfCorrection() 180 0.008098 0.008098 0.008098 1.93% Castro::computeTemp() 30 0.00802 0.00802 0.00802 1.91% FabArray::FillBoundary() 1766 0.007996 0.007996 0.007996 1.91% Castro::construct_old_gravity() 5 0.00788 0.00788 0.00788 1.88% Gravity::get_old_grav_vector() 5 0.007857 0.007857 0.007857 1.87% MLMG::mgVcycle_down::1 36 0.007801 0.007801 0.007801 1.86% FabArray::ParallelCopy() 380 0.007655 0.007655 0.007655 1.82% FillBoundary_nowait() 1766 0.007587 0.007587 0.007587 1.81% FabArray::ParallelCopy_nowait() 380 0.007507 0.007507 0.007507 1.79% MLMG::mgVcycle_down::2 36 0.007491 0.007491 0.007491 1.78% MLMG::mgVcycle_down::3 36 0.007117 0.007117 0.007117 1.70% MLMG::mgVcycle_down::4 36 0.006836 0.006836 0.006836 1.63% Castro::post_restart() 1 0.006489 0.006489 0.006489 1.55% MLCellLinOp::defineAuxData() 6 0.006456 0.006456 0.006456 1.54% Castro::do_new_sources() 5 0.006354 0.006354 0.006354 1.51% Gravity::multilevel_solve_for_new_phi() 1 0.006107 0.006107 0.006107 1.46% Gravity::actual_multilevel_solve() 1 0.006084 0.006084 0.006084 1.45% Gravity::fill_multipole_BCs() 6 0.00606 0.00606 0.00606 1.44% MLMG::mgVcycle_up::2 36 0.005829 0.005829 0.005829 1.39% MLMG::addInterpCorrection() 180 0.00579 0.00579 0.00579 1.38% MLMG::mgVcycle_up::1 36 0.005747 0.005747 0.005747 1.37% amrex::average_down 180 0.005578 0.005578 0.005578 1.33% CGSolver::sxay() 690 0.00557 0.00557 0.00557 1.33% MLCGSolver::ParallelAllReduce 659 0.005559 0.005559 0.005559 1.32% MLMG::mgVcycle_up::3 36 0.005519 0.005519 0.005519 1.32% MLMG::mgVcycle_up::4 36 0.005471 0.005471 0.005471 1.30% MultiFab::LinComb() 690 0.005348 0.005348 0.005348 1.27% MLPoisson::Fapply() 500 0.004636 0.004636 0.004636 1.10% Castro::do_old_sources() 5 0.004023 0.004023 0.004023 0.96% FabArray::setDomainBndry() 20 0.003802 0.003802 0.003802 0.91% Castro::enforce_min_density() 30 0.003659 0.003659 0.003659 0.87% Castro::post_timestep() 5 0.003582 0.003582 0.003582 0.85% MLMG::prepareForSolve() 6 0.003474 0.003474 0.003474 0.83% MLCellLinOp::solutionResidual() 42 0.003394 0.003394 0.003394 0.81% Castro::estTimeStep() 10 0.003013 0.003013 0.003013 0.72% MLCellLinOp::defineBC() 6 0.002826 0.002826 0.002826 0.67% MLMG::computeResidual() 36 0.002826 0.002826 0.002826 0.67% BndryData::define() 6 0.002722 0.002722 0.002722 0.65% MultiFab::Xpay() 258 0.002661 0.002661 0.002661 0.63% Castro::reset_internal_energy() 30 0.002467 0.002467 0.002467 0.59% Castro::normalize_species() 30 0.002215 0.002215 0.002215 0.53% Castro::computeNewDt() 5 0.001732 0.001732 0.001732 0.41% MLMG::getGradSolution() 6 0.001725 0.001725 0.001725 0.41% MLCellLinOp::compGrad() 6 0.001721 0.001721 0.001721 0.41% Castro::construct_new_source() 25 0.001711 0.001711 0.001711 0.41% Castro::construct_new_gravity_source() 5 0.001545 0.001545 0.001545 0.37% Castro::construct_old_source() 25 0.001347 0.001347 0.001347 0.32% Castro::construct_old_gravity_source() 5 0.001317 0.001317 0.001317 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001207 0.001207 0.001207 0.29% Castro::apply_source_to_state() 10 0.0009247 0.0009247 0.0009247 0.22% MultiFab::Saxpy() 10 0.0009194 0.0009194 0.0009194 0.22% Gravity::swapTimeLevels() 5 0.0008972 0.0008972 0.0008972 0.21% MLMG::ResNormInf() 42 0.0008602 0.0008602 0.0008602 0.20% FabArrayBase::getCPC() 632 0.0008543 0.0008543 0.0008543 0.20% MLCellLinOp::setLevelBC() 6 0.0008384 0.0008384 0.0008384 0.20% Gravity::update_max_rhs() 6 0.0007269 0.0007269 0.0007269 0.17% MLPoisson::prepareForSolve() 6 0.0006998 0.0006998 0.0006998 0.17% MLCellLinOp::prepareForSolve() 6 0.0006951 0.0006951 0.0006951 0.17% FabArray::mult() 22 0.0006528 0.0006528 0.0006528 0.16% MLMG::computeMLResidual() 6 0.0005875 0.0005875 0.0005875 0.14% Castro::enforce_speed_limit() 30 0.0005257 0.0005257 0.0005257 0.13% FabArrayBase::CPC::define() 244 0.0004221 0.0004221 0.0004221 0.10% MultiFab::Copy() 6 0.0003778 0.0003778 0.0003778 0.09% Amr::InitAmr() 1 0.0003775 0.0003775 0.0003775 0.09% FabArrayBase::getFB() 1766 0.0003703 0.0003703 0.0003703 0.09% MultiFab::min() 5 0.0001688 0.0001688 0.0001688 0.04% MLLinOp::define() 6 0.0001656 0.0001656 0.0001656 0.04% MLLinOp::defineGrids() 6 0.0001374 0.0001374 0.0001374 0.03% MLMG::MLResNormInf() 6 0.0001359 0.0001359 0.0001359 0.03% MultiFab::max() 6 0.0001339 0.0001339 0.0001339 0.03% MLMG::MLRhsNormInf() 6 0.0001046 0.0001046 0.0001046 0.02% FabArrayBase::FB::FB() 26 6.003e-05 6.003e-05 6.003e-05 0.01% Castro::create_source_corrector() 5 5.893e-05 5.893e-05 5.893e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.212e-05 3.212e-05 3.212e-05 0.01% Castro::finalize_advance() 5 3.041e-05 3.041e-05 3.041e-05 0.01% Amr::writeSmallPlotFile() 1 2.666e-05 2.666e-05 2.666e-05 0.01% Castro::swap_state_time_levels() 5 2.577e-05 2.577e-05 2.577e-05 0.01% makeSFC 30 2.401e-05 2.401e-05 2.401e-05 0.01% Castro::finalize_do_advance() 5 2.194e-05 2.194e-05 2.194e-05 0.01% DistributionMapping::Distribute() 31 1.017e-05 1.017e-05 1.017e-05 0.00% Amr::initSubcycle() 1 9.866e-06 9.866e-06 9.866e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.245e-06 5.245e-06 5.245e-06 0.00% MLMG::buildFineMask() 6 3.707e-06 3.707e-06 3.707e-06 0.00% Gravity::set_mass_offset() 6 3.002e-06 3.002e-06 3.002e-06 0.00% Castro::retry_advance_ctu() 5 1.817e-06 1.817e-06 1.817e-06 0.00% Castro::FluxRegFineAdd() 5 1.726e-06 1.726e-06 1.726e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.253e-06 1.253e-06 1.253e-06 0.00% AmrLevel::AmrLevel() 1 7.95e-07 7.95e-07 7.95e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-18-ga537fb0516d5) finalized