Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-4-ge4ab0485621d) initialized Starting run at 08:34:44 UTC on 2022-10-07. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-4-ge4ab04856 Microphysics git describe: 22.10-2-gbaa4f033 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051901294 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029676968 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049491223 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052078146 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051100118 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.061085108 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.061240516 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047992724 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.087334361 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.069519847 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058708598 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060014801 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063598405 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047601479 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029689068 seconds Ending run at 08:34:45 UTC on 2022-10-07. Run time = 0.874563091 Run time without initialization = 0.740150712 Average number of zones advanced per microsecond: 3.542 Average number of zones advanced per microsecond per rank: 3.542 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8746 ... 0.8746 ... 0.8746 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2264 0.2264 0.2264 25.88% VisMF::Write(FabArray) 11 0.1988 0.1988 0.1988 22.73% MLCellLinOp::applyBC() 4433 0.07912 0.07912 0.07912 9.05% MLPoisson::Fsmooth() 3280 0.06392 0.06392 0.06392 7.31% MLCGSolver::bicgstab 82 0.02346 0.02346 0.02346 2.68% StateData::FillBoundary(geom) 328 0.02288 0.02288 0.02288 2.62% MultiFab::Dot() 1114 0.02191 0.02191 0.02191 2.51% Castro::normalize_species() 62 0.01553 0.01553 0.01553 1.78% StateDataPhysBCFunct::() 41 0.01495 0.01495 0.01495 1.71% Castro::computeTemp() 63 0.01434 0.01434 0.01434 1.64% MultiFab::LinComb() 1586 0.01412 0.01412 0.01412 1.61% FillBoundary_nowait() 4023 0.01409 0.01409 0.01409 1.61% FabArray::setVal() 1144 0.01402 0.01402 0.01402 1.60% FabArray::ParallelCopy_nowait() 861 0.01286 0.01286 0.01286 1.47% Castro::enforce_min_density() 62 0.01197 0.01197 0.01197 1.37% MLPoisson::Fapply() 1142 0.01158 0.01158 0.01158 1.32% MLCellLinOp::defineAuxData() 11 0.0113 0.0113 0.0113 1.29% Gravity::fill_multipole_BCs() 11 0.01006 0.01006 0.01006 1.15% MLMG::addInterpCorrection() 410 0.007547 0.007547 0.007547 0.86% amrex::average_down 410 0.006773 0.006773 0.006773 0.77% MultiFab::Xpay() 585 0.006504 0.006504 0.006504 0.74% Castro::estTimeStep() 21 0.005521 0.005521 0.005521 0.63% Amr::checkPoint() 3 0.005252 0.005252 0.005252 0.60% Castro::do_advance_ctu() 10 0.004902 0.004902 0.004902 0.56% Castro::reset_internal_energy(MultiFab) 63 0.004244 0.004244 0.004244 0.49% BndryData::define() 11 0.003731 0.003731 0.003731 0.43% Castro::construct_new_gravity_source() 10 0.003417 0.003417 0.003417 0.39% Castro::construct_old_gravity_source() 10 0.002941 0.002941 0.002941 0.34% Amr::writePlotFile() 2 0.002893 0.002893 0.002893 0.33% MLMG::ResNormInf() 93 0.002032 0.002032 0.002032 0.23% Gravity::get_new_grav_vector() 11 0.001912 0.001912 0.001912 0.22% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.21% Castro::expand_state() 10 0.001727 0.001727 0.001727 0.20% Gravity::get_old_grav_vector() 10 0.001717 0.001717 0.001717 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001638 0.001638 0.001638 0.19% MultiFab::Add() 82 0.001623 0.001623 0.001623 0.19% MLCellLinOp::setLevelBC() 11 0.001499 0.001499 0.001499 0.17% Gravity::actual_solve_with_mlmg() 11 0.001444 0.001444 0.001444 0.17% Castro::reset_internal_energy(Fab) 504 0.00144 0.00144 0.00144 0.16% FabArray::mult() 43 0.00132 0.00132 0.00132 0.15% FabArray::setDomainBndry() 41 0.001288 0.001288 0.001288 0.15% Castro::initData() 1 0.001288 0.001288 0.001288 0.15% MLMG::prepareForSolve() 11 0.001224 0.001224 0.001224 0.14% MultiFab::contains_nan() 20 0.001181 0.001181 0.001181 0.14% MLCellLinOp::prepareForSolve() 11 0.001138 0.001138 0.001138 0.13% MLCellLinOp::smooth() 1640 0.001034 0.001034 0.001034 0.12% Castro::enforce_speed_limit() 62 0.001023 0.001023 0.001023 0.12% MLCellLinOp::compGrad() 11 0.000918 0.000918 0.000918 0.10% FabArray::FillBoundary() 4023 0.0009103 0.0009103 0.0009103 0.10% FabArrayBase::getCPC() 1323 0.0007769 0.0007769 0.0007769 0.09% Castro::subcycle_advance_ctu() 10 0.0006945 0.0006945 0.0006945 0.08% FabArrayBase::CPC::define() 454 0.0006763 0.0006763 0.0006763 0.08% FabArrayBase::getFB() 4023 0.0006035 0.0006035 0.0006035 0.07% Amr::InitAmr() 1 0.0004713 0.0004713 0.0004713 0.05% MLCellLinOp::apply() 1142 0.0004677 0.0004677 0.0004677 0.05% Gravity::solve_for_phi() 10 0.0004251 0.0004251 0.0004251 0.05% Gravity::update_max_rhs() 11 0.0004183 0.0004183 0.0004183 0.05% Amr::coarseTimeStep() 10 0.0003661 0.0003661 0.0003661 0.04% CGSolver::sxay() 1586 0.0003468 0.0003468 0.0003468 0.04% MultiFab::Copy() 11 0.0003196 0.0003196 0.0003196 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002958 0.0002958 0.0002958 0.03% FillPatchIterator::Initialize 41 0.0002895 0.0002895 0.0002895 0.03% MLCellLinOp::defineBC() 11 0.000276 0.000276 0.000276 0.03% main() 1 0.0002661 0.0002661 0.0002661 0.03% FabArray::ParallelCopy() 861 0.0002652 0.0002652 0.0002652 0.03% MultiFab::max() 11 0.0002595 0.0002595 0.0002595 0.03% MLCellLinOp::correctionResidual() 492 0.0002281 0.0002281 0.0002281 0.03% MLMG::mgVcycle() 82 0.0002153 0.0002153 0.0002153 0.02% MLMG::MLRhsNormInf() 11 0.0002143 0.0002143 0.0002143 0.02% Castro::construct_new_gravity() 10 0.0002075 0.0002075 0.0002075 0.02% Castro::create_source_corrector() 10 0.0001791 0.0001791 0.0001791 0.02% Amr::timeStep() 10 0.00017 0.00017 0.00017 0.02% MLMG:computeResOfCorrection() 410 0.0001564 0.0001564 0.0001564 0.02% MLLinOp::defineGrids() 11 0.0001545 0.0001545 0.0001545 0.02% StateData::checkPoint() 12 0.0001315 0.0001315 0.0001315 0.02% MLMG::mgVcycle_down::0 82 0.000107 0.000107 0.000107 0.01% Castro::Castro() 1 9.271e-05 9.271e-05 9.271e-05 0.01% MLMG::mgVcycle_down::1 82 9.073e-05 9.073e-05 9.073e-05 0.01% MLMG::mgVcycle_down::2 82 9.001e-05 9.001e-05 9.001e-05 0.01% FabArrayBase::FB::FB() 56 8.619e-05 8.619e-05 8.619e-05 0.01% Castro::initialize_advance() 10 8.418e-05 8.418e-05 8.418e-05 0.01% MLMG::actualBottomSolve() 82 8.379e-05 8.379e-05 8.379e-05 0.01% Castro::clean_state() 62 8.341e-05 8.341e-05 8.341e-05 0.01% MLMG::mgVcycle_down::3 82 7.942e-05 7.942e-05 7.942e-05 0.01% MLMG::mgVcycle_down::4 82 7.81e-05 7.81e-05 7.81e-05 0.01% Castro::advance() 10 7.795e-05 7.795e-05 7.795e-05 0.01% AmrLevel::checkPoint() 3 7.443e-05 7.443e-05 7.443e-05 0.01% MLMG::solve() 11 7.112e-05 7.112e-05 7.112e-05 0.01% MLMG::mgVcycle_up::4 82 7.068e-05 7.068e-05 7.068e-05 0.01% StateData::define() 4 7.005e-05 7.005e-05 7.005e-05 0.01% Castro::initialize_do_advance() 10 6.106e-05 6.106e-05 6.106e-05 0.01% Castro::finalize_advance() 10 5.99e-05 5.99e-05 5.99e-05 0.01% MLMG::oneIter() 82 5.782e-05 5.782e-05 5.782e-05 0.01% MLMG::mgVcycle_up::0 82 5.754e-05 5.754e-05 5.754e-05 0.01% MLMG::mgVcycle_up::2 82 5.458e-05 5.458e-05 5.458e-05 0.01% MLMG::mgVcycle_up::3 82 5.419e-05 5.419e-05 5.419e-05 0.01% MLMG::mgVcycle_up::1 82 5.354e-05 5.354e-05 5.354e-05 0.01% MLCellLinOp::solutionResidual() 93 5.144e-05 5.144e-05 5.144e-05 0.01% Castro::construct_new_source() 50 4.128e-05 4.128e-05 4.128e-05 0.00% Castro::swap_state_time_levels() 10 4.008e-05 4.008e-05 4.008e-05 0.00% MLMG::computeResidual() 82 3.974e-05 3.974e-05 3.974e-05 0.00% Castro::finalize_do_advance() 10 3.49e-05 3.49e-05 3.49e-05 0.00% MLMG::mgVcycle_bottom 82 3.478e-05 3.478e-05 3.478e-05 0.00% Castro::enforce_consistent_e() 1 3.317e-05 3.317e-05 3.317e-05 0.00% Gravity::actual_multilevel_solve() 1 3.129e-05 3.129e-05 3.129e-05 0.00% Castro::initMFs() 1 2.894e-05 2.894e-05 2.894e-05 0.00% FillPatchSingleLevel 41 2.72e-05 2.72e-05 2.72e-05 0.00% makeSFC 55 2.691e-05 2.691e-05 2.691e-05 0.00% Amr::writeSmallPlotFile() 1 2.69e-05 2.69e-05 2.69e-05 0.00% Castro::buildMetrics() 1 2.605e-05 2.605e-05 2.605e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.599e-05 2.599e-05 2.599e-05 0.00% Gravity::swapTimeLevels() 10 2.48e-05 2.48e-05 2.48e-05 0.00% MLLinOp::define() 11 2.422e-05 2.422e-05 2.422e-05 0.00% MLPoisson::define() 11 2.334e-05 2.334e-05 2.334e-05 0.00% Amr::FinalizeInit() 1 2.083e-05 2.083e-05 2.083e-05 0.00% Amr::defBaseLevel() 1 2.062e-05 2.062e-05 2.062e-05 0.00% Castro::construct_old_source() 50 1.819e-05 1.819e-05 1.819e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.746e-05 1.746e-05 1.746e-05 0.00% Castro::do_new_sources() 10 1.746e-05 1.746e-05 1.746e-05 0.00% Castro::do_old_sources() 10 1.687e-05 1.687e-05 1.687e-05 0.00% DistributionMapping::Distribute() 56 1.5e-05 1.5e-05 1.5e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.334e-05 1.334e-05 1.334e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.269e-05 1.269e-05 1.269e-05 0.00% Castro::check_for_nan() 20 1.195e-05 1.195e-05 1.195e-05 0.00% Castro::apply_source_to_state() 20 1.195e-05 1.195e-05 1.195e-05 0.00% Castro::post_timestep() 10 9.712e-06 9.712e-06 9.712e-06 0.00% Castro::construct_old_gravity() 10 9.633e-06 9.633e-06 9.633e-06 0.00% MLMG::computeMLResidual() 11 9.034e-06 9.034e-06 9.034e-06 0.00% Amr::initSubcycle() 1 8.697e-06 8.697e-06 8.697e-06 0.00% MLPoisson::prepareForSolve() 11 8.428e-06 8.428e-06 8.428e-06 0.00% Castro::computeNewDt() 9 7.655e-06 7.655e-06 7.655e-06 0.00% MLMG::getGradSolution() 11 6.411e-06 6.411e-06 6.411e-06 0.00% AmrLevel::checkPointPost() 3 5.572e-06 5.572e-06 5.572e-06 0.00% Amr::InitializeInit() 1 4.877e-06 4.877e-06 4.877e-06 0.00% Castro::retry_advance_ctu() 10 4.48e-06 4.48e-06 4.48e-06 0.00% Gravity::set_mass_offset() 11 4.338e-06 4.338e-06 4.338e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.877e-06 3.877e-06 3.877e-06 0.00% MLMG::MLResNormInf() 11 3.466e-06 3.466e-06 3.466e-06 0.00% Castro::post_init() 1 3.424e-06 3.424e-06 3.424e-06 0.00% Castro::FluxRegCrseInit 10 3.4e-06 3.4e-06 3.4e-06 0.00% Castro::computeInitialDt() 2 3.128e-06 3.128e-06 3.128e-06 0.00% Castro::FluxRegFineAdd() 10 2.896e-06 2.896e-06 2.896e-06 0.00% Amr::init() 1 2.356e-06 2.356e-06 2.356e-06 0.00% AmrLevel::checkPointPre() 3 2.235e-06 2.235e-06 2.235e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.904e-06 1.904e-06 1.904e-06 0.00% Castro::post_regrid() 1 1.382e-06 1.382e-06 1.382e-06 0.00% Amr::initialInit() 1 1.009e-06 1.009e-06 1.009e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8746 0.8746 0.8746 100.00% Amr::coarseTimeStep() 10 0.7102 0.7102 0.7102 81.21% Amr::timeStep() 10 0.6109 0.6109 0.6109 69.85% Castro::advance() 10 0.6031 0.6031 0.6031 68.96% Castro::subcycle_advance_ctu() 10 0.5908 0.5908 0.5908 67.55% Castro::do_advance_ctu() 10 0.5901 0.5901 0.5901 67.47% Gravity::solve_phi_with_mlmg() 11 0.3123 0.3123 0.3123 35.70% Gravity::actual_solve_with_mlmg() 11 0.302 0.302 0.302 34.53% Castro::construct_new_gravity() 10 0.2839 0.2839 0.2839 32.46% MLMG::solve() 11 0.2798 0.2798 0.2798 31.99% Gravity::solve_for_phi() 10 0.2686 0.2686 0.2686 30.71% MLMG::oneIter() 82 0.2651 0.2651 0.2651 30.31% MLMG::mgVcycle() 82 0.2634 0.2634 0.2634 30.12% Castro::construct_ctu_hydro_source() 10 0.2264 0.2264 0.2264 25.88% VisMF::Write(FabArray) 11 0.1988 0.1988 0.1988 22.73% Amr::checkPoint() 3 0.1477 0.1477 0.1477 16.89% AmrLevel::checkPoint() 3 0.1424 0.1424 0.1424 16.28% StateData::checkPoint() 12 0.1423 0.1423 0.1423 16.28% MLCellLinOp::smooth() 1640 0.1356 0.1356 0.1356 15.50% Amr::init() 1 0.1338 0.1338 0.1338 15.30% MLCellLinOp::applyBC() 4433 0.09482 0.09482 0.09482 10.84% MLMG::mgVcycle_bottom 82 0.08045 0.08045 0.08045 9.20% MLMG::actualBottomSolve() 82 0.08041 0.08041 0.08041 9.19% MLCGSolver::bicgstab 82 0.07961 0.07961 0.07961 9.10% MLPoisson::Fsmooth() 3280 0.06392 0.06392 0.06392 7.31% Amr::writePlotFile() 2 0.05952 0.05952 0.05952 6.80% Amr::initialInit() 1 0.05208 0.05208 0.05208 5.96% Castro::clean_state() 62 0.04781 0.04781 0.04781 5.47% Amr::FinalizeInit() 1 0.04776 0.04776 0.04776 5.46% Castro::post_init() 1 0.04637 0.04637 0.04637 5.30% Gravity::multilevel_solve_for_new_phi() 1 0.04413 0.04413 0.04413 5.05% Gravity::actual_multilevel_solve() 1 0.04412 0.04412 0.04412 5.04% FillPatchIterator::Initialize 41 0.04341 0.04341 0.04341 4.96% FillPatchSingleLevel 41 0.04183 0.04183 0.04183 4.78% StateDataPhysBCFunct::() 41 0.03782 0.03782 0.03782 4.32% MLCellLinOp::apply() 1142 0.03575 0.03575 0.03575 4.09% MLMG::mgVcycle_down::0 82 0.03527 0.03527 0.03527 4.03% MLMG::mgVcycle_up::0 82 0.03021 0.03021 0.03021 3.45% StateData::FillBoundary(geom) 328 0.02288 0.02288 0.02288 2.62% MultiFab::Dot() 1114 0.02191 0.02191 0.02191 2.51% Castro::initialize_do_advance() 10 0.02129 0.02129 0.02129 2.43% MLCellLinOp::correctionResidual() 492 0.02098 0.02098 0.02098 2.40% Castro::computeTemp() 63 0.02003 0.02003 0.02003 2.29% MLMG:computeResOfCorrection() 410 0.01811 0.01811 0.01811 2.07% MLPoisson::define() 11 0.01782 0.01782 0.01782 2.04% MLMG::mgVcycle_down::1 82 0.01755 0.01755 0.01755 2.01% Gravity::get_new_grav_vector() 11 0.01726 0.01726 0.01726 1.97% MLMG::mgVcycle_down::2 82 0.01715 0.01715 0.01715 1.96% MLMG::mgVcycle_down::3 82 0.01624 0.01624 0.01624 1.86% FabArray::FillBoundary() 4023 0.01569 0.01569 0.01569 1.79% Castro::normalize_species() 62 0.01553 0.01553 0.01553 1.78% MLMG::mgVcycle_down::4 82 0.01542 0.01542 0.01542 1.76% FillBoundary_nowait() 4023 0.01478 0.01478 0.01478 1.69% Castro::construct_old_gravity() 10 0.01469 0.01469 0.01469 1.68% Gravity::get_old_grav_vector() 10 0.01468 0.01468 0.01468 1.68% CGSolver::sxay() 1586 0.01447 0.01447 0.01447 1.65% MultiFab::LinComb() 1586 0.01412 0.01412 0.01412 1.61% FabArray::setVal() 1144 0.01402 0.01402 0.01402 1.60% FabArray::ParallelCopy() 861 0.01397 0.01397 0.01397 1.60% FabArray::ParallelCopy_nowait() 861 0.0137 0.0137 0.0137 1.57% MLMG::mgVcycle_up::2 82 0.01319 0.01319 0.01319 1.51% MLCGSolver::ParallelAllReduce 1514 0.01307 0.01307 0.01307 1.49% MLMG::mgVcycle_up::1 82 0.01293 0.01293 0.01293 1.48% Castro::expand_state() 10 0.01267 0.01267 0.01267 1.45% MLCellLinOp::defineAuxData() 11 0.01262 0.01262 0.01262 1.44% MLMG::addInterpCorrection() 410 0.01253 0.01253 0.01253 1.43% MLMG::mgVcycle_up::3 82 0.01246 0.01246 0.01246 1.42% MLMG::mgVcycle_up::4 82 0.01237 0.01237 0.01237 1.41% Castro::do_new_sources() 10 0.01226 0.01226 0.01226 1.40% Castro::initialize_advance() 10 0.01221 0.01221 0.01221 1.40% Castro::enforce_min_density() 62 0.01197 0.01197 0.01197 1.37% amrex::average_down 410 0.01178 0.01178 0.01178 1.35% MLPoisson::Fapply() 1142 0.01158 0.01158 0.01158 1.32% Castro::do_old_sources() 10 0.01119 0.01119 0.01119 1.28% Gravity::fill_multipole_BCs() 11 0.01006 0.01006 0.01006 1.15% Castro::post_timestep() 10 0.007604 0.007604 0.007604 0.87% MLCellLinOp::solutionResidual() 93 0.007079 0.007079 0.007079 0.81% MultiFab::Xpay() 585 0.006504 0.006504 0.006504 0.74% MLMG::computeResidual() 82 0.006113 0.006113 0.006113 0.70% Castro::reset_internal_energy(MultiFab) 63 0.005685 0.005685 0.005685 0.65% Castro::estTimeStep() 21 0.005521 0.005521 0.005521 0.63% MLMG::prepareForSolve() 11 0.005233 0.005233 0.005233 0.60% MLCellLinOp::defineBC() 11 0.004938 0.004938 0.004938 0.56% BndryData::define() 11 0.004662 0.004662 0.004662 0.53% Amr::InitializeInit() 1 0.004318 0.004318 0.004318 0.49% Amr::defBaseLevel() 1 0.004313 0.004313 0.004313 0.49% Castro::initData() 1 0.003757 0.003757 0.003757 0.43% Castro::construct_new_source() 50 0.003459 0.003459 0.003459 0.40% Castro::construct_new_gravity_source() 10 0.003417 0.003417 0.003417 0.39% Castro::construct_old_source() 50 0.00296 0.00296 0.00296 0.34% Castro::construct_old_gravity_source() 10 0.002941 0.002941 0.002941 0.34% Castro::computeNewDt() 9 0.00272 0.00272 0.00272 0.31% MLMG::ResNormInf() 93 0.002032 0.002032 0.002032 0.23% Castro::apply_source_to_state() 20 0.001826 0.001826 0.001826 0.21% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001638 0.001638 0.001638 0.19% MultiFab::Add() 82 0.001623 0.001623 0.001623 0.19% MLCellLinOp::setLevelBC() 11 0.001499 0.001499 0.001499 0.17% FabArrayBase::getCPC() 1323 0.001453 0.001453 0.001453 0.17% Castro::reset_internal_energy(Fab) 504 0.00144 0.00144 0.00144 0.16% MLMG::getGradSolution() 11 0.001416 0.001416 0.001416 0.16% MLCellLinOp::compGrad() 11 0.00141 0.00141 0.00141 0.16% FabArray::mult() 43 0.00132 0.00132 0.00132 0.15% FabArray::setDomainBndry() 41 0.001288 0.001288 0.001288 0.15% Castro::check_for_nan() 20 0.001193 0.001193 0.001193 0.14% MultiFab::contains_nan() 20 0.001181 0.001181 0.001181 0.14% Castro::post_regrid() 1 0.001176 0.001176 0.001176 0.13% MLPoisson::prepareForSolve() 11 0.001147 0.001147 0.001147 0.13% MLCellLinOp::prepareForSolve() 11 0.001138 0.001138 0.001138 0.13% Castro::enforce_speed_limit() 62 0.001023 0.001023 0.001023 0.12% MLMG::computeMLResidual() 11 0.001015 0.001015 0.001015 0.12% Gravity::update_max_rhs() 11 0.000825 0.000825 0.000825 0.09% Castro::computeInitialDt() 2 0.000715 0.000715 0.000715 0.08% FabArrayBase::getFB() 4023 0.0006897 0.0006897 0.0006897 0.08% FabArrayBase::CPC::define() 454 0.0006763 0.0006763 0.0006763 0.08% Amr::InitAmr() 1 0.00048 0.00048 0.00048 0.05% Gravity::swapTimeLevels() 10 0.0004608 0.0004608 0.0004608 0.05% Castro::Castro() 1 0.0004469 0.0004469 0.0004469 0.05% MultiFab::Copy() 11 0.0003196 0.0003196 0.0003196 0.04% MLMG::MLResNormInf() 11 0.0002764 0.0002764 0.0002764 0.03% MultiFab::max() 11 0.0002595 0.0002595 0.0002595 0.03% MLLinOp::define() 11 0.0002342 0.0002342 0.0002342 0.03% MLMG::MLRhsNormInf() 11 0.0002143 0.0002143 0.0002143 0.02% MLLinOp::defineGrids() 11 0.00021 0.00021 0.00021 0.02% Castro::create_source_corrector() 10 0.0001791 0.0001791 0.0001791 0.02% Castro::buildMetrics() 1 0.000163 0.000163 0.000163 0.02% FabArrayBase::FB::FB() 56 8.619e-05 8.619e-05 8.619e-05 0.01% AmrLevel::AmrLevel(dm) 1 8.339e-05 8.339e-05 8.339e-05 0.01% StateData::define() 4 7.005e-05 7.005e-05 7.005e-05 0.01% Castro::finalize_advance() 10 6.619e-05 6.619e-05 6.619e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.354e-05 5.354e-05 5.354e-05 0.01% makeSFC 55 4.085e-05 4.085e-05 4.085e-05 0.00% Castro::swap_state_time_levels() 10 4.008e-05 4.008e-05 4.008e-05 0.00% Castro::finalize_do_advance() 10 3.49e-05 3.49e-05 3.49e-05 0.00% Castro::enforce_consistent_e() 1 3.317e-05 3.317e-05 3.317e-05 0.00% Castro::initMFs() 1 2.894e-05 2.894e-05 2.894e-05 0.00% Amr::writeSmallPlotFile() 1 2.69e-05 2.69e-05 2.69e-05 0.00% DistributionMapping::Distribute() 56 1.5e-05 1.5e-05 1.5e-05 0.00% Amr::initSubcycle() 1 8.697e-06 8.697e-06 8.697e-06 0.00% AmrLevel::checkPointPost() 3 5.572e-06 5.572e-06 5.572e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.945e-06 4.945e-06 4.945e-06 0.00% Castro::retry_advance_ctu() 10 4.48e-06 4.48e-06 4.48e-06 0.00% Gravity::set_mass_offset() 11 4.338e-06 4.338e-06 4.338e-06 0.00% Castro::FluxRegCrseInit 10 3.4e-06 3.4e-06 3.4e-06 0.00% Castro::FluxRegFineAdd() 10 2.896e-06 2.896e-06 2.896e-06 0.00% AmrLevel::checkPointPre() 3 2.235e-06 2.235e-06 2.235e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.904e-06 1.904e-06 1.904e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-4-ge4ab0485621d) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-4-ge4ab0485621d) initialized Starting run at 08:34:46 UTC on 2022-10-07. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-4-ge4ab04856 Microphysics git describe: 22.10-2-gbaa4f033 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.457414044 Restart time = 0.049584621 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.055241297 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.054649539 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05741879 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.067656229 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069794508 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031650812 seconds Ending run at 08:34:46 UTC on 2022-10-07. Run time = 0.386960084 Run time without initialization = 0.336816507 Average number of zones advanced per microsecond: 3.891 Average number of zones advanced per microsecond per rank: 3.891 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.387 ... 0.387 ... 0.387 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0861 0.0861 0.0861 22.24% VisMF::Read() 3 0.04131 0.04131 0.04131 10.67% MLCellLinOp::applyBC() 1946 0.0375 0.0375 0.0375 9.69% VisMF::Write(FabArray) 1 0.03011 0.03011 0.03011 7.78% MLPoisson::Fsmooth() 1440 0.02827 0.02827 0.02827 7.30% StateData::FillBoundary(geom) 160 0.02741 0.02741 0.02741 7.08% MLCGSolver::bicgstab 36 0.01078 0.01078 0.01078 2.78% MultiFab::Dot() 484 0.01001 0.01001 0.01001 2.59% Castro::normalize_species() 30 0.007302 0.007302 0.007302 1.89% FabArray::setVal() 537 0.007105 0.007105 0.007105 1.84% Castro::enforce_min_density() 30 0.00672 0.00672 0.00672 1.74% MLCellLinOp::defineAuxData() 6 0.006564 0.006564 0.006564 1.70% FillBoundary_nowait() 1766 0.006558 0.006558 0.006558 1.69% MultiFab::LinComb() 690 0.006479 0.006479 0.006479 1.67% FabArray::ParallelCopy_nowait() 380 0.006476 0.006476 0.006476 1.67% Gravity::fill_multipole_BCs() 6 0.005719 0.005719 0.005719 1.48% Castro::computeTemp() 30 0.005517 0.005517 0.005517 1.43% MLPoisson::Fapply() 500 0.005313 0.005313 0.005313 1.37% Amr::restart() 1 0.003689 0.003689 0.003689 0.95% MLMG::addInterpCorrection() 180 0.003624 0.003624 0.003624 0.94% amrex::average_down 180 0.003261 0.003261 0.003261 0.84% MultiFab::Xpay() 258 0.002963 0.002963 0.002963 0.77% StateDataPhysBCFunct::() 20 0.002828 0.002828 0.002828 0.73% Castro::estTimeStep() 10 0.002766 0.002766 0.002766 0.71% BndryData::define() 6 0.002204 0.002204 0.002204 0.57% Castro::reset_internal_energy(MultiFab) 30 0.002145 0.002145 0.002145 0.55% Castro::do_advance_ctu() 5 0.002115 0.002115 0.002115 0.55% Castro::construct_new_gravity_source() 5 0.001682 0.001682 0.001682 0.43% Amr::writePlotFile() 1 0.001625 0.001625 0.001625 0.42% Castro::construct_old_gravity_source() 5 0.001431 0.001431 0.001431 0.37% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009773 0.0009773 0.0009773 0.25% Castro::reset_internal_energy(Fab) 240 0.000959 0.000959 0.000959 0.25% Gravity::get_old_grav_vector() 5 0.0009372 0.0009372 0.0009372 0.24% MLMG::ResNormInf() 42 0.0009367 0.0009367 0.0009367 0.24% MultiFab::Saxpy() 10 0.000926 0.000926 0.000926 0.24% Gravity::get_new_grav_vector() 5 0.0008927 0.0008927 0.0008927 0.23% Castro::expand_state() 5 0.0008909 0.0008909 0.0008909 0.23% MLCellLinOp::setLevelBC() 6 0.0008527 0.0008527 0.0008527 0.22% MLCellLinOp::smooth() 720 0.0007636 0.0007636 0.0007636 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007608 0.0007608 0.0007608 0.20% MLMG::prepareForSolve() 6 0.0007548 0.0007548 0.0007548 0.20% MultiFab::Add() 36 0.0007252 0.0007252 0.0007252 0.19% MLCellLinOp::prepareForSolve() 6 0.000692 0.000692 0.000692 0.18% FabArray::setDomainBndry() 20 0.0006739 0.0006739 0.0006739 0.17% FabArray::mult() 22 0.0006695 0.0006695 0.0006695 0.17% Castro::enforce_speed_limit() 30 0.0006182 0.0006182 0.0006182 0.16% MultiFab::contains_nan() 10 0.0005979 0.0005979 0.0005979 0.15% MLCellLinOp::compGrad() 6 0.000498 0.000498 0.000498 0.13% Castro::subcycle_advance_ctu() 5 0.0004741 0.0004741 0.0004741 0.12% FabArray::FillBoundary() 1766 0.0004652 0.0004652 0.0004652 0.12% FabArrayBase::CPC::define() 244 0.0004306 0.0004306 0.0004306 0.11% FabArrayBase::getCPC() 632 0.0004167 0.0004167 0.0004167 0.11% Amr::InitAmr() 1 0.0003992 0.0003992 0.0003992 0.10% MLCellLinOp::apply() 500 0.0002847 0.0002847 0.0002847 0.07% main() 1 0.0002548 0.0002548 0.0002548 0.07% FabArrayBase::getFB() 1766 0.0002525 0.0002525 0.0002525 0.07% Gravity::update_max_rhs() 6 0.0002472 0.0002472 0.0002472 0.06% CGSolver::sxay() 690 0.0002374 0.0002374 0.0002374 0.06% Gravity::solve_for_phi() 5 0.0002287 0.0002287 0.0002287 0.06% MultiFab::Copy() 6 0.0001854 0.0001854 0.0001854 0.05% MLCGSolver::ParallelAllReduce 659 0.0001762 0.0001762 0.0001762 0.05% Amr::coarseTimeStep() 5 0.0001716 0.0001716 0.0001716 0.04% FillPatchIterator::Initialize 20 0.0001679 0.0001679 0.0001679 0.04% FabArray::ParallelCopy() 380 0.000159 0.000159 0.000159 0.04% MLCellLinOp::defineBC() 6 0.00015 0.00015 0.00015 0.04% MultiFab::max() 6 0.0001434 0.0001434 0.0001434 0.04% MLCellLinOp::correctionResidual() 216 0.000129 0.000129 0.000129 0.03% Castro::construct_new_gravity() 5 0.0001224 0.0001224 0.0001224 0.03% MLMG::mgVcycle() 36 0.0001205 0.0001205 0.0001205 0.03% MLMG::MLRhsNormInf() 6 0.0001173 0.0001173 0.0001173 0.03% AmrLevel::restart() 1 9.082e-05 9.082e-05 9.082e-05 0.02% Amr::timeStep() 5 8.853e-05 8.853e-05 8.853e-05 0.02% MLMG:computeResOfCorrection() 180 8.346e-05 8.346e-05 8.346e-05 0.02% StateData::restartDoit() 4 8.191e-05 8.191e-05 8.191e-05 0.02% Castro::create_source_corrector() 5 8.074e-05 8.074e-05 8.074e-05 0.02% MLLinOp::defineGrids() 6 7.258e-05 7.258e-05 7.258e-05 0.02% Castro::finalize_advance() 5 6.938e-05 6.938e-05 6.938e-05 0.02% FabArrayBase::FB::FB() 26 6.397e-05 6.397e-05 6.397e-05 0.02% MLMG::mgVcycle_down::4 36 6.262e-05 6.262e-05 6.262e-05 0.02% MLMG::mgVcycle_down::1 36 6.061e-05 6.061e-05 6.061e-05 0.02% Castro::construct_new_source() 25 6.028e-05 6.028e-05 6.028e-05 0.02% MLMG::mgVcycle_down::0 36 6.015e-05 6.015e-05 6.015e-05 0.02% MLMG::mgVcycle_down::2 36 5.974e-05 5.974e-05 5.974e-05 0.02% MLMG::mgVcycle_down::3 36 5.722e-05 5.722e-05 5.722e-05 0.01% Castro::initialize_advance() 5 4.859e-05 4.859e-05 4.859e-05 0.01% Castro::clean_state() 30 4.797e-05 4.797e-05 4.797e-05 0.01% Castro::construct_old_source() 25 4.649e-05 4.649e-05 4.649e-05 0.01% MLMG::actualBottomSolve() 36 4.477e-05 4.477e-05 4.477e-05 0.01% MLMG::mgVcycle_up::4 36 4.027e-05 4.027e-05 4.027e-05 0.01% MLMG::solve() 6 3.906e-05 3.906e-05 3.906e-05 0.01% Castro::buildMetrics() 1 3.767e-05 3.767e-05 3.767e-05 0.01% MLMG::mgVcycle_up::0 36 3.683e-05 3.683e-05 3.683e-05 0.01% MLMG::mgVcycle_up::3 36 3.606e-05 3.606e-05 3.606e-05 0.01% Castro::initialize_do_advance() 5 3.582e-05 3.582e-05 3.582e-05 0.01% MLMG::mgVcycle_up::2 36 3.489e-05 3.489e-05 3.489e-05 0.01% MLMG::mgVcycle_up::1 36 3.371e-05 3.371e-05 3.371e-05 0.01% Castro::post_restart() 1 3.352e-05 3.352e-05 3.352e-05 0.01% MLMG::oneIter() 36 3.346e-05 3.346e-05 3.346e-05 0.01% Castro::advance() 5 3.207e-05 3.207e-05 3.207e-05 0.01% Gravity::actual_multilevel_solve() 1 3.197e-05 3.197e-05 3.197e-05 0.01% Castro::initMFs() 1 3.097e-05 3.097e-05 3.097e-05 0.01% Castro::swap_state_time_levels() 5 2.985e-05 2.985e-05 2.985e-05 0.01% MLCellLinOp::solutionResidual() 42 2.871e-05 2.871e-05 2.871e-05 0.01% Amr::writeSmallPlotFile() 1 2.635e-05 2.635e-05 2.635e-05 0.01% MLLinOp::define() 6 2.255e-05 2.255e-05 2.255e-05 0.01% Castro::finalize_do_advance() 5 2.066e-05 2.066e-05 2.066e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.986e-05 1.986e-05 1.986e-05 0.01% MLMG::computeResidual() 36 1.883e-05 1.883e-05 1.883e-05 0.00% MLMG::mgVcycle_bottom 36 1.703e-05 1.703e-05 1.703e-05 0.00% FillPatchSingleLevel 20 1.627e-05 1.627e-05 1.627e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.578e-05 1.578e-05 1.578e-05 0.00% MLPoisson::define() 6 1.556e-05 1.556e-05 1.556e-05 0.00% makeSFC 30 1.545e-05 1.545e-05 1.545e-05 0.00% Castro::do_new_sources() 5 1.013e-05 1.013e-05 1.013e-05 0.00% Castro::do_old_sources() 5 9.791e-06 9.791e-06 9.791e-06 0.00% Amr::initSubcycle() 1 8.761e-06 8.761e-06 8.761e-06 0.00% DistributionMapping::Distribute() 31 8.684e-06 8.684e-06 8.684e-06 0.00% Castro::apply_source_to_state() 10 6.902e-06 6.902e-06 6.902e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.856e-06 6.856e-06 6.856e-06 0.00% Castro::check_for_nan() 10 6.831e-06 6.831e-06 6.831e-06 0.00% Castro::construct_old_gravity() 5 6.7e-06 6.7e-06 6.7e-06 0.00% Gravity::swapTimeLevels() 5 6.142e-06 6.142e-06 6.142e-06 0.00% Castro::post_timestep() 5 5.977e-06 5.977e-06 5.977e-06 0.00% MLMG::computeMLResidual() 6 5.11e-06 5.11e-06 5.11e-06 0.00% MLPoisson::prepareForSolve() 6 4.95e-06 4.95e-06 4.95e-06 0.00% MLMG::getGradSolution() 6 3.926e-06 3.926e-06 3.926e-06 0.00% Castro::computeNewDt() 5 3.615e-06 3.615e-06 3.615e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.401e-06 3.401e-06 3.401e-06 0.00% MLMG::MLResNormInf() 6 2.596e-06 2.596e-06 2.596e-06 0.00% Gravity::set_mass_offset() 6 2.491e-06 2.491e-06 2.491e-06 0.00% Castro::retry_advance_ctu() 5 2.048e-06 2.048e-06 2.048e-06 0.00% Castro::FluxRegCrseInit 5 1.858e-06 1.858e-06 1.858e-06 0.00% Amr::init() 1 1.134e-06 1.134e-06 1.134e-06 0.00% Castro::FluxRegFineAdd() 5 1.105e-06 1.105e-06 1.105e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.083e-06 1.083e-06 1.083e-06 0.00% AmrLevel::AmrLevel() 1 9.72e-07 9.72e-07 9.72e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.387 0.387 0.387 99.99% Amr::coarseTimeStep() 5 0.3049 0.3049 0.3049 78.79% Amr::timeStep() 5 0.3032 0.3032 0.3032 78.35% Castro::advance() 5 0.2996 0.2996 0.2996 77.42% Castro::subcycle_advance_ctu() 5 0.2929 0.2929 0.2929 75.69% Castro::do_advance_ctu() 5 0.2924 0.2924 0.2924 75.56% Castro::construct_new_gravity() 5 0.1585 0.1585 0.1585 40.94% Gravity::solve_phi_with_mlmg() 6 0.1496 0.1496 0.1496 38.65% Gravity::solve_for_phi() 5 0.1458 0.1458 0.1458 37.67% Gravity::actual_solve_with_mlmg() 6 0.1437 0.1437 0.1437 37.13% MLMG::solve() 6 0.1308 0.1308 0.1308 33.80% MLMG::oneIter() 36 0.1232 0.1232 0.1232 31.82% MLMG::mgVcycle() 36 0.1224 0.1224 0.1224 31.63% Castro::construct_ctu_hydro_source() 5 0.08606 0.08606 0.08606 22.24% MLCellLinOp::smooth() 720 0.06219 0.06219 0.06219 16.07% Amr::init() 1 0.04963 0.04963 0.04963 12.82% Amr::restart() 1 0.04963 0.04963 0.04963 12.82% MLCellLinOp::applyBC() 1946 0.04484 0.04484 0.04484 11.59% AmrLevel::restart() 1 0.04154 0.04154 0.04154 10.73% StateData::restartDoit() 4 0.04144 0.04144 0.04144 10.71% VisMF::Read() 3 0.04131 0.04131 0.04131 10.67% MLMG::mgVcycle_bottom 36 0.03734 0.03734 0.03734 9.65% MLMG::actualBottomSolve() 36 0.03733 0.03733 0.03733 9.64% MLCGSolver::bicgstab 36 0.03694 0.03694 0.03694 9.54% FillPatchIterator::Initialize 20 0.03312 0.03312 0.03312 8.56% FillPatchSingleLevel 20 0.03228 0.03228 0.03228 8.34% Amr::writePlotFile() 1 0.03174 0.03174 0.03174 8.20% StateDataPhysBCFunct::() 20 0.03024 0.03024 0.03024 7.81% VisMF::Write(FabArray) 1 0.03011 0.03011 0.03011 7.78% MLPoisson::Fsmooth() 1440 0.02827 0.02827 0.02827 7.30% StateData::FillBoundary(geom) 160 0.02741 0.02741 0.02741 7.08% Castro::clean_state() 30 0.02331 0.02331 0.02331 6.02% MLCellLinOp::apply() 500 0.01698 0.01698 0.01698 4.39% MLMG::mgVcycle_down::0 36 0.01605 0.01605 0.01605 4.15% MLMG::mgVcycle_up::0 36 0.01378 0.01378 0.01378 3.56% Castro::construct_old_gravity() 5 0.01307 0.01307 0.01307 3.38% Gravity::get_old_grav_vector() 5 0.01306 0.01306 0.01306 3.37% Gravity::get_new_grav_vector() 5 0.01255 0.01255 0.01255 3.24% Castro::initialize_do_advance() 5 0.01085 0.01085 0.01085 2.80% MLPoisson::define() 6 0.01048 0.01048 0.01048 2.71% MultiFab::Dot() 484 0.01001 0.01001 0.01001 2.59% MLCellLinOp::correctionResidual() 216 0.009816 0.009816 0.009816 2.54% Castro::computeTemp() 30 0.00862 0.00862 0.00862 2.23% MLMG:computeResOfCorrection() 180 0.008458 0.008458 0.008458 2.19% MLMG::mgVcycle_down::1 36 0.008238 0.008238 0.008238 2.13% MLMG::mgVcycle_down::2 36 0.007992 0.007992 0.007992 2.07% MLMG::mgVcycle_down::3 36 0.007625 0.007625 0.007625 1.97% MLCellLinOp::defineAuxData() 6 0.00741 0.00741 0.00741 1.91% FabArray::FillBoundary() 1766 0.007339 0.007339 0.007339 1.90% MLMG::mgVcycle_down::4 36 0.007336 0.007336 0.007336 1.90% Castro::normalize_species() 30 0.007302 0.007302 0.007302 1.89% FabArray::setVal() 537 0.007105 0.007105 0.007105 1.84% FabArray::ParallelCopy() 380 0.007034 0.007034 0.007034 1.82% Castro::do_new_sources() 5 0.006905 0.006905 0.006905 1.78% FabArray::ParallelCopy_nowait() 380 0.006875 0.006875 0.006875 1.78% FillBoundary_nowait() 1766 0.006874 0.006874 0.006874 1.78% Castro::enforce_min_density() 30 0.00672 0.00672 0.00672 1.74% CGSolver::sxay() 690 0.006716 0.006716 0.006716 1.74% Castro::initialize_advance() 5 0.006625 0.006625 0.006625 1.71% Castro::expand_state() 5 0.006518 0.006518 0.006518 1.68% MultiFab::LinComb() 690 0.006479 0.006479 0.006479 1.67% MLMG::addInterpCorrection() 180 0.006148 0.006148 0.006148 1.59% MLMG::mgVcycle_up::2 36 0.006145 0.006145 0.006145 1.59% MLMG::mgVcycle_up::1 36 0.006087 0.006087 0.006087 1.57% MLCGSolver::ParallelAllReduce 659 0.00604 0.00604 0.00604 1.56% MLMG::mgVcycle_up::3 36 0.005853 0.005853 0.005853 1.51% MLMG::mgVcycle_up::4 36 0.005833 0.005833 0.005833 1.51% amrex::average_down 180 0.005753 0.005753 0.005753 1.49% Gravity::fill_multipole_BCs() 6 0.005719 0.005719 0.005719 1.48% Castro::do_old_sources() 5 0.005502 0.005502 0.005502 1.42% MLPoisson::Fapply() 500 0.005313 0.005313 0.005313 1.37% Castro::post_restart() 1 0.0042 0.0042 0.0042 1.09% Gravity::multilevel_solve_for_new_phi() 1 0.004064 0.004064 0.004064 1.05% Gravity::actual_multilevel_solve() 1 0.004045 0.004045 0.004045 1.05% Castro::post_timestep() 5 0.003488 0.003488 0.003488 0.90% MLCellLinOp::solutionResidual() 42 0.003393 0.003393 0.003393 0.88% MLMG::prepareForSolve() 6 0.003151 0.003151 0.003151 0.81% Castro::reset_internal_energy(MultiFab) 30 0.003104 0.003104 0.003104 0.80% MultiFab::Xpay() 258 0.002963 0.002963 0.002963 0.77% MLCellLinOp::defineBC() 6 0.002933 0.002933 0.002933 0.76% MLMG::computeResidual() 36 0.002817 0.002817 0.002817 0.73% BndryData::define() 6 0.002783 0.002783 0.002783 0.72% Castro::estTimeStep() 10 0.002766 0.002766 0.002766 0.71% Castro::construct_new_source() 25 0.001742 0.001742 0.001742 0.45% Castro::construct_new_gravity_source() 5 0.001682 0.001682 0.001682 0.43% Castro::computeNewDt() 5 0.001531 0.001531 0.001531 0.40% Castro::construct_old_source() 25 0.001478 0.001478 0.001478 0.38% Castro::construct_old_gravity_source() 5 0.001431 0.001431 0.001431 0.37% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009773 0.0009773 0.0009773 0.25% Castro::reset_internal_energy(Fab) 240 0.000959 0.000959 0.000959 0.25% MLMG::ResNormInf() 42 0.0009367 0.0009367 0.0009367 0.24% Castro::apply_source_to_state() 10 0.0009329 0.0009329 0.0009329 0.24% MultiFab::Saxpy() 10 0.000926 0.000926 0.000926 0.24% MLCellLinOp::setLevelBC() 6 0.0008527 0.0008527 0.0008527 0.22% FabArrayBase::getCPC() 632 0.0008473 0.0008473 0.0008473 0.22% MLMG::getGradSolution() 6 0.0007911 0.0007911 0.0007911 0.20% MLCellLinOp::compGrad() 6 0.0007872 0.0007872 0.0007872 0.20% MultiFab::Add() 36 0.0007252 0.0007252 0.0007252 0.19% MLPoisson::prepareForSolve() 6 0.000697 0.000697 0.000697 0.18% MLCellLinOp::prepareForSolve() 6 0.000692 0.000692 0.000692 0.18% FabArray::setDomainBndry() 20 0.0006739 0.0006739 0.0006739 0.17% FabArray::mult() 22 0.0006695 0.0006695 0.0006695 0.17% Castro::enforce_speed_limit() 30 0.0006182 0.0006182 0.0006182 0.16% Castro::check_for_nan() 10 0.0006048 0.0006048 0.0006048 0.16% MLMG::computeMLResidual() 6 0.0006005 0.0006005 0.0006005 0.16% MultiFab::contains_nan() 10 0.0005979 0.0005979 0.0005979 0.15% Gravity::update_max_rhs() 6 0.0004736 0.0004736 0.0004736 0.12% FabArrayBase::CPC::define() 244 0.0004306 0.0004306 0.0004306 0.11% Amr::InitAmr() 1 0.0004079 0.0004079 0.0004079 0.11% FabArrayBase::getFB() 1766 0.0003165 0.0003165 0.0003165 0.08% Gravity::swapTimeLevels() 5 0.000246 0.000246 0.000246 0.06% MultiFab::Copy() 6 0.0001854 0.0001854 0.0001854 0.05% Castro::buildMetrics() 1 0.0001683 0.0001683 0.0001683 0.04% MLMG::MLResNormInf() 6 0.0001533 0.0001533 0.0001533 0.04% MultiFab::max() 6 0.0001434 0.0001434 0.0001434 0.04% MLLinOp::define() 6 0.0001257 0.0001257 0.0001257 0.03% MLMG::MLRhsNormInf() 6 0.0001173 0.0001173 0.0001173 0.03% MLLinOp::defineGrids() 6 0.0001032 0.0001032 0.0001032 0.03% Castro::create_source_corrector() 5 8.074e-05 8.074e-05 8.074e-05 0.02% Castro::finalize_advance() 5 7.235e-05 7.235e-05 7.235e-05 0.02% FabArrayBase::FB::FB() 26 6.397e-05 6.397e-05 6.397e-05 0.02% Castro::initMFs() 1 3.097e-05 3.097e-05 3.097e-05 0.01% Castro::swap_state_time_levels() 5 2.985e-05 2.985e-05 2.985e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.954e-05 2.954e-05 2.954e-05 0.01% Amr::writeSmallPlotFile() 1 2.635e-05 2.635e-05 2.635e-05 0.01% makeSFC 30 2.268e-05 2.268e-05 2.268e-05 0.01% Castro::finalize_do_advance() 5 2.066e-05 2.066e-05 2.066e-05 0.01% Amr::initSubcycle() 1 8.761e-06 8.761e-06 8.761e-06 0.00% DistributionMapping::Distribute() 31 8.684e-06 8.684e-06 8.684e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.848e-06 4.848e-06 4.848e-06 0.00% Gravity::set_mass_offset() 6 2.491e-06 2.491e-06 2.491e-06 0.00% Castro::retry_advance_ctu() 5 2.048e-06 2.048e-06 2.048e-06 0.00% Castro::FluxRegCrseInit 5 1.858e-06 1.858e-06 1.858e-06 0.00% Castro::FluxRegFineAdd() 5 1.105e-06 1.105e-06 1.105e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.083e-06 1.083e-06 1.083e-06 0.00% AmrLevel::AmrLevel() 1 9.72e-07 9.72e-07 9.72e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-4-ge4ab0485621d) finalized