Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-38-g3d344ec19655) initialized Starting run at 17:57:09 UTC on 2022-04-29. Successfully read inputs file ... Castro git describe: 22.04-48-g0ae74bbc5 AMReX git describe: 22.04-38-g3d344ec19 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.040419546 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.023713072 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047174585 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050627513 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.062214604 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.063434463 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.059185393 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.038026257 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048838698 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049317167 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062046539 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062462129 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060212991 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.037844625 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024019897 seconds Ending run at 17:57:10 UTC on 2022-04-29. Run time = 0.777538918 Run time without initialization = 0.665880478 Average number of zones advanced per microsecond: 3.937 Average number of zones advanced per microsecond per rank: 3.937 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7776 ... 0.7776 ... 0.7776 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1761 0.1761 0.1761 22.65% VisMF::Write(FabArray) 11 0.1576 0.1576 0.1576 20.27% MLCellLinOp::applyBC() 4379 0.07377 0.07377 0.07377 9.49% MLPoisson::Fsmooth() 3240 0.06048 0.06048 0.06048 7.78% StateData::FillBoundary(geom) 328 0.02451 0.02451 0.02451 3.15% MLCGSolver::bicgstab 81 0.02161 0.02161 0.02161 2.78% MultiFab::Dot() 1100 0.02107 0.02107 0.02107 2.71% Castro::normalize_species() 62 0.02058 0.02058 0.02058 2.65% Castro::computeTemp() 63 0.01673 0.01673 0.01673 2.15% Castro::enforce_min_density() 62 0.01526 0.01526 0.01526 1.96% FillBoundary_nowait() 3974 0.01447 0.01447 0.01447 1.86% FabArray::setVal() 1135 0.01348 0.01348 0.01348 1.73% FabArray::ParallelCopy_nowait() 851 0.01333 0.01333 0.01333 1.71% MultiFab::LinComb() 1566 0.01277 0.01277 0.01277 1.64% StateDataPhysBCFunct::() 41 0.0115 0.0115 0.0115 1.48% MLPoisson::Fapply() 1128 0.01077 0.01077 0.01077 1.39% MLCellLinOp::defineAuxData() 11 0.01005 0.01005 0.01005 1.29% Gravity::fill_multipole_BCs() 11 0.008378 0.008378 0.008378 1.08% Castro::estTimeStep() 21 0.008291 0.008291 0.008291 1.07% MLMG::addInterpCorrection() 405 0.006747 0.006747 0.006747 0.87% amrex::average_down 405 0.006219 0.006219 0.006219 0.80% MultiFab::Xpay() 578 0.006143 0.006143 0.006143 0.79% Castro::reset_internal_energy(MultiFab) 63 0.006085 0.006085 0.006085 0.78% Castro::do_advance_ctu() 10 0.005251 0.005251 0.005251 0.68% Amr::checkPoint() 3 0.004071 0.004071 0.004071 0.52% BndryData::define() 11 0.003835 0.003835 0.003835 0.49% Castro::construct_new_gravity_source() 10 0.003172 0.003172 0.003172 0.41% Castro::construct_old_gravity_source() 10 0.002558 0.002558 0.002558 0.33% Amr::writePlotFile() 2 0.002404 0.002404 0.002404 0.31% Castro::enforce_speed_limit() 62 0.002207 0.002207 0.002207 0.28% Gravity::get_new_grav_vector() 11 0.001948 0.001948 0.001948 0.25% MLMG::ResNormInf() 92 0.001927 0.001927 0.001927 0.25% MultiFab::Saxpy() 20 0.001819 0.001819 0.001819 0.23% Castro::expand_state() 10 0.001739 0.001739 0.001739 0.22% Gravity::get_old_grav_vector() 10 0.001737 0.001737 0.001737 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001703 0.001703 0.001703 0.22% MLMG::oneIter() 81 0.001682 0.001682 0.001682 0.22% Castro::reset_internal_energy(Fab) 504 0.001606 0.001606 0.001606 0.21% Gravity::actual_solve_with_mlmg() 11 0.001386 0.001386 0.001386 0.18% FabArray::mult() 43 0.001364 0.001364 0.001364 0.18% MLCellLinOp::setLevelBC() 11 0.001347 0.001347 0.001347 0.17% FabArray::setDomainBndry() 41 0.001332 0.001332 0.001332 0.17% Castro::initData() 1 0.001188 0.001188 0.001188 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.15% MLCellLinOp::prepareForSolve() 11 0.001099 0.001099 0.001099 0.14% MLCellLinOp::smooth() 1620 0.001048 0.001048 0.001048 0.13% MLCellLinOp::compGrad() 11 0.0009215 0.0009215 0.0009215 0.12% Castro::subcycle_advance_ctu() 10 0.0008432 0.0008432 0.0008432 0.11% FabArrayBase::getCPC() 1313 0.0008284 0.0008284 0.0008284 0.11% FabArray::FillBoundary() 3974 0.0008103 0.0008103 0.0008103 0.10% MLMG::prepareForSolve() 11 0.0007772 0.0007772 0.0007772 0.10% FabArrayBase::CPC::define() 454 0.0006816 0.0006816 0.0006816 0.09% FabArrayBase::getFB() 3974 0.0006736 0.0006736 0.0006736 0.09% MLCellLinOp::apply() 1128 0.0004938 0.0004938 0.0004938 0.06% Amr::InitAmr() 1 0.0004577 0.0004577 0.0004577 0.06% CGSolver::sxay() 1566 0.0004222 0.0004222 0.0004222 0.05% Gravity::update_max_rhs() 11 0.0004141 0.0004141 0.0004141 0.05% Gravity::solve_for_phi() 10 0.0004027 0.0004027 0.0004027 0.05% MLLinOp::defineGrids() 11 0.0003871 0.0003871 0.0003871 0.05% MLMG::mgVcycle() 81 0.0003593 0.0003593 0.0003593 0.05% MLCGSolver::ParallelAllReduce 1495 0.0003246 0.0003246 0.0003246 0.04% main() 1 0.0003001 0.0003001 0.0003001 0.04% FabArray::ParallelCopy() 851 0.0002821 0.0002821 0.0002821 0.04% MultiFab::Copy() 11 0.0002689 0.0002689 0.0002689 0.03% MultiFab::max() 11 0.0002643 0.0002643 0.0002643 0.03% FillPatchIterator::Initialize 41 0.0002591 0.0002591 0.0002591 0.03% MLCellLinOp::correctionResidual() 486 0.0002203 0.0002203 0.0002203 0.03% Castro::construct_new_gravity() 10 0.0002099 0.0002099 0.0002099 0.03% MLMG::MLRhsNormInf() 11 0.0002041 0.0002041 0.0002041 0.03% Amr::timeStep() 10 0.0001972 0.0001972 0.0001972 0.03% Amr::coarseTimeStep() 10 0.0001924 0.0001924 0.0001924 0.02% MLCellLinOp::defineBC() 11 0.0001899 0.0001899 0.0001899 0.02% Castro::advance() 10 0.0001504 0.0001504 0.0001504 0.02% MLMG:computeResOfCorrection() 405 0.0001366 0.0001366 0.0001366 0.02% StateData::checkPoint() 12 0.0001278 0.0001278 0.0001278 0.02% MLMG::actualBottomSolve() 81 0.0001143 0.0001143 0.0001143 0.01% Castro::finalize_advance() 10 0.0001034 0.0001034 0.0001034 0.01% Castro::Castro() 1 8.657e-05 8.657e-05 8.657e-05 0.01% FabArrayBase::FB::FB() 56 8.399e-05 8.399e-05 8.399e-05 0.01% MLMG::mgVcycle_down::0 81 8.336e-05 8.336e-05 8.336e-05 0.01% MLMG::solve() 11 8.192e-05 8.192e-05 8.192e-05 0.01% MLMG::mgVcycle_down::1 81 7.883e-05 7.883e-05 7.883e-05 0.01% Castro::initialize_advance() 10 7.707e-05 7.707e-05 7.707e-05 0.01% MLMG::mgVcycle_down::2 81 7.369e-05 7.369e-05 7.369e-05 0.01% Castro::clean_state() 62 7.362e-05 7.362e-05 7.362e-05 0.01% AmrLevel::checkPoint() 3 7.1e-05 7.1e-05 7.1e-05 0.01% MLMG::mgVcycle_down::3 81 6.971e-05 6.971e-05 6.971e-05 0.01% MLMG::mgVcycle_down::4 81 6.935e-05 6.935e-05 6.935e-05 0.01% Castro::initialize_do_advance() 10 6.165e-05 6.165e-05 6.165e-05 0.01% MLMG::mgVcycle_up::4 81 5.744e-05 5.744e-05 5.744e-05 0.01% MLMG::mgVcycle_up::0 81 5.097e-05 5.097e-05 5.097e-05 0.01% MLCellLinOp::solutionResidual() 92 4.922e-05 4.922e-05 4.922e-05 0.01% MLMG::mgVcycle_up::1 81 4.591e-05 4.591e-05 4.591e-05 0.01% MLMG::mgVcycle_up::3 81 4.564e-05 4.564e-05 4.564e-05 0.01% MLMG::mgVcycle_up::2 81 4.514e-05 4.514e-05 4.514e-05 0.01% StateData::define() 4 4.369e-05 4.369e-05 4.369e-05 0.01% MLMG::mgVcycle_bottom 81 3.701e-05 3.701e-05 3.701e-05 0.00% Castro::finalize_do_advance() 10 3.648e-05 3.648e-05 3.648e-05 0.00% Castro::swap_state_time_levels() 10 3.534e-05 3.534e-05 3.534e-05 0.00% MLMG::computeResidual() 81 3.463e-05 3.463e-05 3.463e-05 0.00% Castro::enforce_consistent_e() 1 3.438e-05 3.438e-05 3.438e-05 0.00% Gravity::actual_multilevel_solve() 1 3.131e-05 3.131e-05 3.131e-05 0.00% FillPatchSingleLevel 41 2.908e-05 2.908e-05 2.908e-05 0.00% Amr::writeSmallPlotFile() 1 2.673e-05 2.673e-05 2.673e-05 0.00% Castro::initMFs() 1 2.605e-05 2.605e-05 2.605e-05 0.00% makeSFC 55 2.523e-05 2.523e-05 2.523e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.43e-05 2.43e-05 2.43e-05 0.00% Castro::construct_old_source() 50 2.414e-05 2.414e-05 2.414e-05 0.00% MLLinOp::define() 11 2.401e-05 2.401e-05 2.401e-05 0.00% MLPoisson::define() 11 2.327e-05 2.327e-05 2.327e-05 0.00% Amr::FinalizeInit() 1 2.046e-05 2.046e-05 2.046e-05 0.00% Castro::buildMetrics() 1 2.042e-05 2.042e-05 2.042e-05 0.00% Castro::construct_new_source() 50 1.919e-05 1.919e-05 1.919e-05 0.00% Castro::do_new_sources() 10 1.675e-05 1.675e-05 1.675e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.64e-05 1.64e-05 1.64e-05 0.00% Amr::defBaseLevel() 1 1.632e-05 1.632e-05 1.632e-05 0.00% Castro::do_old_sources() 10 1.54e-05 1.54e-05 1.54e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.441e-05 1.441e-05 1.441e-05 0.00% DistributionMapping::Distribute() 56 1.44e-05 1.44e-05 1.44e-05 0.00% Castro::check_for_nan() 20 1.192e-05 1.192e-05 1.192e-05 0.00% Castro::apply_source_to_state() 20 1.185e-05 1.185e-05 1.185e-05 0.00% Castro::construct_old_gravity() 10 1.011e-05 1.011e-05 1.011e-05 0.00% AmrLevel::AmrLevel(dm) 1 9.82e-06 9.82e-06 9.82e-06 0.00% Amr::initSubcycle() 1 9.782e-06 9.782e-06 9.782e-06 0.00% Gravity::swapTimeLevels() 10 9.439e-06 9.439e-06 9.439e-06 0.00% Castro::post_timestep() 10 9.106e-06 9.106e-06 9.106e-06 0.00% MLPoisson::prepareForSolve() 11 8.28e-06 8.28e-06 8.28e-06 0.00% Castro::computeNewDt() 9 6.726e-06 6.726e-06 6.726e-06 0.00% MLMG::computeMLResidual() 11 6.5e-06 6.5e-06 6.5e-06 0.00% Amr::InitializeInit() 1 6.273e-06 6.273e-06 6.273e-06 0.00% Castro::retry_advance_ctu() 10 5.905e-06 5.905e-06 5.905e-06 0.00% MLMG::getGradSolution() 11 5.823e-06 5.823e-06 5.823e-06 0.00% MLMG::buildFineMask() 11 5.668e-06 5.668e-06 5.668e-06 0.00% MLMG::MLResNormInf() 11 4.959e-06 4.959e-06 4.959e-06 0.00% Castro::post_init() 1 4.795e-06 4.795e-06 4.795e-06 0.00% Gravity::set_mass_offset() 11 4.487e-06 4.487e-06 4.487e-06 0.00% AmrLevel::checkPointPost() 3 4.323e-06 4.323e-06 4.323e-06 0.00% Castro::create_source_corrector() 10 4.24e-06 4.24e-06 4.24e-06 0.00% Castro::FluxRegCrseInit 10 4.059e-06 4.059e-06 4.059e-06 0.00% Castro::FluxRegFineAdd() 10 3.253e-06 3.253e-06 3.253e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.731e-06 2.731e-06 2.731e-06 0.00% Amr::init() 1 2.368e-06 2.368e-06 2.368e-06 0.00% Castro::computeInitialDt() 2 2.362e-06 2.362e-06 2.362e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.216e-06 2.216e-06 2.216e-06 0.00% AmrLevel::checkPointPre() 3 2.105e-06 2.105e-06 2.105e-06 0.00% Castro::post_regrid() 1 1.309e-06 1.309e-06 1.309e-06 0.00% Amr::initialInit() 1 9.43e-07 9.43e-07 9.43e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7776 0.7776 0.7776 100.00% Amr::coarseTimeStep() 10 0.6416 0.6416 0.6416 82.52% Amr::timeStep() 10 0.5611 0.5611 0.5611 72.15% Castro::advance() 10 0.5505 0.5505 0.5505 70.79% Castro::subcycle_advance_ctu() 10 0.5356 0.5356 0.5356 68.88% Castro::do_advance_ctu() 10 0.5347 0.5347 0.5347 68.77% Gravity::solve_phi_with_mlmg() 11 0.2939 0.2939 0.2939 37.79% Gravity::actual_solve_with_mlmg() 11 0.2853 0.2853 0.2853 36.68% Castro::construct_new_gravity() 10 0.27 0.27 0.27 34.72% MLMG::solve() 11 0.2642 0.2642 0.2642 33.98% Gravity::solve_for_phi() 10 0.2545 0.2545 0.2545 32.73% MLMG::oneIter() 81 0.2503 0.2503 0.2503 32.18% MLMG::mgVcycle() 81 0.2486 0.2486 0.2486 31.97% Castro::construct_ctu_hydro_source() 10 0.1761 0.1761 0.1761 22.65% VisMF::Write(FabArray) 11 0.1576 0.1576 0.1576 20.27% MLCellLinOp::smooth() 1620 0.1281 0.1281 0.1281 16.48% Amr::checkPoint() 3 0.1164 0.1164 0.1164 14.97% AmrLevel::checkPoint() 3 0.1123 0.1123 0.1123 14.45% StateData::checkPoint() 12 0.1123 0.1123 0.1123 14.44% Amr::init() 1 0.1111 0.1111 0.1111 14.28% MLCellLinOp::applyBC() 4379 0.08981 0.08981 0.08981 11.55% MLMG::mgVcycle_bottom 81 0.07494 0.07494 0.07494 9.64% MLMG::actualBottomSolve() 81 0.07491 0.07491 0.07491 9.63% MLCGSolver::bicgstab 81 0.07415 0.07415 0.07415 9.54% Castro::clean_state() 62 0.06173 0.06173 0.06173 7.94% MLPoisson::Fsmooth() 3240 0.06048 0.06048 0.06048 7.78% Amr::writePlotFile() 2 0.04786 0.04786 0.04786 6.15% Amr::initialInit() 1 0.04682 0.04682 0.04682 6.02% Amr::FinalizeInit() 1 0.04293 0.04293 0.04293 5.52% FillPatchIterator::Initialize 41 0.04161 0.04161 0.04161 5.35% Castro::post_init() 1 0.0416 0.0416 0.0416 5.35% FillPatchSingleLevel 41 0.04002 0.04002 0.04002 5.15% Gravity::multilevel_solve_for_new_phi() 1 0.03982 0.03982 0.03982 5.12% Gravity::actual_multilevel_solve() 1 0.0398 0.0398 0.0398 5.12% StateDataPhysBCFunct::() 41 0.03602 0.03602 0.03602 4.63% MLMG::mgVcycle_down::0 81 0.03474 0.03474 0.03474 4.47% MLCellLinOp::apply() 1128 0.034 0.034 0.034 4.37% MLMG::mgVcycle_up::0 81 0.02962 0.02962 0.02962 3.81% StateData::FillBoundary(geom) 328 0.02451 0.02451 0.02451 3.15% Castro::computeTemp() 63 0.02442 0.02442 0.02442 3.14% Castro::initialize_do_advance() 10 0.02261 0.02261 0.02261 2.91% MultiFab::Dot() 1100 0.02107 0.02107 0.02107 2.71% Castro::normalize_species() 62 0.02058 0.02058 0.02058 2.65% MLCellLinOp::correctionResidual() 486 0.01989 0.01989 0.01989 2.56% MLMG:computeResOfCorrection() 405 0.01725 0.01725 0.01725 2.22% Gravity::get_new_grav_vector() 11 0.01696 0.01696 0.01696 2.18% MLPoisson::define() 11 0.0169 0.0169 0.0169 2.17% MLMG::mgVcycle_down::1 81 0.01651 0.01651 0.01651 2.12% FabArray::FillBoundary() 3974 0.01604 0.01604 0.01604 2.06% MLMG::mgVcycle_down::2 81 0.01594 0.01594 0.01594 2.05% Castro::enforce_min_density() 62 0.01526 0.01526 0.01526 1.96% FillBoundary_nowait() 3974 0.01523 0.01523 0.01523 1.96% MLMG::mgVcycle_down::3 81 0.01503 0.01503 0.01503 1.93% Castro::construct_old_gravity() 10 0.01466 0.01466 0.01466 1.89% Gravity::get_old_grav_vector() 10 0.01465 0.01465 0.01465 1.88% Castro::initialize_advance() 10 0.01462 0.01462 0.01462 1.88% FabArray::ParallelCopy() 851 0.01449 0.01449 0.01449 1.86% MLMG::mgVcycle_down::4 81 0.01423 0.01423 0.01423 1.83% FabArray::ParallelCopy_nowait() 851 0.01421 0.01421 0.01421 1.83% FabArray::setVal() 1135 0.01348 0.01348 0.01348 1.73% Castro::do_new_sources() 10 0.01346 0.01346 0.01346 1.73% CGSolver::sxay() 1566 0.01319 0.01319 0.01319 1.70% Castro::do_old_sources() 10 0.01306 0.01306 0.01306 1.68% MultiFab::LinComb() 1566 0.01277 0.01277 0.01277 1.64% MLCGSolver::ParallelAllReduce 1495 0.01263 0.01263 0.01263 1.62% MLMG::mgVcycle_up::2 81 0.01225 0.01225 0.01225 1.58% MLMG::mgVcycle_up::1 81 0.01207 0.01207 0.01207 1.55% MLMG::addInterpCorrection() 405 0.01201 0.01201 0.01201 1.55% MLMG::mgVcycle_up::3 81 0.0115 0.0115 0.0115 1.48% amrex::average_down 405 0.01147 0.01147 0.01147 1.47% Castro::expand_state() 10 0.01142 0.01142 0.01142 1.47% MLCellLinOp::defineAuxData() 11 0.01142 0.01142 0.01142 1.47% MLMG::mgVcycle_up::4 81 0.01138 0.01138 0.01138 1.46% MLPoisson::Fapply() 1128 0.01077 0.01077 0.01077 1.39% Castro::post_timestep() 10 0.01039 0.01039 0.01039 1.34% Gravity::fill_multipole_BCs() 11 0.008378 0.008378 0.008378 1.08% Castro::estTimeStep() 21 0.008291 0.008291 0.008291 1.07% Castro::reset_internal_energy(MultiFab) 63 0.007692 0.007692 0.007692 0.99% MLCellLinOp::solutionResidual() 92 0.007087 0.007087 0.007087 0.91% MultiFab::Xpay() 578 0.006143 0.006143 0.006143 0.79% MLMG::computeResidual() 81 0.006093 0.006093 0.006093 0.78% MLCellLinOp::defineBC() 11 0.004995 0.004995 0.004995 0.64% BndryData::define() 11 0.004805 0.004805 0.004805 0.62% MLMG::prepareForSolve() 11 0.004608 0.004608 0.004608 0.59% Castro::computeNewDt() 9 0.003965 0.003965 0.003965 0.51% Amr::InitializeInit() 1 0.003888 0.003888 0.003888 0.50% Amr::defBaseLevel() 1 0.003881 0.003881 0.003881 0.50% Castro::initData() 1 0.003358 0.003358 0.003358 0.43% Castro::construct_new_source() 50 0.003191 0.003191 0.003191 0.41% Castro::construct_new_gravity_source() 10 0.003172 0.003172 0.003172 0.41% Castro::construct_old_source() 50 0.002582 0.002582 0.002582 0.33% Castro::construct_old_gravity_source() 10 0.002558 0.002558 0.002558 0.33% Castro::enforce_speed_limit() 62 0.002207 0.002207 0.002207 0.28% MLMG::ResNormInf() 92 0.001927 0.001927 0.001927 0.25% Castro::apply_source_to_state() 20 0.001831 0.001831 0.001831 0.24% MultiFab::Saxpy() 20 0.001819 0.001819 0.001819 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001703 0.001703 0.001703 0.22% Castro::reset_internal_energy(Fab) 504 0.001606 0.001606 0.001606 0.21% FabArrayBase::getCPC() 1313 0.00151 0.00151 0.00151 0.19% MLMG::getGradSolution() 11 0.001398 0.001398 0.001398 0.18% MLCellLinOp::compGrad() 11 0.001392 0.001392 0.001392 0.18% FabArray::mult() 43 0.001364 0.001364 0.001364 0.18% MLCellLinOp::setLevelBC() 11 0.001347 0.001347 0.001347 0.17% FabArray::setDomainBndry() 41 0.001332 0.001332 0.001332 0.17% Castro::check_for_nan() 20 0.001192 0.001192 0.001192 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.15% MLPoisson::prepareForSolve() 11 0.001107 0.001107 0.001107 0.14% Castro::post_regrid() 1 0.001103 0.001103 0.001103 0.14% MLCellLinOp::prepareForSolve() 11 0.001099 0.001099 0.001099 0.14% MLMG::computeMLResidual() 11 0.001035 0.001035 0.001035 0.13% Gravity::update_max_rhs() 11 0.0008279 0.0008279 0.0008279 0.11% FabArrayBase::getFB() 3974 0.0007576 0.0007576 0.0007576 0.10% Castro::computeInitialDt() 2 0.0006873 0.0006873 0.0006873 0.09% FabArrayBase::CPC::define() 454 0.0006816 0.0006816 0.0006816 0.09% Amr::InitAmr() 1 0.0004675 0.0004675 0.0004675 0.06% MLLinOp::define() 11 0.0004662 0.0004662 0.0004662 0.06% Castro::Castro() 1 0.0004494 0.0004494 0.0004494 0.06% Gravity::swapTimeLevels() 10 0.0004477 0.0004477 0.0004477 0.06% MLLinOp::defineGrids() 11 0.0004422 0.0004422 0.0004422 0.06% MultiFab::Copy() 11 0.0002689 0.0002689 0.0002689 0.03% MultiFab::max() 11 0.0002643 0.0002643 0.0002643 0.03% MLMG::MLResNormInf() 11 0.0002614 0.0002614 0.0002614 0.03% MLMG::MLRhsNormInf() 11 0.0002041 0.0002041 0.0002041 0.03% Castro::buildMetrics() 1 0.0001707 0.0001707 0.0001707 0.02% Castro::finalize_advance() 10 0.0001107 0.0001107 0.0001107 0.01% FabArrayBase::FB::FB() 56 8.399e-05 8.399e-05 8.399e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.351e-05 5.351e-05 5.351e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.288e-05 5.288e-05 5.288e-05 0.01% StateData::define() 4 4.369e-05 4.369e-05 4.369e-05 0.01% makeSFC 55 3.846e-05 3.846e-05 3.846e-05 0.00% Castro::finalize_do_advance() 10 3.648e-05 3.648e-05 3.648e-05 0.00% Castro::swap_state_time_levels() 10 3.534e-05 3.534e-05 3.534e-05 0.00% Castro::enforce_consistent_e() 1 3.438e-05 3.438e-05 3.438e-05 0.00% Amr::writeSmallPlotFile() 1 2.673e-05 2.673e-05 2.673e-05 0.00% Castro::initMFs() 1 2.605e-05 2.605e-05 2.605e-05 0.00% DistributionMapping::Distribute() 56 1.44e-05 1.44e-05 1.44e-05 0.00% Amr::initSubcycle() 1 9.782e-06 9.782e-06 9.782e-06 0.00% Castro::retry_advance_ctu() 10 5.905e-06 5.905e-06 5.905e-06 0.00% MLMG::buildFineMask() 11 5.668e-06 5.668e-06 5.668e-06 0.00% Gravity::set_mass_offset() 11 4.487e-06 4.487e-06 4.487e-06 0.00% AmrLevel::checkPointPost() 3 4.323e-06 4.323e-06 4.323e-06 0.00% Castro::create_source_corrector() 10 4.24e-06 4.24e-06 4.24e-06 0.00% Castro::FluxRegCrseInit 10 4.059e-06 4.059e-06 4.059e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.902e-06 3.902e-06 3.902e-06 0.00% Castro::FluxRegFineAdd() 10 3.253e-06 3.253e-06 3.253e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.216e-06 2.216e-06 2.216e-06 0.00% AmrLevel::checkPointPre() 3 2.105e-06 2.105e-06 2.105e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-38-g3d344ec19655) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-38-g3d344ec19655) initialized Starting run at 17:57:11 UTC on 2022-04-29. Successfully read inputs file ... Castro git describe: 22.04-48-g0ae74bbc5 AMReX git describe: 22.04-38-g3d344ec19 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.432245695 Restart time = 0.04520669 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049474726 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047614071 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061449796 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.06336932 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064150974 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.059884526 seconds Ending run at 17:57:11 UTC on 2022-04-29. Run time = 0.39204768 Run time without initialization = 0.346279732 Average number of zones advanced per microsecond: 3.785 Average number of zones advanced per microsecond per rank: 3.785 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3921 ... 0.3921 ... 0.3921 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0947 0.0947 0.0947 24.16% VisMF::Read() 3 0.03822 0.03822 0.03822 9.75% Amr::writePlotFile() 1 0.0369 0.0369 0.0369 9.41% MLCellLinOp::applyBC() 1946 0.03197 0.03197 0.03197 8.15% MLPoisson::Fsmooth() 1440 0.02597 0.02597 0.02597 6.62% VisMF::Write(FabArray) 1 0.02307 0.02307 0.02307 5.88% StateData::FillBoundary(geom) 160 0.01181 0.01181 0.01181 3.01% MLCGSolver::bicgstab 36 0.009281 0.009281 0.009281 2.37% MultiFab::Dot() 484 0.009017 0.009017 0.009017 2.30% Castro::normalize_species() 30 0.008508 0.008508 0.008508 2.17% Castro::computeTemp() 30 0.007664 0.007664 0.007664 1.95% FabArray::setVal() 537 0.006402 0.006402 0.006402 1.63% FillBoundary_nowait() 1766 0.00628 0.00628 0.00628 1.60% Castro::enforce_min_density() 30 0.006175 0.006175 0.006175 1.57% FabArray::ParallelCopy_nowait() 380 0.006015 0.006015 0.006015 1.53% MultiFab::LinComb() 690 0.005428 0.005428 0.005428 1.38% MLCellLinOp::defineAuxData() 6 0.005362 0.005362 0.005362 1.37% StateDataPhysBCFunct::() 20 0.005245 0.005245 0.005245 1.34% Gravity::fill_multipole_BCs() 6 0.004884 0.004884 0.004884 1.25% MLPoisson::Fapply() 500 0.004645 0.004645 0.004645 1.18% Castro::estTimeStep() 10 0.0035 0.0035 0.0035 0.89% MLMG::addInterpCorrection() 180 0.002912 0.002912 0.002912 0.74% Amr::restart() 1 0.002876 0.002876 0.002876 0.73% amrex::average_down 180 0.002677 0.002677 0.002677 0.68% MultiFab::Xpay() 258 0.002652 0.002652 0.002652 0.68% Castro::enforce_speed_limit() 30 0.002448 0.002448 0.002448 0.62% Castro::do_advance_ctu() 5 0.002233 0.002233 0.002233 0.57% BndryData::define() 6 0.002068 0.002068 0.002068 0.53% Castro::reset_internal_energy(MultiFab) 30 0.002016 0.002016 0.002016 0.51% Castro::construct_new_gravity_source() 5 0.001549 0.001549 0.001549 0.40% Castro::construct_old_gravity_source() 5 0.001323 0.001323 0.001323 0.34% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009238 0.0009238 0.0009238 0.24% MultiFab::Saxpy() 10 0.0009201 0.0009201 0.0009201 0.23% Gravity::get_old_grav_vector() 5 0.000893 0.000893 0.000893 0.23% Castro::expand_state() 5 0.0008717 0.0008717 0.0008717 0.22% Gravity::get_new_grav_vector() 5 0.0008665 0.0008665 0.0008665 0.22% MLMG::ResNormInf() 42 0.0008498 0.0008498 0.0008498 0.22% Castro::reset_internal_energy(Fab) 240 0.0007384 0.0007384 0.0007384 0.19% MLCellLinOp::setLevelBC() 6 0.000737 0.000737 0.000737 0.19% MLMG::oneIter() 36 0.0007282 0.0007282 0.0007282 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007107 0.0007107 0.0007107 0.18% FabArray::mult() 22 0.0006543 0.0006543 0.0006543 0.17% FabArray::setDomainBndry() 20 0.0006505 0.0006505 0.0006505 0.17% MLCellLinOp::prepareForSolve() 6 0.0005925 0.0005925 0.0005925 0.15% MultiFab::contains_nan() 10 0.0005868 0.0005868 0.0005868 0.15% MLCellLinOp::compGrad() 6 0.0004838 0.0004838 0.0004838 0.12% MLCellLinOp::smooth() 720 0.0004631 0.0004631 0.0004631 0.12% MLMG::prepareForSolve() 6 0.0004279 0.0004279 0.0004279 0.11% FabArrayBase::CPC::define() 244 0.0004186 0.0004186 0.0004186 0.11% Amr::InitAmr() 1 0.0003854 0.0003854 0.0003854 0.10% FabArray::FillBoundary() 1766 0.0003796 0.0003796 0.0003796 0.10% FabArrayBase::getCPC() 632 0.0003789 0.0003789 0.0003789 0.10% main() 1 0.0002736 0.0002736 0.0002736 0.07% FabArrayBase::getFB() 1766 0.0002693 0.0002693 0.0002693 0.07% Gravity::update_max_rhs() 6 0.0002246 0.0002246 0.0002246 0.06% MLCellLinOp::apply() 500 0.0002214 0.0002214 0.0002214 0.06% Gravity::solve_for_phi() 5 0.0002095 0.0002095 0.0002095 0.05% CGSolver::sxay() 690 0.000207 0.000207 0.000207 0.05% MLLinOp::defineGrids() 6 0.0001819 0.0001819 0.0001819 0.05% Castro::subcycle_advance_ctu() 5 0.0001737 0.0001737 0.0001737 0.04% MLMG::mgVcycle() 36 0.0001594 0.0001594 0.0001594 0.04% MLCGSolver::ParallelAllReduce 659 0.0001466 0.0001466 0.0001466 0.04% MultiFab::Copy() 6 0.000144 0.000144 0.000144 0.04% FabArray::ParallelCopy() 380 0.0001382 0.0001382 0.0001382 0.04% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.03% FillPatchIterator::Initialize 20 0.0001213 0.0001213 0.0001213 0.03% MLMG::MLRhsNormInf() 6 0.0001062 0.0001062 0.0001062 0.03% Castro::construct_new_gravity() 5 0.0001049 0.0001049 0.0001049 0.03% Amr::coarseTimeStep() 5 0.0001044 0.0001044 0.0001044 0.03% MLCellLinOp::defineBC() 6 0.0001019 0.0001019 0.0001019 0.03% MLCellLinOp::correctionResidual() 216 9.843e-05 9.843e-05 9.843e-05 0.03% Amr::timeStep() 5 9.466e-05 9.466e-05 9.466e-05 0.02% Castro::advance() 5 7.665e-05 7.665e-05 7.665e-05 0.02% AmrLevel::restart() 1 7.633e-05 7.633e-05 7.633e-05 0.02% StateData::restartDoit() 4 7.136e-05 7.136e-05 7.136e-05 0.02% FabArrayBase::FB::FB() 26 6.011e-05 6.011e-05 6.011e-05 0.02% MLMG:computeResOfCorrection() 180 5.511e-05 5.511e-05 5.511e-05 0.01% MLMG::actualBottomSolve() 36 4.724e-05 4.724e-05 4.724e-05 0.01% Castro::create_source_corrector() 5 4.384e-05 4.384e-05 4.384e-05 0.01% Castro::initialize_do_advance() 5 4.181e-05 4.181e-05 4.181e-05 0.01% MLMG::solve() 6 3.952e-05 3.952e-05 3.952e-05 0.01% Castro::clean_state() 30 3.789e-05 3.789e-05 3.789e-05 0.01% Castro::construct_old_source() 25 3.597e-05 3.597e-05 3.597e-05 0.01% MLMG::mgVcycle_down::0 36 3.496e-05 3.496e-05 3.496e-05 0.01% Castro::initialize_advance() 5 3.486e-05 3.486e-05 3.486e-05 0.01% MLMG::mgVcycle_down::1 36 3.371e-05 3.371e-05 3.371e-05 0.01% MLMG::mgVcycle_down::2 36 3.053e-05 3.053e-05 3.053e-05 0.01% Castro::buildMetrics() 1 2.992e-05 2.992e-05 2.992e-05 0.01% MLMG::mgVcycle_down::4 36 2.929e-05 2.929e-05 2.929e-05 0.01% Castro::post_restart() 1 2.908e-05 2.908e-05 2.908e-05 0.01% Gravity::actual_multilevel_solve() 1 2.903e-05 2.903e-05 2.903e-05 0.01% MLMG::mgVcycle_down::3 36 2.901e-05 2.901e-05 2.901e-05 0.01% Castro::initMFs() 1 2.651e-05 2.651e-05 2.651e-05 0.01% Amr::writeSmallPlotFile() 1 2.648e-05 2.648e-05 2.648e-05 0.01% Castro::finalize_advance() 5 2.536e-05 2.536e-05 2.536e-05 0.01% MLMG::mgVcycle_up::4 36 2.516e-05 2.516e-05 2.516e-05 0.01% Castro::swap_state_time_levels() 5 2.46e-05 2.46e-05 2.46e-05 0.01% MLCellLinOp::solutionResidual() 42 2.327e-05 2.327e-05 2.327e-05 0.01% MLMG::mgVcycle_up::0 36 2.236e-05 2.236e-05 2.236e-05 0.01% MLMG::mgVcycle_up::3 36 2.143e-05 2.143e-05 2.143e-05 0.01% MLMG::mgVcycle_up::2 36 2.12e-05 2.12e-05 2.12e-05 0.01% MLLinOp::define() 6 2.027e-05 2.027e-05 2.027e-05 0.01% MLMG::mgVcycle_up::1 36 1.945e-05 1.945e-05 1.945e-05 0.00% Castro::finalize_do_advance() 5 1.935e-05 1.935e-05 1.935e-05 0.00% MLMG::mgVcycle_bottom 36 1.603e-05 1.603e-05 1.603e-05 0.00% Castro::construct_new_source() 25 1.598e-05 1.598e-05 1.598e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.56e-05 1.56e-05 1.56e-05 0.00% makeSFC 30 1.48e-05 1.48e-05 1.48e-05 0.00% FillPatchSingleLevel 20 1.476e-05 1.476e-05 1.476e-05 0.00% MLMG::computeResidual() 36 1.471e-05 1.471e-05 1.471e-05 0.00% MLPoisson::define() 6 1.429e-05 1.429e-05 1.429e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.293e-05 1.293e-05 1.293e-05 0.00% Amr::initSubcycle() 1 9.019e-06 9.019e-06 9.019e-06 0.00% DistributionMapping::Distribute() 31 9.01e-06 9.01e-06 9.01e-06 0.00% Castro::do_new_sources() 5 8.429e-06 8.429e-06 8.429e-06 0.00% Castro::do_old_sources() 5 7.977e-06 7.977e-06 7.977e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.568e-06 7.568e-06 7.568e-06 0.00% Castro::check_for_nan() 10 5.792e-06 5.792e-06 5.792e-06 0.00% Castro::apply_source_to_state() 10 5.689e-06 5.689e-06 5.689e-06 0.00% Castro::construct_old_gravity() 5 5.541e-06 5.541e-06 5.541e-06 0.00% MLPoisson::prepareForSolve() 6 4.762e-06 4.762e-06 4.762e-06 0.00% Castro::post_timestep() 5 4.31e-06 4.31e-06 4.31e-06 0.00% Gravity::swapTimeLevels() 5 3.886e-06 3.886e-06 3.886e-06 0.00% MLMG::buildFineMask() 6 3.256e-06 3.256e-06 3.256e-06 0.00% Castro::computeNewDt() 5 3.157e-06 3.157e-06 3.157e-06 0.00% MLMG::computeMLResidual() 6 2.97e-06 2.97e-06 2.97e-06 0.00% MLMG::getGradSolution() 6 2.851e-06 2.851e-06 2.851e-06 0.00% Gravity::set_mass_offset() 6 2.794e-06 2.794e-06 2.794e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.53e-06 2.53e-06 2.53e-06 0.00% MLMG::MLResNormInf() 6 2.236e-06 2.236e-06 2.236e-06 0.00% Castro::retry_advance_ctu() 5 1.792e-06 1.792e-06 1.792e-06 0.00% Castro::FluxRegCrseInit 5 1.716e-06 1.716e-06 1.716e-06 0.00% Castro::FluxRegFineAdd() 5 1.349e-06 1.349e-06 1.349e-06 0.00% AmrLevel::AmrLevel() 1 1.151e-06 1.151e-06 1.151e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.13e-06 1.13e-06 1.13e-06 0.00% Amr::init() 1 9.39e-07 9.39e-07 9.39e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3921 0.3921 0.3921 100.00% Amr::coarseTimeStep() 5 0.2862 0.2862 0.2862 72.98% Amr::timeStep() 5 0.2841 0.2841 0.2841 72.45% Castro::advance() 5 0.2782 0.2782 0.2782 70.96% Castro::subcycle_advance_ctu() 5 0.2711 0.2711 0.2711 69.14% Castro::do_advance_ctu() 5 0.2709 0.2709 0.2709 69.10% Castro::construct_new_gravity() 5 0.135 0.135 0.135 34.43% Gravity::solve_phi_with_mlmg() 6 0.1307 0.1307 0.1307 33.34% Gravity::solve_for_phi() 5 0.1274 0.1274 0.1274 32.49% Gravity::actual_solve_with_mlmg() 6 0.1257 0.1257 0.1257 32.07% MLMG::solve() 6 0.1144 0.1144 0.1144 29.19% MLMG::oneIter() 36 0.1078 0.1078 0.1078 27.49% MLMG::mgVcycle() 36 0.107 0.107 0.107 27.30% Castro::construct_ctu_hydro_source() 5 0.09472 0.09472 0.09472 24.16% Amr::writePlotFile() 1 0.05997 0.05997 0.05997 15.30% MLCellLinOp::smooth() 720 0.05522 0.05522 0.05522 14.08% Amr::init() 1 0.04524 0.04524 0.04524 11.54% Amr::restart() 1 0.04524 0.04524 0.04524 11.54% MLCellLinOp::applyBC() 1946 0.03896 0.03896 0.03896 9.94% AmrLevel::restart() 1 0.03844 0.03844 0.03844 9.80% StateData::restartDoit() 4 0.03836 0.03836 0.03836 9.78% VisMF::Read() 3 0.03822 0.03822 0.03822 9.75% MLMG::mgVcycle_bottom 36 0.03221 0.03221 0.03221 8.21% MLMG::actualBottomSolve() 36 0.03219 0.03219 0.03219 8.21% MLCGSolver::bicgstab 36 0.03187 0.03187 0.03187 8.13% Castro::clean_state() 30 0.02759 0.02759 0.02759 7.04% MLPoisson::Fsmooth() 1440 0.02597 0.02597 0.02597 6.62% VisMF::Write(FabArray) 1 0.02307 0.02307 0.02307 5.88% FillPatchIterator::Initialize 20 0.01985 0.01985 0.01985 5.06% FillPatchSingleLevel 20 0.01908 0.01908 0.01908 4.87% StateDataPhysBCFunct::() 20 0.01706 0.01706 0.01706 4.35% MLMG::mgVcycle_down::0 36 0.01502 0.01502 0.01502 3.83% MLCellLinOp::apply() 500 0.01478 0.01478 0.01478 3.77% MLMG::mgVcycle_up::0 36 0.0128 0.0128 0.0128 3.27% StateData::FillBoundary(geom) 160 0.01181 0.01181 0.01181 3.01% Castro::initialize_do_advance() 5 0.01077 0.01077 0.01077 2.75% Castro::computeTemp() 30 0.01042 0.01042 0.01042 2.66% MLPoisson::define() 6 0.0091 0.0091 0.0091 2.32% MultiFab::Dot() 484 0.009017 0.009017 0.009017 2.30% MLCellLinOp::correctionResidual() 216 0.008576 0.008576 0.008576 2.19% Castro::normalize_species() 30 0.008508 0.008508 0.008508 2.17% Gravity::get_new_grav_vector() 5 0.007534 0.007534 0.007534 1.92% MLMG:computeResOfCorrection() 180 0.007417 0.007417 0.007417 1.89% Castro::construct_old_gravity() 5 0.0074 0.0074 0.0074 1.89% Gravity::get_old_grav_vector() 5 0.007395 0.007395 0.007395 1.89% MLMG::mgVcycle_down::1 36 0.007076 0.007076 0.007076 1.80% Castro::initialize_advance() 5 0.007027 0.007027 0.007027 1.79% FabArray::FillBoundary() 1766 0.006989 0.006989 0.006989 1.78% MLMG::mgVcycle_down::2 36 0.006821 0.006821 0.006821 1.74% FillBoundary_nowait() 1766 0.00661 0.00661 0.00661 1.69% FabArray::ParallelCopy() 380 0.006554 0.006554 0.006554 1.67% MLMG::mgVcycle_down::3 36 0.006423 0.006423 0.006423 1.64% FabArray::ParallelCopy_nowait() 380 0.006416 0.006416 0.006416 1.64% FabArray::setVal() 537 0.006402 0.006402 0.006402 1.63% Castro::enforce_min_density() 30 0.006175 0.006175 0.006175 1.57% MLMG::mgVcycle_down::4 36 0.006141 0.006141 0.006141 1.57% Castro::do_old_sources() 5 0.006134 0.006134 0.006134 1.56% MLCellLinOp::defineAuxData() 6 0.006121 0.006121 0.006121 1.56% Castro::do_new_sources() 5 0.005889 0.005889 0.005889 1.50% Castro::post_timestep() 5 0.005752 0.005752 0.005752 1.47% CGSolver::sxay() 690 0.005635 0.005635 0.005635 1.44% Castro::expand_state() 5 0.005627 0.005627 0.005627 1.44% MLCGSolver::ParallelAllReduce 659 0.005439 0.005439 0.005439 1.39% MultiFab::LinComb() 690 0.005428 0.005428 0.005428 1.38% MLMG::mgVcycle_up::2 36 0.005287 0.005287 0.005287 1.35% MLMG::mgVcycle_up::1 36 0.005196 0.005196 0.005196 1.33% MLMG::addInterpCorrection() 180 0.005169 0.005169 0.005169 1.32% MLMG::mgVcycle_up::3 36 0.004993 0.004993 0.004993 1.27% amrex::average_down 180 0.004962 0.004962 0.004962 1.27% MLMG::mgVcycle_up::4 36 0.004916 0.004916 0.004916 1.25% Gravity::fill_multipole_BCs() 6 0.004884 0.004884 0.004884 1.25% MLPoisson::Fapply() 500 0.004645 0.004645 0.004645 1.18% Castro::post_restart() 1 0.003733 0.003733 0.003733 0.95% Gravity::multilevel_solve_for_new_phi() 1 0.003612 0.003612 0.003612 0.92% Gravity::actual_multilevel_solve() 1 0.003597 0.003597 0.003597 0.92% Castro::estTimeStep() 10 0.0035 0.0035 0.0035 0.89% MLCellLinOp::solutionResidual() 42 0.003166 0.003166 0.003166 0.81% Castro::reset_internal_energy(MultiFab) 30 0.002755 0.002755 0.002755 0.70% MLCellLinOp::defineBC() 6 0.002731 0.002731 0.002731 0.70% MultiFab::Xpay() 258 0.002652 0.002652 0.002652 0.68% BndryData::define() 6 0.002629 0.002629 0.002629 0.67% MLMG::computeResidual() 36 0.002623 0.002623 0.002623 0.67% MLMG::prepareForSolve() 6 0.002477 0.002477 0.002477 0.63% Castro::enforce_speed_limit() 30 0.002448 0.002448 0.002448 0.62% Castro::computeNewDt() 5 0.001985 0.001985 0.001985 0.51% Castro::construct_new_source() 25 0.001565 0.001565 0.001565 0.40% Castro::construct_new_gravity_source() 5 0.001549 0.001549 0.001549 0.40% Castro::construct_old_source() 25 0.001359 0.001359 0.001359 0.35% Castro::construct_old_gravity_source() 5 0.001323 0.001323 0.001323 0.34% Castro::apply_source_to_state() 10 0.0009258 0.0009258 0.0009258 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009238 0.0009238 0.0009238 0.24% MultiFab::Saxpy() 10 0.0009201 0.0009201 0.0009201 0.23% MLMG::ResNormInf() 42 0.0008498 0.0008498 0.0008498 0.22% FabArrayBase::getCPC() 632 0.0007975 0.0007975 0.0007975 0.20% MLMG::getGradSolution() 6 0.0007441 0.0007441 0.0007441 0.19% MLCellLinOp::compGrad() 6 0.0007412 0.0007412 0.0007412 0.19% Castro::reset_internal_energy(Fab) 240 0.0007384 0.0007384 0.0007384 0.19% MLCellLinOp::setLevelBC() 6 0.000737 0.000737 0.000737 0.19% FabArray::mult() 22 0.0006543 0.0006543 0.0006543 0.17% FabArray::setDomainBndry() 20 0.0006505 0.0006505 0.0006505 0.17% MLPoisson::prepareForSolve() 6 0.0005973 0.0005973 0.0005973 0.15% Castro::check_for_nan() 10 0.0005926 0.0005926 0.0005926 0.15% MLCellLinOp::prepareForSolve() 6 0.0005925 0.0005925 0.0005925 0.15% MultiFab::contains_nan() 10 0.0005868 0.0005868 0.0005868 0.15% MLMG::computeMLResidual() 6 0.0005599 0.0005599 0.0005599 0.14% Gravity::update_max_rhs() 6 0.000437 0.000437 0.000437 0.11% FabArrayBase::CPC::define() 244 0.0004186 0.0004186 0.0004186 0.11% Amr::InitAmr() 1 0.0003944 0.0003944 0.0003944 0.10% FabArrayBase::getFB() 1766 0.0003294 0.0003294 0.0003294 0.08% MLLinOp::define() 6 0.0002333 0.0002333 0.0002333 0.06% Gravity::swapTimeLevels() 5 0.000227 0.000227 0.000227 0.06% MLLinOp::defineGrids() 6 0.000213 0.000213 0.000213 0.05% Castro::buildMetrics() 1 0.0001668 0.0001668 0.0001668 0.04% MultiFab::Copy() 6 0.000144 0.000144 0.000144 0.04% MLMG::MLResNormInf() 6 0.0001339 0.0001339 0.0001339 0.03% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.03% MLMG::MLRhsNormInf() 6 0.0001062 0.0001062 0.0001062 0.03% FabArrayBase::FB::FB() 26 6.011e-05 6.011e-05 6.011e-05 0.02% Castro::create_source_corrector() 5 4.384e-05 4.384e-05 4.384e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.997e-05 2.997e-05 2.997e-05 0.01% Castro::finalize_advance() 5 2.843e-05 2.843e-05 2.843e-05 0.01% Castro::initMFs() 1 2.651e-05 2.651e-05 2.651e-05 0.01% Amr::writeSmallPlotFile() 1 2.648e-05 2.648e-05 2.648e-05 0.01% Castro::swap_state_time_levels() 5 2.46e-05 2.46e-05 2.46e-05 0.01% makeSFC 30 2.24e-05 2.24e-05 2.24e-05 0.01% Castro::finalize_do_advance() 5 1.935e-05 1.935e-05 1.935e-05 0.00% Amr::initSubcycle() 1 9.019e-06 9.019e-06 9.019e-06 0.00% DistributionMapping::Distribute() 31 9.01e-06 9.01e-06 9.01e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.934e-06 3.934e-06 3.934e-06 0.00% MLMG::buildFineMask() 6 3.256e-06 3.256e-06 3.256e-06 0.00% Gravity::set_mass_offset() 6 2.794e-06 2.794e-06 2.794e-06 0.00% Castro::retry_advance_ctu() 5 1.792e-06 1.792e-06 1.792e-06 0.00% Castro::FluxRegCrseInit 5 1.716e-06 1.716e-06 1.716e-06 0.00% Castro::FluxRegFineAdd() 5 1.349e-06 1.349e-06 1.349e-06 0.00% AmrLevel::AmrLevel() 1 1.151e-06 1.151e-06 1.151e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.13e-06 1.13e-06 1.13e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-38-g3d344ec19655) finalized