Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-18-g3d29fd7d0e81) initialized Starting run at 08:29:13 UTC on 2022-08-30. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-18-g3d29fd7d0 Microphysics git describe: 22.08-6-ge3a6cb11 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051340273 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029682044 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048299562 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052278463 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050052202 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.061221276 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.072045975 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047539337 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.07864017 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.066661096 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.055140434 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.048534827 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062807246 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047504923 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029267631 seconds Ending run at 08:29:14 UTC on 2022-08-30. Run time = 0.854104626 Run time without initialization = 0.720628628 Average number of zones advanced per microsecond: 3.638 Average number of zones advanced per microsecond per rank: 3.638 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8541 ... 0.8541 ... 0.8541 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2056 0.2056 0.2056 24.07% VisMF::Write(FabArray) 11 0.1974 0.1974 0.1974 23.11% MLCellLinOp::applyBC() 4433 0.07864 0.07864 0.07864 9.21% MLPoisson::Fsmooth() 3280 0.06321 0.06321 0.06321 7.40% StateData::FillBoundary(geom) 328 0.02426 0.02426 0.02426 2.84% MLCGSolver::bicgstab 82 0.02346 0.02346 0.02346 2.75% MultiFab::Dot() 1114 0.02181 0.02181 0.02181 2.55% StateDataPhysBCFunct::() 41 0.01804 0.01804 0.01804 2.11% Castro::normalize_species() 62 0.01574 0.01574 0.01574 1.84% Castro::computeTemp() 63 0.0142 0.0142 0.0142 1.66% FillBoundary_nowait() 4023 0.01415 0.01415 0.01415 1.66% MultiFab::LinComb() 1586 0.01405 0.01405 0.01405 1.64% FabArray::setVal() 1144 0.01402 0.01402 0.01402 1.64% FabArray::ParallelCopy_nowait() 861 0.01277 0.01277 0.01277 1.50% MLPoisson::Fapply() 1142 0.01154 0.01154 0.01154 1.35% MLCellLinOp::defineAuxData() 11 0.01121 0.01121 0.01121 1.31% Gravity::fill_multipole_BCs() 11 0.01003 0.01003 0.01003 1.17% Castro::enforce_min_density() 62 0.009301 0.009301 0.009301 1.09% MLMG::addInterpCorrection() 410 0.007599 0.007599 0.007599 0.89% amrex::average_down 410 0.006782 0.006782 0.006782 0.79% Castro::estTimeStep() 21 0.006779 0.006779 0.006779 0.79% MultiFab::Xpay() 585 0.00651 0.00651 0.00651 0.76% Amr::checkPoint() 3 0.005139 0.005139 0.005139 0.60% Castro::do_advance_ctu() 10 0.004911 0.004911 0.004911 0.57% Castro::reset_internal_energy(MultiFab) 63 0.004891 0.004891 0.004891 0.57% BndryData::define() 11 0.003681 0.003681 0.003681 0.43% Castro::construct_new_gravity_source() 10 0.003429 0.003429 0.003429 0.40% Castro::construct_old_gravity_source() 10 0.002892 0.002892 0.002892 0.34% Amr::writePlotFile() 2 0.002818 0.002818 0.002818 0.33% MLMG::ResNormInf() 93 0.002121 0.002121 0.002121 0.25% Gravity::get_new_grav_vector() 11 0.001917 0.001917 0.001917 0.22% MultiFab::Saxpy() 20 0.001818 0.001818 0.001818 0.21% Gravity::get_old_grav_vector() 10 0.001744 0.001744 0.001744 0.20% Castro::expand_state() 10 0.00173 0.00173 0.00173 0.20% MultiFab::Add() 82 0.001663 0.001663 0.001663 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00162 0.00162 0.00162 0.19% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.18% Gravity::actual_solve_with_mlmg() 11 0.001433 0.001433 0.001433 0.17% Castro::reset_internal_energy(Fab) 504 0.00143 0.00143 0.00143 0.17% FabArray::mult() 43 0.00132 0.00132 0.00132 0.15% Castro::initData() 1 0.0013 0.0013 0.0013 0.15% FabArray::setDomainBndry() 41 0.001286 0.001286 0.001286 0.15% MLCellLinOp::smooth() 1640 0.001252 0.001252 0.001252 0.15% MLMG::prepareForSolve() 11 0.001221 0.001221 0.001221 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% MLCellLinOp::prepareForSolve() 11 0.001148 0.001148 0.001148 0.13% Castro::enforce_speed_limit() 62 0.001009 0.001009 0.001009 0.12% MLCellLinOp::compGrad() 11 0.0009134 0.0009134 0.0009134 0.11% FabArray::FillBoundary() 4023 0.0008237 0.0008237 0.0008237 0.10% FabArrayBase::getCPC() 1323 0.00076 0.00076 0.00076 0.09% FabArrayBase::CPC::define() 454 0.000654 0.000654 0.000654 0.08% FabArrayBase::getFB() 4023 0.0005929 0.0005929 0.0005929 0.07% MLCellLinOp::apply() 1142 0.000514 0.000514 0.000514 0.06% Amr::InitAmr() 1 0.0005061 0.0005061 0.0005061 0.06% Gravity::solve_for_phi() 10 0.0004376 0.0004376 0.0004376 0.05% Gravity::update_max_rhs() 11 0.0004112 0.0004112 0.0004112 0.05% CGSolver::sxay() 1586 0.0003938 0.0003938 0.0003938 0.05% Amr::coarseTimeStep() 10 0.0003438 0.0003438 0.0003438 0.04% MultiFab::Copy() 11 0.0003219 0.0003219 0.0003219 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002932 0.0002932 0.0002932 0.03% FillPatchIterator::Initialize 41 0.0002854 0.0002854 0.0002854 0.03% MLCellLinOp::defineBC() 11 0.0002798 0.0002798 0.0002798 0.03% FabArray::ParallelCopy() 861 0.0002673 0.0002673 0.0002673 0.03% main() 1 0.0002571 0.0002571 0.0002571 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% MLCellLinOp::correctionResidual() 492 0.0002323 0.0002323 0.0002323 0.03% MLMG::MLRhsNormInf() 11 0.0002155 0.0002155 0.0002155 0.03% Amr::timeStep() 10 0.0002113 0.0002113 0.0002113 0.02% Castro::construct_new_gravity() 10 0.0002007 0.0002007 0.0002007 0.02% MLMG::mgVcycle() 82 0.0001985 0.0001985 0.0001985 0.02% Castro::subcycle_advance_ctu() 10 0.0001693 0.0001693 0.0001693 0.02% MLMG:computeResOfCorrection() 410 0.0001555 0.0001555 0.0001555 0.02% Castro::create_source_corrector() 10 0.0001551 0.0001551 0.0001551 0.02% StateData::checkPoint() 12 0.0001355 0.0001355 0.0001355 0.02% MLLinOp::defineGrids() 11 0.0001338 0.0001338 0.0001338 0.02% MLMG::mgVcycle_down::0 82 0.0001102 0.0001102 0.0001102 0.01% MLMG::mgVcycle_down::1 82 9.413e-05 9.413e-05 9.413e-05 0.01% Castro::Castro() 1 9.199e-05 9.199e-05 9.199e-05 0.01% MLMG::mgVcycle_down::2 82 9.019e-05 9.019e-05 9.019e-05 0.01% MLMG::mgVcycle_down::3 82 8.77e-05 8.77e-05 8.77e-05 0.01% FabArrayBase::FB::FB() 56 8.491e-05 8.491e-05 8.491e-05 0.01% Castro::initialize_advance() 10 8.327e-05 8.327e-05 8.327e-05 0.01% MLMG::mgVcycle_down::4 82 8.242e-05 8.242e-05 8.242e-05 0.01% Castro::clean_state() 62 7.863e-05 7.863e-05 7.863e-05 0.01% MLMG::actualBottomSolve() 82 7.617e-05 7.617e-05 7.617e-05 0.01% AmrLevel::checkPoint() 3 7.423e-05 7.423e-05 7.423e-05 0.01% MLMG::mgVcycle_up::4 82 7.062e-05 7.062e-05 7.062e-05 0.01% MLMG::solve() 11 6.857e-05 6.857e-05 6.857e-05 0.01% Castro::initialize_do_advance() 10 6.365e-05 6.365e-05 6.365e-05 0.01% MLMG::mgVcycle_up::0 82 5.995e-05 5.995e-05 5.995e-05 0.01% MLMG::oneIter() 82 5.916e-05 5.916e-05 5.916e-05 0.01% Castro::construct_new_source() 50 5.856e-05 5.856e-05 5.856e-05 0.01% MLMG::mgVcycle_up::1 82 5.776e-05 5.776e-05 5.776e-05 0.01% MLMG::mgVcycle_up::2 82 5.7e-05 5.7e-05 5.7e-05 0.01% MLMG::mgVcycle_up::3 82 5.685e-05 5.685e-05 5.685e-05 0.01% Castro::advance() 10 5.614e-05 5.614e-05 5.614e-05 0.01% MLCellLinOp::solutionResidual() 93 5.31e-05 5.31e-05 5.31e-05 0.01% Castro::finalize_advance() 10 5.225e-05 5.225e-05 5.225e-05 0.01% StateData::define() 4 4.459e-05 4.459e-05 4.459e-05 0.01% Gravity::actual_multilevel_solve() 1 3.959e-05 3.959e-05 3.959e-05 0.00% Castro::swap_state_time_levels() 10 3.954e-05 3.954e-05 3.954e-05 0.00% MLMG::computeResidual() 82 3.895e-05 3.895e-05 3.895e-05 0.00% Castro::enforce_consistent_e() 1 3.401e-05 3.401e-05 3.401e-05 0.00% MLMG::mgVcycle_bottom 82 3.3e-05 3.3e-05 3.3e-05 0.00% Castro::finalize_do_advance() 10 3.181e-05 3.181e-05 3.181e-05 0.00% MLPoisson::define() 11 3.011e-05 3.011e-05 3.011e-05 0.00% Amr::defBaseLevel() 1 2.944e-05 2.944e-05 2.944e-05 0.00% FillPatchSingleLevel 41 2.758e-05 2.758e-05 2.758e-05 0.00% Castro::initMFs() 1 2.653e-05 2.653e-05 2.653e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.615e-05 2.615e-05 2.615e-05 0.00% Amr::writeSmallPlotFile() 1 2.581e-05 2.581e-05 2.581e-05 0.00% makeSFC 55 2.502e-05 2.502e-05 2.502e-05 0.00% Castro::buildMetrics() 1 2.353e-05 2.353e-05 2.353e-05 0.00% MLLinOp::define() 11 2.315e-05 2.315e-05 2.315e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.266e-05 2.266e-05 2.266e-05 0.00% Amr::FinalizeInit() 1 2.207e-05 2.207e-05 2.207e-05 0.00% Castro::construct_old_source() 50 1.895e-05 1.895e-05 1.895e-05 0.00% Castro::do_new_sources() 10 1.751e-05 1.751e-05 1.751e-05 0.00% Castro::do_old_sources() 10 1.542e-05 1.542e-05 1.542e-05 0.00% DistributionMapping::Distribute() 56 1.452e-05 1.452e-05 1.452e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.334e-05 1.334e-05 1.334e-05 0.00% Castro::check_for_nan() 20 1.241e-05 1.241e-05 1.241e-05 0.00% Castro::apply_source_to_state() 20 1.09e-05 1.09e-05 1.09e-05 0.00% Castro::construct_old_gravity() 10 1.002e-05 1.002e-05 1.002e-05 0.00% MLMG::computeMLResidual() 11 9.813e-06 9.813e-06 9.813e-06 0.00% MLPoisson::prepareForSolve() 11 9.293e-06 9.293e-06 9.293e-06 0.00% Gravity::swapTimeLevels() 10 9.191e-06 9.191e-06 9.191e-06 0.00% Amr::initSubcycle() 1 8.985e-06 8.985e-06 8.985e-06 0.00% Castro::post_timestep() 10 8.872e-06 8.872e-06 8.872e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.808e-06 7.808e-06 7.808e-06 0.00% Amr::InitializeInit() 1 7.214e-06 7.214e-06 7.214e-06 0.00% Castro::computeNewDt() 9 6.568e-06 6.568e-06 6.568e-06 0.00% MLMG::getGradSolution() 11 6.251e-06 6.251e-06 6.251e-06 0.00% AmrLevel::checkPointPost() 3 5.417e-06 5.417e-06 5.417e-06 0.00% Castro::post_init() 1 4.006e-06 4.006e-06 4.006e-06 0.00% Castro::retry_advance_ctu() 10 3.955e-06 3.955e-06 3.955e-06 0.00% MLMG::MLResNormInf() 11 3.747e-06 3.747e-06 3.747e-06 0.00% Gravity::set_mass_offset() 11 3.741e-06 3.741e-06 3.741e-06 0.00% Castro::FluxRegCrseInit 10 3.429e-06 3.429e-06 3.429e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.809e-06 2.809e-06 2.809e-06 0.00% Castro::computeInitialDt() 2 2.715e-06 2.715e-06 2.715e-06 0.00% Amr::init() 1 2.681e-06 2.681e-06 2.681e-06 0.00% Castro::FluxRegFineAdd() 10 2.271e-06 2.271e-06 2.271e-06 0.00% AmrLevel::checkPointPre() 3 1.96e-06 1.96e-06 1.96e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.731e-06 1.731e-06 1.731e-06 0.00% Castro::post_regrid() 1 1.165e-06 1.165e-06 1.165e-06 0.00% Amr::initialInit() 1 1.023e-06 1.023e-06 1.023e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8541 0.8541 0.8541 100.00% Amr::coarseTimeStep() 10 0.6911 0.6911 0.6911 80.92% Amr::timeStep() 10 0.5918 0.5918 0.5918 69.28% Castro::advance() 10 0.584 0.584 0.584 68.37% Castro::subcycle_advance_ctu() 10 0.5725 0.5725 0.5725 67.03% Castro::do_advance_ctu() 10 0.5724 0.5724 0.5724 67.01% Gravity::solve_phi_with_mlmg() 11 0.3111 0.3111 0.3111 36.42% Gravity::actual_solve_with_mlmg() 11 0.3008 0.3008 0.3008 35.22% Castro::construct_new_gravity() 10 0.2828 0.2828 0.2828 33.11% MLMG::solve() 11 0.2788 0.2788 0.2788 32.64% Gravity::solve_for_phi() 10 0.2675 0.2675 0.2675 31.32% MLMG::oneIter() 82 0.264 0.264 0.264 30.91% MLMG::mgVcycle() 82 0.2623 0.2623 0.2623 30.70% Castro::construct_ctu_hydro_source() 10 0.2056 0.2056 0.2056 24.07% VisMF::Write(FabArray) 11 0.1974 0.1974 0.1974 23.11% Amr::checkPoint() 3 0.1465 0.1465 0.1465 17.15% AmrLevel::checkPoint() 3 0.1414 0.1414 0.1414 16.55% StateData::checkPoint() 12 0.1413 0.1413 0.1413 16.54% MLCellLinOp::smooth() 1640 0.1346 0.1346 0.1346 15.76% Amr::init() 1 0.1329 0.1329 0.1329 15.55% MLCellLinOp::applyBC() 4433 0.09429 0.09429 0.09429 11.04% MLMG::mgVcycle_bottom 82 0.08022 0.08022 0.08022 9.39% MLMG::actualBottomSolve() 82 0.08018 0.08018 0.08018 9.39% MLCGSolver::bicgstab 82 0.0794 0.0794 0.0794 9.30% MLPoisson::Fsmooth() 3280 0.06321 0.06321 0.06321 7.40% Amr::writePlotFile() 2 0.05908 0.05908 0.05908 6.92% Amr::initialInit() 1 0.05171 0.05171 0.05171 6.05% FillPatchIterator::Initialize 41 0.04785 0.04785 0.04785 5.60% Amr::FinalizeInit() 1 0.04735 0.04735 0.04735 5.54% FillPatchSingleLevel 41 0.04628 0.04628 0.04628 5.42% Castro::post_init() 1 0.04602 0.04602 0.04602 5.39% Castro::clean_state() 62 0.04576 0.04576 0.04576 5.36% Gravity::multilevel_solve_for_new_phi() 1 0.04405 0.04405 0.04405 5.16% Gravity::actual_multilevel_solve() 1 0.04402 0.04402 0.04402 5.15% StateDataPhysBCFunct::() 41 0.0423 0.0423 0.0423 4.95% MLCellLinOp::apply() 1142 0.03568 0.03568 0.03568 4.18% MLMG::mgVcycle_down::0 82 0.03508 0.03508 0.03508 4.11% MLMG::mgVcycle_up::0 82 0.03002 0.03002 0.03002 3.51% Castro::initialize_do_advance() 10 0.02471 0.02471 0.02471 2.89% StateData::FillBoundary(geom) 328 0.02426 0.02426 0.02426 2.84% MultiFab::Dot() 1114 0.02181 0.02181 0.02181 2.55% MLCellLinOp::correctionResidual() 492 0.02097 0.02097 0.02097 2.45% Castro::computeTemp() 63 0.02053 0.02053 0.02053 2.40% MLMG:computeResOfCorrection() 410 0.01812 0.01812 0.01812 2.12% MLPoisson::define() 11 0.01764 0.01764 0.01764 2.06% MLMG::mgVcycle_down::1 82 0.01748 0.01748 0.01748 2.05% MLMG::mgVcycle_down::2 82 0.01702 0.01702 0.01702 1.99% Gravity::get_new_grav_vector() 11 0.017 0.017 0.017 1.99% Castro::expand_state() 10 0.01699 0.01699 0.01699 1.99% MLMG::mgVcycle_down::3 82 0.0162 0.0162 0.0162 1.90% Castro::normalize_species() 62 0.01574 0.01574 0.01574 1.84% FabArray::FillBoundary() 4023 0.01565 0.01565 0.01565 1.83% MLMG::mgVcycle_down::4 82 0.01532 0.01532 0.01532 1.79% Castro::construct_old_gravity() 10 0.01483 0.01483 0.01483 1.74% FillBoundary_nowait() 4023 0.01483 0.01483 0.01483 1.74% Gravity::get_old_grav_vector() 10 0.01482 0.01482 0.01482 1.74% CGSolver::sxay() 1586 0.01444 0.01444 0.01444 1.69% MultiFab::LinComb() 1586 0.01405 0.01405 0.01405 1.64% FabArray::setVal() 1144 0.01402 0.01402 0.01402 1.64% FabArray::ParallelCopy() 861 0.01385 0.01385 0.01385 1.62% FabArray::ParallelCopy_nowait() 861 0.01358 0.01358 0.01358 1.59% MLMG::mgVcycle_up::2 82 0.01314 0.01314 0.01314 1.54% MLCGSolver::ParallelAllReduce 1514 0.01296 0.01296 0.01296 1.52% MLMG::mgVcycle_up::1 82 0.01286 0.01286 0.01286 1.51% Castro::do_new_sources() 10 0.01278 0.01278 0.01278 1.50% MLCellLinOp::defineAuxData() 11 0.01251 0.01251 0.01251 1.46% MLMG::addInterpCorrection() 410 0.0125 0.0125 0.0125 1.46% MLMG::mgVcycle_up::3 82 0.01244 0.01244 0.01244 1.46% MLMG::mgVcycle_up::4 82 0.01228 0.01228 0.01228 1.44% amrex::average_down 410 0.01178 0.01178 0.01178 1.38% MLPoisson::Fapply() 1142 0.01154 0.01154 0.01154 1.35% Castro::initialize_advance() 10 0.01135 0.01135 0.01135 1.33% Castro::do_old_sources() 10 0.01062 0.01062 0.01062 1.24% Gravity::fill_multipole_BCs() 11 0.01003 0.01003 0.01003 1.17% Castro::enforce_min_density() 62 0.009301 0.009301 0.009301 1.09% Castro::post_timestep() 10 0.007554 0.007554 0.007554 0.88% MLCellLinOp::solutionResidual() 93 0.007087 0.007087 0.007087 0.83% Castro::estTimeStep() 21 0.006779 0.006779 0.006779 0.79% MultiFab::Xpay() 585 0.00651 0.00651 0.00651 0.76% Castro::reset_internal_energy(MultiFab) 63 0.006322 0.006322 0.006322 0.74% MLMG::computeResidual() 82 0.006123 0.006123 0.006123 0.72% MLMG::prepareForSolve() 11 0.005291 0.005291 0.005291 0.62% MLCellLinOp::defineBC() 11 0.004884 0.004884 0.004884 0.57% BndryData::define() 11 0.004604 0.004604 0.004604 0.54% Amr::InitializeInit() 1 0.004351 0.004351 0.004351 0.51% Amr::defBaseLevel() 1 0.004343 0.004343 0.004343 0.51% Castro::initData() 1 0.003824 0.003824 0.003824 0.45% Castro::construct_new_source() 50 0.003487 0.003487 0.003487 0.41% Castro::construct_new_gravity_source() 10 0.003429 0.003429 0.003429 0.40% Castro::computeNewDt() 9 0.003428 0.003428 0.003428 0.40% Castro::construct_old_source() 50 0.002911 0.002911 0.002911 0.34% Castro::construct_old_gravity_source() 10 0.002892 0.002892 0.002892 0.34% MLMG::ResNormInf() 93 0.002121 0.002121 0.002121 0.25% Castro::apply_source_to_state() 20 0.001829 0.001829 0.001829 0.21% MultiFab::Saxpy() 20 0.001818 0.001818 0.001818 0.21% MultiFab::Add() 82 0.001663 0.001663 0.001663 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00162 0.00162 0.00162 0.19% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.18% Castro::reset_internal_energy(Fab) 504 0.00143 0.00143 0.00143 0.17% MLMG::getGradSolution() 11 0.001415 0.001415 0.001415 0.17% FabArrayBase::getCPC() 1323 0.001414 0.001414 0.001414 0.17% MLCellLinOp::compGrad() 11 0.001409 0.001409 0.001409 0.16% FabArray::mult() 43 0.00132 0.00132 0.00132 0.15% FabArray::setDomainBndry() 41 0.001286 0.001286 0.001286 0.15% Castro::check_for_nan() 20 0.001192 0.001192 0.001192 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% MLPoisson::prepareForSolve() 11 0.001157 0.001157 0.001157 0.14% MLCellLinOp::prepareForSolve() 11 0.001148 0.001148 0.001148 0.13% Castro::post_regrid() 1 0.001123 0.001123 0.001123 0.13% MLMG::computeMLResidual() 11 0.001013 0.001013 0.001013 0.12% Castro::enforce_speed_limit() 62 0.001009 0.001009 0.001009 0.12% Gravity::update_max_rhs() 11 0.000811 0.000811 0.000811 0.09% FabArrayBase::getFB() 4023 0.0006778 0.0006778 0.0006778 0.08% Castro::computeInitialDt() 2 0.0006743 0.0006743 0.0006743 0.08% FabArrayBase::CPC::define() 454 0.000654 0.000654 0.000654 0.08% Amr::InitAmr() 1 0.0005151 0.0005151 0.0005151 0.06% Gravity::swapTimeLevels() 10 0.0004398 0.0004398 0.0004398 0.05% Castro::Castro() 1 0.0004339 0.0004339 0.0004339 0.05% MultiFab::Copy() 11 0.0003219 0.0003219 0.0003219 0.04% MLMG::MLResNormInf() 11 0.0002784 0.0002784 0.0002784 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% MLMG::MLRhsNormInf() 11 0.0002155 0.0002155 0.0002155 0.03% MLLinOp::define() 11 0.0002105 0.0002105 0.0002105 0.02% MLLinOp::defineGrids() 11 0.0001873 0.0001873 0.0001873 0.02% Castro::create_source_corrector() 10 0.0001551 0.0001551 0.0001551 0.02% Castro::buildMetrics() 1 0.000153 0.000153 0.000153 0.02% FabArrayBase::FB::FB() 56 8.491e-05 8.491e-05 8.491e-05 0.01% Castro::finalize_advance() 10 5.795e-05 5.795e-05 5.795e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.24e-05 5.24e-05 5.24e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.181e-05 5.181e-05 5.181e-05 0.01% StateData::define() 4 4.459e-05 4.459e-05 4.459e-05 0.01% Castro::swap_state_time_levels() 10 3.954e-05 3.954e-05 3.954e-05 0.00% makeSFC 55 3.847e-05 3.847e-05 3.847e-05 0.00% Castro::enforce_consistent_e() 1 3.401e-05 3.401e-05 3.401e-05 0.00% Castro::finalize_do_advance() 10 3.181e-05 3.181e-05 3.181e-05 0.00% Castro::initMFs() 1 2.653e-05 2.653e-05 2.653e-05 0.00% Amr::writeSmallPlotFile() 1 2.581e-05 2.581e-05 2.581e-05 0.00% DistributionMapping::Distribute() 56 1.452e-05 1.452e-05 1.452e-05 0.00% Amr::initSubcycle() 1 8.985e-06 8.985e-06 8.985e-06 0.00% AmrLevel::checkPointPost() 3 5.417e-06 5.417e-06 5.417e-06 0.00% Castro::retry_advance_ctu() 10 3.955e-06 3.955e-06 3.955e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.878e-06 3.878e-06 3.878e-06 0.00% Gravity::set_mass_offset() 11 3.741e-06 3.741e-06 3.741e-06 0.00% Castro::FluxRegCrseInit 10 3.429e-06 3.429e-06 3.429e-06 0.00% Castro::FluxRegFineAdd() 10 2.271e-06 2.271e-06 2.271e-06 0.00% AmrLevel::checkPointPre() 3 1.96e-06 1.96e-06 1.96e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.731e-06 1.731e-06 1.731e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-18-g3d29fd7d0e81) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-18-g3d29fd7d0e81) initialized Starting run at 08:29:15 UTC on 2022-08-30. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-18-g3d29fd7d0 Microphysics git describe: 22.08-6-ge3a6cb11 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.464953897 Restart time = 0.048775049 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052192163 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050805241 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049745935 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.050413184 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.052142172 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.072508058 seconds Ending run at 08:29:15 UTC on 2022-08-30. Run time = 0.377703523 Run time without initialization = 0.328357549 Average number of zones advanced per microsecond: 3.992 Average number of zones advanced per microsecond per rank: 3.992 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3777 ... 0.3777 ... 0.3777 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0641 0.0641 0.0641 16.96% VisMF::Read() 3 0.04089 0.04089 0.04089 10.83% Amr::writePlotFile() 1 0.03837 0.03837 0.03837 10.16% VisMF::Write(FabArray) 1 0.03426 0.03426 0.03426 9.07% MLCellLinOp::applyBC() 1946 0.03374 0.03374 0.03374 8.93% MLPoisson::Fsmooth() 1440 0.02653 0.02653 0.02653 7.02% StateData::FillBoundary(geom) 160 0.01177 0.01177 0.01177 3.11% MLCGSolver::bicgstab 36 0.009968 0.009968 0.009968 2.64% MultiFab::Dot() 484 0.009204 0.009204 0.009204 2.44% FabArray::setVal() 537 0.006561 0.006561 0.006561 1.74% Castro::computeTemp() 30 0.006356 0.006356 0.006356 1.68% Castro::normalize_species() 30 0.006305 0.006305 0.006305 1.67% FillBoundary_nowait() 1766 0.00618 0.00618 0.00618 1.64% MLCellLinOp::defineAuxData() 6 0.005996 0.005996 0.005996 1.59% MultiFab::LinComb() 690 0.005924 0.005924 0.005924 1.57% StateDataPhysBCFunct::() 20 0.005789 0.005789 0.005789 1.53% FabArray::ParallelCopy_nowait() 380 0.005787 0.005787 0.005787 1.53% Gravity::fill_multipole_BCs() 6 0.00559 0.00559 0.00559 1.48% MLPoisson::Fapply() 500 0.004901 0.004901 0.004901 1.30% Castro::enforce_min_density() 30 0.0042 0.0042 0.0042 1.11% Amr::restart() 1 0.0037 0.0037 0.0037 0.98% Castro::estTimeStep() 10 0.003273 0.003273 0.003273 0.87% MLMG::addInterpCorrection() 180 0.003237 0.003237 0.003237 0.86% amrex::average_down 180 0.002918 0.002918 0.002918 0.77% MultiFab::Xpay() 258 0.002787 0.002787 0.002787 0.74% Castro::reset_internal_energy(MultiFab) 30 0.002224 0.002224 0.002224 0.59% BndryData::define() 6 0.002021 0.002021 0.002021 0.54% Castro::construct_new_gravity_source() 5 0.001681 0.001681 0.001681 0.44% Castro::do_advance_ctu() 5 0.001567 0.001567 0.001567 0.41% Castro::construct_old_gravity_source() 5 0.001522 0.001522 0.001522 0.40% Castro::subcycle_advance_ctu() 5 0.001184 0.001184 0.001184 0.31% MLMG::ResNormInf() 42 0.0009242 0.0009242 0.0009242 0.24% MultiFab::Saxpy() 10 0.0009119 0.0009119 0.0009119 0.24% Gravity::get_old_grav_vector() 5 0.0008934 0.0008934 0.0008934 0.24% Castro::expand_state() 5 0.0008863 0.0008863 0.0008863 0.23% Gravity::get_new_grav_vector() 5 0.00088 0.00088 0.00088 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008659 0.0008659 0.0008659 0.23% MLCellLinOp::setLevelBC() 6 0.0007938 0.0007938 0.0007938 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007436 0.0007436 0.0007436 0.20% Castro::reset_internal_energy(Fab) 240 0.0007287 0.0007287 0.0007287 0.19% MultiFab::Add() 36 0.0007101 0.0007101 0.0007101 0.19% FabArray::mult() 22 0.0006516 0.0006516 0.0006516 0.17% MLMG::prepareForSolve() 6 0.0006499 0.0006499 0.0006499 0.17% FabArray::setDomainBndry() 20 0.0006239 0.0006239 0.0006239 0.17% MLCellLinOp::prepareForSolve() 6 0.0006216 0.0006216 0.0006216 0.16% MultiFab::contains_nan() 10 0.0005825 0.0005825 0.0005825 0.15% MLCellLinOp::smooth() 720 0.0005546 0.0005546 0.0005546 0.15% Castro::enforce_speed_limit() 30 0.0005034 0.0005034 0.0005034 0.13% MLCellLinOp::compGrad() 6 0.0004842 0.0004842 0.0004842 0.13% Amr::InitAmr() 1 0.0003873 0.0003873 0.0003873 0.10% FabArray::FillBoundary() 1766 0.0003828 0.0003828 0.0003828 0.10% FabArrayBase::CPC::define() 244 0.0003815 0.0003815 0.0003815 0.10% FabArrayBase::getCPC() 632 0.0003548 0.0003548 0.0003548 0.09% main() 1 0.0003488 0.0003488 0.0003488 0.09% FabArrayBase::getFB() 1766 0.0002597 0.0002597 0.0002597 0.07% Gravity::update_max_rhs() 6 0.0002544 0.0002544 0.0002544 0.07% Gravity::solve_for_phi() 5 0.0002337 0.0002337 0.0002337 0.06% MLCellLinOp::apply() 500 0.0002301 0.0002301 0.0002301 0.06% CGSolver::sxay() 690 0.0001929 0.0001929 0.0001929 0.05% Amr::coarseTimeStep() 5 0.0001792 0.0001792 0.0001792 0.05% MLCellLinOp::defineBC() 6 0.0001756 0.0001756 0.0001756 0.05% MultiFab::Copy() 6 0.0001697 0.0001697 0.0001697 0.04% MultiFab::max() 6 0.000134 0.000134 0.000134 0.04% FillPatchIterator::Initialize 20 0.0001326 0.0001326 0.0001326 0.04% MLCGSolver::ParallelAllReduce 659 0.0001271 0.0001271 0.0001271 0.03% FabArray::ParallelCopy() 380 0.0001197 0.0001197 0.0001197 0.03% Castro::construct_new_gravity() 5 0.0001145 0.0001145 0.0001145 0.03% Amr::timeStep() 5 0.0001136 0.0001136 0.0001136 0.03% MLMG::MLRhsNormInf() 6 0.0001115 0.0001115 0.0001115 0.03% MLCellLinOp::correctionResidual() 216 0.0001005 0.0001005 0.0001005 0.03% Castro::create_source_corrector() 5 9.666e-05 9.666e-05 9.666e-05 0.03% MLMG::mgVcycle() 36 8.553e-05 8.553e-05 8.553e-05 0.02% AmrLevel::restart() 1 7.655e-05 7.655e-05 7.655e-05 0.02% StateData::restartDoit() 4 7.416e-05 7.416e-05 7.416e-05 0.02% MLLinOp::defineGrids() 6 7.157e-05 7.157e-05 7.157e-05 0.02% MLMG:computeResOfCorrection() 180 7.144e-05 7.144e-05 7.144e-05 0.02% Castro::advance() 5 6.259e-05 6.259e-05 6.259e-05 0.02% FabArrayBase::FB::FB() 26 5.63e-05 5.63e-05 5.63e-05 0.01% MLMG::mgVcycle_down::0 36 4.752e-05 4.752e-05 4.752e-05 0.01% MLMG::mgVcycle_down::1 36 4.454e-05 4.454e-05 4.454e-05 0.01% MLMG::mgVcycle_down::2 36 4.138e-05 4.138e-05 4.138e-05 0.01% Castro::initialize_advance() 5 4.026e-05 4.026e-05 4.026e-05 0.01% MLMG::mgVcycle_down::4 36 3.89e-05 3.89e-05 3.89e-05 0.01% Castro::clean_state() 30 3.88e-05 3.88e-05 3.88e-05 0.01% Castro::finalize_advance() 5 3.827e-05 3.827e-05 3.827e-05 0.01% MLMG::mgVcycle_down::3 36 3.807e-05 3.807e-05 3.807e-05 0.01% MLMG::actualBottomSolve() 36 3.598e-05 3.598e-05 3.598e-05 0.01% MLMG::mgVcycle_up::4 36 3.339e-05 3.339e-05 3.339e-05 0.01% Amr::writeSmallPlotFile() 1 3.334e-05 3.334e-05 3.334e-05 0.01% MLMG::solve() 6 3.31e-05 3.31e-05 3.31e-05 0.01% Castro::buildMetrics() 1 3.218e-05 3.218e-05 3.218e-05 0.01% Castro::initialize_do_advance() 5 3.174e-05 3.174e-05 3.174e-05 0.01% Castro::computeNewDt() 5 3.174e-05 3.174e-05 3.174e-05 0.01% Gravity::actual_multilevel_solve() 1 2.87e-05 2.87e-05 2.87e-05 0.01% Castro::post_restart() 1 2.862e-05 2.862e-05 2.862e-05 0.01% MLMG::mgVcycle_up::0 36 2.775e-05 2.775e-05 2.775e-05 0.01% Castro::swap_state_time_levels() 5 2.775e-05 2.775e-05 2.775e-05 0.01% Castro::initMFs() 1 2.669e-05 2.669e-05 2.669e-05 0.01% MLMG::mgVcycle_up::3 36 2.607e-05 2.607e-05 2.607e-05 0.01% MLMG::oneIter() 36 2.565e-05 2.565e-05 2.565e-05 0.01% MLMG::mgVcycle_up::2 36 2.522e-05 2.522e-05 2.522e-05 0.01% Castro::construct_old_source() 25 2.521e-05 2.521e-05 2.521e-05 0.01% MLMG::mgVcycle_up::1 36 2.341e-05 2.341e-05 2.341e-05 0.01% MLCellLinOp::solutionResidual() 42 2.264e-05 2.264e-05 2.264e-05 0.01% MLPoisson::define() 6 2.183e-05 2.183e-05 2.183e-05 0.01% MLLinOp::define() 6 2.181e-05 2.181e-05 2.181e-05 0.01% Castro::finalize_do_advance() 5 1.88e-05 1.88e-05 1.88e-05 0.00% MLMG::computeResidual() 36 1.761e-05 1.761e-05 1.761e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.737e-05 1.737e-05 1.737e-05 0.00% MLMG::mgVcycle_bottom 36 1.521e-05 1.521e-05 1.521e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.39e-05 1.39e-05 1.39e-05 0.00% makeSFC 30 1.366e-05 1.366e-05 1.366e-05 0.00% FillPatchSingleLevel 20 1.354e-05 1.354e-05 1.354e-05 0.00% Castro::construct_new_source() 25 8.945e-06 8.945e-06 8.945e-06 0.00% Castro::do_new_sources() 5 8.836e-06 8.836e-06 8.836e-06 0.00% Amr::initSubcycle() 1 8.833e-06 8.833e-06 8.833e-06 0.00% Castro::do_old_sources() 5 8.458e-06 8.458e-06 8.458e-06 0.00% DistributionMapping::Distribute() 31 8.371e-06 8.371e-06 8.371e-06 0.00% Castro::check_for_nan() 10 7.142e-06 7.142e-06 7.142e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.98e-06 6.98e-06 6.98e-06 0.00% Castro::construct_old_gravity() 5 5.955e-06 5.955e-06 5.955e-06 0.00% Castro::apply_source_to_state() 10 5.416e-06 5.416e-06 5.416e-06 0.00% Castro::post_timestep() 5 5.267e-06 5.267e-06 5.267e-06 0.00% Gravity::swapTimeLevels() 5 5.167e-06 5.167e-06 5.167e-06 0.00% MLPoisson::prepareForSolve() 6 4.791e-06 4.791e-06 4.791e-06 0.00% MLMG::computeMLResidual() 6 4.566e-06 4.566e-06 4.566e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.048e-06 3.048e-06 3.048e-06 0.00% MLMG::getGradSolution() 6 3.027e-06 3.027e-06 3.027e-06 0.00% MLMG::MLResNormInf() 6 2.201e-06 2.201e-06 2.201e-06 0.00% Castro::FluxRegCrseInit 5 2.094e-06 2.094e-06 2.094e-06 0.00% Gravity::set_mass_offset() 6 2.032e-06 2.032e-06 2.032e-06 0.00% Castro::retry_advance_ctu() 5 1.91e-06 1.91e-06 1.91e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.297e-06 1.297e-06 1.297e-06 0.00% Castro::FluxRegFineAdd() 5 1.182e-06 1.182e-06 1.182e-06 0.00% Amr::init() 1 9.43e-07 9.43e-07 9.43e-07 0.00% AmrLevel::AmrLevel() 1 9.15e-07 9.15e-07 9.15e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3777 0.3777 0.3777 100.00% Amr::coarseTimeStep() 5 0.2555 0.2555 0.2555 67.63% Amr::timeStep() 5 0.2531 0.2531 0.2531 67.00% Castro::advance() 5 0.2502 0.2502 0.2502 66.24% Castro::subcycle_advance_ctu() 5 0.2437 0.2437 0.2437 64.53% Castro::do_advance_ctu() 5 0.2426 0.2426 0.2426 64.21% Castro::construct_new_gravity() 5 0.1413 0.1413 0.1413 37.42% Gravity::solve_phi_with_mlmg() 6 0.137 0.137 0.137 36.26% Gravity::solve_for_phi() 5 0.1335 0.1335 0.1335 35.35% Gravity::actual_solve_with_mlmg() 6 0.1313 0.1313 0.1313 34.75% MLMG::solve() 6 0.1194 0.1194 0.1194 31.61% MLMG::oneIter() 36 0.1123 0.1123 0.1123 29.74% MLMG::mgVcycle() 36 0.1116 0.1116 0.1116 29.54% Amr::writePlotFile() 1 0.07263 0.07263 0.07263 19.23% Castro::construct_ctu_hydro_source() 5 0.06407 0.06407 0.06407 16.96% MLCellLinOp::smooth() 720 0.05721 0.05721 0.05721 15.15% Amr::init() 1 0.04884 0.04884 0.04884 12.93% Amr::restart() 1 0.04884 0.04884 0.04884 12.93% AmrLevel::restart() 1 0.0411 0.0411 0.0411 10.88% StateData::restartDoit() 4 0.04102 0.04102 0.04102 10.86% VisMF::Read() 3 0.04089 0.04089 0.04089 10.83% MLCellLinOp::applyBC() 1946 0.04062 0.04062 0.04062 10.75% VisMF::Write(FabArray) 1 0.03426 0.03426 0.03426 9.07% MLMG::mgVcycle_bottom 36 0.03404 0.03404 0.03404 9.01% MLMG::actualBottomSolve() 36 0.03403 0.03403 0.03403 9.01% MLCGSolver::bicgstab 36 0.03369 0.03369 0.03369 8.92% MLPoisson::Fsmooth() 1440 0.02653 0.02653 0.02653 7.02% Castro::clean_state() 30 0.02036 0.02036 0.02036 5.39% FillPatchIterator::Initialize 20 0.02031 0.02031 0.02031 5.38% FillPatchSingleLevel 20 0.01956 0.01956 0.01956 5.18% StateDataPhysBCFunct::() 20 0.01755 0.01755 0.01755 4.65% MLCellLinOp::apply() 500 0.01536 0.01536 0.01536 4.07% MLMG::mgVcycle_down::0 36 0.01501 0.01501 0.01501 3.97% MLMG::mgVcycle_up::0 36 0.01283 0.01283 0.01283 3.40% StateData::FillBoundary(geom) 160 0.01177 0.01177 0.01177 3.11% MLPoisson::define() 6 0.009571 0.009571 0.009571 2.53% Castro::computeTemp() 30 0.009308 0.009308 0.009308 2.46% MultiFab::Dot() 484 0.009204 0.009204 0.009204 2.44% Castro::initialize_do_advance() 5 0.009046 0.009046 0.009046 2.39% MLCellLinOp::correctionResidual() 216 0.008963 0.008963 0.008963 2.37% MLMG:computeResOfCorrection() 180 0.007738 0.007738 0.007738 2.05% Gravity::get_new_grav_vector() 5 0.00771 0.00771 0.00771 2.04% Castro::construct_old_gravity() 5 0.007505 0.007505 0.007505 1.99% Gravity::get_old_grav_vector() 5 0.007499 0.007499 0.007499 1.99% MLMG::mgVcycle_down::1 36 0.007445 0.007445 0.007445 1.97% MLMG::mgVcycle_down::2 36 0.007208 0.007208 0.007208 1.91% Castro::do_new_sources() 5 0.007207 0.007207 0.007207 1.91% MLMG::mgVcycle_down::3 36 0.006886 0.006886 0.006886 1.82% FabArray::FillBoundary() 1766 0.006879 0.006879 0.006879 1.82% MLCellLinOp::defineAuxData() 6 0.006713 0.006713 0.006713 1.78% MLMG::mgVcycle_down::4 36 0.006584 0.006584 0.006584 1.74% FabArray::setVal() 537 0.006561 0.006561 0.006561 1.74% FillBoundary_nowait() 1766 0.006496 0.006496 0.006496 1.72% Castro::initialize_advance() 5 0.006374 0.006374 0.006374 1.69% Castro::normalize_species() 30 0.006305 0.006305 0.006305 1.67% FabArray::ParallelCopy() 380 0.006275 0.006275 0.006275 1.66% FabArray::ParallelCopy_nowait() 380 0.006155 0.006155 0.006155 1.63% CGSolver::sxay() 690 0.006117 0.006117 0.006117 1.62% MultiFab::LinComb() 690 0.005924 0.005924 0.005924 1.57% Castro::expand_state() 5 0.00563 0.00563 0.00563 1.49% Gravity::fill_multipole_BCs() 6 0.00559 0.00559 0.00559 1.48% MLMG::mgVcycle_up::2 36 0.005546 0.005546 0.005546 1.47% MLCGSolver::ParallelAllReduce 659 0.00552 0.00552 0.00552 1.46% MLMG::mgVcycle_up::1 36 0.005457 0.005457 0.005457 1.44% MLMG::addInterpCorrection() 180 0.005343 0.005343 0.005343 1.41% MLMG::mgVcycle_up::3 36 0.00525 0.00525 0.00525 1.39% MLMG::mgVcycle_up::4 36 0.005244 0.005244 0.005244 1.39% amrex::average_down 180 0.005097 0.005097 0.005097 1.35% MLPoisson::Fapply() 500 0.004901 0.004901 0.004901 1.30% Castro::do_old_sources() 5 0.004739 0.004739 0.004739 1.25% Castro::enforce_min_density() 30 0.0042 0.0042 0.0042 1.11% Castro::post_restart() 1 0.003864 0.003864 0.003864 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003744 0.003744 0.003744 0.99% Gravity::actual_multilevel_solve() 1 0.003727 0.003727 0.003727 0.99% Castro::estTimeStep() 10 0.003273 0.003273 0.003273 0.87% MLCellLinOp::solutionResidual() 42 0.003178 0.003178 0.003178 0.84% Castro::reset_internal_energy(MultiFab) 30 0.002952 0.002952 0.002952 0.78% MLMG::prepareForSolve() 6 0.002811 0.002811 0.002811 0.74% MultiFab::Xpay() 258 0.002787 0.002787 0.002787 0.74% Castro::post_timestep() 5 0.002754 0.002754 0.002754 0.73% MLCellLinOp::defineBC() 6 0.002714 0.002714 0.002714 0.72% MLMG::computeResidual() 36 0.002643 0.002643 0.002643 0.70% BndryData::define() 6 0.002538 0.002538 0.002538 0.67% Castro::computeNewDt() 5 0.002205 0.002205 0.002205 0.58% Castro::construct_new_source() 25 0.00169 0.00169 0.00169 0.45% Castro::construct_new_gravity_source() 5 0.001681 0.001681 0.001681 0.44% Castro::construct_old_source() 25 0.001547 0.001547 0.001547 0.41% Castro::construct_old_gravity_source() 5 0.001522 0.001522 0.001522 0.40% MLMG::ResNormInf() 42 0.0009242 0.0009242 0.0009242 0.24% Castro::apply_source_to_state() 10 0.0009173 0.0009173 0.0009173 0.24% MultiFab::Saxpy() 10 0.0009119 0.0009119 0.0009119 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008659 0.0008659 0.0008659 0.23% MLCellLinOp::setLevelBC() 6 0.0007938 0.0007938 0.0007938 0.21% MLMG::getGradSolution() 6 0.0007552 0.0007552 0.0007552 0.20% MLCellLinOp::compGrad() 6 0.0007521 0.0007521 0.0007521 0.20% FabArrayBase::getCPC() 632 0.0007364 0.0007364 0.0007364 0.19% Castro::reset_internal_energy(Fab) 240 0.0007287 0.0007287 0.0007287 0.19% MultiFab::Add() 36 0.0007101 0.0007101 0.0007101 0.19% FabArray::mult() 22 0.0006516 0.0006516 0.0006516 0.17% MLPoisson::prepareForSolve() 6 0.0006264 0.0006264 0.0006264 0.17% FabArray::setDomainBndry() 20 0.0006239 0.0006239 0.0006239 0.17% MLCellLinOp::prepareForSolve() 6 0.0006216 0.0006216 0.0006216 0.16% Castro::check_for_nan() 10 0.0005897 0.0005897 0.0005897 0.16% MultiFab::contains_nan() 10 0.0005825 0.0005825 0.0005825 0.15% MLMG::computeMLResidual() 6 0.0005576 0.0005576 0.0005576 0.15% Castro::enforce_speed_limit() 30 0.0005034 0.0005034 0.0005034 0.13% Gravity::update_max_rhs() 6 0.0004632 0.0004632 0.0004632 0.12% Amr::InitAmr() 1 0.0003961 0.0003961 0.0003961 0.10% FabArrayBase::CPC::define() 244 0.0003815 0.0003815 0.0003815 0.10% FabArrayBase::getFB() 1766 0.000316 0.000316 0.000316 0.08% Gravity::swapTimeLevels() 5 0.0002223 0.0002223 0.0002223 0.06% MultiFab::Copy() 6 0.0001697 0.0001697 0.0001697 0.04% Castro::buildMetrics() 1 0.0001504 0.0001504 0.0001504 0.04% MLMG::MLResNormInf() 6 0.0001461 0.0001461 0.0001461 0.04% MultiFab::max() 6 0.000134 0.000134 0.000134 0.04% MLLinOp::define() 6 0.0001224 0.0001224 0.0001224 0.03% MLMG::MLRhsNormInf() 6 0.0001115 0.0001115 0.0001115 0.03% MLLinOp::defineGrids() 6 0.0001006 0.0001006 0.0001006 0.03% Castro::create_source_corrector() 5 9.666e-05 9.666e-05 9.666e-05 0.03% FabArrayBase::FB::FB() 26 5.63e-05 5.63e-05 5.63e-05 0.01% Castro::finalize_advance() 5 4.154e-05 4.154e-05 4.154e-05 0.01% Amr::writeSmallPlotFile() 1 3.334e-05 3.334e-05 3.334e-05 0.01% Castro::swap_state_time_levels() 5 2.775e-05 2.775e-05 2.775e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.772e-05 2.772e-05 2.772e-05 0.01% Castro::initMFs() 1 2.669e-05 2.669e-05 2.669e-05 0.01% makeSFC 30 2.074e-05 2.074e-05 2.074e-05 0.01% Castro::finalize_do_advance() 5 1.88e-05 1.88e-05 1.88e-05 0.00% Amr::initSubcycle() 1 8.833e-06 8.833e-06 8.833e-06 0.00% DistributionMapping::Distribute() 31 8.371e-06 8.371e-06 8.371e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.336e-06 4.336e-06 4.336e-06 0.00% Castro::FluxRegCrseInit 5 2.094e-06 2.094e-06 2.094e-06 0.00% Gravity::set_mass_offset() 6 2.032e-06 2.032e-06 2.032e-06 0.00% Castro::retry_advance_ctu() 5 1.91e-06 1.91e-06 1.91e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.297e-06 1.297e-06 1.297e-06 0.00% Castro::FluxRegFineAdd() 5 1.182e-06 1.182e-06 1.182e-06 0.00% AmrLevel::AmrLevel() 1 9.15e-07 9.15e-07 9.15e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-18-g3d29fd7d0e81) finalized