Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-48-gb2b9150ada12) initialized Starting run at 08:29:20 UTC on 2022-06-29. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-48-gb2b9150ad Microphysics git describe: 22.06-7-ga6a75714 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.04443596 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02578319 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048485575 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051062745 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.052442642 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.069330117 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.079072311 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.04123297 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.078345739 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048746564 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049525561 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.06427694 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068643828 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.041058391 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02540168 seconds Ending run at 08:29:21 UTC on 2022-06-29. Run time = 0.841556329 Run time without initialization = 0.718279491 Average number of zones advanced per microsecond: 3.650 Average number of zones advanced per microsecond per rank: 3.650 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8416 ... 0.8416 ... 0.8416 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2031 0.2031 0.2031 24.13% VisMF::Write(FabArray) 11 0.1711 0.1711 0.1711 20.33% MLCellLinOp::applyBC() 4433 0.08106 0.08106 0.08106 9.63% MLPoisson::Fsmooth() 3280 0.06385 0.06385 0.06385 7.59% StateDataPhysBCFunct::() 41 0.02741 0.02741 0.02741 3.26% StateData::FillBoundary(geom) 328 0.0245 0.0245 0.0245 2.91% MLCGSolver::bicgstab 82 0.02441 0.02441 0.02441 2.90% MultiFab::Dot() 1114 0.02268 0.02268 0.02268 2.70% Castro::normalize_species() 62 0.01602 0.01602 0.01602 1.90% MultiFab::LinComb() 1586 0.01454 0.01454 0.01454 1.73% FabArray::setVal() 1144 0.01447 0.01447 0.01447 1.72% FillBoundary_nowait() 4023 0.01433 0.01433 0.01433 1.70% Castro::computeTemp() 63 0.01385 0.01385 0.01385 1.65% FabArray::ParallelCopy_nowait() 861 0.01317 0.01317 0.01317 1.57% Castro::enforce_min_density() 62 0.01262 0.01262 0.01262 1.50% MLPoisson::Fapply() 1142 0.01182 0.01182 0.01182 1.40% MLCellLinOp::defineAuxData() 11 0.01165 0.01165 0.01165 1.38% Gravity::fill_multipole_BCs() 11 0.009424 0.009424 0.009424 1.12% MLMG::addInterpCorrection() 410 0.007539 0.007539 0.007539 0.90% amrex::average_down 410 0.006966 0.006966 0.006966 0.83% MultiFab::Xpay() 585 0.006657 0.006657 0.006657 0.79% Castro::estTimeStep() 21 0.006109 0.006109 0.006109 0.73% Castro::do_advance_ctu() 10 0.004986 0.004986 0.004986 0.59% Amr::checkPoint() 3 0.00438 0.00438 0.00438 0.52% BndryData::define() 11 0.003909 0.003909 0.003909 0.46% Castro::reset_internal_energy(MultiFab) 63 0.003909 0.003909 0.003909 0.46% Castro::construct_new_gravity_source() 10 0.003095 0.003095 0.003095 0.37% Castro::construct_old_gravity_source() 10 0.002678 0.002678 0.002678 0.32% Amr::writePlotFile() 2 0.002558 0.002558 0.002558 0.30% Gravity::get_new_grav_vector() 11 0.001941 0.001941 0.001941 0.23% MLMG::ResNormInf() 93 0.001928 0.001928 0.001928 0.23% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.22% Gravity::get_old_grav_vector() 10 0.001773 0.001773 0.001773 0.21% Castro::expand_state() 10 0.001762 0.001762 0.001762 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001681 0.001681 0.001681 0.20% MLMG::oneIter() 82 0.001678 0.001678 0.001678 0.20% MLCellLinOp::setLevelBC() 11 0.001553 0.001553 0.001553 0.18% Castro::reset_internal_energy(Fab) 504 0.00148 0.00148 0.00148 0.18% Gravity::actual_solve_with_mlmg() 11 0.001334 0.001334 0.001334 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% FabArray::setDomainBndry() 41 0.001313 0.001313 0.001313 0.16% Castro::initData() 1 0.001198 0.001198 0.001198 0.14% MLCellLinOp::prepareForSolve() 11 0.001184 0.001184 0.001184 0.14% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% Castro::enforce_speed_limit() 62 0.001109 0.001109 0.001109 0.13% MLCellLinOp::smooth() 1640 0.001099 0.001099 0.001099 0.13% MLMG::prepareForSolve() 11 0.001036 0.001036 0.001036 0.12% MLCellLinOp::compGrad() 11 0.0009293 0.0009293 0.0009293 0.11% FabArray::FillBoundary() 4023 0.0008033 0.0008033 0.0008033 0.10% FabArrayBase::getCPC() 1323 0.0007949 0.0007949 0.0007949 0.09% FabArrayBase::CPC::define() 454 0.0006711 0.0006711 0.0006711 0.08% FabArrayBase::getFB() 4023 0.0006224 0.0006224 0.0006224 0.07% Amr::InitAmr() 1 0.0004876 0.0004876 0.0004876 0.06% MLCellLinOp::apply() 1142 0.0004466 0.0004466 0.0004466 0.05% Gravity::solve_for_phi() 10 0.0004246 0.0004246 0.0004246 0.05% Gravity::update_max_rhs() 11 0.0004161 0.0004161 0.0004161 0.05% CGSolver::sxay() 1586 0.000413 0.000413 0.000413 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003114 0.0003114 0.0003114 0.04% Amr::coarseTimeStep() 10 0.0003082 0.0003082 0.0003082 0.04% FillPatchIterator::Initialize 41 0.0003059 0.0003059 0.0003059 0.04% main() 1 0.0002963 0.0002963 0.0002963 0.04% MLCellLinOp::defineBC() 11 0.0002812 0.0002812 0.0002812 0.03% FabArray::ParallelCopy() 861 0.0002804 0.0002804 0.0002804 0.03% MultiFab::Copy() 11 0.0002605 0.0002605 0.0002605 0.03% MultiFab::max() 11 0.0002578 0.0002578 0.0002578 0.03% MLCellLinOp::correctionResidual() 492 0.0002228 0.0002228 0.0002228 0.03% Castro::construct_new_gravity() 10 0.0002092 0.0002092 0.0002092 0.02% MLMG::MLRhsNormInf() 11 0.0002022 0.0002022 0.0002022 0.02% Amr::timeStep() 10 0.0001934 0.0001934 0.0001934 0.02% MLMG::mgVcycle() 82 0.0001918 0.0001918 0.0001918 0.02% Castro::create_source_corrector() 10 0.0001734 0.0001734 0.0001734 0.02% MLLinOp::defineGrids() 11 0.0001613 0.0001613 0.0001613 0.02% Castro::subcycle_advance_ctu() 10 0.0001474 0.0001474 0.0001474 0.02% MLMG:computeResOfCorrection() 410 0.0001378 0.0001378 0.0001378 0.02% StateData::checkPoint() 12 0.0001283 0.0001283 0.0001283 0.02% MLMG::actualBottomSolve() 82 0.0001072 0.0001072 0.0001072 0.01% Castro::Castro() 1 0.000104 0.000104 0.000104 0.01% FabArrayBase::FB::FB() 56 9.732e-05 9.732e-05 9.732e-05 0.01% MLMG::mgVcycle_down::0 82 9.583e-05 9.583e-05 9.583e-05 0.01% Castro::initialize_advance() 10 8.585e-05 8.585e-05 8.585e-05 0.01% MLMG::mgVcycle_down::1 82 7.963e-05 7.963e-05 7.963e-05 0.01% MLMG::mgVcycle_down::2 82 7.669e-05 7.669e-05 7.669e-05 0.01% Castro::clean_state() 62 7.45e-05 7.45e-05 7.45e-05 0.01% MLMG::solve() 11 7.353e-05 7.353e-05 7.353e-05 0.01% MLMG::mgVcycle_down::3 82 7.341e-05 7.341e-05 7.341e-05 0.01% AmrLevel::checkPoint() 3 7.326e-05 7.326e-05 7.326e-05 0.01% MLMG::mgVcycle_down::4 82 7.205e-05 7.205e-05 7.205e-05 0.01% Castro::initialize_do_advance() 10 6.524e-05 6.524e-05 6.524e-05 0.01% Castro::finalize_advance() 10 6.253e-05 6.253e-05 6.253e-05 0.01% MLMG::mgVcycle_up::4 82 5.9e-05 5.9e-05 5.9e-05 0.01% Castro::advance() 10 5.705e-05 5.705e-05 5.705e-05 0.01% MLMG::mgVcycle_up::0 82 5.511e-05 5.511e-05 5.511e-05 0.01% MLMG::mgVcycle_up::2 82 5.211e-05 5.211e-05 5.211e-05 0.01% MLMG::mgVcycle_up::1 82 5.169e-05 5.169e-05 5.169e-05 0.01% MLCellLinOp::solutionResidual() 93 5.151e-05 5.151e-05 5.151e-05 0.01% MLMG::mgVcycle_up::3 82 5.082e-05 5.082e-05 5.082e-05 0.01% Castro::construct_new_source() 50 5.055e-05 5.055e-05 5.055e-05 0.01% Gravity::solve_phi_with_mlmg() 11 4.917e-05 4.917e-05 4.917e-05 0.01% Castro::swap_state_time_levels() 10 3.993e-05 3.993e-05 3.993e-05 0.00% Castro::finalize_do_advance() 10 3.878e-05 3.878e-05 3.878e-05 0.00% StateData::define() 4 3.831e-05 3.831e-05 3.831e-05 0.00% MLMG::mgVcycle_bottom 82 3.572e-05 3.572e-05 3.572e-05 0.00% Castro::enforce_consistent_e() 1 3.391e-05 3.391e-05 3.391e-05 0.00% MLMG::computeResidual() 82 3.318e-05 3.318e-05 3.318e-05 0.00% Gravity::actual_multilevel_solve() 1 3.186e-05 3.186e-05 3.186e-05 0.00% FillPatchSingleLevel 41 2.923e-05 2.923e-05 2.923e-05 0.00% Castro::initMFs() 1 2.876e-05 2.876e-05 2.876e-05 0.00% Amr::writeSmallPlotFile() 1 2.664e-05 2.664e-05 2.664e-05 0.00% makeSFC 55 2.637e-05 2.637e-05 2.637e-05 0.00% Castro::buildMetrics() 1 2.525e-05 2.525e-05 2.525e-05 0.00% MLLinOp::define() 11 2.294e-05 2.294e-05 2.294e-05 0.00% Amr::defBaseLevel() 1 2.266e-05 2.266e-05 2.266e-05 0.00% Amr::FinalizeInit() 1 2.21e-05 2.21e-05 2.21e-05 0.00% MLPoisson::define() 11 2.201e-05 2.201e-05 2.201e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.826e-05 1.826e-05 1.826e-05 0.00% Castro::construct_old_source() 50 1.755e-05 1.755e-05 1.755e-05 0.00% Castro::do_new_sources() 10 1.676e-05 1.676e-05 1.676e-05 0.00% Castro::do_old_sources() 10 1.64e-05 1.64e-05 1.64e-05 0.00% DistributionMapping::Distribute() 56 1.471e-05 1.471e-05 1.471e-05 0.00% Castro::check_for_nan() 20 1.277e-05 1.277e-05 1.277e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.272e-05 1.272e-05 1.272e-05 0.00% Castro::apply_source_to_state() 20 1.167e-05 1.167e-05 1.167e-05 0.00% Castro::construct_old_gravity() 10 9.738e-06 9.738e-06 9.738e-06 0.00% Gravity::swapTimeLevels() 10 9.55e-06 9.55e-06 9.55e-06 0.00% Castro::post_timestep() 10 9.331e-06 9.331e-06 9.331e-06 0.00% Amr::initSubcycle() 1 8.841e-06 8.841e-06 8.841e-06 0.00% MLPoisson::prepareForSolve() 11 7.329e-06 7.329e-06 7.329e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.811e-06 6.811e-06 6.811e-06 0.00% Amr::InitializeInit() 1 6.722e-06 6.722e-06 6.722e-06 0.00% Castro::computeNewDt() 9 6.264e-06 6.264e-06 6.264e-06 0.00% MLMG::computeMLResidual() 11 6.189e-06 6.189e-06 6.189e-06 0.00% MLMG::getGradSolution() 11 6.144e-06 6.144e-06 6.144e-06 0.00% MLMG::buildFineMask() 11 5.437e-06 5.437e-06 5.437e-06 0.00% MLMG::MLResNormInf() 11 4.473e-06 4.473e-06 4.473e-06 0.00% Castro::retry_advance_ctu() 10 3.879e-06 3.879e-06 3.879e-06 0.00% AmrLevel::checkPointPost() 3 3.861e-06 3.861e-06 3.861e-06 0.00% Gravity::set_mass_offset() 11 3.741e-06 3.741e-06 3.741e-06 0.00% Castro::post_init() 1 3.735e-06 3.735e-06 3.735e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.563e-06 3.563e-06 3.563e-06 0.00% Amr::init() 1 2.728e-06 2.728e-06 2.728e-06 0.00% Castro::FluxRegCrseInit 10 2.725e-06 2.725e-06 2.725e-06 0.00% Castro::FluxRegFineAdd() 10 2.526e-06 2.526e-06 2.526e-06 0.00% Castro::computeInitialDt() 2 2.47e-06 2.47e-06 2.47e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.922e-06 1.922e-06 1.922e-06 0.00% AmrLevel::checkPointPre() 3 1.758e-06 1.758e-06 1.758e-06 0.00% Amr::initialInit() 1 1.721e-06 1.721e-06 1.721e-06 0.00% Castro::post_regrid() 1 1.324e-06 1.324e-06 1.324e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8416 0.8416 0.8416 100.00% Amr::coarseTimeStep() 10 0.6926 0.6926 0.6926 82.30% Amr::timeStep() 10 0.6059 0.6059 0.6059 72.00% Castro::advance() 10 0.5991 0.5991 0.5991 71.19% Castro::subcycle_advance_ctu() 10 0.5892 0.5892 0.5892 70.01% Castro::do_advance_ctu() 10 0.5891 0.5891 0.5891 70.00% Gravity::solve_phi_with_mlmg() 11 0.3174 0.3174 0.3174 37.71% Gravity::actual_solve_with_mlmg() 11 0.3077 0.3077 0.3077 36.56% Castro::construct_new_gravity() 10 0.288 0.288 0.288 34.22% MLMG::solve() 11 0.285 0.285 0.285 33.86% Gravity::solve_for_phi() 10 0.2727 0.2727 0.2727 32.40% MLMG::oneIter() 82 0.2703 0.2703 0.2703 32.12% MLMG::mgVcycle() 82 0.2687 0.2687 0.2687 31.92% Castro::construct_ctu_hydro_source() 10 0.2031 0.2031 0.2031 24.13% VisMF::Write(FabArray) 11 0.1711 0.1711 0.1711 20.33% MLCellLinOp::smooth() 1640 0.1371 0.1371 0.1371 16.29% Amr::checkPoint() 3 0.1269 0.1269 0.1269 15.08% Amr::init() 1 0.1226 0.1226 0.1226 14.57% AmrLevel::checkPoint() 3 0.1225 0.1225 0.1225 14.56% StateData::checkPoint() 12 0.1224 0.1224 0.1224 14.55% MLCellLinOp::applyBC() 4433 0.09692 0.09692 0.09692 11.52% MLMG::mgVcycle_bottom 82 0.08322 0.08322 0.08322 9.89% MLMG::actualBottomSolve() 82 0.08319 0.08319 0.08319 9.88% MLCGSolver::bicgstab 82 0.08234 0.08234 0.08234 9.78% MLPoisson::Fsmooth() 3280 0.06385 0.06385 0.06385 7.59% FillPatchIterator::Initialize 41 0.05752 0.05752 0.05752 6.83% FillPatchSingleLevel 41 0.0559 0.0559 0.0559 6.64% Amr::initialInit() 1 0.05229 0.05229 0.05229 6.21% StateDataPhysBCFunct::() 41 0.05191 0.05191 0.05191 6.17% Amr::writePlotFile() 2 0.05134 0.05134 0.05134 6.10% Castro::clean_state() 62 0.04837 0.04837 0.04837 5.75% Amr::FinalizeInit() 1 0.04834 0.04834 0.04834 5.74% Castro::post_init() 1 0.04703 0.04703 0.04703 5.59% Gravity::multilevel_solve_for_new_phi() 1 0.04516 0.04516 0.04516 5.37% Gravity::actual_multilevel_solve() 1 0.04514 0.04514 0.04514 5.36% MLCellLinOp::apply() 1142 0.03657 0.03657 0.03657 4.35% MLMG::mgVcycle_down::0 82 0.03573 0.03573 0.03573 4.25% MLMG::mgVcycle_up::0 82 0.03054 0.03054 0.03054 3.63% StateData::FillBoundary(geom) 328 0.0245 0.0245 0.0245 2.91% MultiFab::Dot() 1114 0.02268 0.02268 0.02268 2.70% MLCellLinOp::correctionResidual() 492 0.02145 0.02145 0.02145 2.55% Castro::initialize_do_advance() 10 0.02088 0.02088 0.02088 2.48% Castro::computeTemp() 63 0.01924 0.01924 0.01924 2.29% MLMG:computeResOfCorrection() 410 0.01852 0.01852 0.01852 2.20% MLPoisson::define() 11 0.01841 0.01841 0.01841 2.19% MLMG::mgVcycle_down::1 82 0.01787 0.01787 0.01787 2.12% MLMG::mgVcycle_down::2 82 0.01739 0.01739 0.01739 2.07% Gravity::get_new_grav_vector() 11 0.01691 0.01691 0.01691 2.01% MLMG::mgVcycle_down::3 82 0.01649 0.01649 0.01649 1.96% Castro::normalize_species() 62 0.01602 0.01602 0.01602 1.90% FabArray::FillBoundary() 4023 0.01585 0.01585 0.01585 1.88% MLMG::mgVcycle_down::4 82 0.01571 0.01571 0.01571 1.87% Castro::do_new_sources() 10 0.01534 0.01534 0.01534 1.82% Castro::construct_old_gravity() 10 0.01515 0.01515 0.01515 1.80% Gravity::get_old_grav_vector() 10 0.01514 0.01514 0.01514 1.80% FillBoundary_nowait() 4023 0.01505 0.01505 0.01505 1.79% CGSolver::sxay() 1586 0.01495 0.01495 0.01495 1.78% MultiFab::LinComb() 1586 0.01454 0.01454 0.01454 1.73% FabArray::setVal() 1144 0.01447 0.01447 0.01447 1.72% FabArray::ParallelCopy() 861 0.01428 0.01428 0.01428 1.70% FabArray::ParallelCopy_nowait() 861 0.014 0.014 0.014 1.66% MLCGSolver::ParallelAllReduce 1514 0.01353 0.01353 0.01353 1.61% MLMG::mgVcycle_up::2 82 0.01334 0.01334 0.01334 1.59% MLMG::mgVcycle_up::1 82 0.01307 0.01307 0.01307 1.55% MLCellLinOp::defineAuxData() 11 0.01299 0.01299 0.01299 1.54% MLMG::addInterpCorrection() 410 0.01266 0.01266 0.01266 1.50% MLMG::mgVcycle_up::3 82 0.01262 0.01262 0.01262 1.50% Castro::enforce_min_density() 62 0.01262 0.01262 0.01262 1.50% MLMG::mgVcycle_up::4 82 0.01248 0.01248 0.01248 1.48% amrex::average_down 410 0.01217 0.01217 0.01217 1.45% MLPoisson::Fapply() 1142 0.01182 0.01182 0.01182 1.40% Castro::expand_state() 10 0.01111 0.01111 0.01111 1.32% Castro::do_old_sources() 10 0.01074 0.01074 0.01074 1.28% Castro::initialize_advance() 10 0.00976 0.00976 0.00976 1.16% Gravity::fill_multipole_BCs() 11 0.009424 0.009424 0.009424 1.12% MLCellLinOp::solutionResidual() 93 0.007205 0.007205 0.007205 0.86% MultiFab::Xpay() 585 0.006657 0.006657 0.006657 0.79% Castro::post_timestep() 10 0.006613 0.006613 0.006613 0.79% MLMG::computeResidual() 82 0.006198 0.006198 0.006198 0.74% Castro::estTimeStep() 21 0.006109 0.006109 0.006109 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005389 0.005389 0.005389 0.64% MLMG::prepareForSolve() 11 0.005179 0.005179 0.005179 0.62% MLCellLinOp::defineBC() 11 0.005158 0.005158 0.005158 0.61% BndryData::define() 11 0.004877 0.004877 0.004877 0.58% Amr::InitializeInit() 1 0.00395 0.00395 0.00395 0.47% Amr::defBaseLevel() 1 0.003944 0.003944 0.003944 0.47% Castro::computeNewDt() 9 0.00352 0.00352 0.00352 0.42% Castro::initData() 1 0.003371 0.003371 0.003371 0.40% Castro::construct_new_source() 50 0.003145 0.003145 0.003145 0.37% Castro::construct_new_gravity_source() 10 0.003095 0.003095 0.003095 0.37% Castro::construct_old_source() 50 0.002696 0.002696 0.002696 0.32% Castro::construct_old_gravity_source() 10 0.002678 0.002678 0.002678 0.32% MLMG::ResNormInf() 93 0.001928 0.001928 0.001928 0.23% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.22% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001681 0.001681 0.001681 0.20% MLCellLinOp::setLevelBC() 11 0.001553 0.001553 0.001553 0.18% Castro::reset_internal_energy(Fab) 504 0.00148 0.00148 0.00148 0.18% FabArrayBase::getCPC() 1323 0.001466 0.001466 0.001466 0.17% MLMG::getGradSolution() 11 0.001421 0.001421 0.001421 0.17% MLCellLinOp::compGrad() 11 0.001415 0.001415 0.001415 0.17% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% FabArray::setDomainBndry() 41 0.001313 0.001313 0.001313 0.16% Castro::check_for_nan() 20 0.001195 0.001195 0.001195 0.14% MLPoisson::prepareForSolve() 11 0.001191 0.001191 0.001191 0.14% MLCellLinOp::prepareForSolve() 11 0.001184 0.001184 0.001184 0.14% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% Castro::post_regrid() 1 0.001115 0.001115 0.001115 0.13% Castro::enforce_speed_limit() 62 0.001109 0.001109 0.001109 0.13% MLMG::computeMLResidual() 11 0.001046 0.001046 0.001046 0.12% Gravity::update_max_rhs() 11 0.0008217 0.0008217 0.0008217 0.10% FabArrayBase::getFB() 4023 0.0007197 0.0007197 0.0007197 0.09% FabArrayBase::CPC::define() 454 0.0006711 0.0006711 0.0006711 0.08% Castro::computeInitialDt() 2 0.0006539 0.0006539 0.0006539 0.08% Castro::Castro() 1 0.0005007 0.0005007 0.0005007 0.06% Amr::InitAmr() 1 0.0004965 0.0004965 0.0004965 0.06% Gravity::swapTimeLevels() 10 0.0004401 0.0004401 0.0004401 0.05% MultiFab::Copy() 11 0.0002605 0.0002605 0.0002605 0.03% MultiFab::max() 11 0.0002578 0.0002578 0.0002578 0.03% MLMG::MLResNormInf() 11 0.0002532 0.0002532 0.0002532 0.03% MLLinOp::define() 11 0.0002388 0.0002388 0.0002388 0.03% MLLinOp::defineGrids() 11 0.0002159 0.0002159 0.0002159 0.03% MLMG::MLRhsNormInf() 11 0.0002022 0.0002022 0.0002022 0.02% Castro::buildMetrics() 1 0.0001904 0.0001904 0.0001904 0.02% Castro::create_source_corrector() 10 0.0001734 0.0001734 0.0001734 0.02% FabArrayBase::FB::FB() 56 9.732e-05 9.732e-05 9.732e-05 0.01% Castro::finalize_advance() 10 6.778e-05 6.778e-05 6.778e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.266e-05 5.266e-05 5.266e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.512e-05 4.512e-05 4.512e-05 0.01% Castro::swap_state_time_levels() 10 3.993e-05 3.993e-05 3.993e-05 0.00% makeSFC 55 3.993e-05 3.993e-05 3.993e-05 0.00% Castro::finalize_do_advance() 10 3.878e-05 3.878e-05 3.878e-05 0.00% StateData::define() 4 3.831e-05 3.831e-05 3.831e-05 0.00% Castro::enforce_consistent_e() 1 3.391e-05 3.391e-05 3.391e-05 0.00% Castro::initMFs() 1 2.876e-05 2.876e-05 2.876e-05 0.00% Amr::writeSmallPlotFile() 1 2.664e-05 2.664e-05 2.664e-05 0.00% DistributionMapping::Distribute() 56 1.471e-05 1.471e-05 1.471e-05 0.00% Amr::initSubcycle() 1 8.841e-06 8.841e-06 8.841e-06 0.00% MLMG::buildFineMask() 11 5.437e-06 5.437e-06 5.437e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.709e-06 4.709e-06 4.709e-06 0.00% Castro::retry_advance_ctu() 10 3.879e-06 3.879e-06 3.879e-06 0.00% AmrLevel::checkPointPost() 3 3.861e-06 3.861e-06 3.861e-06 0.00% Gravity::set_mass_offset() 11 3.741e-06 3.741e-06 3.741e-06 0.00% Castro::FluxRegCrseInit 10 2.725e-06 2.725e-06 2.725e-06 0.00% Castro::FluxRegFineAdd() 10 2.526e-06 2.526e-06 2.526e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.922e-06 1.922e-06 1.922e-06 0.00% AmrLevel::checkPointPre() 3 1.758e-06 1.758e-06 1.758e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-48-gb2b9150ada12) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-48-gb2b9150ada12) initialized Starting run at 08:29:22 UTC on 2022-06-29. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-48-gb2b9150ad Microphysics git describe: 22.06-7-ga6a75714 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.464885423 Restart time = 0.046903673 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049396701 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051447493 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.066308335 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066334159 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069845332 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026515393 seconds Ending run at 08:29:22 UTC on 2022-06-29. Run time = 0.377678159 Run time without initialization = 0.330226794 Average number of zones advanced per microsecond: 3.969 Average number of zones advanced per microsecond per rank: 3.969 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3777 ... 0.3777 ... 0.3777 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1074 0.1074 0.1074 28.43% VisMF::Read() 3 0.03948 0.03948 0.03948 10.45% MLCellLinOp::applyBC() 1946 0.03451 0.03451 0.03451 9.14% MLPoisson::Fsmooth() 1440 0.02711 0.02711 0.02711 7.18% VisMF::Write(FabArray) 1 0.02516 0.02516 0.02516 6.66% StateData::FillBoundary(geom) 160 0.01184 0.01184 0.01184 3.13% MLCGSolver::bicgstab 36 0.0102 0.0102 0.0102 2.70% MultiFab::Dot() 484 0.009545 0.009545 0.009545 2.53% Castro::enforce_min_density() 30 0.007761 0.007761 0.007761 2.05% Castro::normalize_species() 30 0.007749 0.007749 0.007749 2.05% FabArray::setVal() 537 0.006737 0.006737 0.006737 1.78% FillBoundary_nowait() 1766 0.006183 0.006183 0.006183 1.64% MLCellLinOp::defineAuxData() 6 0.00618 0.00618 0.00618 1.64% MultiFab::LinComb() 690 0.006102 0.006102 0.006102 1.62% FabArray::ParallelCopy_nowait() 380 0.005871 0.005871 0.005871 1.55% Gravity::fill_multipole_BCs() 6 0.005375 0.005375 0.005375 1.42% Castro::computeTemp() 30 0.005226 0.005226 0.005226 1.38% StateDataPhysBCFunct::() 20 0.00519 0.00519 0.00519 1.37% MLPoisson::Fapply() 500 0.005039 0.005039 0.005039 1.33% MLMG::addInterpCorrection() 180 0.0032 0.0032 0.0032 0.85% Amr::restart() 1 0.00318 0.00318 0.00318 0.84% amrex::average_down 180 0.002957 0.002957 0.002957 0.78% Castro::estTimeStep() 10 0.002909 0.002909 0.002909 0.77% MultiFab::Xpay() 258 0.002855 0.002855 0.002855 0.76% Castro::do_advance_ctu() 5 0.002542 0.002542 0.002542 0.67% BndryData::define() 6 0.002138 0.002138 0.002138 0.57% Castro::reset_internal_energy(MultiFab) 30 0.002053 0.002053 0.002053 0.54% Castro::construct_new_gravity_source() 5 0.00162 0.00162 0.00162 0.43% Amr::writePlotFile() 1 0.001442 0.001442 0.001442 0.38% Castro::construct_old_gravity_source() 5 0.001405 0.001405 0.001405 0.37% MultiFab::Saxpy() 10 0.0009189 0.0009189 0.0009189 0.24% Gravity::get_old_grav_vector() 5 0.0008836 0.0008836 0.0008836 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008819 0.0008819 0.0008819 0.23% Castro::expand_state() 5 0.0008696 0.0008696 0.0008696 0.23% Gravity::get_new_grav_vector() 5 0.0008622 0.0008622 0.0008622 0.23% MLMG::ResNormInf() 42 0.0008415 0.0008415 0.0008415 0.22% MLCellLinOp::setLevelBC() 6 0.0008138 0.0008138 0.0008138 0.22% MLMG::oneIter() 36 0.000727 0.000727 0.000727 0.19% Castro::reset_internal_energy(Fab) 240 0.0007041 0.0007041 0.0007041 0.19% Gravity::actual_solve_with_mlmg() 6 0.0006932 0.0006932 0.0006932 0.18% FabArray::mult() 22 0.0006448 0.0006448 0.0006448 0.17% FabArray::setDomainBndry() 20 0.0006342 0.0006342 0.0006342 0.17% MLCellLinOp::prepareForSolve() 6 0.0006305 0.0006305 0.0006305 0.17% Castro::enforce_speed_limit() 30 0.0005874 0.0005874 0.0005874 0.16% MultiFab::contains_nan() 10 0.0005857 0.0005857 0.0005857 0.16% MLMG::prepareForSolve() 6 0.0005685 0.0005685 0.0005685 0.15% MLCellLinOp::compGrad() 6 0.0004893 0.0004893 0.0004893 0.13% MLCellLinOp::smooth() 720 0.0004416 0.0004416 0.0004416 0.12% FabArrayBase::CPC::define() 244 0.0004085 0.0004085 0.0004085 0.11% Amr::InitAmr() 1 0.000393 0.000393 0.000393 0.10% FabArray::FillBoundary() 1766 0.0003854 0.0003854 0.0003854 0.10% FabArrayBase::getCPC() 632 0.0003722 0.0003722 0.0003722 0.10% FabArrayBase::getFB() 1766 0.0002515 0.0002515 0.0002515 0.07% main() 1 0.000239 0.000239 0.000239 0.06% Gravity::update_max_rhs() 6 0.0002257 0.0002257 0.0002257 0.06% Gravity::solve_for_phi() 5 0.0002032 0.0002032 0.0002032 0.05% MLCellLinOp::apply() 500 0.0001856 0.0001856 0.0001856 0.05% CGSolver::sxay() 690 0.0001799 0.0001799 0.0001799 0.05% Amr::coarseTimeStep() 5 0.0001491 0.0001491 0.0001491 0.04% MLCellLinOp::defineBC() 6 0.0001474 0.0001474 0.0001474 0.04% FillPatchIterator::Initialize 20 0.0001429 0.0001429 0.0001429 0.04% MultiFab::Copy() 6 0.0001368 0.0001368 0.0001368 0.04% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.04% FabArray::ParallelCopy() 380 0.0001286 0.0001286 0.0001286 0.03% MLCGSolver::ParallelAllReduce 659 0.0001257 0.0001257 0.0001257 0.03% Castro::create_source_corrector() 5 0.0001136 0.0001136 0.0001136 0.03% Castro::construct_new_gravity() 5 0.0001072 0.0001072 0.0001072 0.03% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.03% Castro::subcycle_advance_ctu() 5 0.0001029 0.0001029 0.0001029 0.03% Castro::finalize_advance() 5 0.0001001 0.0001001 0.0001001 0.03% MLMG::mgVcycle() 36 9.991e-05 9.991e-05 9.991e-05 0.03% Amr::timeStep() 5 9.589e-05 9.589e-05 9.589e-05 0.03% MLCellLinOp::correctionResidual() 216 9.588e-05 9.588e-05 9.588e-05 0.03% AmrLevel::restart() 1 7.691e-05 7.691e-05 7.691e-05 0.02% StateData::restartDoit() 4 7.073e-05 7.073e-05 7.073e-05 0.02% MLLinOp::defineGrids() 6 6.821e-05 6.821e-05 6.821e-05 0.02% Castro::initialize_advance() 5 5.833e-05 5.833e-05 5.833e-05 0.02% FabArrayBase::FB::FB() 26 5.681e-05 5.681e-05 5.681e-05 0.02% MLMG:computeResOfCorrection() 180 5.599e-05 5.599e-05 5.599e-05 0.01% MLMG::actualBottomSolve() 36 4.361e-05 4.361e-05 4.361e-05 0.01% MLMG::mgVcycle_down::0 36 3.885e-05 3.885e-05 3.885e-05 0.01% Castro::clean_state() 30 3.876e-05 3.876e-05 3.876e-05 0.01% Castro::post_restart() 1 3.852e-05 3.852e-05 3.852e-05 0.01% MLMG::mgVcycle_down::1 36 3.756e-05 3.756e-05 3.756e-05 0.01% MLMG::solve() 6 3.637e-05 3.637e-05 3.637e-05 0.01% MLMG::mgVcycle_down::2 36 3.565e-05 3.565e-05 3.565e-05 0.01% MLMG::mgVcycle_down::4 36 3.308e-05 3.308e-05 3.308e-05 0.01% Castro::buildMetrics() 1 3.245e-05 3.245e-05 3.245e-05 0.01% MLMG::mgVcycle_down::3 36 3.178e-05 3.178e-05 3.178e-05 0.01% Castro::initialize_do_advance() 5 3.178e-05 3.178e-05 3.178e-05 0.01% Gravity::actual_multilevel_solve() 1 2.974e-05 2.974e-05 2.974e-05 0.01% Castro::advance() 5 2.842e-05 2.842e-05 2.842e-05 0.01% MLMG::mgVcycle_up::4 36 2.805e-05 2.805e-05 2.805e-05 0.01% Castro::initMFs() 1 2.65e-05 2.65e-05 2.65e-05 0.01% Castro::swap_state_time_levels() 5 2.648e-05 2.648e-05 2.648e-05 0.01% Amr::writeSmallPlotFile() 1 2.615e-05 2.615e-05 2.615e-05 0.01% MLCellLinOp::solutionResidual() 42 2.363e-05 2.363e-05 2.363e-05 0.01% MLMG::mgVcycle_up::3 36 2.227e-05 2.227e-05 2.227e-05 0.01% MLMG::mgVcycle_up::0 36 2.159e-05 2.159e-05 2.159e-05 0.01% MLMG::mgVcycle_up::2 36 2.089e-05 2.089e-05 2.089e-05 0.01% MLLinOp::define() 6 1.985e-05 1.985e-05 1.985e-05 0.01% MLMG::mgVcycle_up::1 36 1.934e-05 1.934e-05 1.934e-05 0.01% Castro::finalize_do_advance() 5 1.806e-05 1.806e-05 1.806e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.776e-05 1.776e-05 1.776e-05 0.00% MLMG::mgVcycle_bottom 36 1.479e-05 1.479e-05 1.479e-05 0.00% MLMG::computeResidual() 36 1.458e-05 1.458e-05 1.458e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.452e-05 1.452e-05 1.452e-05 0.00% FillPatchSingleLevel 20 1.422e-05 1.422e-05 1.422e-05 0.00% makeSFC 30 1.39e-05 1.39e-05 1.39e-05 0.00% Castro::apply_source_to_state() 10 1.327e-05 1.327e-05 1.327e-05 0.00% MLPoisson::define() 6 1.286e-05 1.286e-05 1.286e-05 0.00% Castro::construct_old_source() 25 1.089e-05 1.089e-05 1.089e-05 0.00% Castro::construct_new_source() 25 9.713e-06 9.713e-06 9.713e-06 0.00% DistributionMapping::Distribute() 31 8.736e-06 8.736e-06 8.736e-06 0.00% Amr::initSubcycle() 1 8.542e-06 8.542e-06 8.542e-06 0.00% Castro::do_old_sources() 5 8.343e-06 8.343e-06 8.343e-06 0.00% Castro::do_new_sources() 5 8.206e-06 8.206e-06 8.206e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.976e-06 6.976e-06 6.976e-06 0.00% Castro::check_for_nan() 10 6.968e-06 6.968e-06 6.968e-06 0.00% Castro::computeNewDt() 5 5.392e-06 5.392e-06 5.392e-06 0.00% Castro::post_timestep() 5 5.295e-06 5.295e-06 5.295e-06 0.00% Castro::construct_old_gravity() 5 5.23e-06 5.23e-06 5.23e-06 0.00% Gravity::swapTimeLevels() 5 4.455e-06 4.455e-06 4.455e-06 0.00% MLPoisson::prepareForSolve() 6 4.223e-06 4.223e-06 4.223e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.51e-06 3.51e-06 3.51e-06 0.00% MLMG::getGradSolution() 6 2.906e-06 2.906e-06 2.906e-06 0.00% MLMG::computeMLResidual() 6 2.867e-06 2.867e-06 2.867e-06 0.00% MLMG::buildFineMask() 6 2.812e-06 2.812e-06 2.812e-06 0.00% MLMG::MLResNormInf() 6 2.366e-06 2.366e-06 2.366e-06 0.00% Gravity::set_mass_offset() 6 1.924e-06 1.924e-06 1.924e-06 0.00% Castro::FluxRegCrseInit 5 1.834e-06 1.834e-06 1.834e-06 0.00% Castro::retry_advance_ctu() 5 1.798e-06 1.798e-06 1.798e-06 0.00% Amr::init() 1 1.2e-06 1.2e-06 1.2e-06 0.00% Castro::FluxRegFineAdd() 5 1.159e-06 1.159e-06 1.159e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.112e-06 1.112e-06 1.112e-06 0.00% AmrLevel::AmrLevel() 1 9.2e-07 9.2e-07 9.2e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3777 0.3777 0.3777 100.00% Amr::coarseTimeStep() 5 0.3035 0.3035 0.3035 80.35% Amr::timeStep() 5 0.3014 0.3014 0.3014 79.81% Castro::advance() 5 0.2985 0.2985 0.2985 79.04% Castro::subcycle_advance_ctu() 5 0.2929 0.2929 0.2929 77.54% Castro::do_advance_ctu() 5 0.2928 0.2928 0.2928 77.51% Castro::construct_new_gravity() 5 0.1432 0.1432 0.1432 37.90% Gravity::solve_phi_with_mlmg() 6 0.1392 0.1392 0.1392 36.84% Gravity::solve_for_phi() 5 0.1356 0.1356 0.1356 35.91% Gravity::actual_solve_with_mlmg() 6 0.1337 0.1337 0.1337 35.39% MLMG::solve() 6 0.1215 0.1215 0.1215 32.17% MLMG::oneIter() 36 0.1145 0.1145 0.1145 30.33% MLMG::mgVcycle() 36 0.1138 0.1138 0.1138 30.13% Castro::construct_ctu_hydro_source() 5 0.1074 0.1074 0.1074 28.43% MLCellLinOp::smooth() 720 0.0582 0.0582 0.0582 15.41% Amr::init() 1 0.04695 0.04695 0.04695 12.43% Amr::restart() 1 0.04695 0.04695 0.04695 12.43% MLCellLinOp::applyBC() 1946 0.04139 0.04139 0.04139 10.96% AmrLevel::restart() 1 0.03968 0.03968 0.03968 10.51% StateData::restartDoit() 4 0.0396 0.0396 0.0396 10.48% VisMF::Read() 3 0.03948 0.03948 0.03948 10.45% MLMG::mgVcycle_bottom 36 0.035 0.035 0.035 9.27% MLMG::actualBottomSolve() 36 0.03498 0.03498 0.03498 9.26% MLCGSolver::bicgstab 36 0.03463 0.03463 0.03463 9.17% MLPoisson::Fsmooth() 1440 0.02711 0.02711 0.02711 7.18% Amr::writePlotFile() 1 0.0266 0.0266 0.0266 7.04% VisMF::Write(FabArray) 1 0.02516 0.02516 0.02516 6.66% Castro::clean_state() 30 0.02412 0.02412 0.02412 6.39% FillPatchIterator::Initialize 20 0.0198 0.0198 0.0198 5.24% FillPatchSingleLevel 20 0.01903 0.01903 0.01903 5.04% StateDataPhysBCFunct::() 20 0.01703 0.01703 0.01703 4.51% MLCellLinOp::apply() 500 0.0157 0.0157 0.0157 4.16% MLMG::mgVcycle_down::0 36 0.01531 0.01531 0.01531 4.05% MLMG::mgVcycle_up::0 36 0.01303 0.01303 0.01303 3.45% StateData::FillBoundary(geom) 160 0.01184 0.01184 0.01184 3.13% Castro::initialize_do_advance() 5 0.01084 0.01084 0.01084 2.87% MLPoisson::define() 6 0.009879 0.009879 0.009879 2.62% MultiFab::Dot() 484 0.009545 0.009545 0.009545 2.53% MLCellLinOp::correctionResidual() 216 0.009159 0.009159 0.009159 2.42% Castro::do_new_sources() 5 0.008674 0.008674 0.008674 2.30% Castro::computeTemp() 30 0.007983 0.007983 0.007983 2.11% MLMG:computeResOfCorrection() 180 0.007909 0.007909 0.007909 2.09% Castro::enforce_min_density() 30 0.007761 0.007761 0.007761 2.05% Castro::normalize_species() 30 0.007749 0.007749 0.007749 2.05% MLMG::mgVcycle_down::1 36 0.007576 0.007576 0.007576 2.01% Castro::construct_old_gravity() 5 0.00744 0.00744 0.00744 1.97% Gravity::get_old_grav_vector() 5 0.007435 0.007435 0.007435 1.97% Gravity::get_new_grav_vector() 5 0.007429 0.007429 0.007429 1.97% MLMG::mgVcycle_down::2 36 0.00736 0.00736 0.00736 1.95% MLMG::mgVcycle_down::3 36 0.006982 0.006982 0.006982 1.85% MLCellLinOp::defineAuxData() 6 0.00693 0.00693 0.00693 1.83% FabArray::FillBoundary() 1766 0.006877 0.006877 0.006877 1.82% FabArray::setVal() 537 0.006737 0.006737 0.006737 1.78% MLMG::mgVcycle_down::4 36 0.006655 0.006655 0.006655 1.76% FillBoundary_nowait() 1766 0.006491 0.006491 0.006491 1.72% FabArray::ParallelCopy() 380 0.006379 0.006379 0.006379 1.69% CGSolver::sxay() 690 0.006282 0.006282 0.006282 1.66% FabArray::ParallelCopy_nowait() 380 0.006251 0.006251 0.006251 1.65% MultiFab::LinComb() 690 0.006102 0.006102 0.006102 1.62% MLCGSolver::ParallelAllReduce 659 0.005747 0.005747 0.005747 1.52% Castro::expand_state() 5 0.005655 0.005655 0.005655 1.50% MLMG::mgVcycle_up::2 36 0.005634 0.005634 0.005634 1.49% MLMG::mgVcycle_up::1 36 0.005547 0.005547 0.005547 1.47% Castro::initialize_advance() 5 0.00551 0.00551 0.00551 1.46% MLMG::addInterpCorrection() 180 0.005387 0.005387 0.005387 1.43% Gravity::fill_multipole_BCs() 6 0.005375 0.005375 0.005375 1.42% MLMG::mgVcycle_up::3 36 0.005333 0.005333 0.005333 1.41% Castro::do_old_sources() 5 0.005298 0.005298 0.005298 1.40% MLMG::mgVcycle_up::4 36 0.00529 0.00529 0.00529 1.40% amrex::average_down 180 0.005164 0.005164 0.005164 1.37% MLPoisson::Fapply() 500 0.005039 0.005039 0.005039 1.33% Castro::post_restart() 1 0.003903 0.003903 0.003903 1.03% Gravity::multilevel_solve_for_new_phi() 1 0.003774 0.003774 0.003774 1.00% Gravity::actual_multilevel_solve() 1 0.003756 0.003756 0.003756 0.99% MLCellLinOp::solutionResidual() 42 0.003218 0.003218 0.003218 0.85% Castro::estTimeStep() 10 0.002909 0.002909 0.002909 0.77% MultiFab::Xpay() 258 0.002855 0.002855 0.002855 0.76% Castro::post_timestep() 5 0.002827 0.002827 0.002827 0.75% MLCellLinOp::defineBC() 6 0.002819 0.002819 0.002819 0.75% Castro::reset_internal_energy(MultiFab) 30 0.002757 0.002757 0.002757 0.73% MLMG::prepareForSolve() 6 0.002748 0.002748 0.002748 0.73% BndryData::define() 6 0.002671 0.002671 0.002671 0.71% MLMG::computeResidual() 36 0.002667 0.002667 0.002667 0.71% Castro::computeNewDt() 5 0.00188 0.00188 0.00188 0.50% Castro::construct_new_source() 25 0.00163 0.00163 0.00163 0.43% Castro::construct_new_gravity_source() 5 0.00162 0.00162 0.00162 0.43% Castro::construct_old_source() 25 0.001416 0.001416 0.001416 0.37% Castro::construct_old_gravity_source() 5 0.001405 0.001405 0.001405 0.37% Castro::apply_source_to_state() 10 0.0009322 0.0009322 0.0009322 0.25% MultiFab::Saxpy() 10 0.0009189 0.0009189 0.0009189 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008819 0.0008819 0.0008819 0.23% MLMG::ResNormInf() 42 0.0008415 0.0008415 0.0008415 0.22% MLCellLinOp::setLevelBC() 6 0.0008138 0.0008138 0.0008138 0.22% FabArrayBase::getCPC() 632 0.0007807 0.0007807 0.0007807 0.21% MLMG::getGradSolution() 6 0.000758 0.000758 0.000758 0.20% MLCellLinOp::compGrad() 6 0.0007551 0.0007551 0.0007551 0.20% Castro::reset_internal_energy(Fab) 240 0.0007041 0.0007041 0.0007041 0.19% FabArray::mult() 22 0.0006448 0.0006448 0.0006448 0.17% MLPoisson::prepareForSolve() 6 0.0006347 0.0006347 0.0006347 0.17% FabArray::setDomainBndry() 20 0.0006342 0.0006342 0.0006342 0.17% MLCellLinOp::prepareForSolve() 6 0.0006305 0.0006305 0.0006305 0.17% Castro::check_for_nan() 10 0.0005927 0.0005927 0.0005927 0.16% Castro::enforce_speed_limit() 30 0.0005874 0.0005874 0.0005874 0.16% MultiFab::contains_nan() 10 0.0005857 0.0005857 0.0005857 0.16% MLMG::computeMLResidual() 6 0.0005692 0.0005692 0.0005692 0.15% Gravity::update_max_rhs() 6 0.0004369 0.0004369 0.0004369 0.12% FabArrayBase::CPC::define() 244 0.0004085 0.0004085 0.0004085 0.11% Amr::InitAmr() 1 0.0004015 0.0004015 0.0004015 0.11% FabArrayBase::getFB() 1766 0.0003083 0.0003083 0.0003083 0.08% Gravity::swapTimeLevels() 5 0.0002376 0.0002376 0.0002376 0.06% Castro::buildMetrics() 1 0.0001533 0.0001533 0.0001533 0.04% MultiFab::Copy() 6 0.0001368 0.0001368 0.0001368 0.04% MLMG::MLResNormInf() 6 0.0001342 0.0001342 0.0001342 0.04% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.04% MLLinOp::define() 6 0.0001171 0.0001171 0.0001171 0.03% Castro::create_source_corrector() 5 0.0001136 0.0001136 0.0001136 0.03% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.03% Castro::finalize_advance() 5 0.0001031 0.0001031 0.0001031 0.03% MLLinOp::defineGrids() 6 9.727e-05 9.727e-05 9.727e-05 0.03% FabArrayBase::FB::FB() 26 5.681e-05 5.681e-05 5.681e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.795e-05 2.795e-05 2.795e-05 0.01% Castro::initMFs() 1 2.65e-05 2.65e-05 2.65e-05 0.01% Castro::swap_state_time_levels() 5 2.648e-05 2.648e-05 2.648e-05 0.01% Amr::writeSmallPlotFile() 1 2.615e-05 2.615e-05 2.615e-05 0.01% makeSFC 30 2.097e-05 2.097e-05 2.097e-05 0.01% Castro::finalize_do_advance() 5 1.806e-05 1.806e-05 1.806e-05 0.00% DistributionMapping::Distribute() 31 8.736e-06 8.736e-06 8.736e-06 0.00% Amr::initSubcycle() 1 8.542e-06 8.542e-06 8.542e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.17e-06 5.17e-06 5.17e-06 0.00% MLMG::buildFineMask() 6 2.812e-06 2.812e-06 2.812e-06 0.00% Gravity::set_mass_offset() 6 1.924e-06 1.924e-06 1.924e-06 0.00% Castro::FluxRegCrseInit 5 1.834e-06 1.834e-06 1.834e-06 0.00% Castro::retry_advance_ctu() 5 1.798e-06 1.798e-06 1.798e-06 0.00% Castro::FluxRegFineAdd() 5 1.159e-06 1.159e-06 1.159e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.112e-06 1.112e-06 1.112e-06 0.00% AmrLevel::AmrLevel() 1 9.2e-07 9.2e-07 9.2e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-48-gb2b9150ada12) finalized