Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-19-g48702b48836d) initialized Starting run at 08:29:09 UTC on 2022-07-29. Successfully read inputs file ... Castro git describe: 22.07-13-g4177c4b30 AMReX git describe: 22.07-19-g48702b488 Microphysics git describe: 22.07-15-ga4952214 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05124794 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029498852 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048750511 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049984428 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051415157 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.049534304 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.065863623 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.059700291 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.079351066 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.067819067 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061182074 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.070956782 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.070257672 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048789463 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029147047 seconds Ending run at 08:29:10 UTC on 2022-07-29. Run time = 0.883661932 Run time without initialization = 0.753545808 Average number of zones advanced per microsecond: 3.479 Average number of zones advanced per microsecond per rank: 3.479 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8837 ... 0.8837 ... 0.8837 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2131 0.2131 0.2131 24.12% VisMF::Write(FabArray) 11 0.2097 0.2097 0.2097 23.73% MLCellLinOp::applyBC() 4379 0.08098 0.08098 0.08098 9.16% MLPoisson::Fsmooth() 3240 0.06368 0.06368 0.06368 7.21% StateData::FillBoundary(geom) 328 0.02743 0.02743 0.02743 3.10% MLCGSolver::bicgstab 81 0.02386 0.02386 0.02386 2.70% MultiFab::Dot() 1100 0.02208 0.02208 0.02208 2.50% Castro::normalize_species() 62 0.01682 0.01682 0.01682 1.90% StateDataPhysBCFunct::() 41 0.01627 0.01627 0.01627 1.84% FillBoundary_nowait() 3974 0.01452 0.01452 0.01452 1.64% MultiFab::LinComb() 1566 0.01443 0.01443 0.01443 1.63% FabArray::setVal() 1135 0.01424 0.01424 0.01424 1.61% Castro::computeTemp() 63 0.01409 0.01409 0.01409 1.59% FabArray::ParallelCopy_nowait() 851 0.01328 0.01328 0.01328 1.50% MLCellLinOp::defineAuxData() 11 0.01205 0.01205 0.01205 1.36% MLPoisson::Fapply() 1128 0.0118 0.0118 0.0118 1.33% Castro::enforce_min_density() 62 0.0103 0.0103 0.0103 1.17% Gravity::fill_multipole_BCs() 11 0.009641 0.009641 0.009641 1.09% MLMG::addInterpCorrection() 405 0.007456 0.007456 0.007456 0.84% Castro::estTimeStep() 21 0.007355 0.007355 0.007355 0.83% amrex::average_down 405 0.006944 0.006944 0.006944 0.79% MultiFab::Xpay() 578 0.00655 0.00655 0.00655 0.74% Amr::checkPoint() 3 0.005838 0.005838 0.005838 0.66% Castro::do_advance_ctu() 10 0.004561 0.004561 0.004561 0.52% BndryData::define() 11 0.003878 0.003878 0.003878 0.44% Castro::reset_internal_energy(MultiFab) 63 0.003785 0.003785 0.003785 0.43% Amr::writePlotFile() 2 0.002882 0.002882 0.002882 0.33% Castro::construct_new_gravity_source() 10 0.002754 0.002754 0.002754 0.31% Castro::construct_old_gravity_source() 10 0.002096 0.002096 0.002096 0.24% Gravity::get_new_grav_vector() 11 0.001982 0.001982 0.001982 0.22% MLMG::ResNormInf() 92 0.001934 0.001934 0.001934 0.22% MultiFab::Saxpy() 20 0.001841 0.001841 0.001841 0.21% Gravity::get_old_grav_vector() 10 0.001799 0.001799 0.001799 0.20% Castro::expand_state() 10 0.001764 0.001764 0.001764 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001731 0.001731 0.001731 0.20% MLMG::oneIter() 81 0.001704 0.001704 0.001704 0.19% Castro::reset_internal_energy(Fab) 504 0.001633 0.001633 0.001633 0.18% MLCellLinOp::setLevelBC() 11 0.001589 0.001589 0.001589 0.18% Gravity::actual_solve_with_mlmg() 11 0.001553 0.001553 0.001553 0.18% FabArray::setDomainBndry() 41 0.001376 0.001376 0.001376 0.16% FabArray::mult() 43 0.001343 0.001343 0.001343 0.15% MLCellLinOp::prepareForSolve() 11 0.00122 0.00122 0.00122 0.14% MultiFab::contains_nan() 20 0.001217 0.001217 0.001217 0.14% Castro::initData() 1 0.001199 0.001199 0.001199 0.14% MLMG::prepareForSolve() 11 0.001159 0.001159 0.001159 0.13% MLCellLinOp::smooth() 1620 0.001117 0.001117 0.001117 0.13% Castro::enforce_speed_limit() 62 0.001062 0.001062 0.001062 0.12% MLCellLinOp::compGrad() 11 0.0009286 0.0009286 0.0009286 0.11% FabArray::FillBoundary() 3974 0.0008657 0.0008657 0.0008657 0.10% FabArrayBase::getCPC() 1313 0.0008629 0.0008629 0.0008629 0.10% FabArrayBase::CPC::define() 454 0.0007455 0.0007455 0.0007455 0.08% Castro::subcycle_advance_ctu() 10 0.0006759 0.0006759 0.0006759 0.08% FabArrayBase::getFB() 3974 0.0006345 0.0006345 0.0006345 0.07% Gravity::update_max_rhs() 11 0.000506 0.000506 0.000506 0.06% Gravity::solve_for_phi() 10 0.0004956 0.0004956 0.0004956 0.06% Amr::InitAmr() 1 0.0004944 0.0004944 0.0004944 0.06% Amr::coarseTimeStep() 10 0.0004724 0.0004724 0.0004724 0.05% MLCellLinOp::apply() 1128 0.0004635 0.0004635 0.0004635 0.05% CGSolver::sxay() 1566 0.0003931 0.0003931 0.0003931 0.04% FillPatchIterator::Initialize 41 0.0003456 0.0003456 0.0003456 0.04% MLCellLinOp::defineBC() 11 0.0003151 0.0003151 0.0003151 0.04% Amr::timeStep() 10 0.0003128 0.0003128 0.0003128 0.04% FabArray::ParallelCopy() 851 0.0003114 0.0003114 0.0003114 0.04% MLCGSolver::ParallelAllReduce 1495 0.0003087 0.0003087 0.0003087 0.03% MultiFab::max() 11 0.0002734 0.0002734 0.0002734 0.03% main() 1 0.0002639 0.0002639 0.0002639 0.03% MLLinOp::defineGrids() 11 0.0002597 0.0002597 0.0002597 0.03% MultiFab::Copy() 11 0.0002539 0.0002539 0.0002539 0.03% Castro::construct_new_gravity() 10 0.0002288 0.0002288 0.0002288 0.03% MLCellLinOp::correctionResidual() 486 0.0002252 0.0002252 0.0002252 0.03% MLMG::mgVcycle() 81 0.0002154 0.0002154 0.0002154 0.02% MLMG::MLRhsNormInf() 11 0.0001984 0.0001984 0.0001984 0.02% StateData::checkPoint() 12 0.0001438 0.0001438 0.0001438 0.02% MLMG:computeResOfCorrection() 405 0.0001295 0.0001295 0.0001295 0.01% Castro::create_source_corrector() 10 0.0001288 0.0001288 0.0001288 0.01% MLMG::actualBottomSolve() 81 0.0001072 0.0001072 0.0001072 0.01% Castro::initialize_advance() 10 0.0001042 0.0001042 0.0001042 0.01% Castro::construct_new_source() 50 0.0001041 0.0001041 0.0001041 0.01% AmrLevel::checkPoint() 3 0.0001038 0.0001038 0.0001038 0.01% Castro::clean_state() 62 0.0001005 0.0001005 0.0001005 0.01% Castro::initialize_do_advance() 10 9.861e-05 9.861e-05 9.861e-05 0.01% MLMG::mgVcycle_down::0 81 9.817e-05 9.817e-05 9.817e-05 0.01% FabArrayBase::FB::FB() 56 9.635e-05 9.635e-05 9.635e-05 0.01% MLMG::mgVcycle_down::1 81 8.885e-05 8.885e-05 8.885e-05 0.01% Castro::Castro() 1 8.531e-05 8.531e-05 8.531e-05 0.01% MLMG::mgVcycle_down::3 81 8.454e-05 8.454e-05 8.454e-05 0.01% MLMG::solve() 11 8.189e-05 8.189e-05 8.189e-05 0.01% MLMG::mgVcycle_down::2 81 8.118e-05 8.118e-05 8.118e-05 0.01% MLMG::mgVcycle_down::4 81 7.507e-05 7.507e-05 7.507e-05 0.01% Castro::finalize_advance() 10 6.7e-05 6.7e-05 6.7e-05 0.01% Castro::advance() 10 6.233e-05 6.233e-05 6.233e-05 0.01% MLMG::mgVcycle_up::4 81 6.194e-05 6.194e-05 6.194e-05 0.01% MLMG::mgVcycle_up::1 81 5.559e-05 5.559e-05 5.559e-05 0.01% MLMG::mgVcycle_up::0 81 5.448e-05 5.448e-05 5.448e-05 0.01% MLCellLinOp::solutionResidual() 92 5.439e-05 5.439e-05 5.439e-05 0.01% MLMG::mgVcycle_up::3 81 5.069e-05 5.069e-05 5.069e-05 0.01% MLMG::mgVcycle_up::2 81 5.046e-05 5.046e-05 5.046e-05 0.01% Castro::swap_state_time_levels() 10 4.394e-05 4.394e-05 4.394e-05 0.00% Castro::finalize_do_advance() 10 4.068e-05 4.068e-05 4.068e-05 0.00% MLMG::computeResidual() 81 4.064e-05 4.064e-05 4.064e-05 0.00% StateData::define() 4 3.782e-05 3.782e-05 3.782e-05 0.00% Castro::enforce_consistent_e() 1 3.524e-05 3.524e-05 3.524e-05 0.00% MLMG::mgVcycle_bottom 81 3.396e-05 3.396e-05 3.396e-05 0.00% MLPoisson::define() 11 3.161e-05 3.161e-05 3.161e-05 0.00% Gravity::actual_multilevel_solve() 1 3.148e-05 3.148e-05 3.148e-05 0.00% FillPatchSingleLevel 41 3.126e-05 3.126e-05 3.126e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.124e-05 3.124e-05 3.124e-05 0.00% makeSFC 55 3.076e-05 3.076e-05 3.076e-05 0.00% MLLinOp::define() 11 2.798e-05 2.798e-05 2.798e-05 0.00% Amr::writeSmallPlotFile() 1 2.617e-05 2.617e-05 2.617e-05 0.00% Amr::defBaseLevel() 1 2.508e-05 2.508e-05 2.508e-05 0.00% Castro::initMFs() 1 2.5e-05 2.5e-05 2.5e-05 0.00% Amr::FinalizeInit() 1 2.332e-05 2.332e-05 2.332e-05 0.00% Castro::construct_old_source() 50 2.319e-05 2.319e-05 2.319e-05 0.00% Castro::do_new_sources() 10 2.309e-05 2.309e-05 2.309e-05 0.00% Castro::buildMetrics() 1 2.206e-05 2.206e-05 2.206e-05 0.00% Castro::check_for_nan() 20 2.073e-05 2.073e-05 2.073e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.924e-05 1.924e-05 1.924e-05 0.00% Castro::apply_source_to_state() 20 1.756e-05 1.756e-05 1.756e-05 0.00% DistributionMapping::Distribute() 56 1.736e-05 1.736e-05 1.736e-05 0.00% Castro::do_old_sources() 10 1.684e-05 1.684e-05 1.684e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.515e-05 1.515e-05 1.515e-05 0.00% Castro::construct_old_gravity() 10 1.459e-05 1.459e-05 1.459e-05 0.00% Castro::post_timestep() 10 1.336e-05 1.336e-05 1.336e-05 0.00% MLPoisson::prepareForSolve() 11 1.173e-05 1.173e-05 1.173e-05 0.00% Gravity::swapTimeLevels() 10 1.134e-05 1.134e-05 1.134e-05 0.00% Amr::initSubcycle() 1 8.721e-06 8.721e-06 8.721e-06 0.00% MLMG::computeMLResidual() 11 7.192e-06 7.192e-06 7.192e-06 0.00% MLMG::buildFineMask() 11 7.171e-06 7.171e-06 7.171e-06 0.00% MLMG::getGradSolution() 11 6.979e-06 6.979e-06 6.979e-06 0.00% Castro::computeNewDt() 9 6.94e-06 6.94e-06 6.94e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.878e-06 6.878e-06 6.878e-06 0.00% Amr::InitializeInit() 1 6.478e-06 6.478e-06 6.478e-06 0.00% Gravity::set_mass_offset() 11 5.486e-06 5.486e-06 5.486e-06 0.00% AmrLevel::checkPointPost() 3 4.854e-06 4.854e-06 4.854e-06 0.00% MLMG::MLResNormInf() 11 4.839e-06 4.839e-06 4.839e-06 0.00% Castro::retry_advance_ctu() 10 4.094e-06 4.094e-06 4.094e-06 0.00% Castro::post_init() 1 3.533e-06 3.533e-06 3.533e-06 0.00% Castro::FluxRegCrseInit 10 3.332e-06 3.332e-06 3.332e-06 0.00% Amr::init() 1 3.104e-06 3.104e-06 3.104e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.817e-06 2.817e-06 2.817e-06 0.00% Castro::FluxRegFineAdd() 10 2.52e-06 2.52e-06 2.52e-06 0.00% Castro::computeInitialDt() 2 2.46e-06 2.46e-06 2.46e-06 0.00% AmrLevel::checkPointPre() 3 2.088e-06 2.088e-06 2.088e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.941e-06 1.941e-06 1.941e-06 0.00% Castro::post_regrid() 1 1.696e-06 1.696e-06 1.696e-06 0.00% Amr::initialInit() 1 9.64e-07 9.64e-07 9.64e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8837 0.8837 0.8837 100.00% Amr::coarseTimeStep() 10 0.7242 0.7242 0.7242 81.95% Amr::timeStep() 10 0.6103 0.6103 0.6103 69.07% Castro::advance() 10 0.6037 0.6037 0.6037 68.32% Castro::subcycle_advance_ctu() 10 0.5937 0.5937 0.5937 67.18% Castro::do_advance_ctu() 10 0.593 0.593 0.593 67.10% Gravity::solve_phi_with_mlmg() 11 0.3171 0.3171 0.3171 35.88% Gravity::actual_solve_with_mlmg() 11 0.3072 0.3072 0.3072 34.76% Castro::construct_new_gravity() 10 0.2945 0.2945 0.2945 33.32% MLMG::solve() 11 0.2836 0.2836 0.2836 32.09% Gravity::solve_for_phi() 10 0.2767 0.2767 0.2767 31.31% MLMG::oneIter() 81 0.2688 0.2688 0.2688 30.41% MLMG::mgVcycle() 81 0.2671 0.2671 0.2671 30.22% Castro::construct_ctu_hydro_source() 10 0.2131 0.2131 0.2131 24.12% VisMF::Write(FabArray) 11 0.2097 0.2097 0.2097 23.73% Amr::checkPoint() 3 0.1599 0.1599 0.1599 18.10% AmrLevel::checkPoint() 3 0.1541 0.1541 0.1541 17.44% StateData::checkPoint() 12 0.154 0.154 0.154 17.42% MLCellLinOp::smooth() 1620 0.1371 0.1371 0.1371 15.51% Amr::init() 1 0.1295 0.1295 0.1295 14.66% MLCellLinOp::applyBC() 4379 0.09709 0.09709 0.09709 10.99% MLMG::mgVcycle_bottom 81 0.08169 0.08169 0.08169 9.24% MLMG::actualBottomSolve() 81 0.08165 0.08165 0.08165 9.24% MLCGSolver::bicgstab 81 0.08084 0.08084 0.08084 9.15% MLPoisson::Fsmooth() 3240 0.06368 0.06368 0.06368 7.21% Amr::writePlotFile() 2 0.05877 0.05877 0.05877 6.65% FillPatchIterator::Initialize 41 0.04961 0.04961 0.04961 5.61% Amr::initialInit() 1 0.04862 0.04862 0.04862 5.50% FillPatchSingleLevel 41 0.04789 0.04789 0.04789 5.42% Castro::clean_state() 62 0.04713 0.04713 0.04713 5.33% Amr::FinalizeInit() 1 0.04483 0.04483 0.04483 5.07% StateDataPhysBCFunct::() 41 0.0437 0.0437 0.0437 4.95% Castro::post_init() 1 0.04348 0.04348 0.04348 4.92% Gravity::multilevel_solve_for_new_phi() 1 0.04095 0.04095 0.04095 4.63% Gravity::actual_multilevel_solve() 1 0.04094 0.04094 0.04094 4.63% MLCellLinOp::apply() 1128 0.03653 0.03653 0.03653 4.13% MLMG::mgVcycle_down::0 81 0.03561 0.03561 0.03561 4.03% MLMG::mgVcycle_up::0 81 0.03078 0.03078 0.03078 3.48% StateData::FillBoundary(geom) 328 0.02743 0.02743 0.02743 3.10% Castro::initialize_do_advance() 10 0.0227 0.0227 0.0227 2.57% MultiFab::Dot() 1100 0.02208 0.02208 0.02208 2.50% MLCellLinOp::correctionResidual() 486 0.02141 0.02141 0.02141 2.42% Gravity::get_new_grav_vector() 11 0.02003 0.02003 0.02003 2.27% Castro::computeTemp() 63 0.01951 0.01951 0.01951 2.21% MLPoisson::define() 11 0.01903 0.01903 0.01903 2.15% MLMG:computeResOfCorrection() 405 0.01849 0.01849 0.01849 2.09% Castro::construct_old_gravity() 10 0.01831 0.01831 0.01831 2.07% Gravity::get_old_grav_vector() 10 0.0183 0.0183 0.0183 2.07% MLMG::mgVcycle_down::1 81 0.01778 0.01778 0.01778 2.01% MLMG::mgVcycle_down::2 81 0.01729 0.01729 0.01729 1.96% Castro::normalize_species() 62 0.01682 0.01682 0.01682 1.90% MLMG::mgVcycle_down::3 81 0.01674 0.01674 0.01674 1.89% FabArray::FillBoundary() 3974 0.01612 0.01612 0.01612 1.82% MLMG::mgVcycle_down::4 81 0.01555 0.01555 0.01555 1.76% FillBoundary_nowait() 3974 0.01525 0.01525 0.01525 1.73% CGSolver::sxay() 1566 0.01483 0.01483 0.01483 1.68% FabArray::ParallelCopy() 851 0.01453 0.01453 0.01453 1.64% MultiFab::LinComb() 1566 0.01443 0.01443 0.01443 1.63% FabArray::setVal() 1135 0.01424 0.01424 0.01424 1.61% FabArray::ParallelCopy_nowait() 851 0.01422 0.01422 0.01422 1.61% Castro::do_new_sources() 10 0.01363 0.01363 0.01363 1.54% MLCellLinOp::defineAuxData() 11 0.0135 0.0135 0.0135 1.53% MLCGSolver::ParallelAllReduce 1495 0.0132 0.0132 0.0132 1.49% MLMG::mgVcycle_up::2 81 0.01317 0.01317 0.01317 1.49% MLMG::mgVcycle_up::1 81 0.01314 0.01314 0.01314 1.49% MLMG::mgVcycle_up::4 81 0.01259 0.01259 0.01259 1.42% MLMG::addInterpCorrection() 405 0.01257 0.01257 0.01257 1.42% MLMG::mgVcycle_up::3 81 0.0125 0.0125 0.0125 1.41% amrex::average_down 405 0.01221 0.01221 0.01221 1.38% Castro::expand_state() 10 0.01205 0.01205 0.01205 1.36% MLPoisson::Fapply() 1128 0.0118 0.0118 0.0118 1.33% Castro::do_old_sources() 10 0.01066 0.01066 0.01066 1.21% Castro::enforce_min_density() 62 0.0103 0.0103 0.0103 1.17% Castro::initialize_advance() 10 0.009895 0.009895 0.009895 1.12% Gravity::fill_multipole_BCs() 11 0.009641 0.009641 0.009641 1.09% Castro::estTimeStep() 21 0.007355 0.007355 0.007355 0.83% MLCellLinOp::solutionResidual() 92 0.007211 0.007211 0.007211 0.82% MultiFab::Xpay() 578 0.00655 0.00655 0.00655 0.74% Castro::post_timestep() 10 0.006322 0.006322 0.006322 0.72% MLMG::computeResidual() 81 0.006185 0.006185 0.006185 0.70% Castro::reset_internal_energy(MultiFab) 63 0.005418 0.005418 0.005418 0.61% MLMG::prepareForSolve() 11 0.005328 0.005328 0.005328 0.60% MLCellLinOp::defineBC() 11 0.005145 0.005145 0.005145 0.58% BndryData::define() 11 0.00483 0.00483 0.00483 0.55% Castro::computeNewDt() 9 0.00424 0.00424 0.00424 0.48% Amr::InitializeInit() 1 0.003789 0.003789 0.003789 0.43% Amr::defBaseLevel() 1 0.003782 0.003782 0.003782 0.43% Castro::initData() 1 0.003289 0.003289 0.003289 0.37% Castro::construct_new_source() 50 0.002858 0.002858 0.002858 0.32% Castro::construct_new_gravity_source() 10 0.002754 0.002754 0.002754 0.31% Castro::construct_old_source() 50 0.00212 0.00212 0.00212 0.24% Castro::construct_old_gravity_source() 10 0.002096 0.002096 0.002096 0.24% MLMG::ResNormInf() 92 0.001934 0.001934 0.001934 0.22% Castro::apply_source_to_state() 20 0.001858 0.001858 0.001858 0.21% MultiFab::Saxpy() 20 0.001841 0.001841 0.001841 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001731 0.001731 0.001731 0.20% Castro::reset_internal_energy(Fab) 504 0.001633 0.001633 0.001633 0.18% FabArrayBase::getCPC() 1313 0.001608 0.001608 0.001608 0.18% MLCellLinOp::setLevelBC() 11 0.001589 0.001589 0.001589 0.18% MLMG::getGradSolution() 11 0.00146 0.00146 0.00146 0.17% MLCellLinOp::compGrad() 11 0.001453 0.001453 0.001453 0.16% FabArray::setDomainBndry() 41 0.001376 0.001376 0.001376 0.16% FabArray::mult() 43 0.001343 0.001343 0.001343 0.15% Castro::check_for_nan() 20 0.001237 0.001237 0.001237 0.14% MLPoisson::prepareForSolve() 11 0.001231 0.001231 0.001231 0.14% MLCellLinOp::prepareForSolve() 11 0.00122 0.00122 0.00122 0.14% MultiFab::contains_nan() 20 0.001217 0.001217 0.001217 0.14% Castro::post_regrid() 1 0.001142 0.001142 0.001142 0.13% MLMG::computeMLResidual() 11 0.001074 0.001074 0.001074 0.12% Castro::enforce_speed_limit() 62 0.001062 0.001062 0.001062 0.12% Gravity::update_max_rhs() 11 0.0009351 0.0009351 0.0009351 0.11% FabArrayBase::CPC::define() 454 0.0007455 0.0007455 0.0007455 0.08% FabArrayBase::getFB() 3974 0.0007309 0.0007309 0.0007309 0.08% Castro::computeInitialDt() 2 0.0006891 0.0006891 0.0006891 0.08% Amr::InitAmr() 1 0.0005032 0.0005032 0.0005032 0.06% Gravity::swapTimeLevels() 10 0.0004502 0.0004502 0.0004502 0.05% Castro::Castro() 1 0.0004198 0.0004198 0.0004198 0.05% MLLinOp::define() 11 0.0003518 0.0003518 0.0003518 0.04% MLLinOp::defineGrids() 11 0.0003239 0.0003239 0.0003239 0.04% MultiFab::max() 11 0.0002734 0.0002734 0.0002734 0.03% MLMG::MLResNormInf() 11 0.0002563 0.0002563 0.0002563 0.03% MultiFab::Copy() 11 0.0002539 0.0002539 0.0002539 0.03% MLMG::MLRhsNormInf() 11 0.0001984 0.0001984 0.0001984 0.02% Castro::buildMetrics() 1 0.0001475 0.0001475 0.0001475 0.02% Castro::create_source_corrector() 10 0.0001288 0.0001288 0.0001288 0.01% FabArrayBase::FB::FB() 56 9.635e-05 9.635e-05 9.635e-05 0.01% Castro::finalize_advance() 10 7.286e-05 7.286e-05 7.286e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.219e-05 6.219e-05 6.219e-05 0.01% makeSFC 55 4.704e-05 4.704e-05 4.704e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.47e-05 4.47e-05 4.47e-05 0.01% Castro::swap_state_time_levels() 10 4.394e-05 4.394e-05 4.394e-05 0.00% Castro::finalize_do_advance() 10 4.068e-05 4.068e-05 4.068e-05 0.00% StateData::define() 4 3.782e-05 3.782e-05 3.782e-05 0.00% Castro::enforce_consistent_e() 1 3.524e-05 3.524e-05 3.524e-05 0.00% Amr::writeSmallPlotFile() 1 2.617e-05 2.617e-05 2.617e-05 0.00% Castro::initMFs() 1 2.5e-05 2.5e-05 2.5e-05 0.00% DistributionMapping::Distribute() 56 1.736e-05 1.736e-05 1.736e-05 0.00% Amr::initSubcycle() 1 8.721e-06 8.721e-06 8.721e-06 0.00% MLMG::buildFineMask() 11 7.171e-06 7.171e-06 7.171e-06 0.00% Gravity::set_mass_offset() 11 5.486e-06 5.486e-06 5.486e-06 0.00% AmrLevel::checkPointPost() 3 4.854e-06 4.854e-06 4.854e-06 0.00% Castro::retry_advance_ctu() 10 4.094e-06 4.094e-06 4.094e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.901e-06 3.901e-06 3.901e-06 0.00% Castro::FluxRegCrseInit 10 3.332e-06 3.332e-06 3.332e-06 0.00% Castro::FluxRegFineAdd() 10 2.52e-06 2.52e-06 2.52e-06 0.00% AmrLevel::checkPointPre() 3 2.088e-06 2.088e-06 2.088e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.941e-06 1.941e-06 1.941e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-19-g48702b48836d) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-19-g48702b48836d) initialized Starting run at 08:29:10 UTC on 2022-07-29. Successfully read inputs file ... Castro git describe: 22.07-13-g4177c4b30 AMReX git describe: 22.07-19-g48702b488 Microphysics git describe: 22.07-15-ga4952214 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.455404007 Restart time = 0.048861825 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052509184 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051638511 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.069012022 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.080950556 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.087369869 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033662332 seconds Ending run at 08:29:11 UTC on 2022-07-29. Run time = 0.425165086 Run time without initialization = 0.375713061 Average number of zones advanced per microsecond: 3.489 Average number of zones advanced per microsecond per rank: 3.489 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4252 ... 0.4252 ... 0.4252 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1026 0.1026 0.1026 24.13% MLCellLinOp::applyBC() 1946 0.04177 0.04177 0.04177 9.82% VisMF::Read() 3 0.04101 0.04101 0.04101 9.64% VisMF::Write(FabArray) 1 0.03104 0.03104 0.03104 7.30% MLPoisson::Fsmooth() 1440 0.03081 0.03081 0.03081 7.25% StateData::FillBoundary(geom) 160 0.0154 0.0154 0.0154 3.62% MLCGSolver::bicgstab 36 0.01218 0.01218 0.01218 2.87% MultiFab::Dot() 484 0.01082 0.01082 0.01082 2.54% Castro::computeTemp() 30 0.00882 0.00882 0.00882 2.07% StateDataPhysBCFunct::() 20 0.008582 0.008582 0.008582 2.02% Castro::enforce_min_density() 30 0.007767 0.007767 0.007767 1.83% MLCellLinOp::defineAuxData() 6 0.007756 0.007756 0.007756 1.82% MLMG::addInterpCorrection() 180 0.007707 0.007707 0.007707 1.81% FillBoundary_nowait() 1766 0.007575 0.007575 0.007575 1.78% FabArray::setVal() 537 0.007566 0.007566 0.007566 1.78% MultiFab::LinComb() 690 0.0071 0.0071 0.0071 1.67% FabArray::ParallelCopy_nowait() 380 0.006999 0.006999 0.006999 1.65% Castro::normalize_species() 30 0.006928 0.006928 0.006928 1.63% MLPoisson::Fapply() 500 0.0059 0.0059 0.0059 1.39% Gravity::fill_multipole_BCs() 6 0.005546 0.005546 0.005546 1.30% Castro::do_advance_ctu() 5 0.003828 0.003828 0.003828 0.90% amrex::average_down 180 0.003633 0.003633 0.003633 0.85% Amr::restart() 1 0.003598 0.003598 0.003598 0.85% MultiFab::Xpay() 258 0.003206 0.003206 0.003206 0.75% Amr::writePlotFile() 1 0.002737 0.002737 0.002737 0.64% Castro::estTimeStep() 10 0.002672 0.002672 0.002672 0.63% BndryData::define() 6 0.0025 0.0025 0.0025 0.59% Castro::construct_new_gravity_source() 5 0.001669 0.001669 0.001669 0.39% Castro::enforce_speed_limit() 30 0.001617 0.001617 0.001617 0.38% Castro::reset_internal_energy(MultiFab) 30 0.00144 0.00144 0.00144 0.34% Castro::construct_old_gravity_source() 5 0.001427 0.001427 0.001427 0.34% Gravity::actual_solve_with_mlmg() 6 0.001132 0.001132 0.001132 0.27% Castro::reset_internal_energy(Fab) 240 0.001128 0.001128 0.001128 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00102 0.00102 0.00102 0.24% Gravity::get_old_grav_vector() 5 0.0009824 0.0009824 0.0009824 0.23% MLCellLinOp::setLevelBC() 6 0.000977 0.000977 0.000977 0.23% Gravity::get_new_grav_vector() 5 0.0009567 0.0009567 0.0009567 0.22% MultiFab::Saxpy() 10 0.000939 0.000939 0.000939 0.22% MLMG::ResNormInf() 42 0.000918 0.000918 0.000918 0.22% Castro::expand_state() 5 0.0009037 0.0009037 0.0009037 0.21% MLMG::prepareForSolve() 6 0.0007952 0.0007952 0.0007952 0.19% MLMG::oneIter() 36 0.0007865 0.0007865 0.0007865 0.18% MLCellLinOp::prepareForSolve() 6 0.0007631 0.0007631 0.0007631 0.18% MLCellLinOp::smooth() 720 0.0007296 0.0007296 0.0007296 0.17% FabArray::setDomainBndry() 20 0.0007081 0.0007081 0.0007081 0.17% FabArray::mult() 22 0.0006776 0.0006776 0.0006776 0.16% MLCellLinOp::compGrad() 6 0.0006393 0.0006393 0.0006393 0.15% MultiFab::contains_nan() 10 0.0006099 0.0006099 0.0006099 0.14% FabArrayBase::getCPC() 632 0.0005966 0.0005966 0.0005966 0.14% FabArray::FillBoundary() 1766 0.0005523 0.0005523 0.0005523 0.13% FabArrayBase::CPC::define() 244 0.0005207 0.0005207 0.0005207 0.12% Amr::InitAmr() 1 0.0004133 0.0004133 0.0004133 0.10% FabArrayBase::getFB() 1766 0.00041 0.00041 0.00041 0.10% Amr::coarseTimeStep() 5 0.0002889 0.0002889 0.0002889 0.07% main() 1 0.0002881 0.0002881 0.0002881 0.07% MLCellLinOp::apply() 500 0.0002864 0.0002864 0.0002864 0.07% Gravity::update_max_rhs() 6 0.0002704 0.0002704 0.0002704 0.06% Gravity::solve_for_phi() 5 0.0002643 0.0002643 0.0002643 0.06% CGSolver::sxay() 690 0.0002554 0.0002554 0.0002554 0.06% MLCellLinOp::defineBC() 6 0.000237 0.000237 0.000237 0.06% Castro::construct_new_source() 25 0.0002036 0.0002036 0.0002036 0.05% Amr::timeStep() 5 0.0001875 0.0001875 0.0001875 0.04% FillPatchIterator::Initialize 20 0.0001866 0.0001866 0.0001866 0.04% MLCGSolver::ParallelAllReduce 659 0.0001798 0.0001798 0.0001798 0.04% FabArray::ParallelCopy() 380 0.0001792 0.0001792 0.0001792 0.04% MultiFab::Copy() 6 0.0001516 0.0001516 0.0001516 0.04% MultiFab::max() 6 0.0001494 0.0001494 0.0001494 0.04% MLCellLinOp::correctionResidual() 216 0.0001368 0.0001368 0.0001368 0.03% MLLinOp::defineGrids() 6 0.0001333 0.0001333 0.0001333 0.03% Castro::construct_new_gravity() 5 0.0001278 0.0001278 0.0001278 0.03% Castro::subcycle_advance_ctu() 5 0.0001222 0.0001222 0.0001222 0.03% MLMG::mgVcycle() 36 0.0001177 0.0001177 0.0001177 0.03% MLMG::MLRhsNormInf() 6 0.0001069 0.0001069 0.0001069 0.03% Castro::construct_old_source() 25 8.931e-05 8.931e-05 8.931e-05 0.02% Castro::create_source_corrector() 5 8.897e-05 8.897e-05 8.897e-05 0.02% FabArrayBase::FB::FB() 26 7.668e-05 7.668e-05 7.668e-05 0.02% MLMG::actualBottomSolve() 36 7.589e-05 7.589e-05 7.589e-05 0.02% Castro::advance() 5 7.555e-05 7.555e-05 7.555e-05 0.02% AmrLevel::restart() 1 7.523e-05 7.523e-05 7.523e-05 0.02% MLMG:computeResOfCorrection() 180 6.875e-05 6.875e-05 6.875e-05 0.02% StateData::restartDoit() 4 6.868e-05 6.868e-05 6.868e-05 0.02% MLMG::mgVcycle_down::1 36 6.868e-05 6.868e-05 6.868e-05 0.02% MLMG::mgVcycle_down::0 36 6.206e-05 6.206e-05 6.206e-05 0.01% Castro::initialize_advance() 5 5.653e-05 5.653e-05 5.653e-05 0.01% Castro::clean_state() 30 5.538e-05 5.538e-05 5.538e-05 0.01% MLMG::mgVcycle_down::2 36 5.457e-05 5.457e-05 5.457e-05 0.01% MLMG::solve() 6 5.029e-05 5.029e-05 5.029e-05 0.01% MLMG::mgVcycle_up::3 36 4.586e-05 4.586e-05 4.586e-05 0.01% MLMG::mgVcycle_down::4 36 4.408e-05 4.408e-05 4.408e-05 0.01% Castro::initialize_do_advance() 5 4.363e-05 4.363e-05 4.363e-05 0.01% MLMG::mgVcycle_up::4 36 4.229e-05 4.229e-05 4.229e-05 0.01% Castro::finalize_advance() 5 4.106e-05 4.106e-05 4.106e-05 0.01% MLMG::mgVcycle_up::2 36 4.093e-05 4.093e-05 4.093e-05 0.01% MLMG::mgVcycle_down::3 36 4.009e-05 4.009e-05 4.009e-05 0.01% Castro::computeNewDt() 5 3.761e-05 3.761e-05 3.761e-05 0.01% MLCellLinOp::solutionResidual() 42 3.684e-05 3.684e-05 3.684e-05 0.01% Castro::initMFs() 1 3.597e-05 3.597e-05 3.597e-05 0.01% Castro::finalize_do_advance() 5 3.536e-05 3.536e-05 3.536e-05 0.01% Castro::post_restart() 1 3.174e-05 3.174e-05 3.174e-05 0.01% Castro::buildMetrics() 1 3.133e-05 3.133e-05 3.133e-05 0.01% Gravity::actual_multilevel_solve() 1 2.945e-05 2.945e-05 2.945e-05 0.01% MLMG::mgVcycle_up::1 36 2.882e-05 2.882e-05 2.882e-05 0.01% MLMG::mgVcycle_up::0 36 2.77e-05 2.77e-05 2.77e-05 0.01% MLMG::mgVcycle_bottom 36 2.733e-05 2.733e-05 2.733e-05 0.01% Castro::swap_state_time_levels() 5 2.717e-05 2.717e-05 2.717e-05 0.01% Amr::writeSmallPlotFile() 1 2.598e-05 2.598e-05 2.598e-05 0.01% MLPoisson::define() 6 2.364e-05 2.364e-05 2.364e-05 0.01% MLLinOp::define() 6 2.304e-05 2.304e-05 2.304e-05 0.01% FillPatchSingleLevel 20 2.224e-05 2.224e-05 2.224e-05 0.01% makeSFC 30 2.127e-05 2.127e-05 2.127e-05 0.01% MLMG::computeResidual() 36 1.908e-05 1.908e-05 1.908e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.732e-05 1.732e-05 1.732e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.71e-05 1.71e-05 1.71e-05 0.00% Castro::do_new_sources() 5 1.619e-05 1.619e-05 1.619e-05 0.00% Castro::do_old_sources() 5 1.476e-05 1.476e-05 1.476e-05 0.00% DistributionMapping::Distribute() 31 1.198e-05 1.198e-05 1.198e-05 0.00% Castro::check_for_nan() 10 1.009e-05 1.009e-05 1.009e-05 0.00% Amr::initSubcycle() 1 8.75e-06 8.75e-06 8.75e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.617e-06 8.617e-06 8.617e-06 0.00% Castro::apply_source_to_state() 10 7.838e-06 7.838e-06 7.838e-06 0.00% Gravity::swapTimeLevels() 5 7.505e-06 7.505e-06 7.505e-06 0.00% Castro::post_timestep() 5 6.231e-06 6.231e-06 6.231e-06 0.00% Castro::construct_old_gravity() 5 6.169e-06 6.169e-06 6.169e-06 0.00% Castro::FluxRegCrseInit 5 5.342e-06 5.342e-06 5.342e-06 0.00% MLMG::getGradSolution() 6 5.047e-06 5.047e-06 5.047e-06 0.00% MLPoisson::prepareForSolve() 6 5.033e-06 5.033e-06 5.033e-06 0.00% MLMG::computeMLResidual() 6 4.884e-06 4.884e-06 4.884e-06 0.00% MLMG::MLResNormInf() 6 4.392e-06 4.392e-06 4.392e-06 0.00% MLMG::buildFineMask() 6 3.419e-06 3.419e-06 3.419e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.084e-06 3.084e-06 3.084e-06 0.00% MLLinOp::makeSubCommunicator() 6 2.816e-06 2.816e-06 2.816e-06 0.00% Gravity::set_mass_offset() 6 2.467e-06 2.467e-06 2.467e-06 0.00% Castro::retry_advance_ctu() 5 1.706e-06 1.706e-06 1.706e-06 0.00% Castro::FluxRegFineAdd() 5 1.251e-06 1.251e-06 1.251e-06 0.00% Amr::init() 1 1.102e-06 1.102e-06 1.102e-06 0.00% AmrLevel::AmrLevel() 1 7.39e-07 7.39e-07 7.39e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4252 0.4252 0.4252 100.00% Amr::coarseTimeStep() 5 0.3418 0.3418 0.3418 80.38% Amr::timeStep() 5 0.3401 0.3401 0.3401 79.99% Castro::advance() 5 0.3354 0.3354 0.3354 78.87% Castro::subcycle_advance_ctu() 5 0.3287 0.3287 0.3287 77.32% Castro::do_advance_ctu() 5 0.3286 0.3286 0.3286 77.29% Castro::construct_new_gravity() 5 0.1763 0.1763 0.1763 41.46% Gravity::solve_phi_with_mlmg() 6 0.1692 0.1692 0.1692 39.79% Gravity::solve_for_phi() 5 0.1657 0.1657 0.1657 38.97% Gravity::actual_solve_with_mlmg() 6 0.1635 0.1635 0.1635 38.45% MLMG::solve() 6 0.1482 0.1482 0.1482 34.85% MLMG::oneIter() 36 0.1399 0.1399 0.1399 32.91% MLMG::mgVcycle() 36 0.1392 0.1392 0.1392 32.73% Castro::construct_ctu_hydro_source() 5 0.1026 0.1026 0.1026 24.13% MLCellLinOp::smooth() 720 0.06875 0.06875 0.06875 16.17% MLCellLinOp::applyBC() 1946 0.05039 0.05039 0.05039 11.85% Amr::init() 1 0.04891 0.04891 0.04891 11.50% Amr::restart() 1 0.04891 0.04891 0.04891 11.50% MLMG::mgVcycle_bottom 36 0.04144 0.04144 0.04144 9.75% MLMG::actualBottomSolve() 36 0.04141 0.04141 0.04141 9.74% AmrLevel::restart() 1 0.04121 0.04121 0.04121 9.69% StateData::restartDoit() 4 0.04112 0.04112 0.04112 9.67% VisMF::Read() 3 0.04101 0.04101 0.04101 9.64% MLCGSolver::bicgstab 36 0.04099 0.04099 0.04099 9.64% Amr::writePlotFile() 1 0.03378 0.03378 0.03378 7.94% VisMF::Write(FabArray) 1 0.03104 0.03104 0.03104 7.30% MLPoisson::Fsmooth() 1440 0.03081 0.03081 0.03081 7.25% Castro::clean_state() 30 0.02775 0.02775 0.02775 6.53% FillPatchIterator::Initialize 20 0.02705 0.02705 0.02705 6.36% FillPatchSingleLevel 20 0.02616 0.02616 0.02616 6.15% StateDataPhysBCFunct::() 20 0.02399 0.02399 0.02399 5.64% MLCellLinOp::apply() 500 0.01906 0.01906 0.01906 4.48% MLMG::mgVcycle_down::0 36 0.01819 0.01819 0.01819 4.28% MLMG::mgVcycle_up::0 36 0.01574 0.01574 0.01574 3.70% StateData::FillBoundary(geom) 160 0.0154 0.0154 0.0154 3.62% Castro::initialize_do_advance() 5 0.01233 0.01233 0.01233 2.90% MLPoisson::define() 6 0.01226 0.01226 0.01226 2.88% Castro::computeTemp() 30 0.01139 0.01139 0.01139 2.68% MLCellLinOp::correctionResidual() 216 0.01089 0.01089 0.01089 2.56% MultiFab::Dot() 484 0.01082 0.01082 0.01082 2.54% Gravity::get_new_grav_vector() 5 0.01046 0.01046 0.01046 2.46% MLMG::addInterpCorrection() 180 0.01043 0.01043 0.01043 2.45% MLMG::mgVcycle_up::3 36 0.01037 0.01037 0.01037 2.44% Castro::construct_old_gravity() 5 0.009414 0.009414 0.009414 2.21% Gravity::get_old_grav_vector() 5 0.009407 0.009407 0.009407 2.21% MLMG:computeResOfCorrection() 180 0.009368 0.009368 0.009368 2.20% MLMG::mgVcycle_down::1 36 0.009314 0.009314 0.009314 2.19% MLCellLinOp::defineAuxData() 6 0.008696 0.008696 0.008696 2.05% MLMG::mgVcycle_down::2 36 0.008669 0.008669 0.008669 2.04% FabArray::FillBoundary() 1766 0.008614 0.008614 0.008614 2.03% MLMG::mgVcycle_down::3 36 0.008328 0.008328 0.008328 1.96% FillBoundary_nowait() 1766 0.008062 0.008062 0.008062 1.90% FabArray::ParallelCopy() 380 0.007768 0.007768 0.007768 1.83% Castro::enforce_min_density() 30 0.007767 0.007767 0.007767 1.83% MLMG::mgVcycle_down::4 36 0.007713 0.007713 0.007713 1.81% FabArray::ParallelCopy_nowait() 380 0.007589 0.007589 0.007589 1.78% FabArray::setVal() 537 0.007566 0.007566 0.007566 1.78% Castro::do_new_sources() 5 0.007388 0.007388 0.007388 1.74% CGSolver::sxay() 690 0.007356 0.007356 0.007356 1.73% MultiFab::LinComb() 690 0.0071 0.0071 0.0071 1.67% Castro::expand_state() 5 0.007035 0.007035 0.007035 1.65% Castro::normalize_species() 30 0.006928 0.006928 0.006928 1.63% Castro::do_old_sources() 5 0.006623 0.006623 0.006623 1.56% MLMG::mgVcycle_up::2 36 0.006589 0.006589 0.006589 1.55% amrex::average_down 180 0.006527 0.006527 0.006527 1.54% MLCGSolver::ParallelAllReduce 659 0.006511 0.006511 0.006511 1.53% Castro::initialize_advance() 5 0.006498 0.006498 0.006498 1.53% MLMG::mgVcycle_up::1 36 0.006428 0.006428 0.006428 1.51% MLMG::mgVcycle_up::4 36 0.006255 0.006255 0.006255 1.47% MLPoisson::Fapply() 500 0.0059 0.0059 0.0059 1.39% Gravity::fill_multipole_BCs() 6 0.005546 0.005546 0.005546 1.30% Castro::post_timestep() 5 0.004578 0.004578 0.004578 1.08% Castro::post_restart() 1 0.003921 0.003921 0.003921 0.92% Gravity::multilevel_solve_for_new_phi() 1 0.003796 0.003796 0.003796 0.89% Gravity::actual_multilevel_solve() 1 0.003779 0.003779 0.003779 0.89% MLCellLinOp::solutionResidual() 42 0.003772 0.003772 0.003772 0.89% MLMG::prepareForSolve() 6 0.003361 0.003361 0.003361 0.79% MLCellLinOp::defineBC() 6 0.003344 0.003344 0.003344 0.79% MultiFab::Xpay() 258 0.003206 0.003206 0.003206 0.75% MLMG::computeResidual() 36 0.00316 0.00316 0.00316 0.74% BndryData::define() 6 0.003107 0.003107 0.003107 0.73% Castro::estTimeStep() 10 0.002672 0.002672 0.002672 0.63% Castro::reset_internal_energy(MultiFab) 30 0.002568 0.002568 0.002568 0.60% Castro::construct_new_source() 25 0.001872 0.001872 0.001872 0.44% Castro::construct_new_gravity_source() 5 0.001669 0.001669 0.001669 0.39% Castro::enforce_speed_limit() 30 0.001617 0.001617 0.001617 0.38% Castro::construct_old_source() 25 0.001516 0.001516 0.001516 0.36% Castro::construct_old_gravity_source() 5 0.001427 0.001427 0.001427 0.34% Castro::computeNewDt() 5 0.001336 0.001336 0.001336 0.31% Castro::reset_internal_energy(Fab) 240 0.001128 0.001128 0.001128 0.27% FabArrayBase::getCPC() 632 0.001117 0.001117 0.001117 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00102 0.00102 0.00102 0.24% MLCellLinOp::setLevelBC() 6 0.000977 0.000977 0.000977 0.23% Castro::apply_source_to_state() 10 0.0009468 0.0009468 0.0009468 0.22% MLMG::getGradSolution() 6 0.0009447 0.0009447 0.0009447 0.22% MLCellLinOp::compGrad() 6 0.0009397 0.0009397 0.0009397 0.22% MultiFab::Saxpy() 10 0.000939 0.000939 0.000939 0.22% MLMG::ResNormInf() 42 0.000918 0.000918 0.000918 0.22% MLPoisson::prepareForSolve() 6 0.0007681 0.0007681 0.0007681 0.18% MLCellLinOp::prepareForSolve() 6 0.0007631 0.0007631 0.0007631 0.18% FabArray::setDomainBndry() 20 0.0007081 0.0007081 0.0007081 0.17% FabArray::mult() 22 0.0006776 0.0006776 0.0006776 0.16% MLMG::computeMLResidual() 6 0.000636 0.000636 0.000636 0.15% Castro::check_for_nan() 10 0.00062 0.00062 0.00062 0.15% MultiFab::contains_nan() 10 0.0006099 0.0006099 0.0006099 0.14% FabArrayBase::CPC::define() 244 0.0005207 0.0005207 0.0005207 0.12% Gravity::update_max_rhs() 6 0.0005025 0.0005025 0.0005025 0.12% FabArrayBase::getFB() 1766 0.0004867 0.0004867 0.0004867 0.11% Amr::InitAmr() 1 0.000422 0.000422 0.000422 0.10% Gravity::swapTimeLevels() 5 0.0002439 0.0002439 0.0002439 0.06% MLLinOp::define() 6 0.0001988 0.0001988 0.0001988 0.05% MLLinOp::defineGrids() 6 0.0001758 0.0001758 0.0001758 0.04% MultiFab::Copy() 6 0.0001516 0.0001516 0.0001516 0.04% MLMG::MLResNormInf() 6 0.0001504 0.0001504 0.0001504 0.04% MultiFab::max() 6 0.0001494 0.0001494 0.0001494 0.04% Castro::buildMetrics() 1 0.0001477 0.0001477 0.0001477 0.03% MLMG::MLRhsNormInf() 6 0.0001069 0.0001069 0.0001069 0.03% Castro::create_source_corrector() 5 8.897e-05 8.897e-05 8.897e-05 0.02% FabArrayBase::FB::FB() 26 7.668e-05 7.668e-05 7.668e-05 0.02% Castro::finalize_advance() 5 4.766e-05 4.766e-05 4.766e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.969e-05 3.969e-05 3.969e-05 0.01% Castro::initMFs() 1 3.597e-05 3.597e-05 3.597e-05 0.01% Castro::finalize_do_advance() 5 3.536e-05 3.536e-05 3.536e-05 0.01% makeSFC 30 3.107e-05 3.107e-05 3.107e-05 0.01% Castro::swap_state_time_levels() 5 2.717e-05 2.717e-05 2.717e-05 0.01% Amr::writeSmallPlotFile() 1 2.598e-05 2.598e-05 2.598e-05 0.01% DistributionMapping::Distribute() 31 1.198e-05 1.198e-05 1.198e-05 0.00% Amr::initSubcycle() 1 8.75e-06 8.75e-06 8.75e-06 0.00% Castro::FluxRegCrseInit 5 5.342e-06 5.342e-06 5.342e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.264e-06 5.264e-06 5.264e-06 0.00% MLMG::buildFineMask() 6 3.419e-06 3.419e-06 3.419e-06 0.00% MLLinOp::makeSubCommunicator() 6 2.816e-06 2.816e-06 2.816e-06 0.00% Gravity::set_mass_offset() 6 2.467e-06 2.467e-06 2.467e-06 0.00% Castro::retry_advance_ctu() 5 1.706e-06 1.706e-06 1.706e-06 0.00% Castro::FluxRegFineAdd() 5 1.251e-06 1.251e-06 1.251e-06 0.00% AmrLevel::AmrLevel() 1 7.39e-07 7.39e-07 7.39e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-19-g48702b48836d) finalized