Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-23-gd92232a0c17b) initialized Starting run at 07:32:19 UTC on 2023-08-10. Successfully read inputs file ... Castro git describe: 23.08-3-g76295aa13 AMReX git describe: 23.08-23-gd92232a0c Microphysics git describe: 23.08-3-g3d2714f8 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.067817926 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.038833524 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.051415305 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052705959 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.076031079 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.076839693 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.071056742 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.1132585 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052169436 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.054157129 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.085221602 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.075929512 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.051162346 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.060724607 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.034113312 seconds Ending run at 07:32:20 UTC on 2023-08-10. Run time = 1.013071682 Run time without initialization = 0.855610307 Average number of zones advanced per microsecond: 3.064 Average number of zones advanced per microsecond per rank: 3.064 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 1.013 ... 1.013 ... 1.013 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.3074 0.3074 0.3074 30.35% Castro::construct_ctu_hydro_source() 10 0.2647 0.2647 0.2647 26.13% MLCellLinOp::applyBC() 4433 0.08061 0.08061 0.08061 7.96% MLPoisson::Fsmooth() 3280 0.0334 0.0334 0.0334 3.30% FillBoundary_nowait() 4023 0.03064 0.03064 0.03064 3.02% StateData::FillBoundary(geom) 328 0.02651 0.02651 0.02651 2.62% amrex::Dot() 1114 0.02135 0.02135 0.02135 2.11% Castro::normalize_species() 62 0.01975 0.01975 0.01975 1.95% Castro::computeTemp() 63 0.01661 0.01661 0.01661 1.64% amrex::Copy() 1029 0.01533 0.01533 0.01533 1.51% FabArray::norminf() 743 0.0149 0.0149 0.0149 1.47% FabArray::ParallelCopy_nowait() 861 0.01405 0.01405 0.01405 1.39% FabArray::setVal() 1144 0.01355 0.01355 0.01355 1.34% Castro::enforce_min_density() 62 0.01182 0.01182 0.01182 1.17% StateDataPhysBCFunct::() 41 0.01163 0.01163 0.01163 1.15% MLPoisson::Fapply() 1142 0.01075 0.01075 0.01075 1.06% MLCellLinOp::defineAuxData() 11 0.01031 0.01031 0.01031 1.02% FabArray::Saxpy() 813 0.008398 0.008398 0.008398 0.83% FabArray::Xpay() 821 0.00833 0.00833 0.00833 0.82% Gravity::fill_multipole_BCs() 11 0.007116 0.007116 0.007116 0.70% MLMG::addInterpCorrection() 410 0.00706 0.00706 0.00706 0.70% amrex::average_down 410 0.006291 0.006291 0.006291 0.62% Castro::estTimeStep() 21 0.005548 0.005548 0.005548 0.55% Castro::reset_internal_energy(MultiFab) 63 0.005509 0.005509 0.005509 0.54% FabArray::LinComb() 557 0.004699 0.004699 0.004699 0.46% amrex::Add() 164 0.004381 0.004381 0.004381 0.43% Amr::checkPoint() 3 0.004317 0.004317 0.004317 0.43% BndryData::define() 11 0.003941 0.003941 0.003941 0.39% Castro::construct_new_gravity_source() 10 0.003145 0.003145 0.003145 0.31% Castro::construct_old_gravity_source() 10 0.002741 0.002741 0.002741 0.27% Amr::writePlotFile() 2 0.002489 0.002489 0.002489 0.25% Castro::enforce_speed_limit() 62 0.002376 0.002376 0.002376 0.23% MLCGSolver::bicgstab 82 0.002291 0.002291 0.002291 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001845 0.001845 0.001845 0.18% Gravity::actual_solve_with_mlmg() 11 0.001778 0.001778 0.001778 0.18% Castro::reset_internal_energy(Fab) 504 0.001635 0.001635 0.001635 0.16% MLCellLinOp::setLevelBC() 11 0.001467 0.001467 0.001467 0.14% check_for_negative_density() 10 0.001445 0.001445 0.001445 0.14% FabArray::setDomainBndry() 41 0.001388 0.001388 0.001388 0.14% FabArray::mult() 43 0.001379 0.001379 0.001379 0.14% MultiFab::contains_nan() 20 0.001241 0.001241 0.001241 0.12% Castro::initData() 1 0.00122 0.00122 0.00122 0.12% MLCellLinOp::prepareForSolve() 11 0.001206 0.001206 0.001206 0.12% MLCellLinOp::smooth() 1640 0.001136 0.001136 0.001136 0.11% MLCellLinOp::compGrad() 11 0.0009639 0.0009639 0.0009639 0.10% MLMG::prepareForSolve() 11 0.0009619 0.0009619 0.0009619 0.09% FabArray::FillBoundary() 4023 0.0009104 0.0009104 0.0009104 0.09% FabArrayBase::getCPC() 1323 0.000817 0.000817 0.000817 0.08% FabArrayBase::CPC::define() 454 0.0007794 0.0007794 0.0007794 0.08% FabArrayBase::getFB() 4023 0.0006532 0.0006532 0.0006532 0.06% Gravity::get_new_grav_vector() 11 0.0005885 0.0005885 0.0005885 0.06% Amr::InitAmr() 1 0.000521 0.000521 0.000521 0.05% Amr::coarseTimeStep() 10 0.0005135 0.0005135 0.0005135 0.05% Gravity::get_old_grav_vector() 10 0.000493 0.000493 0.000493 0.05% MLCellLinOp::apply() 1142 0.0004874 0.0004874 0.0004874 0.05% MLLinOp::defineGrids() 11 0.0004873 0.0004873 0.0004873 0.05% MLMG::mgVcycle() 82 0.000418 0.000418 0.000418 0.04% AmrLevel::FillPatch() 41 0.0003576 0.0003576 0.0003576 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003398 0.0003398 0.0003398 0.03% main() 1 0.000309 0.000309 0.000309 0.03% Amr::timeStep() 10 0.0002845 0.0002845 0.0002845 0.03% FabArray::ParallelCopy() 861 0.0002739 0.0002739 0.0002739 0.03% MultiFab::max() 11 0.0002698 0.0002698 0.0002698 0.03% Castro::subcycle_advance_ctu() 10 0.0002695 0.0002695 0.0002695 0.03% MLCellLinOp::correctionResidual() 492 0.0002511 0.0002511 0.0002511 0.02% MLCellLinOp::defineBC() 11 0.0002482 0.0002482 0.0002482 0.02% FillPatchIterator::Initialize 41 0.0002437 0.0002437 0.0002437 0.02% Gravity::solve_for_phi() 10 0.0001669 0.0001669 0.0001669 0.02% StateData::checkPoint() 12 0.0001649 0.0001649 0.0001649 0.02% Castro::create_source_corrector() 10 0.0001634 0.0001634 0.0001634 0.02% Castro::do_new_sources() 10 0.0001582 0.0001582 0.0001582 0.02% Castro::construct_new_source() 50 0.000152 0.000152 0.000152 0.02% Gravity::update_max_rhs() 11 0.0001457 0.0001457 0.0001457 0.01% MLMG:computeResOfCorrection() 410 0.0001186 0.0001186 0.0001186 0.01% MLMG::mgVcycle_down::0 82 0.0001093 0.0001093 0.0001093 0.01% Castro::advance() 10 0.0001078 0.0001078 0.0001078 0.01% FabArrayBase::FB::FB() 56 0.0001044 0.0001044 0.0001044 0.01% MLMG::actualBottomSolve() 82 0.0001037 0.0001037 0.0001037 0.01% Castro::Castro() 1 9.517e-05 9.517e-05 9.517e-05 0.01% MLMG::mgVcycle_down::1 82 9.126e-05 9.126e-05 9.126e-05 0.01% MLMG::mgVcycle_down::2 82 8.832e-05 8.832e-05 8.832e-05 0.01% Castro::initialize_advance() 10 8.709e-05 8.709e-05 8.709e-05 0.01% AmrLevel::checkPoint() 3 8.584e-05 8.584e-05 8.584e-05 0.01% MLMG::solve() 11 8.549e-05 8.549e-05 8.549e-05 0.01% Castro::finalize_advance() 10 8.485e-05 8.485e-05 8.485e-05 0.01% MLMG::mgVcycle_down::3 82 8.454e-05 8.454e-05 8.454e-05 0.01% MLMG::mgVcycle_down::4 82 8.433e-05 8.433e-05 8.433e-05 0.01% Castro::do_advance_ctu() 10 7.947e-05 7.947e-05 7.947e-05 0.01% Castro::clean_state() 62 7.807e-05 7.807e-05 7.807e-05 0.01% MLMG::mgVcycle_up::4 82 6.799e-05 6.799e-05 6.799e-05 0.01% Castro::initialize_do_advance() 10 6.668e-05 6.668e-05 6.668e-05 0.01% MLMG::mgVcycle_up::0 82 6.423e-05 6.423e-05 6.423e-05 0.01% FillPatchIterator::FillFromLevel0() 41 6.108e-05 6.108e-05 6.108e-05 0.01% MLMG::oneIter() 82 6.095e-05 6.095e-05 6.095e-05 0.01% MLMG::mgVcycle_up::3 82 5.685e-05 5.685e-05 5.685e-05 0.01% MLCellLinOp::solutionResidual() 93 5.645e-05 5.645e-05 5.645e-05 0.01% MLMG::mgVcycle_up::1 82 5.414e-05 5.414e-05 5.414e-05 0.01% MLMG::mgVcycle_up::2 82 5.366e-05 5.366e-05 5.366e-05 0.01% Castro::finalize_do_advance() 10 5.044e-05 5.044e-05 5.044e-05 0.00% Castro::swap_state_time_levels() 10 4.266e-05 4.266e-05 4.266e-05 0.00% Castro::construct_new_gravity() 10 4.204e-05 4.204e-05 4.204e-05 0.00% FillPatchSingleLevel 41 4.111e-05 4.111e-05 4.111e-05 0.00% MLMG::ResNormInf() 93 3.935e-05 3.935e-05 3.935e-05 0.00% MLMG::mgVcycle_bottom 82 3.932e-05 3.932e-05 3.932e-05 0.00% MLPoisson::define() 11 3.642e-05 3.642e-05 3.642e-05 0.00% StateData::define() 4 3.619e-05 3.619e-05 3.619e-05 0.00% Castro::enforce_consistent_e() 1 3.465e-05 3.465e-05 3.465e-05 0.00% MLMG::computeResidual() 82 3.397e-05 3.397e-05 3.397e-05 0.00% makeSFC 55 3.151e-05 3.151e-05 3.151e-05 0.00% Amr::FinalizeInit() 1 3.03e-05 3.03e-05 3.03e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.952e-05 2.952e-05 2.952e-05 0.00% Amr::writeSmallPlotFile() 1 2.649e-05 2.649e-05 2.649e-05 0.00% Castro::initMFs() 1 2.573e-05 2.573e-05 2.573e-05 0.00% Castro::do_old_sources() 10 2.364e-05 2.364e-05 2.364e-05 0.00% Castro::construct_old_source() 50 2.236e-05 2.236e-05 2.236e-05 0.00% DistributionMapping::Distribute() 56 1.921e-05 1.921e-05 1.921e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.91e-05 1.91e-05 1.91e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.907e-05 1.907e-05 1.907e-05 0.00% Castro::buildMetrics() 1 1.838e-05 1.838e-05 1.838e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.738e-05 1.738e-05 1.738e-05 0.00% MLPoisson::prepareForSolve() 11 1.702e-05 1.702e-05 1.702e-05 0.00% Castro::check_for_nan() 20 1.633e-05 1.633e-05 1.633e-05 0.00% MLLinOp::define() 11 1.456e-05 1.456e-05 1.456e-05 0.00% Amr::defBaseLevel() 1 1.423e-05 1.423e-05 1.423e-05 0.00% Castro::construct_old_gravity() 10 1.418e-05 1.418e-05 1.418e-05 0.00% Castro::apply_source_to_state() 20 1.338e-05 1.338e-05 1.338e-05 0.00% Castro::post_timestep() 10 1.233e-05 1.233e-05 1.233e-05 0.00% MLMG::computeMLResidual() 11 1.125e-05 1.125e-05 1.125e-05 0.00% Gravity::swapTimeLevels() 10 1.116e-05 1.116e-05 1.116e-05 0.00% MLMG::getGradSolution() 11 1.033e-05 1.033e-05 1.033e-05 0.00% Gravity::actual_multilevel_solve() 1 9.431e-06 9.431e-06 9.431e-06 0.00% Amr::initSubcycle() 1 9.372e-06 9.372e-06 9.372e-06 0.00% Castro::computeNewDt() 9 7.687e-06 7.687e-06 7.687e-06 0.00% AmrLevel::checkPointPost() 3 6.551e-06 6.551e-06 6.551e-06 0.00% Gravity::set_mass_offset() 11 6.536e-06 6.536e-06 6.536e-06 0.00% Castro::expand_state() 10 6.532e-06 6.532e-06 6.532e-06 0.00% Castro::retry_advance_ctu() 10 5.45e-06 5.45e-06 5.45e-06 0.00% Castro::post_init() 1 4.951e-06 4.951e-06 4.951e-06 0.00% Amr::InitializeInit() 1 4.946e-06 4.946e-06 4.946e-06 0.00% Castro::FluxRegCrseInit 10 4.82e-06 4.82e-06 4.82e-06 0.00% MLMG::MLRhsNormInf() 11 4.174e-06 4.174e-06 4.174e-06 0.00% MLMG::MLResNormInf() 11 3.904e-06 3.904e-06 3.904e-06 0.00% Castro::computeInitialDt() 2 3.039e-06 3.039e-06 3.039e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.952e-06 2.952e-06 2.952e-06 0.00% Amr::init() 1 2.8e-06 2.8e-06 2.8e-06 0.00% Castro::FluxRegFineAdd() 10 2.47e-06 2.47e-06 2.47e-06 0.00% AmrLevel::checkPointPre() 3 2.214e-06 2.214e-06 2.214e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.967e-06 1.967e-06 1.967e-06 0.00% Amr::initialInit() 1 1.31e-06 1.31e-06 1.31e-06 0.00% Castro::post_regrid() 1 1.078e-06 1.078e-06 1.078e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 1.013 1.013 1.013 100.00% Amr::coarseTimeStep() 10 0.8213 0.8213 0.8213 81.07% Amr::timeStep() 10 0.6437 0.6437 0.6437 63.54% Castro::advance() 10 0.6326 0.6326 0.6326 62.44% Castro::subcycle_advance_ctu() 10 0.6185 0.6185 0.6185 61.05% Castro::do_advance_ctu() 10 0.6182 0.6182 0.6182 61.02% VisMF::Write(FabArray) 11 0.3074 0.3074 0.3074 30.35% Gravity::solve_phi_with_mlmg() 11 0.2953 0.2953 0.2953 29.15% Gravity::actual_solve_with_mlmg() 11 0.2877 0.2877 0.2877 28.40% Castro::construct_ctu_hydro_source() 10 0.2749 0.2749 0.2749 27.14% Castro::construct_new_gravity() 10 0.27 0.27 0.27 26.65% MLMG::solve() 11 0.2653 0.2653 0.2653 26.19% Gravity::solve_for_phi() 10 0.2535 0.2535 0.2535 25.03% MLMG::oneIter() 82 0.2504 0.2504 0.2504 24.71% MLMG::mgVcycle() 82 0.2467 0.2467 0.2467 24.35% Amr::checkPoint() 3 0.242 0.242 0.242 23.88% AmrLevel::checkPoint() 3 0.2376 0.2376 0.2376 23.46% StateData::checkPoint() 12 0.2375 0.2375 0.2375 23.45% Amr::init() 1 0.1568 0.1568 0.1568 15.48% MLCellLinOp::smooth() 1640 0.1204 0.1204 0.1204 11.89% MLCellLinOp::applyBC() 4433 0.1129 0.1129 0.1129 11.15% MLMG::mgVcycle_bottom 82 0.07676 0.07676 0.07676 7.58% MLMG::actualBottomSolve() 82 0.07672 0.07672 0.07672 7.57% MLCGSolver::bicgstab 82 0.07597 0.07597 0.07597 7.50% Amr::writePlotFile() 2 0.07308 0.07308 0.07308 7.21% Castro::clean_state() 62 0.05697 0.05697 0.05697 5.62% Amr::initialInit() 1 0.05 0.05 0.05 4.94% AmrLevel::FillPatch() 41 0.04829 0.04829 0.04829 4.77% Amr::FinalizeInit() 1 0.04586 0.04586 0.04586 4.53% Castro::post_init() 1 0.04441 0.04441 0.04441 4.38% FillPatchIterator::Initialize 41 0.04406 0.04406 0.04406 4.35% FillPatchIterator::FillFromLevel0() 41 0.04243 0.04243 0.04243 4.19% FillPatchSingleLevel 41 0.04237 0.04237 0.04237 4.18% Gravity::multilevel_solve_for_new_phi() 1 0.04228 0.04228 0.04228 4.17% Gravity::actual_multilevel_solve() 1 0.04226 0.04226 0.04226 4.17% StateDataPhysBCFunct::() 41 0.03814 0.03814 0.03814 3.76% MLCellLinOp::apply() 1142 0.03776 0.03776 0.03776 3.73% MLMG::mgVcycle_down::0 82 0.03433 0.03433 0.03433 3.39% MLPoisson::Fsmooth() 3280 0.0334 0.0334 0.0334 3.30% FabArray::FillBoundary() 4023 0.0323 0.0323 0.0323 3.19% FillBoundary_nowait() 4023 0.03139 0.03139 0.03139 3.10% StateData::FillBoundary(geom) 328 0.02651 0.02651 0.02651 2.62% MLMG::mgVcycle_up::0 82 0.02591 0.02591 0.02591 2.56% Castro::computeTemp() 63 0.02376 0.02376 0.02376 2.34% MLCellLinOp::correctionResidual() 492 0.02298 0.02298 0.02298 2.27% amrex::Dot() 1114 0.02135 0.02135 0.02135 2.11% Castro::initialize_do_advance() 10 0.02106 0.02106 0.02106 2.08% MLMG:computeResOfCorrection() 410 0.02018 0.02018 0.02018 1.99% Castro::normalize_species() 62 0.01975 0.01975 0.01975 1.95% Castro::do_old_sources() 10 0.01924 0.01924 0.01924 1.90% Gravity::get_new_grav_vector() 11 0.01826 0.01826 0.01826 1.80% MLPoisson::define() 11 0.01764 0.01764 0.01764 1.74% MLMG::mgVcycle_down::1 82 0.01659 0.01659 0.01659 1.64% MLMG::mgVcycle_down::2 82 0.01541 0.01541 0.01541 1.52% Castro::construct_old_gravity() 10 0.01535 0.01535 0.01535 1.52% Gravity::get_old_grav_vector() 10 0.01534 0.01534 0.01534 1.51% amrex::Copy() 1029 0.01533 0.01533 0.01533 1.51% FabArray::ParallelCopy() 861 0.01524 0.01524 0.01524 1.50% MLMG::mgVcycle_down::3 82 0.01507 0.01507 0.01507 1.49% FabArray::ParallelCopy_nowait() 861 0.01496 0.01496 0.01496 1.48% MLMG::mgVcycle_down::4 82 0.01491 0.01491 0.01491 1.47% FabArray::norminf() 743 0.0149 0.0149 0.0149 1.47% Castro::do_new_sources() 10 0.01408 0.01408 0.01408 1.39% FabArray::setVal() 1144 0.01355 0.01355 0.01355 1.34% Castro::initialize_advance() 10 0.01339 0.01339 0.01339 1.32% MLCGSolver::ParallelAllReduce 1514 0.01285 0.01285 0.01285 1.27% MLMG::addInterpCorrection() 410 0.01256 0.01256 0.01256 1.24% MLMG::mgVcycle_up::4 82 0.012 0.012 0.012 1.18% MLMG::mgVcycle_up::1 82 0.012 0.012 0.012 1.18% amrex::average_down 410 0.01184 0.01184 0.01184 1.17% Castro::expand_state() 10 0.01182 0.01182 0.01182 1.17% Castro::enforce_min_density() 62 0.01182 0.01182 0.01182 1.17% MLCellLinOp::defineAuxData() 11 0.01177 0.01177 0.01177 1.16% MLMG::mgVcycle_up::2 82 0.01175 0.01175 0.01175 1.16% MLMG::mgVcycle_up::3 82 0.01154 0.01154 0.01154 1.14% Castro::post_timestep() 10 0.01086 0.01086 0.01086 1.07% MLPoisson::Fapply() 1142 0.01075 0.01075 0.01075 1.06% FabArray::Saxpy() 813 0.008398 0.008398 0.008398 0.83% FabArray::Xpay() 821 0.00833 0.00833 0.00833 0.82% MLCellLinOp::solutionResidual() 93 0.007409 0.007409 0.007409 0.73% Gravity::fill_multipole_BCs() 11 0.007383 0.007383 0.007383 0.73% Castro::reset_internal_energy(MultiFab) 63 0.007143 0.007143 0.007143 0.71% MLMG::computeResidual() 82 0.006329 0.006329 0.006329 0.62% Castro::estTimeStep() 21 0.005548 0.005548 0.005548 0.55% MLCellLinOp::defineBC() 11 0.005256 0.005256 0.005256 0.52% BndryData::define() 11 0.005008 0.005008 0.005008 0.49% MLMG::prepareForSolve() 11 0.004972 0.004972 0.004972 0.49% FabArray::LinComb() 557 0.004699 0.004699 0.004699 0.46% amrex::Add() 164 0.004381 0.004381 0.004381 0.43% Amr::InitializeInit() 1 0.004141 0.004141 0.004141 0.41% Amr::defBaseLevel() 1 0.004136 0.004136 0.004136 0.41% Castro::initData() 1 0.003613 0.003613 0.003613 0.36% Castro::construct_new_source() 50 0.003297 0.003297 0.003297 0.33% Castro::construct_new_gravity_source() 10 0.003145 0.003145 0.003145 0.31% Castro::construct_old_source() 50 0.002764 0.002764 0.002764 0.27% Castro::construct_old_gravity_source() 10 0.002741 0.002741 0.002741 0.27% Castro::computeNewDt() 9 0.002505 0.002505 0.002505 0.25% Castro::finalize_do_advance() 10 0.002494 0.002494 0.002494 0.25% Castro::enforce_speed_limit() 62 0.002376 0.002376 0.002376 0.23% MLMG::ResNormInf() 93 0.002212 0.002212 0.002212 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001845 0.001845 0.001845 0.18% Castro::apply_source_to_state() 20 0.001834 0.001834 0.001834 0.18% Castro::reset_internal_energy(Fab) 504 0.001635 0.001635 0.001635 0.16% FabArrayBase::getCPC() 1323 0.001596 0.001596 0.001596 0.16% MLCellLinOp::setLevelBC() 11 0.001467 0.001467 0.001467 0.14% MLMG::getGradSolution() 11 0.001464 0.001464 0.001464 0.14% MLCellLinOp::compGrad() 11 0.001453 0.001453 0.001453 0.14% check_for_negative_density() 10 0.001445 0.001445 0.001445 0.14% FabArray::setDomainBndry() 41 0.001388 0.001388 0.001388 0.14% FabArray::mult() 43 0.001379 0.001379 0.001379 0.14% Castro::check_for_nan() 20 0.001258 0.001258 0.001258 0.12% Castro::post_regrid() 1 0.001248 0.001248 0.001248 0.12% MultiFab::contains_nan() 20 0.001241 0.001241 0.001241 0.12% MLPoisson::prepareForSolve() 11 0.001223 0.001223 0.001223 0.12% MLCellLinOp::prepareForSolve() 11 0.001206 0.001206 0.001206 0.12% MLMG::computeMLResidual() 11 0.001125 0.001125 0.001125 0.11% Gravity::update_max_rhs() 11 0.000891 0.000891 0.000891 0.09% FabArrayBase::CPC::define() 454 0.0007794 0.0007794 0.0007794 0.08% FabArrayBase::getFB() 4023 0.0007576 0.0007576 0.0007576 0.07% Castro::finalize_advance() 10 0.0006227 0.0006227 0.0006227 0.06% Castro::computeInitialDt() 2 0.0006096 0.0006096 0.0006096 0.06% MLLinOp::define() 11 0.0005723 0.0005723 0.0005723 0.06% MLLinOp::defineGrids() 11 0.0005578 0.0005578 0.0005578 0.06% Amr::InitAmr() 1 0.0005304 0.0005304 0.0005304 0.05% Gravity::swapTimeLevels() 10 0.0004509 0.0004509 0.0004509 0.04% Castro::Castro() 1 0.0004502 0.0004502 0.0004502 0.04% MLMG::MLResNormInf() 11 0.0002989 0.0002989 0.0002989 0.03% MultiFab::max() 11 0.0002698 0.0002698 0.0002698 0.03% MLMG::MLRhsNormInf() 11 0.000223 0.000223 0.000223 0.02% Castro::create_source_corrector() 10 0.0001634 0.0001634 0.0001634 0.02% Castro::buildMetrics() 1 0.0001584 0.0001584 0.0001584 0.02% FabArrayBase::FB::FB() 56 0.0001044 0.0001044 0.0001044 0.01% MLLinOp::makeAgglomeratedDMap 11 6.851e-05 6.851e-05 6.851e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.356e-05 5.356e-05 5.356e-05 0.01% makeSFC 55 4.941e-05 4.941e-05 4.941e-05 0.00% Castro::swap_state_time_levels() 10 4.266e-05 4.266e-05 4.266e-05 0.00% StateData::define() 4 3.619e-05 3.619e-05 3.619e-05 0.00% Castro::enforce_consistent_e() 1 3.465e-05 3.465e-05 3.465e-05 0.00% Amr::writeSmallPlotFile() 1 2.649e-05 2.649e-05 2.649e-05 0.00% Castro::initMFs() 1 2.573e-05 2.573e-05 2.573e-05 0.00% DistributionMapping::Distribute() 56 1.921e-05 1.921e-05 1.921e-05 0.00% Amr::initSubcycle() 1 9.372e-06 9.372e-06 9.372e-06 0.00% AmrLevel::checkPointPost() 3 6.551e-06 6.551e-06 6.551e-06 0.00% Gravity::set_mass_offset() 11 6.536e-06 6.536e-06 6.536e-06 0.00% Castro::retry_advance_ctu() 10 5.45e-06 5.45e-06 5.45e-06 0.00% Castro::FluxRegCrseInit 10 4.82e-06 4.82e-06 4.82e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.26e-06 4.26e-06 4.26e-06 0.00% Castro::FluxRegFineAdd() 10 2.47e-06 2.47e-06 2.47e-06 0.00% AmrLevel::checkPointPre() 3 2.214e-06 2.214e-06 2.214e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.967e-06 1.967e-06 1.967e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 86 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 122 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 924 KiB 39 MiB Castro::initialize_do_advance() 80 80 23 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2028 KiB 28 MiB Castro::initialize_advance() 80 80 14 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7609 KiB 14 MiB MLMG::prepareForSolve() 660 660 3222 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 187 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 155 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7532 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1671 B 2048 KiB Gravity::solve_for_phi() 80 80 511 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 85 KiB 2048 KiB BndryData::define() 1056 1056 294 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 187 KiB 671 KiB Castro::estTimeStep() 21 21 2669 B 480 KiB VisMF::Write(FabArray) 656 656 3012 B 320 KiB Castro::normalize_species() 62 62 6354 B 320 KiB amrex::average_down 1067 1067 1189 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1058 B 257 KiB amrex::Dot() 1360 1360 3092 B 160 KiB FabArray::norminf() 907 907 2167 B 160 KiB check_for_negative_density() 10 10 223 B 160 KiB Castro::initData() 1 1 24 B 160 KiB MultiFab::max() 11 11 40 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 24 B 20 KiB MLPoisson::Fsmooth() 132 132 3149 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 39 B 10 KiB FillBoundary_nowait() 760 760 262 B 9648 B MLCellLinOp::applyBC() 8866 8866 202 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3779 B 6144 B StateData::FillBoundary(geom) 1992 1992 39 B 2688 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 109 B 1472 B MLCellLinOp::defineBC() 66 66 331 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 573 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 46 KiB 8192 KiB VisMF::Write(FabArray) 744 744 592 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3149 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 39 B 10 KiB FillBoundary_nowait() 760 760 262 B 9648 B MLCellLinOp::applyBC() 4433 4433 200 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3779 B 6144 B Gravity::get_new_grav_vector() 3 3 2921 B 3072 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B StateData::FillBoundary(geom) 1992 1992 39 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 268 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 22 B 400 B FabArray::norminf() 907 907 16 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2422 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-23-gd92232a0c17b) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-23-gd92232a0c17b) initialized Starting run at 07:32:20 UTC on 2023-08-10. Successfully read inputs file ... Castro git describe: 23.08-3-g76295aa13 AMReX git describe: 23.08-23-gd92232a0c Microphysics git describe: 23.08-3-g3d2714f8 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.598858203 Restart time = 0.045038125 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052132592 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.06995126 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.083162617 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.070871616 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059452237 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033721703 seconds Ending run at 07:32:21 UTC on 2023-08-10. Run time = 0.415360463 Run time without initialization = 0.369737599 Average number of zones advanced per microsecond: 3.545 Average number of zones advanced per microsecond per rank: 3.545 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.4154 ... 0.4154 ... 0.4154 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1488 0.1488 0.1488 35.81% VisMF::Read() 3 0.03882 0.03882 0.03882 9.35% MLCellLinOp::applyBC() 1946 0.03335 0.03335 0.03335 8.03% VisMF::Write(FabArray) 1 0.03209 0.03209 0.03209 7.73% MLPoisson::Fsmooth() 1440 0.0139 0.0139 0.0139 3.35% FillBoundary_nowait() 1766 0.01262 0.01262 0.01262 3.04% StateData::FillBoundary(geom) 160 0.01228 0.01228 0.01228 2.96% Castro::normalize_species() 30 0.01096 0.01096 0.01096 2.64% amrex::Dot() 484 0.008811 0.008811 0.008811 2.12% Castro::computeTemp() 30 0.007549 0.007549 0.007549 1.82% amrex::Copy() 463 0.006975 0.006975 0.006975 1.68% Castro::enforce_min_density() 30 0.006318 0.006318 0.006318 1.52% FabArray::setVal() 537 0.006288 0.006288 0.006288 1.51% FabArray::norminf() 326 0.006228 0.006228 0.006228 1.50% FabArray::ParallelCopy_nowait() 380 0.00607 0.00607 0.00607 1.46% MLCellLinOp::defineAuxData() 6 0.005205 0.005205 0.005205 1.25% MLPoisson::Fapply() 500 0.004454 0.004454 0.004454 1.07% StateDataPhysBCFunct::() 20 0.004161 0.004161 0.004161 1.00% Gravity::fill_multipole_BCs() 6 0.003654 0.003654 0.003654 0.88% FabArray::Saxpy() 355 0.00361 0.00361 0.00361 0.87% FabArray::Xpay() 361 0.003508 0.003508 0.003508 0.84% MLMG::addInterpCorrection() 180 0.002933 0.002933 0.002933 0.71% amrex::average_down 180 0.002609 0.002609 0.002609 0.63% Amr::restart() 1 0.00249 0.00249 0.00249 0.60% Castro::estTimeStep() 10 0.00242 0.00242 0.00242 0.58% BndryData::define() 6 0.002015 0.002015 0.002015 0.49% Castro::reset_internal_energy(MultiFab) 30 0.002004 0.002004 0.002004 0.48% FabArray::LinComb() 242 0.001927 0.001927 0.001927 0.46% amrex::Add() 72 0.001831 0.001831 0.001831 0.44% Castro::construct_new_gravity_source() 5 0.00157 0.00157 0.00157 0.38% Amr::writePlotFile() 1 0.001468 0.001468 0.001468 0.35% Castro::construct_old_gravity_source() 5 0.001433 0.001433 0.001433 0.35% Castro::enforce_speed_limit() 30 0.001189 0.001189 0.001189 0.29% MLCGSolver::bicgstab 36 0.0009372 0.0009372 0.0009372 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009185 0.0009185 0.0009185 0.22% Castro::reset_internal_energy(Fab) 240 0.0008856 0.0008856 0.0008856 0.21% Castro::finalize_do_advance() 5 0.000843 0.000843 0.000843 0.20% check_for_negative_density() 5 0.0008089 0.0008089 0.0008089 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007808 0.0007808 0.0007808 0.19% MLCellLinOp::setLevelBC() 6 0.0007436 0.0007436 0.0007436 0.18% FabArray::mult() 22 0.0006464 0.0006464 0.0006464 0.16% FabArray::setDomainBndry() 20 0.0006345 0.0006345 0.0006345 0.15% MLCellLinOp::prepareForSolve() 6 0.0006066 0.0006066 0.0006066 0.15% MultiFab::contains_nan() 10 0.0005892 0.0005892 0.0005892 0.14% MLCellLinOp::compGrad() 6 0.0004867 0.0004867 0.0004867 0.12% MLMG::prepareForSolve() 6 0.0004686 0.0004686 0.0004686 0.11% MLCellLinOp::smooth() 720 0.0004456 0.0004456 0.0004456 0.11% FabArrayBase::CPC::define() 244 0.0004141 0.0004141 0.0004141 0.10% Amr::InitAmr() 1 0.0003996 0.0003996 0.0003996 0.10% FabArray::FillBoundary() 1766 0.0003668 0.0003668 0.0003668 0.09% FabArrayBase::getCPC() 632 0.0003608 0.0003608 0.0003608 0.09% main() 1 0.0002773 0.0002773 0.0002773 0.07% FabArrayBase::getFB() 1766 0.0002613 0.0002613 0.0002613 0.06% Gravity::get_old_grav_vector() 5 0.0002573 0.0002573 0.0002573 0.06% Gravity::get_new_grav_vector() 5 0.0002326 0.0002326 0.0002326 0.06% Amr::coarseTimeStep() 5 0.0002038 0.0002038 0.0002038 0.05% MLLinOp::defineGrids() 6 0.0001952 0.0001952 0.0001952 0.05% MLCellLinOp::apply() 500 0.0001816 0.0001816 0.0001816 0.04% AmrLevel::FillPatch() 20 0.0001679 0.0001679 0.0001679 0.04% MLMG::mgVcycle() 36 0.0001501 0.0001501 0.0001501 0.04% MultiFab::max() 6 0.0001358 0.0001358 0.0001358 0.03% MLCGSolver::ParallelAllReduce 659 0.0001278 0.0001278 0.0001278 0.03% Castro::construct_new_source() 25 0.0001097 0.0001097 0.0001097 0.03% MLCellLinOp::defineBC() 6 0.000109 0.000109 0.000109 0.03% FabArray::ParallelCopy() 380 0.0001075 0.0001075 0.0001075 0.03% FillPatchIterator::Initialize 20 0.0001022 0.0001022 0.0001022 0.02% Castro::do_new_sources() 5 0.000102 0.000102 0.000102 0.02% MLCellLinOp::correctionResidual() 216 9.348e-05 9.348e-05 9.348e-05 0.02% Castro::create_source_corrector() 5 8.372e-05 8.372e-05 8.372e-05 0.02% Castro::subcycle_advance_ctu() 5 7.564e-05 7.564e-05 7.564e-05 0.02% Amr::timeStep() 5 7.557e-05 7.557e-05 7.557e-05 0.02% AmrLevel::restart() 1 7.437e-05 7.437e-05 7.437e-05 0.02% Gravity::update_max_rhs() 6 6.462e-05 6.462e-05 6.462e-05 0.02% StateData::restartDoit() 4 6.233e-05 6.233e-05 6.233e-05 0.02% FabArrayBase::FB::FB() 26 5.869e-05 5.869e-05 5.869e-05 0.01% Gravity::solve_for_phi() 5 5.185e-05 5.185e-05 5.185e-05 0.01% Castro::initialize_do_advance() 5 5.012e-05 5.012e-05 5.012e-05 0.01% MLMG:computeResOfCorrection() 180 4.974e-05 4.974e-05 4.974e-05 0.01% MLMG::actualBottomSolve() 36 4.237e-05 4.237e-05 4.237e-05 0.01% MLMG::mgVcycle_down::0 36 4.021e-05 4.021e-05 4.021e-05 0.01% Castro::advance() 5 3.855e-05 3.855e-05 3.855e-05 0.01% MLMG::mgVcycle_down::1 36 3.745e-05 3.745e-05 3.745e-05 0.01% Castro::initialize_advance() 5 3.736e-05 3.736e-05 3.736e-05 0.01% MLMG::mgVcycle_down::2 36 3.628e-05 3.628e-05 3.628e-05 0.01% MLMG::mgVcycle_down::4 36 3.553e-05 3.553e-05 3.553e-05 0.01% MLMG::mgVcycle_down::3 36 3.535e-05 3.535e-05 3.535e-05 0.01% Castro::do_old_sources() 5 3.462e-05 3.462e-05 3.462e-05 0.01% Castro::post_timestep() 5 3.318e-05 3.318e-05 3.318e-05 0.01% Castro::construct_old_source() 25 3.247e-05 3.247e-05 3.247e-05 0.01% Castro::clean_state() 30 3.231e-05 3.231e-05 3.231e-05 0.01% MLMG::solve() 6 3.218e-05 3.218e-05 3.218e-05 0.01% MLMG::mgVcycle_up::4 36 2.854e-05 2.854e-05 2.854e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.775e-05 2.775e-05 2.775e-05 0.01% Castro::finalize_advance() 5 2.768e-05 2.768e-05 2.768e-05 0.01% Castro::initMFs() 1 2.639e-05 2.639e-05 2.639e-05 0.01% Amr::writeSmallPlotFile() 1 2.594e-05 2.594e-05 2.594e-05 0.01% Castro::buildMetrics() 1 2.592e-05 2.592e-05 2.592e-05 0.01% MLMG::oneIter() 36 2.422e-05 2.422e-05 2.422e-05 0.01% MLMG::mgVcycle_up::0 36 2.403e-05 2.403e-05 2.403e-05 0.01% MLMG::mgVcycle_up::3 36 2.36e-05 2.36e-05 2.36e-05 0.01% Castro::do_advance_ctu() 5 2.308e-05 2.308e-05 2.308e-05 0.01% Castro::swap_state_time_levels() 5 2.292e-05 2.292e-05 2.292e-05 0.01% MLMG::mgVcycle_up::2 36 2.272e-05 2.272e-05 2.272e-05 0.01% MLMG::mgVcycle_up::1 36 2.234e-05 2.234e-05 2.234e-05 0.01% MLCellLinOp::solutionResidual() 42 2.133e-05 2.133e-05 2.133e-05 0.01% Castro::post_restart() 1 2.027e-05 2.027e-05 2.027e-05 0.00% MLPoisson::define() 6 1.835e-05 1.835e-05 1.835e-05 0.00% MLMG::ResNormInf() 42 1.78e-05 1.78e-05 1.78e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.545e-05 1.545e-05 1.545e-05 0.00% MLMG::mgVcycle_bottom 36 1.507e-05 1.507e-05 1.507e-05 0.00% FillPatchSingleLevel 20 1.473e-05 1.473e-05 1.473e-05 0.00% MLMG::computeResidual() 36 1.428e-05 1.428e-05 1.428e-05 0.00% makeSFC 30 1.286e-05 1.286e-05 1.286e-05 0.00% Castro::construct_new_gravity() 5 1.222e-05 1.222e-05 1.222e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.162e-05 1.162e-05 1.162e-05 0.00% MLPoisson::prepareForSolve() 6 9.149e-06 9.149e-06 9.149e-06 0.00% DistributionMapping::Distribute() 31 9.145e-06 9.145e-06 9.145e-06 0.00% Amr::initSubcycle() 1 8.596e-06 8.596e-06 8.596e-06 0.00% Gravity::actual_multilevel_solve() 1 7.654e-06 7.654e-06 7.654e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.263e-06 7.263e-06 7.263e-06 0.00% Castro::construct_old_gravity() 5 5.956e-06 5.956e-06 5.956e-06 0.00% MLLinOp::define() 6 5.732e-06 5.732e-06 5.732e-06 0.00% Castro::check_for_nan() 10 5.713e-06 5.713e-06 5.713e-06 0.00% Castro::apply_source_to_state() 10 5.135e-06 5.135e-06 5.135e-06 0.00% Gravity::swapTimeLevels() 5 3.951e-06 3.951e-06 3.951e-06 0.00% MLMG::computeMLResidual() 6 3.657e-06 3.657e-06 3.657e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.618e-06 3.618e-06 3.618e-06 0.00% Castro::computeNewDt() 5 3.251e-06 3.251e-06 3.251e-06 0.00% MLMG::getGradSolution() 6 3.162e-06 3.162e-06 3.162e-06 0.00% Castro::expand_state() 5 2.433e-06 2.433e-06 2.433e-06 0.00% MLMG::MLResNormInf() 6 2.241e-06 2.241e-06 2.241e-06 0.00% MLMG::MLRhsNormInf() 6 2.178e-06 2.178e-06 2.178e-06 0.00% Gravity::set_mass_offset() 6 2.059e-06 2.059e-06 2.059e-06 0.00% Castro::retry_advance_ctu() 5 1.823e-06 1.823e-06 1.823e-06 0.00% Castro::FluxRegCrseInit 5 1.477e-06 1.477e-06 1.477e-06 0.00% Amr::init() 1 1.211e-06 1.211e-06 1.211e-06 0.00% Castro::FluxRegFineAdd() 5 1.091e-06 1.091e-06 1.091e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.072e-06 1.072e-06 1.072e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4154 0.4154 0.4154 100.00% Amr::coarseTimeStep() 5 0.3358 0.3358 0.3358 80.83% Amr::timeStep() 5 0.3341 0.3341 0.3341 80.44% Castro::advance() 5 0.3296 0.3296 0.3296 79.34% Castro::subcycle_advance_ctu() 5 0.3218 0.3218 0.3218 77.48% Castro::do_advance_ctu() 5 0.3217 0.3217 0.3217 77.46% Castro::construct_ctu_hydro_source() 5 0.1552 0.1552 0.1552 37.37% Castro::construct_new_gravity() 5 0.1299 0.1299 0.1299 31.28% Gravity::solve_phi_with_mlmg() 6 0.1252 0.1252 0.1252 30.15% Gravity::solve_for_phi() 5 0.1222 0.1222 0.1222 29.42% Gravity::actual_solve_with_mlmg() 6 0.1213 0.1213 0.1213 29.21% MLMG::solve() 6 0.1102 0.1102 0.1102 26.52% MLMG::oneIter() 36 0.1033 0.1033 0.1033 24.87% MLMG::mgVcycle() 36 0.1018 0.1018 0.1018 24.50% MLCellLinOp::smooth() 720 0.04974 0.04974 0.04974 11.98% MLCellLinOp::applyBC() 1946 0.04665 0.04665 0.04665 11.23% Amr::init() 1 0.04508 0.04508 0.04508 10.85% Amr::restart() 1 0.04508 0.04508 0.04508 10.85% AmrLevel::restart() 1 0.03903 0.03903 0.03903 9.39% StateData::restartDoit() 4 0.03895 0.03895 0.03895 9.38% VisMF::Read() 3 0.03882 0.03882 0.03882 9.35% Amr::writePlotFile() 1 0.03381 0.03381 0.03381 8.14% VisMF::Write(FabArray) 1 0.03209 0.03209 0.03209 7.73% MLMG::mgVcycle_bottom 36 0.0316 0.0316 0.0316 7.61% MLMG::actualBottomSolve() 36 0.03158 0.03158 0.03158 7.60% MLCGSolver::bicgstab 36 0.03128 0.03128 0.03128 7.53% Castro::clean_state() 30 0.02894 0.02894 0.02894 6.97% AmrLevel::FillPatch() 20 0.02132 0.02132 0.02132 5.13% FillPatchIterator::Initialize 20 0.01926 0.01926 0.01926 4.64% FillPatchIterator::FillFromLevel0() 20 0.01852 0.01852 0.01852 4.46% FillPatchSingleLevel 20 0.0185 0.0185 0.0185 4.45% StateDataPhysBCFunct::() 20 0.01644 0.01644 0.01644 3.96% MLCellLinOp::apply() 500 0.01563 0.01563 0.01563 3.76% MLMG::mgVcycle_down::0 36 0.01423 0.01423 0.01423 3.43% MLPoisson::Fsmooth() 1440 0.0139 0.0139 0.0139 3.35% FabArray::FillBoundary() 1766 0.01331 0.01331 0.01331 3.20% FillBoundary_nowait() 1766 0.01294 0.01294 0.01294 3.12% StateData::FillBoundary(geom) 160 0.01228 0.01228 0.01228 2.96% Castro::normalize_species() 30 0.01096 0.01096 0.01096 2.64% MLMG::mgVcycle_up::0 36 0.01077 0.01077 0.01077 2.59% Castro::computeTemp() 30 0.01044 0.01044 0.01044 2.51% Castro::initialize_do_advance() 5 0.01024 0.01024 0.01024 2.47% MLCellLinOp::correctionResidual() 216 0.009492 0.009492 0.009492 2.29% Castro::do_old_sources() 5 0.009405 0.009405 0.009405 2.26% MLPoisson::define() 6 0.008873 0.008873 0.008873 2.14% amrex::Dot() 484 0.008811 0.008811 0.008811 2.12% MLMG:computeResOfCorrection() 180 0.008343 0.008343 0.008343 2.01% Gravity::get_new_grav_vector() 5 0.007625 0.007625 0.007625 1.84% Castro::initialize_advance() 5 0.007406 0.007406 0.007406 1.78% Castro::construct_old_gravity() 5 0.00734 0.00734 0.00734 1.77% Gravity::get_old_grav_vector() 5 0.007334 0.007334 0.007334 1.77% Castro::do_new_sources() 5 0.007236 0.007236 0.007236 1.74% amrex::Copy() 463 0.006975 0.006975 0.006975 1.68% MLMG::mgVcycle_down::1 36 0.006808 0.006808 0.006808 1.64% FabArray::ParallelCopy() 380 0.006576 0.006576 0.006576 1.58% FabArray::ParallelCopy_nowait() 380 0.006468 0.006468 0.006468 1.56% MLMG::mgVcycle_down::2 36 0.006379 0.006379 0.006379 1.54% Castro::enforce_min_density() 30 0.006318 0.006318 0.006318 1.52% FabArray::setVal() 537 0.006288 0.006288 0.006288 1.51% FabArray::norminf() 326 0.006228 0.006228 0.006228 1.50% MLMG::mgVcycle_down::3 36 0.00621 0.00621 0.00621 1.49% MLMG::mgVcycle_down::4 36 0.00614 0.00614 0.00614 1.48% MLCellLinOp::defineAuxData() 6 0.005939 0.005939 0.005939 1.43% MLCGSolver::ParallelAllReduce 659 0.005332 0.005332 0.005332 1.28% MLMG::addInterpCorrection() 180 0.005178 0.005178 0.005178 1.25% MLMG::mgVcycle_up::4 36 0.004969 0.004969 0.004969 1.20% Castro::expand_state() 5 0.00496 0.00496 0.00496 1.19% MLMG::mgVcycle_up::1 36 0.004924 0.004924 0.004924 1.19% amrex::average_down 180 0.004902 0.004902 0.004902 1.18% MLMG::mgVcycle_up::2 36 0.004829 0.004829 0.004829 1.16% MLMG::mgVcycle_up::3 36 0.004765 0.004765 0.004765 1.15% Castro::post_timestep() 5 0.004519 0.004519 0.004519 1.09% MLPoisson::Fapply() 500 0.004454 0.004454 0.004454 1.07% Gravity::fill_multipole_BCs() 6 0.003796 0.003796 0.003796 0.91% FabArray::Saxpy() 355 0.00361 0.00361 0.00361 0.87% FabArray::Xpay() 361 0.003508 0.003508 0.003508 0.84% Castro::post_restart() 1 0.003384 0.003384 0.003384 0.81% Gravity::multilevel_solve_for_new_phi() 1 0.003267 0.003267 0.003267 0.79% Gravity::actual_multilevel_solve() 1 0.003252 0.003252 0.003252 0.78% MLCellLinOp::solutionResidual() 42 0.003196 0.003196 0.003196 0.77% Castro::reset_internal_energy(MultiFab) 30 0.002889 0.002889 0.002889 0.70% MLCellLinOp::defineBC() 6 0.002686 0.002686 0.002686 0.65% MLMG::computeResidual() 36 0.002646 0.002646 0.002646 0.64% BndryData::define() 6 0.002577 0.002577 0.002577 0.62% MLMG::prepareForSolve() 6 0.00253 0.00253 0.00253 0.61% Castro::estTimeStep() 10 0.00242 0.00242 0.00242 0.58% FabArray::LinComb() 242 0.001927 0.001927 0.001927 0.46% Castro::finalize_do_advance() 5 0.001849 0.001849 0.001849 0.45% amrex::Add() 72 0.001831 0.001831 0.001831 0.44% Castro::construct_new_source() 25 0.00168 0.00168 0.00168 0.40% Castro::construct_new_gravity_source() 5 0.00157 0.00157 0.00157 0.38% Castro::construct_old_source() 25 0.001466 0.001466 0.001466 0.35% Castro::construct_old_gravity_source() 5 0.001433 0.001433 0.001433 0.35% Castro::computeNewDt() 5 0.001417 0.001417 0.001417 0.34% Castro::enforce_speed_limit() 30 0.001189 0.001189 0.001189 0.29% MLMG::ResNormInf() 42 0.0009556 0.0009556 0.0009556 0.23% Castro::apply_source_to_state() 10 0.0009198 0.0009198 0.0009198 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009185 0.0009185 0.0009185 0.22% Castro::reset_internal_energy(Fab) 240 0.0008856 0.0008856 0.0008856 0.21% check_for_negative_density() 5 0.0008089 0.0008089 0.0008089 0.19% FabArrayBase::getCPC() 632 0.000775 0.000775 0.000775 0.19% MLMG::getGradSolution() 6 0.0007517 0.0007517 0.0007517 0.18% MLCellLinOp::compGrad() 6 0.0007485 0.0007485 0.0007485 0.18% MLCellLinOp::setLevelBC() 6 0.0007436 0.0007436 0.0007436 0.18% FabArray::mult() 22 0.0006464 0.0006464 0.0006464 0.16% FabArray::setDomainBndry() 20 0.0006345 0.0006345 0.0006345 0.15% MLPoisson::prepareForSolve() 6 0.0006157 0.0006157 0.0006157 0.15% MLCellLinOp::prepareForSolve() 6 0.0006066 0.0006066 0.0006066 0.15% Castro::check_for_nan() 10 0.0005949 0.0005949 0.0005949 0.14% MultiFab::contains_nan() 10 0.0005892 0.0005892 0.0005892 0.14% MLMG::computeMLResidual() 6 0.0005681 0.0005681 0.0005681 0.14% Gravity::update_max_rhs() 6 0.0004458 0.0004458 0.0004458 0.11% FabArrayBase::CPC::define() 244 0.0004141 0.0004141 0.0004141 0.10% Amr::InitAmr() 1 0.0004082 0.0004082 0.0004082 0.10% FabArrayBase::getFB() 1766 0.0003199 0.0003199 0.0003199 0.08% Castro::finalize_advance() 5 0.0002829 0.0002829 0.0002829 0.07% MLLinOp::define() 6 0.0002294 0.0002294 0.0002294 0.06% Gravity::swapTimeLevels() 5 0.0002244 0.0002244 0.0002244 0.05% MLLinOp::defineGrids() 6 0.0002237 0.0002237 0.0002237 0.05% Castro::buildMetrics() 1 0.000153 0.000153 0.000153 0.04% MLMG::MLResNormInf() 6 0.0001509 0.0001509 0.0001509 0.04% MultiFab::max() 6 0.0001358 0.0001358 0.0001358 0.03% MLMG::MLRhsNormInf() 6 0.0001161 0.0001161 0.0001161 0.03% Castro::create_source_corrector() 5 8.372e-05 8.372e-05 8.372e-05 0.02% FabArrayBase::FB::FB() 26 5.869e-05 5.869e-05 5.869e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.743e-05 2.743e-05 2.743e-05 0.01% Castro::initMFs() 1 2.639e-05 2.639e-05 2.639e-05 0.01% Amr::writeSmallPlotFile() 1 2.594e-05 2.594e-05 2.594e-05 0.01% Castro::swap_state_time_levels() 5 2.292e-05 2.292e-05 2.292e-05 0.01% makeSFC 30 2.017e-05 2.017e-05 2.017e-05 0.00% DistributionMapping::Distribute() 31 9.145e-06 9.145e-06 9.145e-06 0.00% Amr::initSubcycle() 1 8.596e-06 8.596e-06 8.596e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.454e-06 5.454e-06 5.454e-06 0.00% Gravity::set_mass_offset() 6 2.059e-06 2.059e-06 2.059e-06 0.00% Castro::retry_advance_ctu() 5 1.823e-06 1.823e-06 1.823e-06 0.00% Castro::FluxRegCrseInit 5 1.477e-06 1.477e-06 1.477e-06 0.00% Castro::FluxRegFineAdd() 5 1.091e-06 1.091e-06 1.091e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.072e-06 1.072e-06 1.072e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 193 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 167 MiB 692 MiB Castro::initMFs() 48 48 61 MiB 68 MiB Castro::swap_state_time_levels() 32 32 49 MiB 55 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB FillPatchIterator::Initialize 160 160 982 KiB 39 MiB Castro::initialize_do_advance() 40 40 30 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2277 KiB 28 MiB Castro::initialize_advance() 40 40 18 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6803 KiB 14 MiB MLMG::prepareForSolve() 361 361 3256 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 183 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6798 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 15 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2079 B 2048 KiB Gravity::solve_for_phi() 40 40 601 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 15 KiB 2048 KiB BndryData::define() 576 576 299 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 192 KiB 671 KiB Castro::estTimeStep() 10 10 2774 B 480 KiB VisMF::Write(FabArray) 112 112 1207 B 320 KiB Castro::normalize_species() 30 30 8596 B 320 KiB amrex::average_down 469 469 1161 B 257 KiB MLMG::addInterpCorrection() 468 468 1073 B 257 KiB amrex::Dot() 592 592 3120 B 160 KiB FabArray::norminf() 398 398 2213 B 160 KiB check_for_negative_density() 5 5 308 B 160 KiB MultiFab::max() 6 6 50 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 28 B 20 KiB MLPoisson::Fsmooth() 60 60 3167 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 263 B 9648 B MLCellLinOp::applyBC() 3892 3892 208 B 9344 B amrex::Copy() 56 56 6008 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 39 B 2208 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 324 324 109 B 1472 B MLCellLinOp::defineBC() 36 36 336 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1435 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 106 KiB 8192 KiB VisMF::Write(FabArray) 120 120 238 KiB 3584 KiB VisMF::Read() 24 24 181 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3167 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 263 B 9648 B MLCellLinOp::applyBC() 1946 1946 207 B 9328 B amrex::Copy() 56 56 6008 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2693 B 3072 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B StateData::FillBoundary(geom) 960 960 39 B 2208 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 513 B 1296 B amrex::average_down 37 37 227 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 22 B 400 B FabArray::norminf() 398 398 16 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2422 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-23-gd92232a0c17b) finalized