Initializing CUDA... CUDA initialized with 1 device. AMReX (23.06-25-g239d4d899334) initialized Starting run at 08:13:15 UTC on 2023-06-21. Successfully read inputs file ... Castro git describe: 23.06-36-g90afb40ca AMReX git describe: 23.06-25-g239d4d899 Microphysics git describe: 23.06-29-gde505749 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058338837 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.033104982 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04465089 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048328334 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.067287384 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.069528071 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.05845904 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057252384 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.057946686 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.076201922 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.076692738 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057395159 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059726314 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057875711 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033053696 seconds Ending run at 08:13:16 UTC on 2023-06-21. Run time = 0.90407357 Run time without initialization = 0.765079283 Average number of zones advanced per microsecond: 3.426 Average number of zones advanced per microsecond per rank: 3.426 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9041 ... 0.9041 ... 0.9041 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2638 0.2638 0.2638 29.18% VisMF::Write(FabArray) 11 0.2327 0.2327 0.2327 25.74% MLCellLinOp::applyBC() 4433 0.07496 0.07496 0.07496 8.29% MLPoisson::Fsmooth() 3280 0.03143 0.03143 0.03143 3.48% FillBoundary_nowait() 4023 0.03035 0.03035 0.03035 3.36% StateData::FillBoundary(geom) 328 0.02409 0.02409 0.02409 2.66% amrex::Dot() 1114 0.01986 0.01986 0.01986 2.20% Castro::normalize_species() 62 0.01848 0.01848 0.01848 2.04% Castro::computeTemp() 63 0.01501 0.01501 0.01501 1.66% amrex::Copy() 1029 0.01459 0.01459 0.01459 1.61% FabArray::norminf() 743 0.01386 0.01386 0.01386 1.53% FabArray::ParallelCopy_nowait() 861 0.01305 0.01305 0.01305 1.44% FabArray::setVal() 1144 0.01279 0.01279 0.01279 1.41% Castro::enforce_min_density() 62 0.01049 0.01049 0.01049 1.16% StateDataPhysBCFunct::() 41 0.01008 0.01008 0.01008 1.11% MLPoisson::Fapply() 1142 0.01007 0.01007 0.01007 1.11% MLCellLinOp::defineAuxData() 11 0.009488 0.009488 0.009488 1.05% FabArray::Saxpy() 813 0.007939 0.007939 0.007939 0.88% FabArray::Xpay() 821 0.007881 0.007881 0.007881 0.87% MLMG::addInterpCorrection() 410 0.006598 0.006598 0.006598 0.73% amrex::average_down 410 0.005857 0.005857 0.005857 0.65% Gravity::fill_multipole_BCs() 11 0.005676 0.005676 0.005676 0.63% Castro::reset_internal_energy(MultiFab) 63 0.004634 0.004634 0.004634 0.51% Castro::estTimeStep() 21 0.004614 0.004614 0.004614 0.51% FabArray::LinComb() 557 0.0044 0.0044 0.0044 0.49% amrex::Add() 164 0.004328 0.004328 0.004328 0.48% Amr::checkPoint() 3 0.004032 0.004032 0.004032 0.45% BndryData::define() 11 0.003483 0.003483 0.003483 0.39% Castro::construct_new_gravity_source() 10 0.002697 0.002697 0.002697 0.30% Amr::writePlotFile() 2 0.002376 0.002376 0.002376 0.26% Castro::enforce_speed_limit() 62 0.002323 0.002323 0.002323 0.26% Castro::construct_old_gravity_source() 10 0.002144 0.002144 0.002144 0.24% MLCGSolver::bicgstab 82 0.002131 0.002131 0.002131 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001647 0.001647 0.001647 0.18% Castro::reset_internal_energy(Fab) 504 0.001531 0.001531 0.001531 0.17% Gravity::actual_solve_with_mlmg() 11 0.001478 0.001478 0.001478 0.16% check_for_negative_density() 10 0.001438 0.001438 0.001438 0.16% MLCellLinOp::setLevelBC() 11 0.001349 0.001349 0.001349 0.15% FabArray::mult() 43 0.001302 0.001302 0.001302 0.14% FabArray::setDomainBndry() 41 0.001263 0.001263 0.001263 0.14% MultiFab::contains_nan() 20 0.001205 0.001205 0.001205 0.13% Castro::initData() 1 0.001147 0.001147 0.001147 0.13% MLCellLinOp::smooth() 1640 0.001135 0.001135 0.001135 0.13% MLCellLinOp::prepareForSolve() 11 0.001097 0.001097 0.001097 0.12% MLCellLinOp::compGrad() 11 0.0008987 0.0008987 0.0008987 0.10% MLMG::prepareForSolve() 11 0.0008675 0.0008675 0.0008675 0.10% FabArray::FillBoundary() 4023 0.0007939 0.0007939 0.0007939 0.09% FabArrayBase::getCPC() 1323 0.0007572 0.0007572 0.0007572 0.08% FabArrayBase::CPC::define() 454 0.0006876 0.0006876 0.0006876 0.08% Gravity::get_new_grav_vector() 11 0.0006332 0.0006332 0.0006332 0.07% FabArrayBase::getFB() 4023 0.0005867 0.0005867 0.0005867 0.06% Gravity::get_old_grav_vector() 10 0.0005478 0.0005478 0.0005478 0.06% Amr::InitAmr() 1 0.0004623 0.0004623 0.0004623 0.05% MLCellLinOp::apply() 1142 0.0004533 0.0004533 0.0004533 0.05% Amr::coarseTimeStep() 10 0.0003781 0.0003781 0.0003781 0.04% MLLinOp::defineGrids() 11 0.0003496 0.0003496 0.0003496 0.04% MLMG::mgVcycle() 82 0.0003417 0.0003417 0.0003417 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003186 0.0003186 0.0003186 0.04% main() 1 0.0002916 0.0002916 0.0002916 0.03% MultiFab::max() 11 0.0002544 0.0002544 0.0002544 0.03% FabArray::ParallelCopy() 861 0.0002504 0.0002504 0.0002504 0.03% FillPatchIterator::Initialize 41 0.0002401 0.0002401 0.0002401 0.03% MLCellLinOp::defineBC() 11 0.0002128 0.0002128 0.0002128 0.02% MLCellLinOp::correctionResidual() 492 0.0002044 0.0002044 0.0002044 0.02% Castro::subcycle_advance_ctu() 10 0.0001893 0.0001893 0.0001893 0.02% Castro::create_source_corrector() 10 0.0001678 0.0001678 0.0001678 0.02% Amr::timeStep() 10 0.0001559 0.0001559 0.0001559 0.02% StateData::checkPoint() 12 0.0001415 0.0001415 0.0001415 0.02% Gravity::update_max_rhs() 11 0.0001282 0.0001282 0.0001282 0.01% MLMG:computeResOfCorrection() 410 0.0001262 0.0001262 0.0001262 0.01% Gravity::solve_for_phi() 10 0.0001219 0.0001219 0.0001219 0.01% Castro::advance() 10 9.394e-05 9.394e-05 9.394e-05 0.01% Castro::do_old_sources() 10 9.165e-05 9.165e-05 9.165e-05 0.01% MLMG::mgVcycle_down::0 82 8.785e-05 8.785e-05 8.785e-05 0.01% Castro::Castro() 1 8.753e-05 8.753e-05 8.753e-05 0.01% FabArrayBase::FB::FB() 56 8.731e-05 8.731e-05 8.731e-05 0.01% MLMG::actualBottomSolve() 82 8.504e-05 8.504e-05 8.504e-05 0.01% MLMG::mgVcycle_down::1 82 8.356e-05 8.356e-05 8.356e-05 0.01% Castro::expand_state() 10 8.028e-05 8.028e-05 8.028e-05 0.01% MLMG::mgVcycle_down::2 82 8.018e-05 8.018e-05 8.018e-05 0.01% AmrLevel::checkPoint() 3 7.922e-05 7.922e-05 7.922e-05 0.01% MLMG::mgVcycle_down::3 82 7.704e-05 7.704e-05 7.704e-05 0.01% Castro::clean_state() 62 7.644e-05 7.644e-05 7.644e-05 0.01% MLMG::mgVcycle_down::4 82 7.637e-05 7.637e-05 7.637e-05 0.01% MLMG::solve() 11 7.496e-05 7.496e-05 7.496e-05 0.01% Castro::do_new_sources() 10 7.388e-05 7.388e-05 7.388e-05 0.01% Castro::initialize_advance() 10 7.385e-05 7.385e-05 7.385e-05 0.01% Castro::finalize_advance() 10 6.821e-05 6.821e-05 6.821e-05 0.01% MLMG::mgVcycle_up::4 82 6.445e-05 6.445e-05 6.445e-05 0.01% Castro::initialize_do_advance() 10 6.024e-05 6.024e-05 6.024e-05 0.01% MLMG::mgVcycle_up::0 82 5.668e-05 5.668e-05 5.668e-05 0.01% MLMG::mgVcycle_up::1 82 5.503e-05 5.503e-05 5.503e-05 0.01% MLMG::oneIter() 82 5.487e-05 5.487e-05 5.487e-05 0.01% MLMG::mgVcycle_up::2 82 5.273e-05 5.273e-05 5.273e-05 0.01% MLMG::mgVcycle_up::3 82 5.224e-05 5.224e-05 5.224e-05 0.01% MLCellLinOp::solutionResidual() 93 4.752e-05 4.752e-05 4.752e-05 0.01% StateData::define() 4 4.315e-05 4.315e-05 4.315e-05 0.00% Castro::do_advance_ctu() 10 4.25e-05 4.25e-05 4.25e-05 0.00% Castro::finalize_do_advance() 10 4.194e-05 4.194e-05 4.194e-05 0.00% Castro::swap_state_time_levels() 10 4.027e-05 4.027e-05 4.027e-05 0.00% MLMG::ResNormInf() 93 3.766e-05 3.766e-05 3.766e-05 0.00% Castro::post_timestep() 10 3.751e-05 3.751e-05 3.751e-05 0.00% Castro::enforce_consistent_e() 1 3.305e-05 3.305e-05 3.305e-05 0.00% MLMG::computeResidual() 82 3.231e-05 3.231e-05 3.231e-05 0.00% MLMG::mgVcycle_bottom 82 3.23e-05 3.23e-05 3.23e-05 0.00% FillPatchSingleLevel 41 3.088e-05 3.088e-05 3.088e-05 0.00% MLPoisson::define() 11 2.827e-05 2.827e-05 2.827e-05 0.00% Castro::construct_new_gravity() 10 2.704e-05 2.704e-05 2.704e-05 0.00% Castro::initMFs() 1 2.646e-05 2.646e-05 2.646e-05 0.00% Amr::writeSmallPlotFile() 1 2.519e-05 2.519e-05 2.519e-05 0.00% Amr::defBaseLevel() 1 2.464e-05 2.464e-05 2.464e-05 0.00% makeSFC 55 2.448e-05 2.448e-05 2.448e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.316e-05 2.316e-05 2.316e-05 0.00% Castro::construct_old_gravity() 10 2.144e-05 2.144e-05 2.144e-05 0.00% Amr::FinalizeInit() 1 2.132e-05 2.132e-05 2.132e-05 0.00% Castro::buildMetrics() 1 1.836e-05 1.836e-05 1.836e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.829e-05 1.829e-05 1.829e-05 0.00% Castro::construct_old_source() 50 1.772e-05 1.772e-05 1.772e-05 0.00% Castro::construct_new_source() 50 1.742e-05 1.742e-05 1.742e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.686e-05 1.686e-05 1.686e-05 0.00% MLLinOp::define() 11 1.678e-05 1.678e-05 1.678e-05 0.00% DistributionMapping::Distribute() 56 1.625e-05 1.625e-05 1.625e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.455e-05 1.455e-05 1.455e-05 0.00% Castro::check_for_nan() 20 1.204e-05 1.204e-05 1.204e-05 0.00% Castro::apply_source_to_state() 20 9.693e-06 9.693e-06 9.693e-06 0.00% MLPoisson::prepareForSolve() 11 8.659e-06 8.659e-06 8.659e-06 0.00% Amr::initSubcycle() 1 8.585e-06 8.585e-06 8.585e-06 0.00% Gravity::swapTimeLevels() 10 8.427e-06 8.427e-06 8.427e-06 0.00% Gravity::actual_multilevel_solve() 1 8.35e-06 8.35e-06 8.35e-06 0.00% MLMG::computeMLResidual() 11 7.83e-06 7.83e-06 7.83e-06 0.00% Castro::computeNewDt() 9 6.324e-06 6.324e-06 6.324e-06 0.00% MLMG::getGradSolution() 11 6.106e-06 6.106e-06 6.106e-06 0.00% AmrLevel::checkPointPost() 3 5.495e-06 5.495e-06 5.495e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.003e-06 5.003e-06 5.003e-06 0.00% Amr::InitializeInit() 1 4.834e-06 4.834e-06 4.834e-06 0.00% Castro::post_init() 1 4.583e-06 4.583e-06 4.583e-06 0.00% Castro::retry_advance_ctu() 10 4.297e-06 4.297e-06 4.297e-06 0.00% Gravity::set_mass_offset() 11 3.862e-06 3.862e-06 3.862e-06 0.00% MLMG::MLRhsNormInf() 11 3.786e-06 3.786e-06 3.786e-06 0.00% MLMG::MLResNormInf() 11 3.725e-06 3.725e-06 3.725e-06 0.00% Castro::computeInitialDt() 2 2.99e-06 2.99e-06 2.99e-06 0.00% Castro::FluxRegCrseInit 10 2.78e-06 2.78e-06 2.78e-06 0.00% Amr::init() 1 2.494e-06 2.494e-06 2.494e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.965e-06 1.965e-06 1.965e-06 0.00% AmrLevel::checkPointPre() 3 1.839e-06 1.839e-06 1.839e-06 0.00% Castro::FluxRegFineAdd() 10 1.753e-06 1.753e-06 1.753e-06 0.00% Amr::initialInit() 1 1.202e-06 1.202e-06 1.202e-06 0.00% Castro::post_regrid() 1 9.36e-07 9.36e-07 9.36e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9041 0.9041 0.9041 100.00% Amr::coarseTimeStep() 10 0.7318 0.7318 0.7318 80.94% Amr::timeStep() 10 0.6136 0.6136 0.6136 67.87% Castro::advance() 10 0.6039 0.6039 0.6039 66.79% Castro::subcycle_advance_ctu() 10 0.5913 0.5913 0.5913 65.40% Castro::do_advance_ctu() 10 0.5911 0.5911 0.5911 65.38% Gravity::solve_phi_with_mlmg() 11 0.2754 0.2754 0.2754 30.46% Castro::construct_ctu_hydro_source() 10 0.2733 0.2733 0.2733 30.23% Gravity::actual_solve_with_mlmg() 11 0.2692 0.2692 0.2692 29.78% Castro::construct_new_gravity() 10 0.2508 0.2508 0.2508 27.74% MLMG::solve() 11 0.2491 0.2491 0.2491 27.55% Gravity::solve_for_phi() 10 0.2359 0.2359 0.2359 26.09% MLMG::oneIter() 82 0.2349 0.2349 0.2349 25.98% VisMF::Write(FabArray) 11 0.2327 0.2327 0.2327 25.74% MLMG::mgVcycle() 82 0.2312 0.2312 0.2312 25.58% Amr::checkPoint() 3 0.1736 0.1736 0.1736 19.20% AmrLevel::checkPoint() 3 0.1696 0.1696 0.1696 18.76% StateData::checkPoint() 12 0.1695 0.1695 0.1695 18.75% Amr::init() 1 0.1384 0.1384 0.1384 15.31% MLCellLinOp::smooth() 1640 0.114 0.114 0.114 12.61% MLCellLinOp::applyBC() 4433 0.1068 0.1068 0.1068 11.81% MLMG::mgVcycle_bottom 82 0.07119 0.07119 0.07119 7.87% MLMG::actualBottomSolve() 82 0.07116 0.07116 0.07116 7.87% MLCGSolver::bicgstab 82 0.07049 0.07049 0.07049 7.80% Amr::writePlotFile() 2 0.06628 0.06628 0.06628 7.33% Castro::clean_state() 62 0.05191 0.05191 0.05191 5.74% Amr::initialInit() 1 0.04682 0.04682 0.04682 5.18% Amr::FinalizeInit() 1 0.04292 0.04292 0.04292 4.75% Castro::post_init() 1 0.04168 0.04168 0.04168 4.61% Gravity::multilevel_solve_for_new_phi() 1 0.0399 0.0399 0.0399 4.41% Gravity::actual_multilevel_solve() 1 0.03988 0.03988 0.03988 4.41% FillPatchIterator::Initialize 41 0.03976 0.03976 0.03976 4.40% FillPatchSingleLevel 41 0.03826 0.03826 0.03826 4.23% MLCellLinOp::apply() 1142 0.03543 0.03543 0.03543 3.92% StateDataPhysBCFunct::() 41 0.03417 0.03417 0.03417 3.78% MLMG::mgVcycle_down::0 82 0.03306 0.03306 0.03306 3.66% FabArray::FillBoundary() 4023 0.03181 0.03181 0.03181 3.52% MLPoisson::Fsmooth() 3280 0.03143 0.03143 0.03143 3.48% FillBoundary_nowait() 4023 0.03102 0.03102 0.03102 3.43% MLMG::mgVcycle_up::0 82 0.02513 0.02513 0.02513 2.78% StateData::FillBoundary(geom) 328 0.02409 0.02409 0.02409 2.66% MLCellLinOp::correctionResidual() 492 0.02158 0.02158 0.02158 2.39% Castro::computeTemp() 63 0.02118 0.02118 0.02118 2.34% Castro::initialize_do_advance() 10 0.0201 0.0201 0.0201 2.22% amrex::Dot() 1114 0.01986 0.01986 0.01986 2.20% MLMG:computeResOfCorrection() 410 0.01903 0.01903 0.01903 2.11% Castro::normalize_species() 62 0.01848 0.01848 0.01848 2.04% Castro::do_old_sources() 10 0.01719 0.01719 0.01719 1.90% Gravity::get_new_grav_vector() 11 0.01634 0.01634 0.01634 1.81% MLPoisson::define() 11 0.01592 0.01592 0.01592 1.76% MLMG::mgVcycle_down::1 82 0.0154 0.0154 0.0154 1.70% amrex::Copy() 1029 0.01459 0.01459 0.01459 1.61% Castro::construct_old_gravity() 10 0.01452 0.01452 0.01452 1.61% Gravity::get_old_grav_vector() 10 0.0145 0.0145 0.0145 1.60% MLMG::mgVcycle_down::2 82 0.01431 0.01431 0.01431 1.58% FabArray::ParallelCopy() 861 0.01411 0.01411 0.01411 1.56% MLMG::mgVcycle_down::3 82 0.01397 0.01397 0.01397 1.55% FabArray::norminf() 743 0.01386 0.01386 0.01386 1.53% FabArray::ParallelCopy_nowait() 861 0.01386 0.01386 0.01386 1.53% MLMG::mgVcycle_down::4 82 0.01381 0.01381 0.01381 1.53% FabArray::setVal() 1144 0.01279 0.01279 0.01279 1.41% Castro::do_new_sources() 10 0.01239 0.01239 0.01239 1.37% MLCGSolver::ParallelAllReduce 1514 0.01194 0.01194 0.01194 1.32% Castro::initialize_advance() 10 0.01189 0.01189 0.01189 1.31% MLMG::addInterpCorrection() 410 0.0116 0.0116 0.0116 1.28% MLMG::mgVcycle_up::4 82 0.0112 0.0112 0.0112 1.24% MLMG::mgVcycle_up::1 82 0.0112 0.0112 0.0112 1.24% MLMG::mgVcycle_up::2 82 0.01091 0.01091 0.01091 1.21% amrex::average_down 410 0.0109 0.0109 0.0109 1.21% MLCellLinOp::defineAuxData() 11 0.01083 0.01083 0.01083 1.20% MLMG::mgVcycle_up::3 82 0.01072 0.01072 0.01072 1.19% Castro::expand_state() 10 0.01052 0.01052 0.01052 1.16% Castro::enforce_min_density() 62 0.01049 0.01049 0.01049 1.16% MLPoisson::Fapply() 1142 0.01007 0.01007 0.01007 1.11% Castro::post_timestep() 10 0.009547 0.009547 0.009547 1.06% FabArray::Saxpy() 813 0.007939 0.007939 0.007939 0.88% FabArray::Xpay() 821 0.007881 0.007881 0.007881 0.87% MLCellLinOp::solutionResidual() 93 0.007235 0.007235 0.007235 0.80% MLMG::computeResidual() 82 0.006275 0.006275 0.006275 0.69% Castro::reset_internal_energy(MultiFab) 63 0.006166 0.006166 0.006166 0.68% Gravity::fill_multipole_BCs() 11 0.005925 0.005925 0.005925 0.66% MLCellLinOp::defineBC() 11 0.004641 0.004641 0.004641 0.51% Castro::estTimeStep() 21 0.004614 0.004614 0.004614 0.51% MLMG::prepareForSolve() 11 0.00454 0.00454 0.00454 0.50% BndryData::define() 11 0.004428 0.004428 0.004428 0.49% FabArray::LinComb() 557 0.0044 0.0044 0.0044 0.49% amrex::Add() 164 0.004328 0.004328 0.004328 0.48% Amr::InitializeInit() 1 0.003905 0.003905 0.003905 0.43% Amr::defBaseLevel() 1 0.0039 0.0039 0.0039 0.43% Castro::initData() 1 0.003364 0.003364 0.003364 0.37% Castro::construct_new_source() 50 0.002714 0.002714 0.002714 0.30% Castro::construct_new_gravity_source() 10 0.002697 0.002697 0.002697 0.30% Castro::enforce_speed_limit() 62 0.002323 0.002323 0.002323 0.26% Castro::computeNewDt() 9 0.002222 0.002222 0.002222 0.25% Castro::construct_old_source() 50 0.002161 0.002161 0.002161 0.24% Castro::construct_old_gravity_source() 10 0.002144 0.002144 0.002144 0.24% MLMG::ResNormInf() 93 0.002077 0.002077 0.002077 0.23% Castro::finalize_do_advance() 10 0.001886 0.001886 0.001886 0.21% Castro::apply_source_to_state() 20 0.00181 0.00181 0.00181 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001647 0.001647 0.001647 0.18% Castro::reset_internal_energy(Fab) 504 0.001531 0.001531 0.001531 0.17% FabArrayBase::getCPC() 1323 0.001445 0.001445 0.001445 0.16% check_for_negative_density() 10 0.001438 0.001438 0.001438 0.16% MLMG::getGradSolution() 11 0.001368 0.001368 0.001368 0.15% MLCellLinOp::compGrad() 11 0.001362 0.001362 0.001362 0.15% MLCellLinOp::setLevelBC() 11 0.001349 0.001349 0.001349 0.15% FabArray::mult() 43 0.001302 0.001302 0.001302 0.14% FabArray::setDomainBndry() 41 0.001263 0.001263 0.001263 0.14% Castro::check_for_nan() 20 0.001217 0.001217 0.001217 0.13% MultiFab::contains_nan() 20 0.001205 0.001205 0.001205 0.13% MLPoisson::prepareForSolve() 11 0.001106 0.001106 0.001106 0.12% MLCellLinOp::prepareForSolve() 11 0.001097 0.001097 0.001097 0.12% Castro::post_regrid() 1 0.001064 0.001064 0.001064 0.12% MLMG::computeMLResidual() 11 0.001001 0.001001 0.001001 0.11% Gravity::update_max_rhs() 11 0.0008187 0.0008187 0.0008187 0.09% FabArrayBase::CPC::define() 454 0.0006876 0.0006876 0.0006876 0.08% FabArrayBase::getFB() 4023 0.000674 0.000674 0.000674 0.07% Castro::finalize_advance() 10 0.0005907 0.0005907 0.0005907 0.07% Castro::computeInitialDt() 2 0.0005567 0.0005567 0.0005567 0.06% Amr::InitAmr() 1 0.0004708 0.0004708 0.0004708 0.05% Castro::Castro() 1 0.0004453 0.0004453 0.0004453 0.05% Gravity::swapTimeLevels() 10 0.0004378 0.0004378 0.0004378 0.05% MLLinOp::define() 11 0.000422 0.000422 0.000422 0.05% MLLinOp::defineGrids() 11 0.0004052 0.0004052 0.0004052 0.04% MLMG::MLResNormInf() 11 0.0002791 0.0002791 0.0002791 0.03% MultiFab::max() 11 0.0002544 0.0002544 0.0002544 0.03% MLMG::MLRhsNormInf() 11 0.0002149 0.0002149 0.0002149 0.02% Castro::buildMetrics() 1 0.0001708 0.0001708 0.0001708 0.02% Castro::create_source_corrector() 10 0.0001678 0.0001678 0.0001678 0.02% FabArrayBase::FB::FB() 56 8.731e-05 8.731e-05 8.731e-05 0.01% AmrLevel::AmrLevel(dm) 1 6.001e-05 6.001e-05 6.001e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.366e-05 5.366e-05 5.366e-05 0.01% StateData::define() 4 4.315e-05 4.315e-05 4.315e-05 0.00% Castro::swap_state_time_levels() 10 4.027e-05 4.027e-05 4.027e-05 0.00% makeSFC 55 3.911e-05 3.911e-05 3.911e-05 0.00% Castro::enforce_consistent_e() 1 3.305e-05 3.305e-05 3.305e-05 0.00% Castro::initMFs() 1 2.646e-05 2.646e-05 2.646e-05 0.00% Amr::writeSmallPlotFile() 1 2.519e-05 2.519e-05 2.519e-05 0.00% DistributionMapping::Distribute() 56 1.625e-05 1.625e-05 1.625e-05 0.00% Amr::initSubcycle() 1 8.585e-06 8.585e-06 8.585e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 6.619e-06 6.619e-06 6.619e-06 0.00% AmrLevel::checkPointPost() 3 5.495e-06 5.495e-06 5.495e-06 0.00% Castro::retry_advance_ctu() 10 4.297e-06 4.297e-06 4.297e-06 0.00% Gravity::set_mass_offset() 11 3.862e-06 3.862e-06 3.862e-06 0.00% Castro::FluxRegCrseInit 10 2.78e-06 2.78e-06 2.78e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.965e-06 1.965e-06 1.965e-06 0.00% AmrLevel::checkPointPre() 3 1.839e-06 1.839e-06 1.839e-06 0.00% Castro::FluxRegFineAdd() 10 1.753e-06 1.753e-06 1.753e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 87 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 135 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 935 KiB 39 MiB Castro::initialize_do_advance() 80 80 25 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2060 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7608 KiB 14 MiB MLMG::prepareForSolve() 660 660 3390 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 187 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 164 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7530 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 10 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1723 B 2048 KiB Gravity::solve_for_phi() 80 80 533 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 90 KiB 2048 KiB BndryData::define() 1056 1056 309 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 196 KiB 671 KiB Castro::estTimeStep() 21 21 2487 B 480 KiB VisMF::Write(FabArray) 656 656 3259 B 320 KiB Castro::normalize_species() 62 62 6654 B 320 KiB amrex::average_down 1067 1067 1220 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1098 B 257 KiB amrex::Dot() 1360 1360 3233 B 160 KiB FabArray::norminf() 907 907 2258 B 160 KiB check_for_negative_density() 10 10 251 B 160 KiB Castro::initData() 1 1 26 B 160 KiB MultiFab::max() 11 11 43 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 26 B 20 KiB MLPoisson::Fsmooth() 132 132 3315 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 293 B 9648 B MLCellLinOp::applyBC() 8866 8866 215 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3861 B 6144 B StateData::FillBoundary(geom) 1992 1992 35 B 2592 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 113 B 1472 B MLCellLinOp::defineBC() 66 66 348 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 574 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 48 KiB 8192 KiB VisMF::Write(FabArray) 744 744 592 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3315 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 293 B 9648 B MLCellLinOp::applyBC() 4433 4433 214 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3861 B 6144 B Gravity::get_new_grav_vector() 3 3 2913 B 3072 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B StateData::FillBoundary(geom) 1992 1992 35 B 2592 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 282 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 23 B 400 B FabArray::norminf() 907 907 16 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2424 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.06-25-g239d4d899334) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.06-25-g239d4d899334) initialized Starting run at 08:13:17 UTC on 2023-06-21. Successfully read inputs file ... Castro git describe: 23.06-36-g90afb40ca AMReX git describe: 23.06-25-g239d4d899 Microphysics git describe: 23.06-29-gde505749 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.484522011 Restart time = 0.045773677 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048657134 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.064091984 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.075125769 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.073527267 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.053325418 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032712521 seconds Ending run at 08:13:17 UTC on 2023-06-21. Run time = 0.394217017 Run time without initialization = 0.347881992 Average number of zones advanced per microsecond: 3.768 Average number of zones advanced per microsecond per rank: 3.768 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3942 ... 0.3942 ... 0.3942 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1352 0.1352 0.1352 34.31% VisMF::Read() 3 0.03955 0.03955 0.03955 10.03% MLCellLinOp::applyBC() 1946 0.03233 0.03233 0.03233 8.20% VisMF::Write(FabArray) 1 0.0311 0.0311 0.0311 7.89% MLPoisson::Fsmooth() 1440 0.01354 0.01354 0.01354 3.43% FillBoundary_nowait() 1766 0.0126 0.0126 0.0126 3.20% StateData::FillBoundary(geom) 160 0.01198 0.01198 0.01198 3.04% amrex::Dot() 484 0.008476 0.008476 0.008476 2.15% Castro::normalize_species() 30 0.007849 0.007849 0.007849 1.99% amrex::Copy() 463 0.006821 0.006821 0.006821 1.73% Castro::computeTemp() 30 0.006446 0.006446 0.006446 1.63% FabArray::setVal() 537 0.006069 0.006069 0.006069 1.54% FabArray::norminf() 326 0.005972 0.005972 0.005972 1.51% FabArray::ParallelCopy_nowait() 380 0.005952 0.005952 0.005952 1.51% Castro::enforce_min_density() 30 0.005337 0.005337 0.005337 1.35% MLCellLinOp::defineAuxData() 6 0.005138 0.005138 0.005138 1.30% StateDataPhysBCFunct::() 20 0.004779 0.004779 0.004779 1.21% MLPoisson::Fapply() 500 0.0043 0.0043 0.0043 1.09% FabArray::Saxpy() 355 0.003524 0.003524 0.003524 0.89% FabArray::Xpay() 361 0.003397 0.003397 0.003397 0.86% Castro::estTimeStep() 10 0.003396 0.003396 0.003396 0.86% Gravity::fill_multipole_BCs() 6 0.003363 0.003363 0.003363 0.85% MLMG::addInterpCorrection() 180 0.002853 0.002853 0.002853 0.72% amrex::average_down 180 0.002526 0.002526 0.002526 0.64% Amr::restart() 1 0.002507 0.002507 0.002507 0.64% Castro::reset_internal_energy(MultiFab) 30 0.002133 0.002133 0.002133 0.54% BndryData::define() 6 0.001909 0.001909 0.001909 0.48% FabArray::LinComb() 242 0.001845 0.001845 0.001845 0.47% amrex::Add() 72 0.001832 0.001832 0.001832 0.46% Castro::construct_new_gravity_source() 5 0.001566 0.001566 0.001566 0.40% Amr::writePlotFile() 1 0.001454 0.001454 0.001454 0.37% Castro::construct_old_gravity_source() 5 0.001444 0.001444 0.001444 0.37% Castro::subcycle_advance_ctu() 5 0.001289 0.001289 0.001289 0.33% Castro::enforce_speed_limit() 30 0.001098 0.001098 0.001098 0.28% MLCGSolver::bicgstab 36 0.0009544 0.0009544 0.0009544 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000885 0.000885 0.000885 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007744 0.0007744 0.0007744 0.20% MLCellLinOp::setLevelBC() 6 0.0007289 0.0007289 0.0007289 0.18% check_for_negative_density() 5 0.0007105 0.0007105 0.0007105 0.18% Castro::reset_internal_energy(Fab) 240 0.0006989 0.0006989 0.0006989 0.18% FabArray::mult() 22 0.0006367 0.0006367 0.0006367 0.16% FabArray::setDomainBndry() 20 0.0006363 0.0006363 0.0006363 0.16% MLCellLinOp::prepareForSolve() 6 0.0005898 0.0005898 0.0005898 0.15% MultiFab::contains_nan() 10 0.0005858 0.0005858 0.0005858 0.15% MLCellLinOp::compGrad() 6 0.0004848 0.0004848 0.0004848 0.12% MLMG::prepareForSolve() 6 0.0004839 0.0004839 0.0004839 0.12% MLCellLinOp::smooth() 720 0.0004625 0.0004625 0.0004625 0.12% FabArrayBase::CPC::define() 244 0.000442 0.000442 0.000442 0.11% Amr::InitAmr() 1 0.0003821 0.0003821 0.0003821 0.10% FabArrayBase::getCPC() 632 0.0003479 0.0003479 0.0003479 0.09% FabArray::FillBoundary() 1766 0.0003449 0.0003449 0.0003449 0.09% Gravity::get_old_grav_vector() 5 0.0003085 0.0003085 0.0003085 0.08% Gravity::get_new_grav_vector() 5 0.0002738 0.0002738 0.0002738 0.07% main() 1 0.0002729 0.0002729 0.0002729 0.07% FabArrayBase::getFB() 1766 0.0002524 0.0002524 0.0002524 0.06% Amr::coarseTimeStep() 5 0.0002051 0.0002051 0.0002051 0.05% MLCellLinOp::apply() 500 0.0001911 0.0001911 0.0001911 0.05% MLLinOp::defineGrids() 6 0.0001751 0.0001751 0.0001751 0.04% MLMG::mgVcycle() 36 0.0001468 0.0001468 0.0001468 0.04% Castro::construct_new_source() 25 0.0001331 0.0001331 0.0001331 0.03% MultiFab::max() 6 0.0001325 0.0001325 0.0001325 0.03% MLCGSolver::ParallelAllReduce 659 0.0001224 0.0001224 0.0001224 0.03% FillPatchIterator::Initialize 20 0.0001138 0.0001138 0.0001138 0.03% MLCellLinOp::defineBC() 6 0.0001093 0.0001093 0.0001093 0.03% FabArray::ParallelCopy() 380 0.0001092 0.0001092 0.0001092 0.03% Amr::timeStep() 5 9.281e-05 9.281e-05 9.281e-05 0.02% AmrLevel::restart() 1 9.141e-05 9.141e-05 9.141e-05 0.02% MLCellLinOp::correctionResidual() 216 8.669e-05 8.669e-05 8.669e-05 0.02% Castro::create_source_corrector() 5 8.206e-05 8.206e-05 8.206e-05 0.02% StateData::restartDoit() 4 6.973e-05 6.973e-05 6.973e-05 0.02% Castro::finalize_do_advance() 5 6.341e-05 6.341e-05 6.341e-05 0.02% Gravity::update_max_rhs() 6 6.051e-05 6.051e-05 6.051e-05 0.02% FabArrayBase::FB::FB() 26 5.816e-05 5.816e-05 5.816e-05 0.01% Castro::do_new_sources() 5 5.731e-05 5.731e-05 5.731e-05 0.01% Castro::advance() 5 5.222e-05 5.222e-05 5.222e-05 0.01% Gravity::solve_for_phi() 5 5.197e-05 5.197e-05 5.197e-05 0.01% MLMG:computeResOfCorrection() 180 5.039e-05 5.039e-05 5.039e-05 0.01% Castro::initialize_do_advance() 5 4.905e-05 4.905e-05 4.905e-05 0.01% Castro::do_old_sources() 5 4.627e-05 4.627e-05 4.627e-05 0.01% Castro::expand_state() 5 4.062e-05 4.062e-05 4.062e-05 0.01% MLMG::mgVcycle_down::0 36 3.97e-05 3.97e-05 3.97e-05 0.01% MLMG::actualBottomSolve() 36 3.889e-05 3.889e-05 3.889e-05 0.01% Castro::finalize_advance() 5 3.797e-05 3.797e-05 3.797e-05 0.01% MLMG::mgVcycle_down::1 36 3.689e-05 3.689e-05 3.689e-05 0.01% Castro::initMFs() 1 3.687e-05 3.687e-05 3.687e-05 0.01% MLMG::mgVcycle_down::4 36 3.482e-05 3.482e-05 3.482e-05 0.01% MLMG::mgVcycle_down::2 36 3.478e-05 3.478e-05 3.478e-05 0.01% MLMG::solve() 6 3.444e-05 3.444e-05 3.444e-05 0.01% Castro::initialize_advance() 5 3.408e-05 3.408e-05 3.408e-05 0.01% MLMG::mgVcycle_down::3 36 3.383e-05 3.383e-05 3.383e-05 0.01% Castro::clean_state() 30 3.289e-05 3.289e-05 3.289e-05 0.01% Castro::post_timestep() 5 3.175e-05 3.175e-05 3.175e-05 0.01% MLMG::mgVcycle_up::4 36 2.917e-05 2.917e-05 2.917e-05 0.01% Amr::writeSmallPlotFile() 1 2.735e-05 2.735e-05 2.735e-05 0.01% Castro::buildMetrics() 1 2.718e-05 2.718e-05 2.718e-05 0.01% MLMG::mgVcycle_up::0 36 2.459e-05 2.459e-05 2.459e-05 0.01% Castro::swap_state_time_levels() 5 2.401e-05 2.401e-05 2.401e-05 0.01% MLMG::oneIter() 36 2.305e-05 2.305e-05 2.305e-05 0.01% MLMG::mgVcycle_up::3 36 2.259e-05 2.259e-05 2.259e-05 0.01% MLMG::mgVcycle_up::2 36 2.25e-05 2.25e-05 2.25e-05 0.01% MLMG::mgVcycle_up::1 36 2.21e-05 2.21e-05 2.21e-05 0.01% MLCellLinOp::solutionResidual() 42 2.123e-05 2.123e-05 2.123e-05 0.01% MLPoisson::define() 6 2.067e-05 2.067e-05 2.067e-05 0.01% Castro::do_advance_ctu() 5 2.063e-05 2.063e-05 2.063e-05 0.01% makeSFC 30 2.052e-05 2.052e-05 2.052e-05 0.01% Castro::post_restart() 1 2.034e-05 2.034e-05 2.034e-05 0.01% MLMG::ResNormInf() 42 1.869e-05 1.869e-05 1.869e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.594e-05 1.594e-05 1.594e-05 0.00% MLMG::computeResidual() 36 1.377e-05 1.377e-05 1.377e-05 0.00% Castro::construct_new_gravity() 5 1.37e-05 1.37e-05 1.37e-05 0.00% MLMG::mgVcycle_bottom 36 1.338e-05 1.338e-05 1.338e-05 0.00% FillPatchSingleLevel 20 1.317e-05 1.317e-05 1.317e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.18e-05 1.18e-05 1.18e-05 0.00% MLLinOp::define() 6 1.128e-05 1.128e-05 1.128e-05 0.00% Castro::construct_old_source() 25 9.05e-06 9.05e-06 9.05e-06 0.00% DistributionMapping::Distribute() 31 8.457e-06 8.457e-06 8.457e-06 0.00% Amr::initSubcycle() 1 8.351e-06 8.351e-06 8.351e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.684e-06 7.684e-06 7.684e-06 0.00% Gravity::actual_multilevel_solve() 1 7.577e-06 7.577e-06 7.577e-06 0.00% Castro::check_for_nan() 10 5.696e-06 5.696e-06 5.696e-06 0.00% Castro::apply_source_to_state() 10 5.195e-06 5.195e-06 5.195e-06 0.00% Castro::construct_old_gravity() 5 5.099e-06 5.099e-06 5.099e-06 0.00% MLPoisson::prepareForSolve() 6 4.737e-06 4.737e-06 4.737e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.41e-06 4.41e-06 4.41e-06 0.00% Gravity::swapTimeLevels() 5 4.18e-06 4.18e-06 4.18e-06 0.00% MLMG::computeMLResidual() 6 3.792e-06 3.792e-06 3.792e-06 0.00% MLMG::getGradSolution() 6 3.065e-06 3.065e-06 3.065e-06 0.00% Castro::computeNewDt() 5 3.007e-06 3.007e-06 3.007e-06 0.00% Castro::retry_advance_ctu() 5 2.704e-06 2.704e-06 2.704e-06 0.00% MLMG::MLResNormInf() 6 2.226e-06 2.226e-06 2.226e-06 0.00% MLMG::MLRhsNormInf() 6 2.067e-06 2.067e-06 2.067e-06 0.00% Gravity::set_mass_offset() 6 2.007e-06 2.007e-06 2.007e-06 0.00% Castro::FluxRegCrseInit 5 1.804e-06 1.804e-06 1.804e-06 0.00% Castro::FluxRegFineAdd() 5 1.454e-06 1.454e-06 1.454e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.154e-06 1.154e-06 1.154e-06 0.00% Amr::init() 1 1.027e-06 1.027e-06 1.027e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3942 0.3942 0.3942 100.00% Amr::coarseTimeStep() 5 0.3149 0.3149 0.3149 79.88% Amr::timeStep() 5 0.3123 0.3123 0.3123 79.20% Castro::advance() 5 0.3081 0.3081 0.3081 78.15% Castro::subcycle_advance_ctu() 5 0.3018 0.3018 0.3018 76.55% Castro::do_advance_ctu() 5 0.3005 0.3005 0.3005 76.22% Castro::construct_ctu_hydro_source() 5 0.1399 0.1399 0.1399 35.50% Castro::construct_new_gravity() 5 0.1265 0.1265 0.1265 32.09% Gravity::solve_phi_with_mlmg() 6 0.1217 0.1217 0.1217 30.87% Gravity::solve_for_phi() 5 0.1187 0.1187 0.1187 30.11% Gravity::actual_solve_with_mlmg() 6 0.1181 0.1181 0.1181 29.96% MLMG::solve() 6 0.1072 0.1072 0.1072 27.18% MLMG::oneIter() 36 0.1004 0.1004 0.1004 25.48% MLMG::mgVcycle() 36 0.09888 0.09888 0.09888 25.08% MLCellLinOp::smooth() 720 0.04862 0.04862 0.04862 12.33% Amr::init() 1 0.04582 0.04582 0.04582 11.62% Amr::restart() 1 0.04581 0.04581 0.04581 11.62% MLCellLinOp::applyBC() 1946 0.04558 0.04558 0.04558 11.56% AmrLevel::restart() 1 0.03977 0.03977 0.03977 10.09% StateData::restartDoit() 4 0.03967 0.03967 0.03967 10.06% VisMF::Read() 3 0.03955 0.03955 0.03955 10.03% Amr::writePlotFile() 1 0.0328 0.0328 0.0328 8.32% VisMF::Write(FabArray) 1 0.0311 0.0311 0.0311 7.89% MLMG::mgVcycle_bottom 36 0.03042 0.03042 0.03042 7.72% MLMG::actualBottomSolve() 36 0.0304 0.0304 0.0304 7.71% MLCGSolver::bicgstab 36 0.03011 0.03011 0.03011 7.64% Castro::clean_state() 30 0.02359 0.02359 0.02359 5.98% FillPatchIterator::Initialize 20 0.01957 0.01957 0.01957 4.96% FillPatchSingleLevel 20 0.01882 0.01882 0.01882 4.77% StateDataPhysBCFunct::() 20 0.01676 0.01676 0.01676 4.25% MLCellLinOp::apply() 500 0.0152 0.0152 0.0152 3.86% MLMG::mgVcycle_down::0 36 0.01408 0.01408 0.01408 3.57% MLPoisson::Fsmooth() 1440 0.01354 0.01354 0.01354 3.43% FabArray::FillBoundary() 1766 0.01326 0.01326 0.01326 3.36% FillBoundary_nowait() 1766 0.01291 0.01291 0.01291 3.28% StateData::FillBoundary(geom) 160 0.01198 0.01198 0.01198 3.04% MLMG::mgVcycle_up::0 36 0.01061 0.01061 0.01061 2.69% Castro::initialize_do_advance() 5 0.009722 0.009722 0.009722 2.47% Castro::do_old_sources() 5 0.009617 0.009617 0.009617 2.44% Castro::computeTemp() 30 0.009278 0.009278 0.009278 2.35% MLCellLinOp::correctionResidual() 216 0.009208 0.009208 0.009208 2.34% MLPoisson::define() 6 0.008684 0.008684 0.008684 2.20% amrex::Dot() 484 0.008476 0.008476 0.008476 2.15% MLMG:computeResOfCorrection() 180 0.008101 0.008101 0.008101 2.05% Castro::normalize_species() 30 0.007849 0.007849 0.007849 1.99% Gravity::get_new_grav_vector() 5 0.007697 0.007697 0.007697 1.95% Castro::construct_old_gravity() 5 0.007388 0.007388 0.007388 1.87% Gravity::get_old_grav_vector() 5 0.007383 0.007383 0.007383 1.87% amrex::Copy() 463 0.006821 0.006821 0.006821 1.73% MLMG::mgVcycle_down::1 36 0.006611 0.006611 0.006611 1.68% FabArray::ParallelCopy() 380 0.006452 0.006452 0.006452 1.64% FabArray::ParallelCopy_nowait() 380 0.006343 0.006343 0.006343 1.61% MLMG::mgVcycle_down::2 36 0.006144 0.006144 0.006144 1.56% FabArray::setVal() 537 0.006069 0.006069 0.006069 1.54% MLMG::mgVcycle_down::3 36 0.006019 0.006019 0.006019 1.53% FabArray::norminf() 326 0.005972 0.005972 0.005972 1.51% Castro::initialize_advance() 5 0.005939 0.005939 0.005939 1.51% MLMG::mgVcycle_down::4 36 0.005921 0.005921 0.005921 1.50% MLCellLinOp::defineAuxData() 6 0.005875 0.005875 0.005875 1.49% Castro::do_new_sources() 5 0.005851 0.005851 0.005851 1.48% Castro::enforce_min_density() 30 0.005337 0.005337 0.005337 1.35% MLCGSolver::ParallelAllReduce 659 0.005122 0.005122 0.005122 1.30% Castro::expand_state() 5 0.005057 0.005057 0.005057 1.28% MLMG::addInterpCorrection() 180 0.005043 0.005043 0.005043 1.28% MLMG::mgVcycle_up::4 36 0.004821 0.004821 0.004821 1.22% MLMG::mgVcycle_up::1 36 0.004814 0.004814 0.004814 1.22% amrex::average_down 180 0.004742 0.004742 0.004742 1.20% MLMG::mgVcycle_up::2 36 0.004698 0.004698 0.004698 1.19% MLMG::mgVcycle_up::3 36 0.004604 0.004604 0.004604 1.17% MLPoisson::Fapply() 500 0.0043 0.0043 0.0043 1.09% Castro::post_timestep() 5 0.004053 0.004053 0.004053 1.03% FabArray::Saxpy() 355 0.003524 0.003524 0.003524 0.89% Gravity::fill_multipole_BCs() 6 0.003507 0.003507 0.003507 0.89% FabArray::Xpay() 361 0.003397 0.003397 0.003397 0.86% Castro::estTimeStep() 10 0.003396 0.003396 0.003396 0.86% Castro::post_restart() 1 0.00335 0.00335 0.00335 0.85% Gravity::multilevel_solve_for_new_phi() 1 0.003243 0.003243 0.003243 0.82% Gravity::actual_multilevel_solve() 1 0.003227 0.003227 0.003227 0.82% MLCellLinOp::solutionResidual() 42 0.003164 0.003164 0.003164 0.80% Castro::reset_internal_energy(MultiFab) 30 0.002832 0.002832 0.002832 0.72% MLMG::computeResidual() 36 0.00262 0.00262 0.00262 0.66% MLCellLinOp::defineBC() 6 0.002565 0.002565 0.002565 0.65% MLMG::prepareForSolve() 6 0.002472 0.002472 0.002472 0.63% Castro::computeNewDt() 5 0.002469 0.002469 0.002469 0.63% BndryData::define() 6 0.002456 0.002456 0.002456 0.62% FabArray::LinComb() 242 0.001845 0.001845 0.001845 0.47% amrex::Add() 72 0.001832 0.001832 0.001832 0.46% Castro::construct_new_source() 25 0.001699 0.001699 0.001699 0.43% Castro::construct_new_gravity_source() 5 0.001566 0.001566 0.001566 0.40% Castro::construct_old_source() 25 0.001453 0.001453 0.001453 0.37% Castro::construct_old_gravity_source() 5 0.001444 0.001444 0.001444 0.37% Castro::enforce_speed_limit() 30 0.001098 0.001098 0.001098 0.28% Castro::finalize_do_advance() 5 0.0009939 0.0009939 0.0009939 0.25% MLMG::ResNormInf() 42 0.0009245 0.0009245 0.0009245 0.23% Castro::apply_source_to_state() 10 0.0009213 0.0009213 0.0009213 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000885 0.000885 0.000885 0.22% FabArrayBase::getCPC() 632 0.0007899 0.0007899 0.0007899 0.20% MLMG::getGradSolution() 6 0.0007455 0.0007455 0.0007455 0.19% MLCellLinOp::compGrad() 6 0.0007424 0.0007424 0.0007424 0.19% MLCellLinOp::setLevelBC() 6 0.0007289 0.0007289 0.0007289 0.18% check_for_negative_density() 5 0.0007105 0.0007105 0.0007105 0.18% Castro::reset_internal_energy(Fab) 240 0.0006989 0.0006989 0.0006989 0.18% FabArray::mult() 22 0.0006367 0.0006367 0.0006367 0.16% FabArray::setDomainBndry() 20 0.0006363 0.0006363 0.0006363 0.16% MLPoisson::prepareForSolve() 6 0.0005945 0.0005945 0.0005945 0.15% Castro::check_for_nan() 10 0.0005915 0.0005915 0.0005915 0.15% MLCellLinOp::prepareForSolve() 6 0.0005898 0.0005898 0.0005898 0.15% MultiFab::contains_nan() 10 0.0005858 0.0005858 0.0005858 0.15% MLMG::computeMLResidual() 6 0.0005619 0.0005619 0.0005619 0.14% FabArrayBase::CPC::define() 244 0.000442 0.000442 0.000442 0.11% Gravity::update_max_rhs() 6 0.0004348 0.0004348 0.0004348 0.11% Amr::InitAmr() 1 0.0003904 0.0003904 0.0003904 0.10% Castro::finalize_advance() 5 0.0003131 0.0003131 0.0003131 0.08% FabArrayBase::getFB() 1766 0.0003105 0.0003105 0.0003105 0.08% MLLinOp::define() 6 0.0002228 0.0002228 0.0002228 0.06% Gravity::swapTimeLevels() 5 0.0002225 0.0002225 0.0002225 0.06% MLLinOp::defineGrids() 6 0.0002115 0.0002115 0.0002115 0.05% MLMG::MLResNormInf() 6 0.0001504 0.0001504 0.0001504 0.04% Castro::buildMetrics() 1 0.0001482 0.0001482 0.0001482 0.04% MultiFab::max() 6 0.0001325 0.0001325 0.0001325 0.03% MLMG::MLRhsNormInf() 6 0.0001146 0.0001146 0.0001146 0.03% Castro::create_source_corrector() 5 8.206e-05 8.206e-05 8.206e-05 0.02% FabArrayBase::FB::FB() 26 5.816e-05 5.816e-05 5.816e-05 0.01% Castro::initMFs() 1 3.687e-05 3.687e-05 3.687e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.526e-05 3.526e-05 3.526e-05 0.01% makeSFC 30 2.757e-05 2.757e-05 2.757e-05 0.01% Amr::writeSmallPlotFile() 1 2.735e-05 2.735e-05 2.735e-05 0.01% Castro::swap_state_time_levels() 5 2.401e-05 2.401e-05 2.401e-05 0.01% DistributionMapping::Distribute() 31 8.457e-06 8.457e-06 8.457e-06 0.00% Amr::initSubcycle() 1 8.351e-06 8.351e-06 8.351e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.816e-06 5.816e-06 5.816e-06 0.00% Castro::retry_advance_ctu() 5 2.704e-06 2.704e-06 2.704e-06 0.00% Gravity::set_mass_offset() 6 2.007e-06 2.007e-06 2.007e-06 0.00% Castro::FluxRegCrseInit 5 1.804e-06 1.804e-06 1.804e-06 0.00% Castro::FluxRegFineAdd() 5 1.454e-06 1.454e-06 1.454e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.154e-06 1.154e-06 1.154e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 205 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 157 MiB 692 MiB Castro::initMFs() 48 48 60 MiB 68 MiB Castro::swap_state_time_levels() 32 32 49 MiB 55 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB FillPatchIterator::Initialize 160 160 1051 KiB 39 MiB Castro::initialize_do_advance() 40 40 29 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2326 KiB 28 MiB Castro::initialize_advance() 40 40 18 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6746 KiB 14 MiB MLMG::prepareForSolve() 361 361 3338 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 194 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 199 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6742 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 14 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2124 B 2048 KiB Gravity::solve_for_phi() 40 40 615 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 16 KiB 2048 KiB BndryData::define() 576 576 306 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 197 KiB 671 KiB Castro::estTimeStep() 10 10 4090 B 480 KiB VisMF::Write(FabArray) 112 112 1230 B 320 KiB Castro::normalize_species() 30 30 6476 B 320 KiB amrex::average_down 469 469 1193 B 257 KiB MLMG::addInterpCorrection() 468 468 1099 B 257 KiB amrex::Dot() 592 592 3157 B 160 KiB FabArray::norminf() 398 398 2230 B 160 KiB check_for_negative_density() 5 5 286 B 160 KiB MultiFab::max() 6 6 51 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3245 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 277 B 9648 B MLCellLinOp::applyBC() 3892 3892 213 B 9344 B amrex::Copy() 56 56 5972 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 44 B 2880 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 324 324 111 B 1472 B MLCellLinOp::defineBC() 36 36 345 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1384 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 111 KiB 8192 KiB VisMF::Write(FabArray) 120 120 242 KiB 3584 KiB VisMF::Read() 24 24 195 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3245 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 277 B 9648 B MLCellLinOp::applyBC() 1946 1946 211 B 9328 B amrex::Copy() 56 56 5972 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2668 B 3072 B StateData::FillBoundary(geom) 960 960 45 B 2880 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 518 B 1296 B amrex::average_down 37 37 225 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 398 398 16 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2424 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.06-25-g239d4d899334) finalized