Initializing AMReX (24.02-6-g4fc7ef352fe1)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.02-6-g4fc7ef352fe1) initialized Starting run at 09:39:16 UTC on 2024-02-08. Successfully read inputs file ... Castro git describe: 24.02-3-g09b5fe593 AMReX git describe: 24.02-6-g4fc7ef352 Microphysics git describe: 24.02-7-g0ad950aa reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.046228924 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025572132 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.068182055 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.076061787 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.081626453 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.068133036 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.05257525 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.046004913 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.073939382 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.075719845 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05691241 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057221469 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.07817325 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.044785913 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025562113 seconds Ending run at 09:39:17 UTC on 2024-02-08. Run time = 0.933160772 Run time without initialization = 0.805612138 Average number of zones advanced per microsecond: 3.254 Average number of zones advanced per microsecond per rank: 3.254 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9332 ... 0.9332 ... 0.9332 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.3021 0.3021 0.3021 32.38% VisMF::Write(FabArray) 11 0.1796 0.1796 0.1796 19.25% MLCellLinOp::applyBC() 4351 0.0816 0.0816 0.0816 8.74% MLPoisson::Fsmooth() 3280 0.03378 0.03378 0.03378 3.62% FillBoundary_nowait() 3941 0.03247 0.03247 0.03247 3.48% StateData::FillBoundary(geom) 328 0.0269 0.0269 0.0269 2.88% Castro::normalize_species() 62 0.02274 0.02274 0.02274 2.44% amrex::Dot() 1114 0.02166 0.02166 0.02166 2.32% FabArray::norminf() 1061 0.0203 0.0203 0.0203 2.18% Castro::computeTemp() 63 0.01632 0.01632 0.01632 1.75% FabArray::ParallelCopy_nowait() 861 0.01379 0.01379 0.01379 1.48% FabArray::setVal() 1062 0.01353 0.01353 0.01353 1.45% FabArray::Saxpy() 1370 0.0134 0.0134 0.0134 1.44% Castro::enforce_min_density() 62 0.01293 0.01293 0.01293 1.39% StateDataPhysBCFunct::() 41 0.01128 0.01128 0.01128 1.21% amrex::Copy() 472 0.01088 0.01088 0.01088 1.17% MLCellLinOp::defineAuxData() 11 0.01048 0.01048 0.01048 1.12% MLPoisson::Fapply() 1060 0.01045 0.01045 0.01045 1.12% Gravity::fill_multipole_BCs() 11 0.0092 0.0092 0.0092 0.99% FabArray::Xpay() 739 0.00799 0.00799 0.00799 0.86% MLMG::addInterpCorrection() 410 0.007104 0.007104 0.007104 0.76% Castro::estTimeStep() 21 0.006531 0.006531 0.006531 0.70% amrex::average_down 410 0.006229 0.006229 0.006229 0.67% Amr::checkPoint() 3 0.005928 0.005928 0.005928 0.64% Castro::reset_internal_energy(MultiFab) 63 0.004926 0.004926 0.004926 0.53% BndryData::define() 11 0.004057 0.004057 0.004057 0.43% amrex::Add() 82 0.003676 0.003676 0.003676 0.39% Castro::construct_new_gravity_source() 10 0.003471 0.003471 0.003471 0.37% Castro::construct_old_gravity_source() 10 0.002882 0.002882 0.002882 0.31% Castro::enforce_speed_limit() 62 0.002283 0.002283 0.002283 0.24% check_for_negative_density() 10 0.00219 0.00219 0.00219 0.23% Amr::writePlotFile() 2 0.002093 0.002093 0.002093 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001797 0.001797 0.001797 0.19% Castro::reset_internal_energy(Fab) 504 0.00174 0.00174 0.00174 0.19% MLCellLinOp::setLevelBC() 11 0.001623 0.001623 0.001623 0.17% MLCGSolver::bicgstab 82 0.001615 0.001615 0.001615 0.17% Gravity::actual_solve_with_mlmg() 11 0.001582 0.001582 0.001582 0.17% Castro::initData() 1 0.001541 0.001541 0.001541 0.17% FabArray::mult() 43 0.001389 0.001389 0.001389 0.15% FabArray::setDomainBndry() 41 0.001386 0.001386 0.001386 0.15% MLCellLinOp::prepareForSolve() 11 0.001355 0.001355 0.001355 0.15% MultiFab::contains_nan() 20 0.001312 0.001312 0.001312 0.14% MLCellLinOp::compGrad() 11 0.001109 0.001109 0.001109 0.12% MLCellLinOp::smooth() 1640 0.001075 0.001075 0.001075 0.12% MLMG::prepareForSolve() 11 0.001 0.001 0.001 0.11% FabArray::FillBoundary() 3941 0.0008032 0.0008032 0.0008032 0.09% FabArrayBase::getCPC() 1323 0.0008013 0.0008013 0.0008013 0.09% FabArrayBase::CPC::define() 454 0.0007197 0.0007197 0.0007197 0.08% Gravity::get_new_grav_vector() 11 0.0006197 0.0006197 0.0006197 0.07% FabArrayBase::getFB() 3941 0.0006192 0.0006192 0.0006192 0.07% Amr::InitAmr() 1 0.0005664 0.0005664 0.0005664 0.06% Gravity::get_old_grav_vector() 10 0.0004843 0.0004843 0.0004843 0.05% Amr::coarseTimeStep() 10 0.000421 0.000421 0.000421 0.05% MLCellLinOp::apply() 1060 0.0004196 0.0004196 0.0004196 0.04% AmrLevel::FillPatch() 41 0.0004065 0.0004065 0.0004065 0.04% MLCGSolver::ParallelAllReduce 1832 0.0003367 0.0003367 0.0003367 0.04% MultiFab::max() 11 0.0003294 0.0003294 0.0003294 0.04% MLCellLinOp::defineBC() 11 0.0002903 0.0002903 0.0002903 0.03% main() 1 0.00029 0.00029 0.00029 0.03% FabArray::ParallelCopy() 861 0.0002335 0.0002335 0.0002335 0.03% FillPatchIterator::Initialize 41 0.0002171 0.0002171 0.0002171 0.02% Castro::subcycle_advance_ctu() 10 0.0002031 0.0002031 0.0002031 0.02% MLMG::mgVcycle() 82 0.0001986 0.0001986 0.0001986 0.02% Castro::create_source_corrector() 10 0.0001763 0.0001763 0.0001763 0.02% MLCellLinOp::correctionResidual() 410 0.0001706 0.0001706 0.0001706 0.02% MLLinOp::defineGrids() 11 0.0001676 0.0001676 0.0001676 0.02% Amr::timeStep() 10 0.0001601 0.0001601 0.0001601 0.02% Gravity::update_max_rhs() 11 0.0001352 0.0001352 0.0001352 0.01% StateData::checkPoint() 12 0.0001351 0.0001351 0.0001351 0.01% Gravity::solve_for_phi() 10 0.0001213 0.0001213 0.0001213 0.01% MLMG:computeResOfCorrection() 410 0.000116 0.000116 0.000116 0.01% FabArrayBase::FB::FB() 56 9.858e-05 9.858e-05 9.858e-05 0.01% Castro::construct_new_source() 50 9.735e-05 9.735e-05 9.735e-05 0.01% Castro::advance() 10 9.491e-05 9.491e-05 9.491e-05 0.01% Castro::Castro() 1 8.926e-05 8.926e-05 8.926e-05 0.01% MLMG::actualBottomSolve() 82 8.891e-05 8.891e-05 8.891e-05 0.01% MLMG::mgVcycle_down::0 82 8.683e-05 8.683e-05 8.683e-05 0.01% MLMG::mgVcycle_down::1 82 8.105e-05 8.105e-05 8.105e-05 0.01% Castro::initialize_advance() 10 7.935e-05 7.935e-05 7.935e-05 0.01% MLMG::mgVcycle_down::2 82 7.776e-05 7.776e-05 7.776e-05 0.01% MLMG::mgVcycle_down::4 82 7.709e-05 7.709e-05 7.709e-05 0.01% AmrLevel::checkPoint() 3 7.654e-05 7.654e-05 7.654e-05 0.01% MLMG::solve() 11 7.647e-05 7.647e-05 7.647e-05 0.01% Castro::clean_state() 62 7.482e-05 7.482e-05 7.482e-05 0.01% MLMG::mgVcycle_down::3 82 7.143e-05 7.143e-05 7.143e-05 0.01% Castro::finalize_advance() 10 6.869e-05 6.869e-05 6.869e-05 0.01% Castro::post_timestep() 10 6.587e-05 6.587e-05 6.587e-05 0.01% Castro::enforce_consistent_e() 1 6.406e-05 6.406e-05 6.406e-05 0.01% Castro::initialize_do_advance() 10 6.212e-05 6.212e-05 6.212e-05 0.01% MLMG::mgVcycle_up::4 82 5.567e-05 5.567e-05 5.567e-05 0.01% MLMG::oneIter() 82 5.425e-05 5.425e-05 5.425e-05 0.01% Castro::do_advance_ctu() 10 5.085e-05 5.085e-05 5.085e-05 0.01% FillPatchIterator::FillFromLevel0() 41 5.054e-05 5.054e-05 5.054e-05 0.01% Castro::do_new_sources() 10 5.033e-05 5.033e-05 5.033e-05 0.01% MLCellLinOp::solutionResidual() 93 5.025e-05 5.025e-05 5.025e-05 0.01% MLMG::mgVcycle_up::0 82 4.752e-05 4.752e-05 4.752e-05 0.01% MLMG::mgVcycle_up::3 82 4.621e-05 4.621e-05 4.621e-05 0.00% MLMG::mgVcycle_up::1 82 4.536e-05 4.536e-05 4.536e-05 0.00% MLMG::mgVcycle_up::2 82 4.397e-05 4.397e-05 4.397e-05 0.00% StateData::define() 4 4.335e-05 4.335e-05 4.335e-05 0.00% Castro::finalize_do_advance() 10 4.245e-05 4.245e-05 4.245e-05 0.00% MLMG::ResNormInf() 93 3.695e-05 3.695e-05 3.695e-05 0.00% Castro::swap_state_time_levels() 10 3.529e-05 3.529e-05 3.529e-05 0.00% MLMG::mgVcycle_bottom 82 3.443e-05 3.443e-05 3.443e-05 0.00% MLMG::computeResidual() 82 3.394e-05 3.394e-05 3.394e-05 0.00% FillPatchSingleLevel 41 3.214e-05 3.214e-05 3.214e-05 0.00% Amr::writeSmallPlotFile() 1 3.161e-05 3.161e-05 3.161e-05 0.00% Amr::defBaseLevel() 1 3.156e-05 3.156e-05 3.156e-05 0.00% Castro::construct_new_gravity() 10 3.135e-05 3.135e-05 3.135e-05 0.00% Castro::initMFs() 1 3.126e-05 3.126e-05 3.126e-05 0.00% makeSFC 55 2.553e-05 2.553e-05 2.553e-05 0.00% Castro::buildMetrics() 1 2.396e-05 2.396e-05 2.396e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.303e-05 2.303e-05 2.303e-05 0.00% Castro::do_old_sources() 10 2.115e-05 2.115e-05 2.115e-05 0.00% MLPoisson::define() 11 2.104e-05 2.104e-05 2.104e-05 0.00% Amr::FinalizeInit() 1 1.993e-05 1.993e-05 1.993e-05 0.00% DistributionMapping::Distribute() 56 1.857e-05 1.857e-05 1.857e-05 0.00% Castro::construct_old_source() 50 1.832e-05 1.832e-05 1.832e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.709e-05 1.709e-05 1.709e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.555e-05 1.555e-05 1.555e-05 0.00% MLPoisson::prepareForSolve() 11 1.498e-05 1.498e-05 1.498e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.398e-05 1.398e-05 1.398e-05 0.00% MLLinOp::define() 11 1.247e-05 1.247e-05 1.247e-05 0.00% Castro::check_for_nan() 20 1.184e-05 1.184e-05 1.184e-05 0.00% Castro::apply_source_to_state() 20 1.161e-05 1.161e-05 1.161e-05 0.00% Castro::construct_old_gravity() 10 1.147e-05 1.147e-05 1.147e-05 0.00% Castro::post_init() 1 1.032e-05 1.032e-05 1.032e-05 0.00% Amr::initSubcycle() 1 9.46e-06 9.46e-06 9.46e-06 0.00% MLMG::computeMLResidual() 11 9.259e-06 9.259e-06 9.259e-06 0.00% Gravity::swapTimeLevels() 10 8.661e-06 8.661e-06 8.661e-06 0.00% Gravity::actual_multilevel_solve() 1 7.867e-06 7.867e-06 7.867e-06 0.00% Castro::computeNewDt() 9 6.743e-06 6.743e-06 6.743e-06 0.00% MLMG::getGradSolution() 11 5.998e-06 5.998e-06 5.998e-06 0.00% Castro::expand_state() 10 5.862e-06 5.862e-06 5.862e-06 0.00% Amr::InitializeInit() 1 4.972e-06 4.972e-06 4.972e-06 0.00% Castro::retry_advance_ctu() 10 4.861e-06 4.861e-06 4.861e-06 0.00% Gravity::set_mass_offset() 11 4.599e-06 4.599e-06 4.599e-06 0.00% AmrLevel::checkPointPost() 3 4.549e-06 4.549e-06 4.549e-06 0.00% MLMG::MLRhsNormInf() 11 4.223e-06 4.223e-06 4.223e-06 0.00% MLMG::MLResNormInf() 11 3.829e-06 3.829e-06 3.829e-06 0.00% Castro::FluxRegCrseInit 10 2.68e-06 2.68e-06 2.68e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.624e-06 2.624e-06 2.624e-06 0.00% Castro::computeInitialDt() 2 2.563e-06 2.563e-06 2.563e-06 0.00% AmrLevel::checkPointPre() 3 2.475e-06 2.475e-06 2.475e-06 0.00% Amr::init() 1 2.473e-06 2.473e-06 2.473e-06 0.00% Castro::FluxRegFineAdd() 10 2.272e-06 2.272e-06 2.272e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.044e-06 2.044e-06 2.044e-06 0.00% Castro::post_regrid() 1 1.236e-06 1.236e-06 1.236e-06 0.00% Amr::initialInit() 1 1.003e-06 1.003e-06 1.003e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9332 0.9332 0.9332 100.00% Amr::coarseTimeStep() 10 0.7798 0.7798 0.7798 83.56% Amr::timeStep() 10 0.6849 0.6849 0.6849 73.40% Castro::advance() 10 0.6734 0.6734 0.6734 72.16% Castro::subcycle_advance_ctu() 10 0.659 0.659 0.659 70.62% Castro::do_advance_ctu() 10 0.6588 0.6588 0.6588 70.60% Castro::construct_ctu_hydro_source() 10 0.3141 0.3141 0.3141 33.66% Gravity::solve_phi_with_mlmg() 11 0.2991 0.2991 0.2991 32.05% Gravity::actual_solve_with_mlmg() 11 0.2894 0.2894 0.2894 31.01% Castro::construct_new_gravity() 10 0.2698 0.2698 0.2698 28.92% MLMG::solve() 11 0.267 0.267 0.267 28.61% Gravity::solve_for_phi() 10 0.2536 0.2536 0.2536 27.18% MLMG::oneIter() 82 0.2513 0.2513 0.2513 26.93% MLMG::mgVcycle() 82 0.2476 0.2476 0.2476 26.53% VisMF::Write(FabArray) 11 0.1796 0.1796 0.1796 19.25% Amr::checkPoint() 3 0.1371 0.1371 0.1371 14.69% AmrLevel::checkPoint() 3 0.1312 0.1312 0.1312 14.06% StateData::checkPoint() 12 0.1311 0.1311 0.1311 14.05% Amr::init() 1 0.1269 0.1269 0.1269 13.59% MLCellLinOp::smooth() 1640 0.1239 0.1239 0.1239 13.28% MLCellLinOp::applyBC() 4351 0.1156 0.1156 0.1156 12.39% MLMG::mgVcycle_bottom 82 0.07392 0.07392 0.07392 7.92% MLMG::actualBottomSolve() 82 0.07388 0.07388 0.07388 7.92% MLCGSolver::bicgstab 82 0.07306 0.07306 0.07306 7.83% Castro::clean_state() 62 0.06004 0.06004 0.06004 6.43% Amr::initialInit() 1 0.05494 0.05494 0.05494 5.89% Amr::writePlotFile() 2 0.05125 0.05125 0.05125 5.49% Amr::FinalizeInit() 1 0.04983 0.04983 0.04983 5.34% Castro::post_init() 1 0.04826 0.04826 0.04826 5.17% AmrLevel::FillPatch() 41 0.04822 0.04822 0.04822 5.17% Gravity::multilevel_solve_for_new_phi() 1 0.04588 0.04588 0.04588 4.92% Gravity::actual_multilevel_solve() 1 0.04586 0.04586 0.04586 4.91% FillPatchIterator::Initialize 41 0.04394 0.04394 0.04394 4.71% FillPatchIterator::FillFromLevel0() 41 0.04234 0.04234 0.04234 4.54% FillPatchSingleLevel 41 0.04229 0.04229 0.04229 4.53% StateDataPhysBCFunct::() 41 0.03818 0.03818 0.03818 4.09% MLCellLinOp::apply() 1060 0.03693 0.03693 0.03693 3.96% MLMG::mgVcycle_down::0 82 0.03573 0.03573 0.03573 3.83% FabArray::FillBoundary() 3941 0.034 0.034 0.034 3.64% MLPoisson::Fsmooth() 3280 0.03378 0.03378 0.03378 3.62% FillBoundary_nowait() 3941 0.03319 0.03319 0.03319 3.56% MLMG::mgVcycle_up::0 82 0.027 0.027 0.027 2.89% StateData::FillBoundary(geom) 328 0.0269 0.0269 0.0269 2.88% Castro::computeTemp() 63 0.02298 0.02298 0.02298 2.46% Castro::normalize_species() 62 0.02274 0.02274 0.02274 2.44% amrex::Dot() 1114 0.02166 0.02166 0.02166 2.32% Castro::initialize_do_advance() 10 0.0213 0.0213 0.0213 2.28% MLMG:computeResOfCorrection() 410 0.0209 0.0209 0.0209 2.24% MLCellLinOp::correctionResidual() 410 0.02079 0.02079 0.02079 2.23% FabArray::norminf() 1061 0.0203 0.0203 0.0203 2.18% Castro::do_old_sources() 10 0.01911 0.01911 0.01911 2.05% Gravity::get_new_grav_vector() 11 0.01813 0.01813 0.01813 1.94% MLPoisson::define() 11 0.01757 0.01757 0.01757 1.88% MLMG::mgVcycle_down::1 82 0.01687 0.01687 0.01687 1.81% MLMG::mgVcycle_down::2 82 0.01563 0.01563 0.01563 1.67% Castro::construct_old_gravity() 10 0.01545 0.01545 0.01545 1.66% Gravity::get_old_grav_vector() 10 0.01544 0.01544 0.01544 1.65% Castro::do_new_sources() 10 0.01534 0.01534 0.01534 1.64% MLMG::mgVcycle_down::3 82 0.01523 0.01523 0.01523 1.63% MLMG::mgVcycle_down::4 82 0.01512 0.01512 0.01512 1.62% FabArray::ParallelCopy() 861 0.01485 0.01485 0.01485 1.59% FabArray::ParallelCopy_nowait() 861 0.01462 0.01462 0.01462 1.57% Castro::initialize_advance() 10 0.01361 0.01361 0.01361 1.46% FabArray::setVal() 1062 0.01353 0.01353 0.01353 1.45% FabArray::Saxpy() 1370 0.0134 0.0134 0.0134 1.44% MLCGSolver::ParallelAllReduce 1832 0.01302 0.01302 0.01302 1.40% Castro::enforce_min_density() 62 0.01293 0.01293 0.01293 1.39% MLMG::addInterpCorrection() 410 0.01249 0.01249 0.01249 1.34% MLMG::mgVcycle_up::1 82 0.01221 0.01221 0.01221 1.31% MLMG::mgVcycle_up::4 82 0.01207 0.01207 0.01207 1.29% Castro::expand_state() 10 0.01196 0.01196 0.01196 1.28% MLCellLinOp::defineAuxData() 11 0.01194 0.01194 0.01194 1.28% MLMG::mgVcycle_up::2 82 0.01192 0.01192 0.01192 1.28% MLMG::mgVcycle_up::3 82 0.01165 0.01165 0.01165 1.25% amrex::average_down 410 0.01162 0.01162 0.01162 1.25% Castro::post_timestep() 10 0.0114 0.0114 0.0114 1.22% amrex::Copy() 472 0.01088 0.01088 0.01088 1.17% MLPoisson::Fapply() 1060 0.01045 0.01045 0.01045 1.12% Gravity::fill_multipole_BCs() 11 0.009437 0.009437 0.009437 1.01% FabArray::Xpay() 739 0.00799 0.00799 0.00799 0.86% MLCellLinOp::solutionResidual() 93 0.007815 0.007815 0.007815 0.84% Castro::reset_internal_energy(MultiFab) 63 0.006665 0.006665 0.006665 0.71% Castro::estTimeStep() 21 0.006531 0.006531 0.006531 0.70% MLMG::computeResidual() 82 0.006479 0.006479 0.006479 0.69% MLCellLinOp::defineBC() 11 0.00538 0.00538 0.00538 0.58% MLMG::prepareForSolve() 11 0.00529 0.00529 0.00529 0.57% Amr::InitializeInit() 1 0.005116 0.005116 0.005116 0.55% Amr::defBaseLevel() 1 0.005111 0.005111 0.005111 0.55% BndryData::define() 11 0.005089 0.005089 0.005089 0.55% Castro::initData() 1 0.004439 0.004439 0.004439 0.48% amrex::Add() 82 0.003676 0.003676 0.003676 0.39% Castro::construct_new_source() 50 0.003568 0.003568 0.003568 0.38% Castro::construct_new_gravity_source() 10 0.003471 0.003471 0.003471 0.37% Castro::construct_old_source() 50 0.0029 0.0029 0.0029 0.31% Castro::construct_old_gravity_source() 10 0.002882 0.002882 0.002882 0.31% Castro::computeNewDt() 9 0.002827 0.002827 0.002827 0.30% Castro::finalize_do_advance() 10 0.002701 0.002701 0.002701 0.29% Castro::enforce_speed_limit() 62 0.002283 0.002283 0.002283 0.24% MLMG::ResNormInf() 93 0.002253 0.002253 0.002253 0.24% check_for_negative_density() 10 0.00219 0.00219 0.00219 0.23% Castro::apply_source_to_state() 20 0.00188 0.00188 0.00188 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001797 0.001797 0.001797 0.19% Castro::reset_internal_energy(Fab) 504 0.00174 0.00174 0.00174 0.19% MLCellLinOp::setLevelBC() 11 0.001623 0.001623 0.001623 0.17% MLMG::getGradSolution() 11 0.001615 0.001615 0.001615 0.17% MLCellLinOp::compGrad() 11 0.001609 0.001609 0.001609 0.17% FabArrayBase::getCPC() 1323 0.001521 0.001521 0.001521 0.16% FabArray::mult() 43 0.001389 0.001389 0.001389 0.15% FabArray::setDomainBndry() 41 0.001386 0.001386 0.001386 0.15% MLMG::computeMLResidual() 11 0.001379 0.001379 0.001379 0.15% MLPoisson::prepareForSolve() 11 0.00137 0.00137 0.00137 0.15% MLCellLinOp::prepareForSolve() 11 0.001355 0.001355 0.001355 0.15% Castro::check_for_nan() 20 0.001324 0.001324 0.001324 0.14% MultiFab::contains_nan() 20 0.001312 0.001312 0.001312 0.14% Castro::post_regrid() 1 0.001265 0.001265 0.001265 0.14% Castro::computeInitialDt() 2 0.001055 0.001055 0.001055 0.11% Gravity::update_max_rhs() 11 0.001006 0.001006 0.001006 0.11% FabArrayBase::CPC::define() 454 0.0007197 0.0007197 0.0007197 0.08% FabArrayBase::getFB() 3941 0.0007178 0.0007178 0.0007178 0.08% Castro::finalize_advance() 10 0.0006074 0.0006074 0.0006074 0.07% Castro::Castro() 1 0.0005779 0.0005779 0.0005779 0.06% Amr::InitAmr() 1 0.0005758 0.0005758 0.0005758 0.06% Gravity::swapTimeLevels() 10 0.0004547 0.0004547 0.0004547 0.05% MLMG::MLResNormInf() 11 0.0003428 0.0003428 0.0003428 0.04% MultiFab::max() 11 0.0003294 0.0003294 0.0003294 0.04% Castro::buildMetrics() 1 0.0002837 0.0002837 0.0002837 0.03% MLLinOp::define() 11 0.000239 0.000239 0.000239 0.03% MLMG::MLRhsNormInf() 11 0.0002368 0.0002368 0.0002368 0.03% MLLinOp::defineGrids() 11 0.0002265 0.0002265 0.0002265 0.02% Castro::create_source_corrector() 10 0.0001763 0.0001763 0.0001763 0.02% FabArrayBase::FB::FB() 56 9.858e-05 9.858e-05 9.858e-05 0.01% Castro::enforce_consistent_e() 1 6.406e-05 6.406e-05 6.406e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.89e-05 5.89e-05 5.89e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.691e-05 5.691e-05 5.691e-05 0.01% StateData::define() 4 4.335e-05 4.335e-05 4.335e-05 0.00% makeSFC 55 4.292e-05 4.292e-05 4.292e-05 0.00% Castro::swap_state_time_levels() 10 3.529e-05 3.529e-05 3.529e-05 0.00% Amr::writeSmallPlotFile() 1 3.161e-05 3.161e-05 3.161e-05 0.00% Castro::initMFs() 1 3.126e-05 3.126e-05 3.126e-05 0.00% DistributionMapping::Distribute() 56 1.857e-05 1.857e-05 1.857e-05 0.00% Amr::initSubcycle() 1 9.46e-06 9.46e-06 9.46e-06 0.00% Castro::retry_advance_ctu() 10 4.861e-06 4.861e-06 4.861e-06 0.00% Gravity::set_mass_offset() 11 4.599e-06 4.599e-06 4.599e-06 0.00% AmrLevel::checkPointPost() 3 4.549e-06 4.549e-06 4.549e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.804e-06 3.804e-06 3.804e-06 0.00% Castro::FluxRegCrseInit 10 2.68e-06 2.68e-06 2.68e-06 0.00% AmrLevel::checkPointPre() 3 2.475e-06 2.475e-06 2.475e-06 0.00% Castro::FluxRegFineAdd() 10 2.272e-06 2.272e-06 2.272e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.044e-06 2.044e-06 2.044e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 91 MiB 9042 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1003 KiB 39 MiB Castro::initialize_do_advance() 80 80 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1534 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7618 KiB 14 MiB MLMG::prepareForSolve() 660 660 3518 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 201 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 169 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7517 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 16 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2078 B 2048 KiB Gravity::solve_for_phi() 80 80 556 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 100 KiB 2048 KiB BndryData::define() 1056 1056 321 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 205 KiB 671 KiB Castro::estTimeStep() 21 21 3415 B 480 KiB VisMF::Write(FabArray) 656 656 3308 B 320 KiB Castro::normalize_species() 62 62 7934 B 320 KiB amrex::average_down 1067 1067 1236 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1141 B 257 KiB amrex::Dot() 1360 1360 3429 B 160 KiB FabArray::norminf() 1143 1143 3367 B 160 KiB check_for_negative_density() 10 10 372 B 160 KiB Castro::initData() 1 1 50 B 160 KiB MultiFab::max() 11 11 55 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3434 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 306 B 9648 B MLCellLinOp::applyBC() 8702 8702 217 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3929 B 6144 B StateData::FillBoundary(geom) 1992 1992 41 B 2928 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 362 B 1248 B MLCGSolver::bicgstab 410 410 94 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 499 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 57 KiB 8192 KiB VisMF::Write(FabArray) 744 744 422 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3434 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 306 B 9648 B MLCellLinOp::applyBC() 4351 4351 216 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3929 B 6144 B Gravity::get_new_grav_vector() 3 3 2893 B 3072 B StateData::FillBoundary(geom) 1992 1992 42 B 2928 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 271 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 293 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 1143 1143 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2189 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.02-6-g4fc7ef352fe1) finalized Initializing AMReX (24.02-6-g4fc7ef352fe1)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.02-6-g4fc7ef352fe1) initialized Starting run at 09:39:18 UTC on 2024-02-08. Successfully read inputs file ... Castro git describe: 24.02-3-g09b5fe593 AMReX git describe: 24.02-6-g4fc7ef352 Microphysics git describe: 24.02-7-g0ad950aa reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.520190277 Restart time = 0.074166263 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.070605094 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048407542 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072926827 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.084532834 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069554952 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029182285 seconds Ending run at 09:39:19 UTC on 2024-02-08. Run time = 0.450530277 Run time without initialization = 0.375670893 Average number of zones advanced per microsecond: 3.489 Average number of zones advanced per microsecond per rank: 3.489 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.4506 ... 0.4506 ... 0.4506 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1442 0.1442 0.1442 31.99% VisMF::Read() 3 0.06233 0.06233 0.06233 13.83% MLCellLinOp::applyBC() 1910 0.03573 0.03573 0.03573 7.93% VisMF::Write(FabArray) 1 0.02652 0.02652 0.02652 5.89% MLPoisson::Fsmooth() 1440 0.01492 0.01492 0.01492 3.31% StateData::FillBoundary(geom) 160 0.01368 0.01368 0.01368 3.04% StateDataPhysBCFunct::() 20 0.01341 0.01341 0.01341 2.98% FillBoundary_nowait() 1730 0.01338 0.01338 0.01338 2.97% Castro::normalize_species() 30 0.009973 0.009973 0.009973 2.21% amrex::Dot() 484 0.009215 0.009215 0.009215 2.05% FabArray::norminf() 465 0.008691 0.008691 0.008691 1.93% FabArray::setVal() 501 0.006591 0.006591 0.006591 1.46% Castro::computeTemp() 30 0.006295 0.006295 0.006295 1.40% FabArray::ParallelCopy_nowait() 380 0.006255 0.006255 0.006255 1.39% FabArray::Saxpy() 597 0.005898 0.005898 0.005898 1.31% MLCellLinOp::defineAuxData() 6 0.005726 0.005726 0.005726 1.27% amrex::Copy() 221 0.00555 0.00555 0.00555 1.23% Gravity::fill_multipole_BCs() 6 0.005444 0.005444 0.005444 1.21% Amr::restart() 1 0.004966 0.004966 0.004966 1.10% MLPoisson::Fapply() 464 0.004528 0.004528 0.004528 1.01% Castro::enforce_min_density() 30 0.004483 0.004483 0.004483 0.99% FabArray::Xpay() 325 0.003523 0.003523 0.003523 0.78% MLMG::addInterpCorrection() 180 0.003132 0.003132 0.003132 0.70% Castro::estTimeStep() 10 0.003008 0.003008 0.003008 0.67% amrex::average_down 180 0.00274 0.00274 0.00274 0.61% Amr::writePlotFile() 1 0.002494 0.002494 0.002494 0.55% Castro::enforce_speed_limit() 30 0.002254 0.002254 0.002254 0.50% BndryData::define() 6 0.002237 0.002237 0.002237 0.50% Castro::reset_internal_energy(MultiFab) 30 0.002002 0.002002 0.002002 0.44% Castro::construct_new_gravity_source() 5 0.001908 0.001908 0.001908 0.42% amrex::Add() 36 0.001587 0.001587 0.001587 0.35% Castro::construct_old_gravity_source() 5 0.00153 0.00153 0.00153 0.34% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009859 0.0009859 0.0009859 0.22% check_for_negative_density() 5 0.0009742 0.0009742 0.0009742 0.22% MLCellLinOp::setLevelBC() 6 0.0009112 0.0009112 0.0009112 0.20% Gravity::actual_solve_with_mlmg() 6 0.0008634 0.0008634 0.0008634 0.19% Castro::reset_internal_energy(Fab) 240 0.0007846 0.0007846 0.0007846 0.17% MLCellLinOp::prepareForSolve() 6 0.0007677 0.0007677 0.0007677 0.17% MLCGSolver::bicgstab 36 0.0007552 0.0007552 0.0007552 0.17% FabArray::setDomainBndry() 20 0.0007305 0.0007305 0.0007305 0.16% FabArray::mult() 22 0.0007091 0.0007091 0.0007091 0.16% MultiFab::contains_nan() 10 0.0006588 0.0006588 0.0006588 0.15% MLCellLinOp::compGrad() 6 0.0006146 0.0006146 0.0006146 0.14% MLMG::prepareForSolve() 6 0.000572 0.000572 0.000572 0.13% MLCellLinOp::smooth() 720 0.0005101 0.0005101 0.0005101 0.11% Amr::InitAmr() 1 0.0005083 0.0005083 0.0005083 0.11% FabArrayBase::CPC::define() 244 0.0004141 0.0004141 0.0004141 0.09% FabArray::FillBoundary() 1730 0.0003803 0.0003803 0.0003803 0.08% FabArrayBase::getCPC() 632 0.0003688 0.0003688 0.0003688 0.08% Gravity::get_old_grav_vector() 5 0.0003525 0.0003525 0.0003525 0.08% main() 1 0.0002896 0.0002896 0.0002896 0.06% Gravity::get_new_grav_vector() 5 0.0002745 0.0002745 0.0002745 0.06% FabArrayBase::getFB() 1730 0.0002538 0.0002538 0.0002538 0.06% AmrLevel::FillPatch() 20 0.0002301 0.0002301 0.0002301 0.05% MultiFab::max() 6 0.0002164 0.0002164 0.0002164 0.05% Amr::coarseTimeStep() 5 0.0001961 0.0001961 0.0001961 0.04% MLCellLinOp::apply() 464 0.0001856 0.0001856 0.0001856 0.04% MLCellLinOp::defineBC() 6 0.0001618 0.0001618 0.0001618 0.04% MLCGSolver::ParallelAllReduce 798 0.0001547 0.0001547 0.0001547 0.03% FillPatchIterator::Initialize 20 0.0001181 0.0001181 0.0001181 0.03% FabArray::ParallelCopy() 380 0.0001083 0.0001083 0.0001083 0.02% MLLinOp::defineGrids() 6 0.0001001 0.0001001 0.0001001 0.02% Castro::subcycle_advance_ctu() 5 9.565e-05 9.565e-05 9.565e-05 0.02% Amr::timeStep() 5 9.515e-05 9.515e-05 9.515e-05 0.02% Castro::initialize_do_advance() 5 9.282e-05 9.282e-05 9.282e-05 0.02% Castro::do_advance_ctu() 5 9.139e-05 9.139e-05 9.139e-05 0.02% MLMG::mgVcycle() 36 8.224e-05 8.224e-05 8.224e-05 0.02% Castro::create_source_corrector() 5 7.842e-05 7.842e-05 7.842e-05 0.02% MLCellLinOp::correctionResidual() 180 7.72e-05 7.72e-05 7.72e-05 0.02% Castro::advance() 5 7.679e-05 7.679e-05 7.679e-05 0.02% AmrLevel::restart() 1 7.542e-05 7.542e-05 7.542e-05 0.02% Gravity::update_max_rhs() 6 7.381e-05 7.381e-05 7.381e-05 0.02% StateData::restartDoit() 4 7.006e-05 7.006e-05 7.006e-05 0.02% FabArrayBase::FB::FB() 26 6.253e-05 6.253e-05 6.253e-05 0.01% Gravity::solve_for_phi() 5 5.353e-05 5.353e-05 5.353e-05 0.01% MLMG:computeResOfCorrection() 180 5.217e-05 5.217e-05 5.217e-05 0.01% Castro::finalize_do_advance() 5 5.146e-05 5.146e-05 5.146e-05 0.01% Castro::construct_old_source() 25 4.739e-05 4.739e-05 4.739e-05 0.01% Castro::finalize_advance() 5 4.097e-05 4.097e-05 4.097e-05 0.01% MLMG::mgVcycle_down::0 36 3.931e-05 3.931e-05 3.931e-05 0.01% Castro::initialize_advance() 5 3.926e-05 3.926e-05 3.926e-05 0.01% MLMG::actualBottomSolve() 36 3.894e-05 3.894e-05 3.894e-05 0.01% MLMG::mgVcycle_down::1 36 3.754e-05 3.754e-05 3.754e-05 0.01% MLMG::solve() 6 3.603e-05 3.603e-05 3.603e-05 0.01% Castro::do_new_sources() 5 3.577e-05 3.577e-05 3.577e-05 0.01% MLMG::mgVcycle_down::2 36 3.397e-05 3.397e-05 3.397e-05 0.01% MLMG::mgVcycle_down::4 36 3.279e-05 3.279e-05 3.279e-05 0.01% Amr::writeSmallPlotFile() 1 3.276e-05 3.276e-05 3.276e-05 0.01% Castro::clean_state() 30 3.222e-05 3.222e-05 3.222e-05 0.01% MLMG::mgVcycle_down::3 36 3.19e-05 3.19e-05 3.19e-05 0.01% Castro::swap_state_time_levels() 5 3.179e-05 3.179e-05 3.179e-05 0.01% FillPatchSingleLevel 20 3.156e-05 3.156e-05 3.156e-05 0.01% Castro::construct_new_source() 25 3.118e-05 3.118e-05 3.118e-05 0.01% Castro::post_restart() 1 3.028e-05 3.028e-05 3.028e-05 0.01% Castro::buildMetrics() 1 3.024e-05 3.024e-05 3.024e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.956e-05 2.956e-05 2.956e-05 0.01% Castro::initMFs() 1 2.646e-05 2.646e-05 2.646e-05 0.01% MLMG::mgVcycle_up::4 36 2.59e-05 2.59e-05 2.59e-05 0.01% MLCellLinOp::solutionResidual() 42 2.301e-05 2.301e-05 2.301e-05 0.01% MLMG::mgVcycle_up::0 36 2.279e-05 2.279e-05 2.279e-05 0.01% MLMG::oneIter() 36 2.263e-05 2.263e-05 2.263e-05 0.01% MLMG::mgVcycle_up::3 36 2.126e-05 2.126e-05 2.126e-05 0.00% MLMG::mgVcycle_up::2 36 2.111e-05 2.111e-05 2.111e-05 0.00% MLMG::mgVcycle_up::1 36 2.017e-05 2.017e-05 2.017e-05 0.00% Castro::construct_new_gravity() 5 1.895e-05 1.895e-05 1.895e-05 0.00% MLMG::ResNormInf() 42 1.856e-05 1.856e-05 1.856e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.827e-05 1.827e-05 1.827e-05 0.00% MLMG::computeResidual() 36 1.595e-05 1.595e-05 1.595e-05 0.00% MLMG::mgVcycle_bottom 36 1.521e-05 1.521e-05 1.521e-05 0.00% MLPoisson::define() 6 1.487e-05 1.487e-05 1.487e-05 0.00% makeSFC 30 1.339e-05 1.339e-05 1.339e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.304e-05 1.304e-05 1.304e-05 0.00% Castro::do_old_sources() 5 1.204e-05 1.204e-05 1.204e-05 0.00% Amr::initSubcycle() 1 1.005e-05 1.005e-05 1.005e-05 0.00% MLPoisson::prepareForSolve() 6 9.645e-06 9.645e-06 9.645e-06 0.00% DistributionMapping::Distribute() 31 9.245e-06 9.245e-06 9.245e-06 0.00% Gravity::actual_multilevel_solve() 1 8.435e-06 8.435e-06 8.435e-06 0.00% Castro::check_for_nan() 10 7.58e-06 7.58e-06 7.58e-06 0.00% Castro::construct_old_gravity() 5 7.438e-06 7.438e-06 7.438e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.38e-06 7.38e-06 7.38e-06 0.00% MLLinOp::define() 6 6.432e-06 6.432e-06 6.432e-06 0.00% Castro::apply_source_to_state() 10 6.016e-06 6.016e-06 6.016e-06 0.00% Gravity::set_mass_offset() 6 5.171e-06 5.171e-06 5.171e-06 0.00% Castro::post_timestep() 5 4.804e-06 4.804e-06 4.804e-06 0.00% Gravity::swapTimeLevels() 5 4.48e-06 4.48e-06 4.48e-06 0.00% MLMG::computeMLResidual() 6 4.284e-06 4.284e-06 4.284e-06 0.00% MLMG::getGradSolution() 6 3.616e-06 3.616e-06 3.616e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.531e-06 3.531e-06 3.531e-06 0.00% Castro::retry_advance_ctu() 5 3.53e-06 3.53e-06 3.53e-06 0.00% Castro::computeNewDt() 5 3.388e-06 3.388e-06 3.388e-06 0.00% Castro::expand_state() 5 2.878e-06 2.878e-06 2.878e-06 0.00% Castro::FluxRegCrseInit 5 2.763e-06 2.763e-06 2.763e-06 0.00% MLMG::MLResNormInf() 6 2.65e-06 2.65e-06 2.65e-06 0.00% Castro::FluxRegFineAdd() 5 2.617e-06 2.617e-06 2.617e-06 0.00% MLMG::MLRhsNormInf() 6 2.291e-06 2.291e-06 2.291e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.042e-06 1.042e-06 1.042e-06 0.00% Amr::init() 1 9.98e-07 9.98e-07 9.98e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4505 0.4505 0.4505 99.99% Amr::coarseTimeStep() 5 0.3462 0.3462 0.3462 76.84% Amr::timeStep() 5 0.344 0.344 0.344 76.35% Castro::advance() 5 0.3389 0.3389 0.3389 75.21% Castro::subcycle_advance_ctu() 5 0.331 0.331 0.331 73.46% Castro::do_advance_ctu() 5 0.3309 0.3309 0.3309 73.44% Castro::construct_ctu_hydro_source() 5 0.1495 0.1495 0.1495 33.19% Castro::construct_new_gravity() 5 0.1446 0.1446 0.1446 32.09% Gravity::solve_phi_with_mlmg() 6 0.1343 0.1343 0.1343 29.81% Gravity::actual_solve_with_mlmg() 6 0.1286 0.1286 0.1286 28.54% Gravity::solve_for_phi() 5 0.1286 0.1286 0.1286 28.53% MLMG::solve() 6 0.1163 0.1163 0.1163 25.81% MLMG::oneIter() 36 0.1085 0.1085 0.1085 24.08% MLMG::mgVcycle() 36 0.1069 0.1069 0.1069 23.72% Amr::init() 1 0.07421 0.07421 0.07421 16.47% Amr::restart() 1 0.07421 0.07421 0.07421 16.47% AmrLevel::restart() 1 0.0627 0.0627 0.0627 13.92% StateData::restartDoit() 4 0.06262 0.06262 0.06262 13.90% VisMF::Read() 3 0.06233 0.06233 0.06233 13.83% MLCellLinOp::smooth() 720 0.05362 0.05362 0.05362 11.90% MLCellLinOp::applyBC() 1910 0.04981 0.04981 0.04981 11.06% AmrLevel::FillPatch() 20 0.03232 0.03232 0.03232 7.17% MLMG::mgVcycle_bottom 36 0.03167 0.03167 0.03167 7.03% MLMG::actualBottomSolve() 36 0.03166 0.03166 0.03166 7.03% MLCGSolver::bicgstab 36 0.03129 0.03129 0.03129 6.94% FillPatchIterator::Initialize 20 0.03008 0.03008 0.03008 6.68% Amr::writePlotFile() 1 0.02927 0.02927 0.02927 6.50% FillPatchIterator::FillFromLevel0() 20 0.02923 0.02923 0.02923 6.49% FillPatchSingleLevel 20 0.0292 0.0292 0.0292 6.48% StateDataPhysBCFunct::() 20 0.02709 0.02709 0.02709 6.01% VisMF::Write(FabArray) 1 0.02652 0.02652 0.02652 5.89% Castro::clean_state() 30 0.02582 0.02582 0.02582 5.73% MLCellLinOp::apply() 464 0.01605 0.01605 0.01605 3.56% Gravity::get_new_grav_vector() 5 0.01592 0.01592 0.01592 3.53% MLMG::mgVcycle_down::0 36 0.0152 0.0152 0.0152 3.37% MLPoisson::Fsmooth() 1440 0.01492 0.01492 0.01492 3.31% FabArray::FillBoundary() 1730 0.01408 0.01408 0.01408 3.12% FillBoundary_nowait() 1730 0.0137 0.0137 0.0137 3.04% StateData::FillBoundary(geom) 160 0.01368 0.01368 0.01368 3.04% MLMG::mgVcycle_up::0 36 0.01154 0.01154 0.01154 2.56% Castro::initialize_do_advance() 5 0.01036 0.01036 0.01036 2.30% Castro::normalize_species() 30 0.009973 0.009973 0.009973 2.21% MLPoisson::define() 6 0.009653 0.009653 0.009653 2.14% Castro::do_old_sources() 5 0.009465 0.009465 0.009465 2.10% amrex::Dot() 484 0.009215 0.009215 0.009215 2.05% Castro::computeTemp() 30 0.009082 0.009082 0.009082 2.02% MLMG:computeResOfCorrection() 180 0.008991 0.008991 0.008991 2.00% MLCellLinOp::correctionResidual() 180 0.008939 0.008939 0.008939 1.98% Castro::construct_old_gravity() 5 0.008759 0.008759 0.008759 1.94% Gravity::get_old_grav_vector() 5 0.008751 0.008751 0.008751 1.94% FabArray::norminf() 465 0.008691 0.008691 0.008691 1.93% MLMG::mgVcycle_down::1 36 0.007658 0.007658 0.007658 1.70% Castro::initialize_advance() 5 0.007513 0.007513 0.007513 1.67% MLMG::mgVcycle_down::2 36 0.006832 0.006832 0.006832 1.52% FabArray::ParallelCopy() 380 0.006756 0.006756 0.006756 1.50% FabArray::ParallelCopy_nowait() 380 0.006648 0.006648 0.006648 1.48% MLMG::mgVcycle_down::3 36 0.006627 0.006627 0.006627 1.47% FabArray::setVal() 501 0.006591 0.006591 0.006591 1.46% MLMG::mgVcycle_down::4 36 0.006552 0.006552 0.006552 1.45% Castro::do_new_sources() 5 0.006552 0.006552 0.006552 1.45% MLCellLinOp::defineAuxData() 6 0.006511 0.006511 0.006511 1.44% Castro::post_restart() 1 0.006363 0.006363 0.006363 1.41% Castro::expand_state() 5 0.006099 0.006099 0.006099 1.35% Gravity::multilevel_solve_for_new_phi() 1 0.005984 0.005984 0.005984 1.33% Gravity::actual_multilevel_solve() 1 0.005966 0.005966 0.005966 1.32% FabArray::Saxpy() 597 0.005898 0.005898 0.005898 1.31% MLCGSolver::ParallelAllReduce 798 0.00561 0.00561 0.00561 1.25% Gravity::fill_multipole_BCs() 6 0.005577 0.005577 0.005577 1.24% amrex::Copy() 221 0.00555 0.00555 0.00555 1.23% MLMG::addInterpCorrection() 180 0.005456 0.005456 0.005456 1.21% MLMG::mgVcycle_up::1 36 0.00529 0.00529 0.00529 1.17% MLMG::mgVcycle_up::4 36 0.005263 0.005263 0.005263 1.17% MLMG::mgVcycle_up::2 36 0.005157 0.005157 0.005157 1.14% amrex::average_down 180 0.005084 0.005084 0.005084 1.13% Castro::post_timestep() 5 0.005041 0.005041 0.005041 1.12% MLMG::mgVcycle_up::3 36 0.005026 0.005026 0.005026 1.12% MLPoisson::Fapply() 464 0.004528 0.004528 0.004528 1.01% Castro::enforce_min_density() 30 0.004483 0.004483 0.004483 0.99% MLCellLinOp::solutionResidual() 42 0.003624 0.003624 0.003624 0.80% FabArray::Xpay() 325 0.003523 0.003523 0.003523 0.78% Castro::estTimeStep() 10 0.003008 0.003008 0.003008 0.67% MLCellLinOp::defineBC() 6 0.002991 0.002991 0.002991 0.66% MLMG::prepareForSolve() 6 0.002935 0.002935 0.002935 0.65% BndryData::define() 6 0.002829 0.002829 0.002829 0.63% MLMG::computeResidual() 36 0.002827 0.002827 0.002827 0.63% Castro::reset_internal_energy(MultiFab) 30 0.002786 0.002786 0.002786 0.62% Castro::enforce_speed_limit() 30 0.002254 0.002254 0.002254 0.50% Castro::computeNewDt() 5 0.001989 0.001989 0.001989 0.44% Castro::construct_new_source() 25 0.001939 0.001939 0.001939 0.43% Castro::construct_new_gravity_source() 5 0.001908 0.001908 0.001908 0.42% amrex::Add() 36 0.001587 0.001587 0.001587 0.35% Castro::construct_old_source() 25 0.001577 0.001577 0.001577 0.35% Castro::construct_old_gravity_source() 5 0.00153 0.00153 0.00153 0.34% Castro::finalize_do_advance() 5 0.001074 0.001074 0.001074 0.24% MLMG::ResNormInf() 42 0.001025 0.001025 0.001025 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009859 0.0009859 0.0009859 0.22% check_for_negative_density() 5 0.0009742 0.0009742 0.0009742 0.22% Castro::apply_source_to_state() 10 0.0009705 0.0009705 0.0009705 0.22% MLCellLinOp::setLevelBC() 6 0.0009112 0.0009112 0.0009112 0.20% MLMG::getGradSolution() 6 0.000899 0.000899 0.000899 0.20% MLCellLinOp::compGrad() 6 0.0008954 0.0008954 0.0008954 0.20% MLMG::computeMLResidual() 6 0.0008173 0.0008173 0.0008173 0.18% Castro::reset_internal_energy(Fab) 240 0.0007846 0.0007846 0.0007846 0.17% FabArrayBase::getCPC() 632 0.0007829 0.0007829 0.0007829 0.17% MLPoisson::prepareForSolve() 6 0.0007774 0.0007774 0.0007774 0.17% MLCellLinOp::prepareForSolve() 6 0.0007677 0.0007677 0.0007677 0.17% Gravity::update_max_rhs() 6 0.0007515 0.0007515 0.0007515 0.17% FabArray::setDomainBndry() 20 0.0007305 0.0007305 0.0007305 0.16% FabArray::mult() 22 0.0007091 0.0007091 0.0007091 0.16% Castro::check_for_nan() 10 0.0006664 0.0006664 0.0006664 0.15% MultiFab::contains_nan() 10 0.0006588 0.0006588 0.0006588 0.15% Amr::InitAmr() 1 0.0005184 0.0005184 0.0005184 0.12% FabArrayBase::CPC::define() 244 0.0004141 0.0004141 0.0004141 0.09% FabArrayBase::getFB() 1730 0.0003164 0.0003164 0.0003164 0.07% Castro::finalize_advance() 5 0.0003159 0.0003159 0.0003159 0.07% Gravity::swapTimeLevels() 5 0.0002353 0.0002353 0.0002353 0.05% MultiFab::max() 6 0.0002164 0.0002164 0.0002164 0.05% MLMG::MLResNormInf() 6 0.0001963 0.0001963 0.0001963 0.04% Castro::buildMetrics() 1 0.0001568 0.0001568 0.0001568 0.03% MLLinOp::define() 6 0.0001361 0.0001361 0.0001361 0.03% MLLinOp::defineGrids() 6 0.0001297 0.0001297 0.0001297 0.03% MLMG::MLRhsNormInf() 6 0.0001253 0.0001253 0.0001253 0.03% Castro::create_source_corrector() 5 7.842e-05 7.842e-05 7.842e-05 0.02% FabArrayBase::FB::FB() 26 6.253e-05 6.253e-05 6.253e-05 0.01% Amr::writeSmallPlotFile() 1 3.276e-05 3.276e-05 3.276e-05 0.01% Castro::swap_state_time_levels() 5 3.179e-05 3.179e-05 3.179e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.853e-05 2.853e-05 2.853e-05 0.01% Castro::initMFs() 1 2.646e-05 2.646e-05 2.646e-05 0.01% makeSFC 30 2.115e-05 2.115e-05 2.115e-05 0.00% Amr::initSubcycle() 1 1.005e-05 1.005e-05 1.005e-05 0.00% DistributionMapping::Distribute() 31 9.245e-06 9.245e-06 9.245e-06 0.00% Gravity::set_mass_offset() 6 5.171e-06 5.171e-06 5.171e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.01e-06 5.01e-06 5.01e-06 0.00% Castro::retry_advance_ctu() 5 3.53e-06 3.53e-06 3.53e-06 0.00% Castro::FluxRegCrseInit 5 2.763e-06 2.763e-06 2.763e-06 0.00% Castro::FluxRegFineAdd() 5 2.617e-06 2.617e-06 2.617e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.042e-06 1.042e-06 1.042e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 177 MiB 9042 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1239 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1736 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6427 KiB 14 MiB MLMG::prepareForSolve() 361 361 3167 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 201 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 362 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6414 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 20 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3291 B 2048 KiB Gravity::solve_for_phi() 40 40 583 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 27 KiB 2048 KiB BndryData::define() 576 576 292 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 187 KiB 671 KiB Castro::estTimeStep() 10 10 3109 B 480 KiB VisMF::Write(FabArray) 112 112 2222 B 320 KiB Castro::normalize_species() 30 30 7200 B 320 KiB amrex::average_down 469 469 1091 B 257 KiB MLMG::addInterpCorrection() 468 468 1043 B 257 KiB amrex::Dot() 592 592 3017 B 160 KiB FabArray::norminf() 501 501 2980 B 160 KiB check_for_negative_density() 5 5 347 B 160 KiB MultiFab::max() 6 6 74 B 160 KiB FabArray::setVal() 66 66 17 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3068 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 43 B 10 KiB FillBoundary_nowait() 336 336 258 B 9648 B MLCellLinOp::applyBC() 3820 3820 198 B 9344 B amrex::Copy() 56 56 5546 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 48 B 2544 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 328 B 1248 B MLCGSolver::bicgstab 180 180 83 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1063 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 82 KiB 8192 KiB VisMF::Write(FabArray) 120 120 156 KiB 3584 KiB VisMF::Read() 24 24 208 KiB 3000 KiB FabArray::setVal() 66 66 17 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3068 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 43 B 10 KiB FillBoundary_nowait() 336 336 258 B 9648 B MLCellLinOp::applyBC() 1910 1910 197 B 9328 B amrex::Copy() 56 56 5546 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2521 B 3072 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B StateData::FillBoundary(geom) 960 960 49 B 2544 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 491 B 1296 B amrex::average_down 37 37 203 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 22 B 400 B FabArray::norminf() 501 501 8 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2189 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.02-6-g4fc7ef352fe1) finalized