Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-31-g435281069858) initialized Starting run at 07:26:29 UTC on 2023-08-18. Successfully read inputs file ... Castro git describe: 23.08-6-g41b0e2684 AMReX git describe: 23.08-31-g435281069 Microphysics git describe: 23.08-11-g9ecfd8d2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.048608164 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02706026 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050575146 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050601782 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072264893 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.067579145 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.05639752 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047338344 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.077148291 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.078151493 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047951291 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.046604822 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069197252 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.046571891 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026947365 seconds Ending run at 07:26:30 UTC on 2023-08-18. Run time = 0.867172559 Run time without initialization = 0.737928018 Average number of zones advanced per microsecond: 3.552 Average number of zones advanced per microsecond per rank: 3.552 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8672 ... 0.8672 ... 0.8672 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2517 0.2517 0.2517 29.02% VisMF::Write(FabArray) 11 0.1892 0.1892 0.1892 21.82% MLCellLinOp::applyBC() 4433 0.07564 0.07564 0.07564 8.72% MLPoisson::Fsmooth() 3280 0.03199 0.03199 0.03199 3.69% FillBoundary_nowait() 4023 0.03155 0.03155 0.03155 3.64% StateData::FillBoundary(geom) 328 0.02467 0.02467 0.02467 2.84% amrex::Dot() 1114 0.02032 0.02032 0.02032 2.34% Castro::normalize_species() 62 0.02019 0.02019 0.02019 2.33% amrex::Copy() 1029 0.01475 0.01475 0.01475 1.70% Castro::computeTemp() 63 0.01456 0.01456 0.01456 1.68% FabArray::norminf() 743 0.0142 0.0142 0.0142 1.64% FabArray::setVal() 1144 0.0131 0.0131 0.0131 1.51% FabArray::ParallelCopy_nowait() 861 0.01309 0.01309 0.01309 1.51% Castro::enforce_min_density() 62 0.01201 0.01201 0.01201 1.39% StateDataPhysBCFunct::() 41 0.01199 0.01199 0.01199 1.38% MLPoisson::Fapply() 1142 0.01018 0.01018 0.01018 1.17% MLCellLinOp::defineAuxData() 11 0.009666 0.009666 0.009666 1.11% Gravity::fill_multipole_BCs() 11 0.009001 0.009001 0.009001 1.04% FabArray::Saxpy() 813 0.00814 0.00814 0.00814 0.94% FabArray::Xpay() 821 0.008007 0.008007 0.008007 0.92% MLMG::addInterpCorrection() 410 0.006678 0.006678 0.006678 0.77% amrex::average_down 410 0.005934 0.005934 0.005934 0.68% Castro::estTimeStep() 21 0.00586 0.00586 0.00586 0.68% Castro::reset_internal_energy(MultiFab) 63 0.005592 0.005592 0.005592 0.64% Amr::checkPoint() 3 0.004852 0.004852 0.004852 0.56% FabArray::LinComb() 557 0.00444 0.00444 0.00444 0.51% amrex::Add() 164 0.004396 0.004396 0.004396 0.51% BndryData::define() 11 0.00362 0.00362 0.00362 0.42% Castro::construct_new_gravity_source() 10 0.00332 0.00332 0.00332 0.38% Castro::construct_old_gravity_source() 10 0.002782 0.002782 0.002782 0.32% Castro::enforce_speed_limit() 62 0.002461 0.002461 0.002461 0.28% MLCGSolver::bicgstab 82 0.00227 0.00227 0.00227 0.26% Amr::writePlotFile() 2 0.001987 0.001987 0.001987 0.23% check_for_negative_density() 10 0.001755 0.001755 0.001755 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001682 0.001682 0.001682 0.19% Castro::reset_internal_energy(Fab) 504 0.001626 0.001626 0.001626 0.19% Gravity::actual_solve_with_mlmg() 11 0.001512 0.001512 0.001512 0.17% Castro::initData() 1 0.001495 0.001495 0.001495 0.17% MLCellLinOp::setLevelBC() 11 0.001415 0.001415 0.001415 0.16% FabArray::mult() 43 0.001352 0.001352 0.001352 0.16% FabArray::setDomainBndry() 41 0.001302 0.001302 0.001302 0.15% MultiFab::contains_nan() 20 0.00126 0.00126 0.00126 0.15% MLCellLinOp::prepareForSolve() 11 0.001236 0.001236 0.001236 0.14% MLCellLinOp::smooth() 1640 0.00112 0.00112 0.00112 0.13% MLCellLinOp::compGrad() 11 0.001036 0.001036 0.001036 0.12% MLMG::prepareForSolve() 11 0.0009294 0.0009294 0.0009294 0.11% FabArray::FillBoundary() 4023 0.0007949 0.0007949 0.0007949 0.09% FabArrayBase::getCPC() 1323 0.0007616 0.0007616 0.0007616 0.09% FabArrayBase::CPC::define() 454 0.0006567 0.0006567 0.0006567 0.08% FabArrayBase::getFB() 4023 0.0006243 0.0006243 0.0006243 0.07% Gravity::get_new_grav_vector() 11 0.0005916 0.0005916 0.0005916 0.07% Amr::InitAmr() 1 0.0004942 0.0004942 0.0004942 0.06% Gravity::get_old_grav_vector() 10 0.0004693 0.0004693 0.0004693 0.05% MLCellLinOp::apply() 1142 0.0004639 0.0004639 0.0004639 0.05% AmrLevel::FillPatch() 41 0.0003321 0.0003321 0.0003321 0.04% Amr::coarseTimeStep() 10 0.0003218 0.0003218 0.0003218 0.04% MultiFab::max() 11 0.0003195 0.0003195 0.0003195 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003029 0.0003029 0.0003029 0.03% main() 1 0.0002787 0.0002787 0.0002787 0.03% FabArray::ParallelCopy() 861 0.0002533 0.0002533 0.0002533 0.03% Castro::subcycle_advance_ctu() 10 0.0002361 0.0002361 0.0002361 0.03% MLCellLinOp::defineBC() 11 0.0002237 0.0002237 0.0002237 0.03% FillPatchIterator::Initialize 41 0.0002165 0.0002165 0.0002165 0.02% MLCellLinOp::correctionResidual() 492 0.0002108 0.0002108 0.0002108 0.02% MLMG::mgVcycle() 82 0.0001984 0.0001984 0.0001984 0.02% MLLinOp::defineGrids() 11 0.0001496 0.0001496 0.0001496 0.02% Castro::create_source_corrector() 10 0.0001495 0.0001495 0.0001495 0.02% Amr::timeStep() 10 0.0001471 0.0001471 0.0001471 0.02% Gravity::solve_for_phi() 10 0.0001426 0.0001426 0.0001426 0.02% Gravity::update_max_rhs() 11 0.0001257 0.0001257 0.0001257 0.01% StateData::checkPoint() 12 0.0001181 0.0001181 0.0001181 0.01% MLMG:computeResOfCorrection() 410 0.0001176 0.0001176 0.0001176 0.01% Castro::Castro() 1 9.446e-05 9.446e-05 9.446e-05 0.01% Castro::advance() 10 9.335e-05 9.335e-05 9.335e-05 0.01% MLMG::actualBottomSolve() 82 9.172e-05 9.172e-05 9.172e-05 0.01% FabArrayBase::FB::FB() 56 8.788e-05 8.788e-05 8.788e-05 0.01% MLMG::mgVcycle_down::0 82 8.786e-05 8.786e-05 8.786e-05 0.01% MLMG::solve() 11 8.165e-05 8.165e-05 8.165e-05 0.01% MLMG::mgVcycle_down::2 82 7.972e-05 7.972e-05 7.972e-05 0.01% MLMG::mgVcycle_down::1 82 7.897e-05 7.897e-05 7.897e-05 0.01% Castro::initialize_advance() 10 7.675e-05 7.675e-05 7.675e-05 0.01% MLMG::mgVcycle_down::4 82 7.625e-05 7.625e-05 7.625e-05 0.01% MLMG::mgVcycle_down::3 82 7.424e-05 7.424e-05 7.424e-05 0.01% Castro::enforce_consistent_e() 1 7.303e-05 7.303e-05 7.303e-05 0.01% Castro::clean_state() 62 7.17e-05 7.17e-05 7.17e-05 0.01% AmrLevel::checkPoint() 3 6.734e-05 6.734e-05 6.734e-05 0.01% Castro::construct_new_source() 50 6.683e-05 6.683e-05 6.683e-05 0.01% MLMG::mgVcycle_up::4 82 6.125e-05 6.125e-05 6.125e-05 0.01% Castro::finalize_advance() 10 6.03e-05 6.03e-05 6.03e-05 0.01% Castro::initialize_do_advance() 10 5.785e-05 5.785e-05 5.785e-05 0.01% MLMG::mgVcycle_up::3 82 5.147e-05 5.147e-05 5.147e-05 0.01% Castro::do_new_sources() 10 5.071e-05 5.071e-05 5.071e-05 0.01% MLMG::oneIter() 82 5.047e-05 5.047e-05 5.047e-05 0.01% MLMG::mgVcycle_up::2 82 5.04e-05 5.04e-05 5.04e-05 0.01% MLMG::mgVcycle_up::0 82 5.032e-05 5.032e-05 5.032e-05 0.01% MLCellLinOp::solutionResidual() 93 4.935e-05 4.935e-05 4.935e-05 0.01% Castro::do_advance_ctu() 10 4.829e-05 4.829e-05 4.829e-05 0.01% MLMG::mgVcycle_up::1 82 4.716e-05 4.716e-05 4.716e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.604e-05 4.604e-05 4.604e-05 0.01% Castro::finalize_do_advance() 10 4.047e-05 4.047e-05 4.047e-05 0.00% Castro::swap_state_time_levels() 10 3.844e-05 3.844e-05 3.844e-05 0.00% Castro::post_timestep() 10 3.782e-05 3.782e-05 3.782e-05 0.00% StateData::define() 4 3.708e-05 3.708e-05 3.708e-05 0.00% Amr::defBaseLevel() 1 3.663e-05 3.663e-05 3.663e-05 0.00% MLMG::computeResidual() 82 3.577e-05 3.577e-05 3.577e-05 0.00% MLMG::mgVcycle_bottom 82 3.561e-05 3.561e-05 3.561e-05 0.00% MLMG::ResNormInf() 93 3.366e-05 3.366e-05 3.366e-05 0.00% FillPatchSingleLevel 41 3.301e-05 3.301e-05 3.301e-05 0.00% Castro::initMFs() 1 2.843e-05 2.843e-05 2.843e-05 0.00% Amr::writeSmallPlotFile() 1 2.646e-05 2.646e-05 2.646e-05 0.00% Castro::construct_new_gravity() 10 2.603e-05 2.603e-05 2.603e-05 0.00% Castro::buildMetrics() 1 2.492e-05 2.492e-05 2.492e-05 0.00% makeSFC 55 2.307e-05 2.307e-05 2.307e-05 0.00% Castro::do_old_sources() 10 2.234e-05 2.234e-05 2.234e-05 0.00% MLPoisson::define() 11 2.188e-05 2.188e-05 2.188e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.123e-05 2.123e-05 2.123e-05 0.00% Amr::FinalizeInit() 1 1.949e-05 1.949e-05 1.949e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.77e-05 1.77e-05 1.77e-05 0.00% Castro::construct_old_source() 50 1.74e-05 1.74e-05 1.74e-05 0.00% DistributionMapping::Distribute() 56 1.699e-05 1.699e-05 1.699e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.548e-05 1.548e-05 1.548e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.245e-05 1.245e-05 1.245e-05 0.00% Castro::check_for_nan() 20 1.208e-05 1.208e-05 1.208e-05 0.00% MLPoisson::prepareForSolve() 11 1.201e-05 1.201e-05 1.201e-05 0.00% Castro::construct_old_gravity() 10 1.155e-05 1.155e-05 1.155e-05 0.00% Castro::apply_source_to_state() 20 1.1e-05 1.1e-05 1.1e-05 0.00% MLLinOp::define() 11 1.056e-05 1.056e-05 1.056e-05 0.00% Castro::post_init() 1 9.626e-06 9.626e-06 9.626e-06 0.00% Amr::initSubcycle() 1 9.296e-06 9.296e-06 9.296e-06 0.00% Gravity::swapTimeLevels() 10 8.921e-06 8.921e-06 8.921e-06 0.00% Gravity::actual_multilevel_solve() 1 8.12e-06 8.12e-06 8.12e-06 0.00% MLMG::computeMLResidual() 11 7.969e-06 7.969e-06 7.969e-06 0.00% Castro::computeNewDt() 9 7.089e-06 7.089e-06 7.089e-06 0.00% AmrLevel::checkPointPost() 3 5.861e-06 5.861e-06 5.861e-06 0.00% Castro::expand_state() 10 5.578e-06 5.578e-06 5.578e-06 0.00% MLMG::getGradSolution() 11 5.478e-06 5.478e-06 5.478e-06 0.00% Castro::retry_advance_ctu() 10 4.64e-06 4.64e-06 4.64e-06 0.00% Amr::InitializeInit() 1 4.579e-06 4.579e-06 4.579e-06 0.00% MLMG::MLRhsNormInf() 11 4.006e-06 4.006e-06 4.006e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.73e-06 3.73e-06 3.73e-06 0.00% MLMG::MLResNormInf() 11 3.52e-06 3.52e-06 3.52e-06 0.00% Gravity::set_mass_offset() 11 3.41e-06 3.41e-06 3.41e-06 0.00% Castro::computeInitialDt() 2 3.096e-06 3.096e-06 3.096e-06 0.00% Amr::init() 1 2.846e-06 2.846e-06 2.846e-06 0.00% Castro::FluxRegCrseInit 10 2.845e-06 2.845e-06 2.845e-06 0.00% Castro::FluxRegFineAdd() 10 2.149e-06 2.149e-06 2.149e-06 0.00% AmrLevel::checkPointPre() 3 2.049e-06 2.049e-06 2.049e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.011e-06 2.011e-06 2.011e-06 0.00% Castro::post_regrid() 1 1.496e-06 1.496e-06 1.496e-06 0.00% Amr::initialInit() 1 1.103e-06 1.103e-06 1.103e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8672 0.8672 0.8672 100.00% Amr::coarseTimeStep() 10 0.7108 0.7108 0.7108 81.96% Amr::timeStep() 10 0.6131 0.6131 0.6131 70.70% Castro::advance() 10 0.6029 0.6029 0.6029 69.53% Castro::subcycle_advance_ctu() 10 0.5887 0.5887 0.5887 67.89% Castro::do_advance_ctu() 10 0.5885 0.5885 0.5885 67.86% Gravity::solve_phi_with_mlmg() 11 0.2836 0.2836 0.2836 32.70% Gravity::actual_solve_with_mlmg() 11 0.2741 0.2741 0.2741 31.61% Castro::construct_ctu_hydro_source() 10 0.263 0.263 0.263 30.33% Castro::construct_new_gravity() 10 0.2554 0.2554 0.2554 29.45% MLMG::solve() 11 0.2536 0.2536 0.2536 29.25% Gravity::solve_for_phi() 10 0.2401 0.2401 0.2401 27.69% MLMG::oneIter() 82 0.2389 0.2389 0.2389 27.55% MLMG::mgVcycle() 82 0.2353 0.2353 0.2353 27.13% VisMF::Write(FabArray) 11 0.1892 0.1892 0.1892 21.82% Amr::checkPoint() 3 0.1426 0.1426 0.1426 16.45% AmrLevel::checkPoint() 3 0.1378 0.1378 0.1378 15.89% StateData::checkPoint() 12 0.1377 0.1377 0.1377 15.88% Amr::init() 1 0.1286 0.1286 0.1286 14.83% MLCellLinOp::smooth() 1640 0.1161 0.1161 0.1161 13.38% MLCellLinOp::applyBC() 4433 0.1087 0.1087 0.1087 12.53% MLMG::mgVcycle_bottom 82 0.07278 0.07278 0.07278 8.39% MLMG::actualBottomSolve() 82 0.07274 0.07274 0.07274 8.39% MLCGSolver::bicgstab 82 0.07205 0.07205 0.07205 8.31% Castro::clean_state() 62 0.05552 0.05552 0.05552 6.40% Amr::writePlotFile() 2 0.05412 0.05412 0.05412 6.24% Amr::initialInit() 1 0.05283 0.05283 0.05283 6.09% Amr::FinalizeInit() 1 0.04767 0.04767 0.04767 5.50% AmrLevel::FillPatch() 41 0.04637 0.04637 0.04637 5.35% Castro::post_init() 1 0.04619 0.04619 0.04619 5.33% Gravity::multilevel_solve_for_new_phi() 1 0.04389 0.04389 0.04389 5.06% Gravity::actual_multilevel_solve() 1 0.04387 0.04387 0.04387 5.06% FillPatchIterator::Initialize 41 0.04223 0.04223 0.04223 4.87% FillPatchIterator::FillFromLevel0() 41 0.04071 0.04071 0.04071 4.69% FillPatchSingleLevel 41 0.04067 0.04067 0.04067 4.69% StateDataPhysBCFunct::() 41 0.03666 0.03666 0.03666 4.23% MLCellLinOp::apply() 1142 0.03592 0.03592 0.03592 4.14% MLMG::mgVcycle_down::0 82 0.03394 0.03394 0.03394 3.91% FabArray::FillBoundary() 4023 0.03306 0.03306 0.03306 3.81% FillBoundary_nowait() 4023 0.03226 0.03226 0.03226 3.72% MLPoisson::Fsmooth() 3280 0.03199 0.03199 0.03199 3.69% MLMG::mgVcycle_up::0 82 0.02552 0.02552 0.02552 2.94% StateData::FillBoundary(geom) 328 0.02467 0.02467 0.02467 2.84% MLCellLinOp::correctionResidual() 492 0.02186 0.02186 0.02186 2.52% Castro::computeTemp() 63 0.02178 0.02178 0.02178 2.51% Castro::initialize_do_advance() 10 0.02136 0.02136 0.02136 2.46% amrex::Dot() 1114 0.02032 0.02032 0.02032 2.34% Castro::normalize_species() 62 0.02019 0.02019 0.02019 2.33% MLMG:computeResOfCorrection() 410 0.01927 0.01927 0.01927 2.22% Castro::do_old_sources() 10 0.01837 0.01837 0.01837 2.12% Gravity::get_new_grav_vector() 11 0.01709 0.01709 0.01709 1.97% MLPoisson::define() 11 0.01605 0.01605 0.01605 1.85% MLMG::mgVcycle_down::1 82 0.01563 0.01563 0.01563 1.80% amrex::Copy() 1029 0.01475 0.01475 0.01475 1.70% Castro::construct_old_gravity() 10 0.01466 0.01466 0.01466 1.69% Gravity::get_old_grav_vector() 10 0.01465 0.01465 0.01465 1.69% MLMG::mgVcycle_down::2 82 0.01445 0.01445 0.01445 1.67% FabArray::norminf() 743 0.0142 0.0142 0.0142 1.64% MLMG::mgVcycle_down::3 82 0.01414 0.01414 0.01414 1.63% FabArray::ParallelCopy() 861 0.01414 0.01414 0.01414 1.63% MLMG::mgVcycle_down::4 82 0.01401 0.01401 0.01401 1.62% FabArray::ParallelCopy_nowait() 861 0.01388 0.01388 0.01388 1.60% Castro::initialize_advance() 10 0.01352 0.01352 0.01352 1.56% FabArray::setVal() 1144 0.0131 0.0131 0.0131 1.51% Castro::do_new_sources() 10 0.01244 0.01244 0.01244 1.43% MLCGSolver::ParallelAllReduce 1514 0.01221 0.01221 0.01221 1.41% Castro::expand_state() 10 0.01205 0.01205 0.01205 1.39% Castro::enforce_min_density() 62 0.01201 0.01201 0.01201 1.39% MLMG::addInterpCorrection() 410 0.01172 0.01172 0.01172 1.35% MLMG::mgVcycle_up::4 82 0.01132 0.01132 0.01132 1.31% MLMG::mgVcycle_up::1 82 0.0113 0.0113 0.0113 1.30% MLMG::mgVcycle_up::2 82 0.01109 0.01109 0.01109 1.28% amrex::average_down 410 0.01105 0.01105 0.01105 1.27% MLCellLinOp::defineAuxData() 11 0.01102 0.01102 0.01102 1.27% MLMG::mgVcycle_up::3 82 0.01088 0.01088 0.01088 1.26% MLPoisson::Fapply() 1142 0.01018 0.01018 0.01018 1.17% Castro::post_timestep() 10 0.01006 0.01006 0.01006 1.16% Gravity::fill_multipole_BCs() 11 0.009228 0.009228 0.009228 1.06% FabArray::Saxpy() 813 0.00814 0.00814 0.00814 0.94% FabArray::Xpay() 821 0.008007 0.008007 0.008007 0.92% MLCellLinOp::solutionResidual() 93 0.007409 0.007409 0.007409 0.85% Castro::reset_internal_energy(MultiFab) 63 0.007219 0.007219 0.007219 0.83% MLMG::computeResidual() 82 0.006159 0.006159 0.006159 0.71% Castro::estTimeStep() 21 0.00586 0.00586 0.00586 0.68% Amr::InitializeInit() 1 0.00516 0.00516 0.00516 0.60% Amr::defBaseLevel() 1 0.005156 0.005156 0.005156 0.59% MLMG::prepareForSolve() 11 0.004835 0.004835 0.004835 0.56% MLCellLinOp::defineBC() 11 0.004793 0.004793 0.004793 0.55% BndryData::define() 11 0.00457 0.00457 0.00457 0.53% Castro::initData() 1 0.004479 0.004479 0.004479 0.52% FabArray::LinComb() 557 0.00444 0.00444 0.00444 0.51% amrex::Add() 164 0.004396 0.004396 0.004396 0.51% Castro::construct_new_source() 50 0.003387 0.003387 0.003387 0.39% Castro::construct_new_gravity_source() 10 0.00332 0.00332 0.00332 0.38% Castro::construct_old_source() 50 0.0028 0.0028 0.0028 0.32% Castro::construct_old_gravity_source() 10 0.002782 0.002782 0.002782 0.32% Castro::computeNewDt() 9 0.002559 0.002559 0.002559 0.30% Castro::enforce_speed_limit() 62 0.002461 0.002461 0.002461 0.28% Castro::finalize_do_advance() 10 0.002317 0.002317 0.002317 0.27% MLMG::ResNormInf() 93 0.002137 0.002137 0.002137 0.25% Castro::apply_source_to_state() 20 0.00185 0.00185 0.00185 0.21% check_for_negative_density() 10 0.001755 0.001755 0.001755 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001682 0.001682 0.001682 0.19% Castro::reset_internal_energy(Fab) 504 0.001626 0.001626 0.001626 0.19% MLMG::getGradSolution() 11 0.001513 0.001513 0.001513 0.17% MLCellLinOp::compGrad() 11 0.001508 0.001508 0.001508 0.17% FabArrayBase::getCPC() 1323 0.001418 0.001418 0.001418 0.16% MLCellLinOp::setLevelBC() 11 0.001415 0.001415 0.001415 0.16% FabArray::mult() 43 0.001352 0.001352 0.001352 0.16% FabArray::setDomainBndry() 41 0.001302 0.001302 0.001302 0.15% MLMG::computeMLResidual() 11 0.001294 0.001294 0.001294 0.15% Castro::check_for_nan() 20 0.001272 0.001272 0.001272 0.15% MultiFab::contains_nan() 20 0.00126 0.00126 0.00126 0.15% MLPoisson::prepareForSolve() 11 0.001248 0.001248 0.001248 0.14% MLCellLinOp::prepareForSolve() 11 0.001236 0.001236 0.001236 0.14% Castro::post_regrid() 1 0.001185 0.001185 0.001185 0.14% Castro::computeInitialDt() 2 0.001035 0.001035 0.001035 0.12% Gravity::update_max_rhs() 11 0.0009643 0.0009643 0.0009643 0.11% FabArrayBase::getFB() 4023 0.0007122 0.0007122 0.0007122 0.08% FabArrayBase::CPC::define() 454 0.0006567 0.0006567 0.0006567 0.08% Castro::finalize_advance() 10 0.0005828 0.0005828 0.0005828 0.07% Castro::Castro() 1 0.0005825 0.0005825 0.0005825 0.07% Amr::InitAmr() 1 0.0005035 0.0005035 0.0005035 0.06% Gravity::swapTimeLevels() 10 0.000432 0.000432 0.000432 0.05% MLMG::MLResNormInf() 11 0.0003243 0.0003243 0.0003243 0.04% MultiFab::max() 11 0.0003195 0.0003195 0.0003195 0.04% Castro::buildMetrics() 1 0.0002894 0.0002894 0.0002894 0.03% MLMG::MLRhsNormInf() 11 0.0002178 0.0002178 0.0002178 0.03% MLLinOp::define() 11 0.0002134 0.0002134 0.0002134 0.02% MLLinOp::defineGrids() 11 0.0002028 0.0002028 0.0002028 0.02% Castro::create_source_corrector() 10 0.0001495 0.0001495 0.0001495 0.02% FabArrayBase::FB::FB() 56 8.788e-05 8.788e-05 8.788e-05 0.01% Castro::enforce_consistent_e() 1 7.303e-05 7.303e-05 7.303e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.255e-05 5.255e-05 5.255e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.118e-05 5.118e-05 5.118e-05 0.01% makeSFC 55 3.873e-05 3.873e-05 3.873e-05 0.00% Castro::swap_state_time_levels() 10 3.844e-05 3.844e-05 3.844e-05 0.00% StateData::define() 4 3.708e-05 3.708e-05 3.708e-05 0.00% Castro::initMFs() 1 2.843e-05 2.843e-05 2.843e-05 0.00% Amr::writeSmallPlotFile() 1 2.646e-05 2.646e-05 2.646e-05 0.00% DistributionMapping::Distribute() 56 1.699e-05 1.699e-05 1.699e-05 0.00% Amr::initSubcycle() 1 9.296e-06 9.296e-06 9.296e-06 0.00% AmrLevel::checkPointPost() 3 5.861e-06 5.861e-06 5.861e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.064e-06 5.064e-06 5.064e-06 0.00% Castro::retry_advance_ctu() 10 4.64e-06 4.64e-06 4.64e-06 0.00% Gravity::set_mass_offset() 11 3.41e-06 3.41e-06 3.41e-06 0.00% Castro::FluxRegCrseInit 10 2.845e-06 2.845e-06 2.845e-06 0.00% Castro::FluxRegFineAdd() 10 2.149e-06 2.149e-06 2.149e-06 0.00% AmrLevel::checkPointPre() 3 2.049e-06 2.049e-06 2.049e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.011e-06 2.011e-06 2.011e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 120 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 135 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1058 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1756 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7621 KiB 14 MiB MLMG::prepareForSolve() 660 660 3597 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 204 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 172 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7513 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 18 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2152 B 2048 KiB Gravity::solve_for_phi() 80 80 566 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 103 KiB 2048 KiB BndryData::define() 1056 1056 328 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 209 KiB 671 KiB Castro::estTimeStep() 21 21 3297 B 480 KiB VisMF::Write(FabArray) 656 656 3453 B 320 KiB Castro::normalize_species() 62 62 7586 B 320 KiB amrex::average_down 1067 1067 1274 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1152 B 257 KiB amrex::Dot() 1360 1360 3445 B 160 KiB FabArray::norminf() 907 907 2420 B 160 KiB check_for_negative_density() 10 10 319 B 160 KiB Castro::initData() 1 1 55 B 160 KiB MultiFab::max() 11 11 57 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 28 B 20 KiB MLPoisson::Fsmooth() 132 132 3513 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 45 B 10 KiB FillBoundary_nowait() 760 760 319 B 9648 B MLCellLinOp::applyBC() 8866 8866 226 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3912 B 6144 B StateData::FillBoundary(geom) 1992 1992 47 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 120 B 1472 B MLCellLinOp::defineBC() 66 66 370 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 604 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 40 KiB 8192 KiB VisMF::Write(FabArray) 744 744 491 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3513 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 45 B 10 KiB FillBoundary_nowait() 760 760 319 B 9648 B MLCellLinOp::applyBC() 4433 4433 224 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3912 B 6144 B Gravity::get_new_grav_vector() 3 3 2886 B 3072 B StateData::FillBoundary(geom) 1992 1992 47 B 3024 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 270 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 299 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 907 907 17 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-31-g435281069858) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-31-g435281069858) initialized Starting run at 07:26:31 UTC on 2023-08-18. Successfully read inputs file ... Castro git describe: 23.08-6-g41b0e2684 AMReX git describe: 23.08-31-g435281069 Microphysics git describe: 23.08-11-g9ecfd8d2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.474018263 Restart time = 0.048972047 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.055137306 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057963833 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.078787724 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.080123539 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.055253942 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.058316386 seconds Ending run at 07:26:31 UTC on 2023-08-18. Run time = 0.435508501 Run time without initialization = 0.385986579 Average number of zones advanced per microsecond: 3.396 Average number of zones advanced per microsecond per rank: 3.396 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.4355 ... 0.4355 ... 0.4355 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1356 0.1356 0.1356 31.14% VisMF::Read() 3 0.03774 0.03774 0.03774 8.67% MLCellLinOp::applyBC() 1946 0.03281 0.03281 0.03281 7.53% Amr::writePlotFile() 1 0.03107 0.03107 0.03107 7.13% VisMF::Write(FabArray) 1 0.02701 0.02701 0.02701 6.20% MLPoisson::Fsmooth() 1440 0.01377 0.01377 0.01377 3.16% StateDataPhysBCFunct::() 20 0.01362 0.01362 0.01362 3.13% FillBoundary_nowait() 1766 0.01314 0.01314 0.01314 3.02% StateData::FillBoundary(geom) 160 0.01136 0.01136 0.01136 2.61% Castro::normalize_species() 30 0.009027 0.009027 0.009027 2.07% amrex::Dot() 484 0.008729 0.008729 0.008729 2.00% amrex::Copy() 463 0.007096 0.007096 0.007096 1.63% Castro::computeTemp() 30 0.006351 0.006351 0.006351 1.46% FabArray::setVal() 537 0.006332 0.006332 0.006332 1.45% FabArray::norminf() 326 0.006179 0.006179 0.006179 1.42% FabArray::ParallelCopy_nowait() 380 0.006013 0.006013 0.006013 1.38% Gravity::fill_multipole_BCs() 6 0.005332 0.005332 0.005332 1.22% MLCellLinOp::defineAuxData() 6 0.005221 0.005221 0.005221 1.20% Castro::enforce_min_density() 30 0.005017 0.005017 0.005017 1.15% Amr::restart() 1 0.004689 0.004689 0.004689 1.08% MLPoisson::Fapply() 500 0.004418 0.004418 0.004418 1.01% FabArray::Saxpy() 355 0.003663 0.003663 0.003663 0.84% FabArray::Xpay() 361 0.003493 0.003493 0.003493 0.80% Castro::estTimeStep() 10 0.003175 0.003175 0.003175 0.73% MLMG::addInterpCorrection() 180 0.002978 0.002978 0.002978 0.68% amrex::average_down 180 0.002589 0.002589 0.002589 0.59% Castro::reset_internal_energy(MultiFab) 30 0.002222 0.002222 0.002222 0.51% FabArray::LinComb() 242 0.002195 0.002195 0.002195 0.50% BndryData::define() 6 0.002002 0.002002 0.002002 0.46% amrex::Add() 72 0.00187 0.00187 0.00187 0.43% Castro::construct_new_gravity_source() 5 0.001718 0.001718 0.001718 0.39% Castro::construct_old_gravity_source() 5 0.001399 0.001399 0.001399 0.32% Castro::subcycle_advance_ctu() 5 0.001276 0.001276 0.001276 0.29% MLCGSolver::bicgstab 36 0.0009984 0.0009984 0.0009984 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009203 0.0009203 0.0009203 0.21% check_for_negative_density() 5 0.0008695 0.0008695 0.0008695 0.20% Gravity::actual_solve_with_mlmg() 6 0.0008096 0.0008096 0.0008096 0.19% Castro::reset_internal_energy(Fab) 240 0.0007927 0.0007927 0.0007927 0.18% MLCellLinOp::setLevelBC() 6 0.0007714 0.0007714 0.0007714 0.18% MLCellLinOp::prepareForSolve() 6 0.0006904 0.0006904 0.0006904 0.16% FabArray::mult() 22 0.0006785 0.0006785 0.0006785 0.16% FabArray::setDomainBndry() 20 0.0006606 0.0006606 0.0006606 0.15% MultiFab::contains_nan() 10 0.0006517 0.0006517 0.0006517 0.15% Castro::initialize_do_advance() 5 0.0005863 0.0005863 0.0005863 0.13% MLCellLinOp::compGrad() 6 0.0005724 0.0005724 0.0005724 0.13% MLMG::prepareForSolve() 6 0.0005218 0.0005218 0.0005218 0.12% MLCellLinOp::smooth() 720 0.0005065 0.0005065 0.0005065 0.12% Castro::enforce_speed_limit() 30 0.0004633 0.0004633 0.0004633 0.11% FabArrayBase::CPC::define() 244 0.0004042 0.0004042 0.0004042 0.09% Amr::InitAmr() 1 0.0003907 0.0003907 0.0003907 0.09% FabArray::FillBoundary() 1766 0.0003504 0.0003504 0.0003504 0.08% FabArrayBase::getCPC() 632 0.0003477 0.0003477 0.0003477 0.08% Gravity::get_old_grav_vector() 5 0.0002894 0.0002894 0.0002894 0.07% FabArrayBase::getFB() 1766 0.0002628 0.0002628 0.0002628 0.06% main() 1 0.0002555 0.0002555 0.0002555 0.06% Gravity::get_new_grav_vector() 5 0.0002382 0.0002382 0.0002382 0.05% MLCellLinOp::apply() 500 0.0002099 0.0002099 0.0002099 0.05% MultiFab::max() 6 0.0001987 0.0001987 0.0001987 0.05% AmrLevel::FillPatch() 20 0.0001637 0.0001637 0.0001637 0.04% Amr::coarseTimeStep() 5 0.0001534 0.0001534 0.0001534 0.04% MLCGSolver::ParallelAllReduce 659 0.0001392 0.0001392 0.0001392 0.03% FabArray::ParallelCopy() 380 0.0001215 0.0001215 0.0001215 0.03% MLCellLinOp::defineBC() 6 0.0001182 0.0001182 0.0001182 0.03% FillPatchIterator::Initialize 20 0.0001067 0.0001067 0.0001067 0.02% Castro::construct_new_source() 25 9.747e-05 9.747e-05 9.747e-05 0.02% MLCellLinOp::correctionResidual() 216 9.499e-05 9.499e-05 9.499e-05 0.02% MLLinOp::defineGrids() 6 8.558e-05 8.558e-05 8.558e-05 0.02% MLMG::mgVcycle() 36 8.489e-05 8.489e-05 8.489e-05 0.02% Amr::timeStep() 5 8.304e-05 8.304e-05 8.304e-05 0.02% Castro::create_source_corrector() 5 7.954e-05 7.954e-05 7.954e-05 0.02% Castro::do_new_sources() 5 7.193e-05 7.193e-05 7.193e-05 0.02% Gravity::solve_for_phi() 5 7.042e-05 7.042e-05 7.042e-05 0.02% AmrLevel::restart() 1 6.822e-05 6.822e-05 6.822e-05 0.02% Gravity::update_max_rhs() 6 6.373e-05 6.373e-05 6.373e-05 0.01% StateData::restartDoit() 4 6.18e-05 6.18e-05 6.18e-05 0.01% FabArrayBase::FB::FB() 26 5.657e-05 5.657e-05 5.657e-05 0.01% Castro::advance() 5 5.435e-05 5.435e-05 5.435e-05 0.01% Castro::construct_old_source() 25 5.42e-05 5.42e-05 5.42e-05 0.01% MLMG:computeResOfCorrection() 180 5.377e-05 5.377e-05 5.377e-05 0.01% Castro::finalize_do_advance() 5 4.574e-05 4.574e-05 4.574e-05 0.01% MLMG::mgVcycle_down::0 36 4.353e-05 4.353e-05 4.353e-05 0.01% MLMG::solve() 6 4.118e-05 4.118e-05 4.118e-05 0.01% MLMG::actualBottomSolve() 36 3.97e-05 3.97e-05 3.97e-05 0.01% MLMG::mgVcycle_down::1 36 3.521e-05 3.521e-05 3.521e-05 0.01% Castro::initialize_advance() 5 3.503e-05 3.503e-05 3.503e-05 0.01% Castro::do_old_sources() 5 3.423e-05 3.423e-05 3.423e-05 0.01% MLMG::mgVcycle_down::2 36 3.397e-05 3.397e-05 3.397e-05 0.01% MLMG::mgVcycle_down::4 36 3.394e-05 3.394e-05 3.394e-05 0.01% Castro::clean_state() 30 3.314e-05 3.314e-05 3.314e-05 0.01% Castro::post_timestep() 5 3.21e-05 3.21e-05 3.21e-05 0.01% MLMG::mgVcycle_down::3 36 3.201e-05 3.201e-05 3.201e-05 0.01% Castro::finalize_advance() 5 3.155e-05 3.155e-05 3.155e-05 0.01% MLMG::mgVcycle_up::4 36 2.832e-05 2.832e-05 2.832e-05 0.01% Amr::writeSmallPlotFile() 1 2.796e-05 2.796e-05 2.796e-05 0.01% Castro::post_restart() 1 2.712e-05 2.712e-05 2.712e-05 0.01% Castro::buildMetrics() 1 2.697e-05 2.697e-05 2.697e-05 0.01% Castro::do_advance_ctu() 5 2.508e-05 2.508e-05 2.508e-05 0.01% Castro::swap_state_time_levels() 5 2.479e-05 2.479e-05 2.479e-05 0.01% MLMG::oneIter() 36 2.368e-05 2.368e-05 2.368e-05 0.01% Castro::initMFs() 1 2.366e-05 2.366e-05 2.366e-05 0.01% MLCellLinOp::solutionResidual() 42 2.285e-05 2.285e-05 2.285e-05 0.01% MLMG::mgVcycle_up::3 36 2.262e-05 2.262e-05 2.262e-05 0.01% MLMG::mgVcycle_up::0 36 2.218e-05 2.218e-05 2.218e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.104e-05 2.104e-05 2.104e-05 0.00% MLMG::mgVcycle_up::2 36 2.093e-05 2.093e-05 2.093e-05 0.00% MLMG::mgVcycle_up::1 36 2.018e-05 2.018e-05 2.018e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.662e-05 1.662e-05 1.662e-05 0.00% MLMG::ResNormInf() 42 1.639e-05 1.639e-05 1.639e-05 0.00% MLMG::computeResidual() 36 1.547e-05 1.547e-05 1.547e-05 0.00% FillPatchSingleLevel 20 1.542e-05 1.542e-05 1.542e-05 0.00% MLMG::mgVcycle_bottom 36 1.536e-05 1.536e-05 1.536e-05 0.00% MLPoisson::define() 6 1.456e-05 1.456e-05 1.456e-05 0.00% makeSFC 30 1.284e-05 1.284e-05 1.284e-05 0.00% Castro::construct_new_gravity() 5 1.277e-05 1.277e-05 1.277e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.256e-05 1.256e-05 1.256e-05 0.00% DistributionMapping::Distribute() 31 9.23e-06 9.23e-06 9.23e-06 0.00% Amr::initSubcycle() 1 8.799e-06 8.799e-06 8.799e-06 0.00% MLPoisson::prepareForSolve() 6 8.662e-06 8.662e-06 8.662e-06 0.00% Gravity::actual_multilevel_solve() 1 7.863e-06 7.863e-06 7.863e-06 0.00% Castro::check_for_nan() 10 7.58e-06 7.58e-06 7.58e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.928e-06 6.928e-06 6.928e-06 0.00% Castro::construct_old_gravity() 5 6.01e-06 6.01e-06 6.01e-06 0.00% Castro::apply_source_to_state() 10 5.937e-06 5.937e-06 5.937e-06 0.00% MLLinOp::define() 6 5.392e-06 5.392e-06 5.392e-06 0.00% Gravity::swapTimeLevels() 5 4.519e-06 4.519e-06 4.519e-06 0.00% MLMG::computeMLResidual() 6 4.074e-06 4.074e-06 4.074e-06 0.00% Castro::computeNewDt() 5 3.901e-06 3.901e-06 3.901e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.82e-06 2.82e-06 2.82e-06 0.00% MLMG::getGradSolution() 6 2.756e-06 2.756e-06 2.756e-06 0.00% Castro::expand_state() 5 2.687e-06 2.687e-06 2.687e-06 0.00% Castro::retry_advance_ctu() 5 2.61e-06 2.61e-06 2.61e-06 0.00% MLMG::MLRhsNormInf() 6 2.434e-06 2.434e-06 2.434e-06 0.00% MLMG::MLResNormInf() 6 2.38e-06 2.38e-06 2.38e-06 0.00% Gravity::set_mass_offset() 6 2.129e-06 2.129e-06 2.129e-06 0.00% Castro::FluxRegCrseInit 5 1.615e-06 1.615e-06 1.615e-06 0.00% Castro::FluxRegFineAdd() 5 1.246e-06 1.246e-06 1.246e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.071e-06 1.071e-06 1.071e-06 0.00% Amr::init() 1 8.04e-07 8.04e-07 8.04e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4355 0.4355 0.4355 100.00% Amr::coarseTimeStep() 5 0.3274 0.3274 0.3274 75.18% Amr::timeStep() 5 0.325 0.325 0.325 74.62% Castro::advance() 5 0.3208 0.3208 0.3208 73.65% Castro::subcycle_advance_ctu() 5 0.3136 0.3136 0.3136 72.00% Castro::do_advance_ctu() 5 0.3123 0.3123 0.3123 71.71% Castro::construct_ctu_hydro_source() 5 0.1408 0.1408 0.1408 32.32% Castro::construct_new_gravity() 5 0.1322 0.1322 0.1322 30.34% Gravity::solve_phi_with_mlmg() 6 0.127 0.127 0.127 29.15% Gravity::solve_for_phi() 5 0.1215 0.1215 0.1215 27.90% Gravity::actual_solve_with_mlmg() 6 0.1214 0.1214 0.1214 27.88% MLMG::solve() 6 0.1102 0.1102 0.1102 25.31% MLMG::oneIter() 36 0.103 0.103 0.103 23.65% MLMG::mgVcycle() 36 0.1015 0.1015 0.1015 23.30% Amr::writePlotFile() 1 0.0584 0.0584 0.0584 13.41% MLCellLinOp::smooth() 720 0.04961 0.04961 0.04961 11.39% Amr::init() 1 0.04902 0.04902 0.04902 11.25% Amr::restart() 1 0.04902 0.04902 0.04902 11.25% MLCellLinOp::applyBC() 1946 0.04662 0.04662 0.04662 10.70% AmrLevel::restart() 1 0.03808 0.03808 0.03808 8.74% StateData::restartDoit() 4 0.038 0.038 0.038 8.73% VisMF::Read() 3 0.03774 0.03774 0.03774 8.67% MLMG::mgVcycle_bottom 36 0.03162 0.03162 0.03162 7.26% MLMG::actualBottomSolve() 36 0.0316 0.0316 0.0316 7.26% MLCGSolver::bicgstab 36 0.0313 0.0313 0.0313 7.19% AmrLevel::FillPatch() 20 0.02991 0.02991 0.02991 6.87% FillPatchIterator::Initialize 20 0.02786 0.02786 0.02786 6.40% FillPatchIterator::FillFromLevel0() 20 0.0271 0.0271 0.0271 6.22% FillPatchSingleLevel 20 0.02708 0.02708 0.02708 6.22% VisMF::Write(FabArray) 1 0.02701 0.02701 0.02701 6.20% StateDataPhysBCFunct::() 20 0.02499 0.02499 0.02499 5.74% Castro::clean_state() 30 0.02391 0.02391 0.02391 5.49% MLCellLinOp::apply() 500 0.01564 0.01564 0.01564 3.59% MLMG::mgVcycle_down::0 36 0.0145 0.0145 0.0145 3.33% FabArray::FillBoundary() 1766 0.01381 0.01381 0.01381 3.17% MLPoisson::Fsmooth() 1440 0.01377 0.01377 0.01377 3.16% FillBoundary_nowait() 1766 0.01346 0.01346 0.01346 3.09% Castro::initialize_do_advance() 5 0.01155 0.01155 0.01155 2.65% StateData::FillBoundary(geom) 160 0.01136 0.01136 0.01136 2.61% MLMG::mgVcycle_up::0 36 0.01084 0.01084 0.01084 2.49% Gravity::get_new_grav_vector() 5 0.01055 0.01055 0.01055 2.42% Castro::construct_old_gravity() 5 0.01015 0.01015 0.01015 2.33% Gravity::get_old_grav_vector() 5 0.01014 0.01014 0.01014 2.33% Castro::do_old_sources() 5 0.009692 0.009692 0.009692 2.23% MLCellLinOp::correctionResidual() 216 0.009437 0.009437 0.009437 2.17% Castro::computeTemp() 30 0.009366 0.009366 0.009366 2.15% Castro::normalize_species() 30 0.009027 0.009027 0.009027 2.07% MLPoisson::define() 6 0.008763 0.008763 0.008763 2.01% amrex::Dot() 484 0.008729 0.008729 0.008729 2.00% MLMG:computeResOfCorrection() 180 0.008316 0.008316 0.008316 1.91% amrex::Copy() 463 0.007096 0.007096 0.007096 1.63% MLMG::mgVcycle_down::1 36 0.006844 0.006844 0.006844 1.57% Castro::initialize_advance() 5 0.006831 0.006831 0.006831 1.57% Castro::expand_state() 5 0.006655 0.006655 0.006655 1.53% Castro::do_new_sources() 5 0.006547 0.006547 0.006547 1.50% FabArray::ParallelCopy() 380 0.006518 0.006518 0.006518 1.50% FabArray::ParallelCopy_nowait() 380 0.006397 0.006397 0.006397 1.47% FabArray::setVal() 537 0.006332 0.006332 0.006332 1.45% MLMG::mgVcycle_down::2 36 0.006258 0.006258 0.006258 1.44% FabArray::norminf() 326 0.006179 0.006179 0.006179 1.42% Castro::post_restart() 1 0.006083 0.006083 0.006083 1.40% MLMG::mgVcycle_down::3 36 0.006071 0.006071 0.006071 1.39% MLMG::mgVcycle_down::4 36 0.006014 0.006014 0.006014 1.38% MLCellLinOp::defineAuxData() 6 0.005957 0.005957 0.005957 1.37% Gravity::multilevel_solve_for_new_phi() 1 0.005728 0.005728 0.005728 1.32% Gravity::actual_multilevel_solve() 1 0.005711 0.005711 0.005711 1.31% Gravity::fill_multipole_BCs() 6 0.005449 0.005449 0.005449 1.25% MLCGSolver::ParallelAllReduce 659 0.005295 0.005295 0.005295 1.22% MLMG::addInterpCorrection() 180 0.005188 0.005188 0.005188 1.19% Castro::enforce_min_density() 30 0.005017 0.005017 0.005017 1.15% MLMG::mgVcycle_up::4 36 0.004912 0.004912 0.004912 1.13% MLMG::mgVcycle_up::1 36 0.004876 0.004876 0.004876 1.12% amrex::average_down 180 0.004826 0.004826 0.004826 1.11% MLMG::mgVcycle_up::2 36 0.004769 0.004769 0.004769 1.09% MLMG::mgVcycle_up::3 36 0.004691 0.004691 0.004691 1.08% MLPoisson::Fapply() 500 0.004418 0.004418 0.004418 1.01% Castro::post_timestep() 5 0.004158 0.004158 0.004158 0.95% FabArray::Saxpy() 355 0.003663 0.003663 0.003663 0.84% FabArray::Xpay() 361 0.003493 0.003493 0.003493 0.80% MLCellLinOp::solutionResidual() 42 0.003389 0.003389 0.003389 0.78% Castro::estTimeStep() 10 0.003175 0.003175 0.003175 0.73% Castro::reset_internal_energy(MultiFab) 30 0.003015 0.003015 0.003015 0.69% MLCellLinOp::defineBC() 6 0.002672 0.002672 0.002672 0.61% MLMG::prepareForSolve() 6 0.002664 0.002664 0.002664 0.61% MLMG::computeResidual() 36 0.002646 0.002646 0.002646 0.61% BndryData::define() 6 0.002554 0.002554 0.002554 0.59% Castro::computeNewDt() 5 0.002242 0.002242 0.002242 0.51% FabArray::LinComb() 242 0.002195 0.002195 0.002195 0.50% amrex::Add() 72 0.00187 0.00187 0.00187 0.43% Castro::construct_new_source() 25 0.001816 0.001816 0.001816 0.42% Castro::construct_new_gravity_source() 5 0.001718 0.001718 0.001718 0.39% Castro::construct_old_source() 25 0.001453 0.001453 0.001453 0.33% Castro::construct_old_gravity_source() 5 0.001399 0.001399 0.001399 0.32% Castro::finalize_do_advance() 5 0.0009827 0.0009827 0.0009827 0.23% MLMG::ResNormInf() 42 0.0009763 0.0009763 0.0009763 0.22% Castro::apply_source_to_state() 10 0.0009553 0.0009553 0.0009553 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009203 0.0009203 0.0009203 0.21% check_for_negative_density() 5 0.0008695 0.0008695 0.0008695 0.20% MLMG::getGradSolution() 6 0.0008444 0.0008444 0.0008444 0.19% MLCellLinOp::compGrad() 6 0.0008417 0.0008417 0.0008417 0.19% Castro::reset_internal_energy(Fab) 240 0.0007927 0.0007927 0.0007927 0.18% MLCellLinOp::setLevelBC() 6 0.0007714 0.0007714 0.0007714 0.18% MLMG::computeMLResidual() 6 0.000763 0.000763 0.000763 0.18% FabArrayBase::getCPC() 632 0.0007519 0.0007519 0.0007519 0.17% MLPoisson::prepareForSolve() 6 0.0006991 0.0006991 0.0006991 0.16% Gravity::update_max_rhs() 6 0.0006907 0.0006907 0.0006907 0.16% MLCellLinOp::prepareForSolve() 6 0.0006904 0.0006904 0.0006904 0.16% FabArray::mult() 22 0.0006785 0.0006785 0.0006785 0.16% FabArray::setDomainBndry() 20 0.0006606 0.0006606 0.0006606 0.15% Castro::check_for_nan() 10 0.0006593 0.0006593 0.0006593 0.15% MultiFab::contains_nan() 10 0.0006517 0.0006517 0.0006517 0.15% Castro::enforce_speed_limit() 30 0.0004633 0.0004633 0.0004633 0.11% FabArrayBase::CPC::define() 244 0.0004042 0.0004042 0.0004042 0.09% Amr::InitAmr() 1 0.0003995 0.0003995 0.0003995 0.09% FabArrayBase::getFB() 1766 0.0003194 0.0003194 0.0003194 0.07% Castro::finalize_advance() 5 0.0002902 0.0002902 0.0002902 0.07% Gravity::swapTimeLevels() 5 0.00023 0.00023 0.00023 0.05% MultiFab::max() 6 0.0001987 0.0001987 0.0001987 0.05% MLMG::MLResNormInf() 6 0.0001849 0.0001849 0.0001849 0.04% Castro::buildMetrics() 1 0.0001448 0.0001448 0.0001448 0.03% MLLinOp::define() 6 0.0001196 0.0001196 0.0001196 0.03% MLMG::MLRhsNormInf() 6 0.0001164 0.0001164 0.0001164 0.03% MLLinOp::defineGrids() 6 0.0001142 0.0001142 0.0001142 0.03% Castro::create_source_corrector() 5 7.954e-05 7.954e-05 7.954e-05 0.02% FabArrayBase::FB::FB() 26 5.657e-05 5.657e-05 5.657e-05 0.01% Amr::writeSmallPlotFile() 1 2.796e-05 2.796e-05 2.796e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.759e-05 2.759e-05 2.759e-05 0.01% Castro::swap_state_time_levels() 5 2.479e-05 2.479e-05 2.479e-05 0.01% Castro::initMFs() 1 2.366e-05 2.366e-05 2.366e-05 0.01% makeSFC 30 2.066e-05 2.066e-05 2.066e-05 0.00% DistributionMapping::Distribute() 31 9.23e-06 9.23e-06 9.23e-06 0.00% Amr::initSubcycle() 1 8.799e-06 8.799e-06 8.799e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.229e-06 4.229e-06 4.229e-06 0.00% Castro::retry_advance_ctu() 5 2.61e-06 2.61e-06 2.61e-06 0.00% Gravity::set_mass_offset() 6 2.129e-06 2.129e-06 2.129e-06 0.00% Castro::FluxRegCrseInit 5 1.615e-06 1.615e-06 1.615e-06 0.00% Castro::FluxRegFineAdd() 5 1.246e-06 1.246e-06 1.246e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.071e-06 1.071e-06 1.071e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 191 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 139 MiB 692 MiB Castro::initMFs() 48 48 61 MiB 68 MiB Castro::swap_state_time_levels() 32 32 49 MiB 55 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB FillPatchIterator::Initialize 160 160 1292 KiB 39 MiB Castro::initialize_do_advance() 40 40 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1840 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6823 KiB 14 MiB MLMG::prepareForSolve() 361 361 3107 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 241 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 248 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6810 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 21 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3147 B 2048 KiB Gravity::solve_for_phi() 40 40 570 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 26 KiB 2048 KiB BndryData::define() 576 576 286 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 183 KiB 671 KiB Castro::estTimeStep() 10 10 3466 B 480 KiB VisMF::Write(FabArray) 112 112 1243 B 320 KiB Castro::normalize_species() 30 30 6747 B 320 KiB amrex::average_down 469 469 1109 B 257 KiB MLMG::addInterpCorrection() 468 468 1019 B 257 KiB amrex::Dot() 592 592 2942 B 160 KiB FabArray::norminf() 398 398 2091 B 160 KiB check_for_negative_density() 5 5 312 B 160 KiB MultiFab::max() 6 6 71 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3013 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 44 B 10 KiB FillBoundary_nowait() 336 336 263 B 9648 B MLCellLinOp::applyBC() 3892 3892 197 B 9344 B amrex::Copy() 56 56 5885 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 99 B 4288 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCGSolver::bicgstab 324 324 104 B 1472 B MLCellLinOp::defineBC() 36 36 321 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1183 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 81 KiB 8192 KiB VisMF::Write(FabArray) 120 120 177 KiB 3584 KiB VisMF::Read() 24 24 168 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3013 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 44 B 10 KiB FillBoundary_nowait() 336 336 263 B 9648 B MLCellLinOp::applyBC() 1946 1946 196 B 9328 B amrex::Copy() 56 56 5885 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 99 B 4288 B Gravity::get_old_grav_vector() 3 3 2675 B 3072 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 500 B 1296 B amrex::average_down 37 37 225 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 21 B 400 B FabArray::norminf() 398 398 15 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2528 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-31-g435281069858) finalized