Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-18-gd6f8b2cce0fb) initialized Starting run at 10:13:34 UTC on 2023-01-16. Successfully read inputs file ... Castro git describe: 23.01-14-g78178b78e AMReX git describe: 23.01-18-gd6f8b2cce Microphysics git describe: 23.01-3-g1e475055 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.059409093 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.034131101 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048694736 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049738639 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.058495074 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059966293 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.063133742 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057641458 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.064794216 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.064378923 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059477913 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059279283 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061811548 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057775584 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033715297 seconds Ending run at 10:13:35 UTC on 2023-01-16. Run time = 0.882159241 Run time without initialization = 0.739552371 Average number of zones advanced per microsecond: 3.545 Average number of zones advanced per microsecond per rank: 3.545 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8822 ... 0.8822 ... 0.8822 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2359 0.2359 0.2359 26.74% Castro::construct_ctu_hydro_source() 10 0.2182 0.2182 0.2182 24.73% MLCellLinOp::applyBC() 4433 0.07713 0.07713 0.07713 8.74% MLPoisson::Fsmooth() 3280 0.03268 0.03268 0.03268 3.70% FillBoundary_nowait() 4023 0.03188 0.03188 0.03188 3.61% StateData::FillBoundary(geom) 328 0.02378 0.02378 0.02378 2.70% amrex::Dot() 1114 0.02058 0.02058 0.02058 2.33% StateDataPhysBCFunct::() 41 0.01792 0.01792 0.01792 2.03% Castro::normalize_species() 62 0.01586 0.01586 0.01586 1.80% Castro::computeTemp() 63 0.01545 0.01545 0.01545 1.75% amrex::Copy() 1029 0.01484 0.01484 0.01484 1.68% FabArray::norminf() 743 0.01445 0.01445 0.01445 1.64% Castro::enforce_min_density() 62 0.01412 0.01412 0.01412 1.60% FabArray::ParallelCopy_nowait() 861 0.01329 0.01329 0.01329 1.51% FabArray::setVal() 1144 0.01319 0.01319 0.01319 1.49% MLPoisson::Fapply() 1142 0.01047 0.01047 0.01047 1.19% MLCellLinOp::defineAuxData() 11 0.009684 0.009684 0.009684 1.10% FabArray::Saxpy() 813 0.008146 0.008146 0.008146 0.92% FabArray::Xpay() 821 0.008146 0.008146 0.008146 0.92% Gravity::fill_multipole_BCs() 11 0.00669 0.00669 0.00669 0.76% MLMG::addInterpCorrection() 410 0.006629 0.006629 0.006629 0.75% amrex::average_down 410 0.005844 0.005844 0.005844 0.66% FabArray::LinComb() 557 0.004593 0.004593 0.004593 0.52% Castro::estTimeStep() 21 0.004591 0.004591 0.004591 0.52% Castro::reset_internal_energy(MultiFab) 63 0.004496 0.004496 0.004496 0.51% amrex::Add() 164 0.004325 0.004325 0.004325 0.49% Amr::checkPoint() 3 0.003988 0.003988 0.003988 0.45% BndryData::define() 11 0.003687 0.003687 0.003687 0.42% Castro::do_advance_ctu() 10 0.003622 0.003622 0.003622 0.41% Castro::construct_new_gravity_source() 10 0.003295 0.003295 0.003295 0.37% Castro::construct_old_gravity_source() 10 0.002766 0.002766 0.002766 0.31% Amr::writePlotFile() 2 0.002374 0.002374 0.002374 0.27% MLCGSolver::bicgstab 82 0.00209 0.00209 0.00209 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001677 0.001677 0.001677 0.19% Castro::reset_internal_energy(Fab) 504 0.001549 0.001549 0.001549 0.18% Gravity::actual_solve_with_mlmg() 11 0.001429 0.001429 0.001429 0.16% MLCellLinOp::setLevelBC() 11 0.001392 0.001392 0.001392 0.16% FabArray::mult() 43 0.001339 0.001339 0.001339 0.15% FabArray::setDomainBndry() 41 0.001309 0.001309 0.001309 0.15% Castro::initData() 1 0.001257 0.001257 0.001257 0.14% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.13% MLCellLinOp::prepareForSolve() 11 0.001126 0.001126 0.001126 0.13% MLCellLinOp::smooth() 1640 0.001119 0.001119 0.001119 0.13% Castro::enforce_speed_limit() 62 0.000983 0.000983 0.000983 0.11% MLCellLinOp::compGrad() 11 0.0009183 0.0009183 0.0009183 0.10% MLMG::prepareForSolve() 11 0.0008413 0.0008413 0.0008413 0.10% FabArray::FillBoundary() 4023 0.0008006 0.0008006 0.0008006 0.09% Castro::subcycle_advance_ctu() 10 0.00077 0.00077 0.00077 0.09% FabArrayBase::getCPC() 1323 0.0007246 0.0007246 0.0007246 0.08% FabArrayBase::CPC::define() 454 0.0006963 0.0006963 0.0006963 0.08% FabArrayBase::getFB() 4023 0.0006076 0.0006076 0.0006076 0.07% Gravity::get_new_grav_vector() 11 0.0006058 0.0006058 0.0006058 0.07% Gravity::get_old_grav_vector() 10 0.000526 0.000526 0.000526 0.06% Amr::InitAmr() 1 0.0004778 0.0004778 0.0004778 0.05% MLCellLinOp::apply() 1142 0.0004539 0.0004539 0.0004539 0.05% MLMG::mgVcycle() 82 0.0003989 0.0003989 0.0003989 0.05% Amr::coarseTimeStep() 10 0.000332 0.000332 0.000332 0.04% main() 1 0.0003028 0.0003028 0.0003028 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002781 0.0002781 0.0002781 0.03% MultiFab::max() 11 0.0002597 0.0002597 0.0002597 0.03% Castro::construct_new_source() 50 0.0002494 0.0002494 0.0002494 0.03% FabArray::ParallelCopy() 861 0.00024 0.00024 0.00024 0.03% MLCellLinOp::correctionResidual() 492 0.0002192 0.0002192 0.0002192 0.02% MLCellLinOp::defineBC() 11 0.0002008 0.0002008 0.0002008 0.02% FillPatchIterator::Initialize 41 0.0001944 0.0001944 0.0001944 0.02% MLLinOp::defineGrids() 11 0.0001679 0.0001679 0.0001679 0.02% Amr::timeStep() 10 0.0001631 0.0001631 0.0001631 0.02% StateData::checkPoint() 12 0.0001349 0.0001349 0.0001349 0.02% Gravity::solve_for_phi() 10 0.0001316 0.0001316 0.0001316 0.01% MLMG:computeResOfCorrection() 410 0.0001135 0.0001135 0.0001135 0.01% Gravity::update_max_rhs() 11 0.0001051 0.0001051 0.0001051 0.01% MLMG::mgVcycle_down::0 82 9.462e-05 9.462e-05 9.462e-05 0.01% MLMG::actualBottomSolve() 82 8.763e-05 8.763e-05 8.763e-05 0.01% Castro::clean_state() 62 8.493e-05 8.493e-05 8.493e-05 0.01% FabArrayBase::FB::FB() 56 8.189e-05 8.189e-05 8.189e-05 0.01% Castro::Castro() 1 7.976e-05 7.976e-05 7.976e-05 0.01% MLMG::mgVcycle_down::1 82 7.555e-05 7.555e-05 7.555e-05 0.01% AmrLevel::checkPoint() 3 7.496e-05 7.496e-05 7.496e-05 0.01% Castro::expand_state() 10 7.461e-05 7.461e-05 7.461e-05 0.01% MLMG::mgVcycle_down::2 82 7.212e-05 7.212e-05 7.212e-05 0.01% MLMG::solve() 11 7.056e-05 7.056e-05 7.056e-05 0.01% MLMG::mgVcycle_down::4 82 6.957e-05 6.957e-05 6.957e-05 0.01% MLMG::mgVcycle_down::3 82 6.924e-05 6.924e-05 6.924e-05 0.01% Castro::finalize_advance() 10 6.563e-05 6.563e-05 6.563e-05 0.01% Castro::initialize_advance() 10 6.252e-05 6.252e-05 6.252e-05 0.01% Castro::advance() 10 5.974e-05 5.974e-05 5.974e-05 0.01% MLMG::mgVcycle_up::4 82 5.601e-05 5.601e-05 5.601e-05 0.01% Castro::initialize_do_advance() 10 5.399e-05 5.399e-05 5.399e-05 0.01% MLMG::oneIter() 82 5.368e-05 5.368e-05 5.368e-05 0.01% MLMG::mgVcycle_up::0 82 5.062e-05 5.062e-05 5.062e-05 0.01% MLCellLinOp::solutionResidual() 93 4.895e-05 4.895e-05 4.895e-05 0.01% MLMG::mgVcycle_up::1 82 4.787e-05 4.787e-05 4.787e-05 0.01% MLMG::mgVcycle_up::3 82 4.77e-05 4.77e-05 4.77e-05 0.01% MLMG::mgVcycle_up::2 82 4.585e-05 4.585e-05 4.585e-05 0.01% Amr::InitializeInit() 1 4.508e-05 4.508e-05 4.508e-05 0.01% Castro::swap_state_time_levels() 10 3.385e-05 3.385e-05 3.385e-05 0.00% Castro::enforce_consistent_e() 1 3.379e-05 3.379e-05 3.379e-05 0.00% MLMG::ResNormInf() 93 3.278e-05 3.278e-05 3.278e-05 0.00% Castro::finalize_do_advance() 10 3.27e-05 3.27e-05 3.27e-05 0.00% MLMG::mgVcycle_bottom 82 3.033e-05 3.033e-05 3.033e-05 0.00% Amr::writeSmallPlotFile() 1 2.978e-05 2.978e-05 2.978e-05 0.00% MLMG::computeResidual() 82 2.831e-05 2.831e-05 2.831e-05 0.00% FillPatchSingleLevel 41 2.781e-05 2.781e-05 2.781e-05 0.00% StateData::define() 4 2.651e-05 2.651e-05 2.651e-05 0.00% makeSFC 55 2.483e-05 2.483e-05 2.483e-05 0.00% Castro::construct_new_gravity() 10 2.348e-05 2.348e-05 2.348e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.23e-05 2.23e-05 2.23e-05 0.00% MLPoisson::define() 11 2.18e-05 2.18e-05 2.18e-05 0.00% Amr::FinalizeInit() 1 2.009e-05 2.009e-05 2.009e-05 0.00% Castro::initMFs() 1 1.889e-05 1.889e-05 1.889e-05 0.00% Castro::construct_old_source() 50 1.834e-05 1.834e-05 1.834e-05 0.00% Amr::defBaseLevel() 1 1.787e-05 1.787e-05 1.787e-05 0.00% Castro::do_new_sources() 10 1.744e-05 1.744e-05 1.744e-05 0.00% Castro::buildMetrics() 1 1.671e-05 1.671e-05 1.671e-05 0.00% DistributionMapping::Distribute() 56 1.552e-05 1.552e-05 1.552e-05 0.00% Castro::do_old_sources() 10 1.503e-05 1.503e-05 1.503e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.435e-05 1.435e-05 1.435e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.423e-05 1.423e-05 1.423e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.335e-05 1.335e-05 1.335e-05 0.00% Castro::check_for_nan() 20 1.179e-05 1.179e-05 1.179e-05 0.00% Castro::apply_source_to_state() 20 1.032e-05 1.032e-05 1.032e-05 0.00% Castro::post_timestep() 10 1.024e-05 1.024e-05 1.024e-05 0.00% Castro::post_init() 1 1.002e-05 1.002e-05 1.002e-05 0.00% Castro::create_source_corrector() 10 9.875e-06 9.875e-06 9.875e-06 0.00% Castro::construct_old_gravity() 10 9.726e-06 9.726e-06 9.726e-06 0.00% MLLinOp::define() 11 9.588e-06 9.588e-06 9.588e-06 0.00% Gravity::swapTimeLevels() 10 9.466e-06 9.466e-06 9.466e-06 0.00% Amr::initSubcycle() 1 8.647e-06 8.647e-06 8.647e-06 0.00% MLPoisson::prepareForSolve() 11 8.271e-06 8.271e-06 8.271e-06 0.00% MLMG::computeMLResidual() 11 7.497e-06 7.497e-06 7.497e-06 0.00% Gravity::actual_multilevel_solve() 1 7.474e-06 7.474e-06 7.474e-06 0.00% Castro::computeNewDt() 9 6.257e-06 6.257e-06 6.257e-06 0.00% MLMG::getGradSolution() 11 5.686e-06 5.686e-06 5.686e-06 0.00% Castro::retry_advance_ctu() 10 4.851e-06 4.851e-06 4.851e-06 0.00% AmrLevel::checkPointPost() 3 4.17e-06 4.17e-06 4.17e-06 0.00% Gravity::set_mass_offset() 11 3.969e-06 3.969e-06 3.969e-06 0.00% MLMG::MLRhsNormInf() 11 3.894e-06 3.894e-06 3.894e-06 0.00% MLMG::MLResNormInf() 11 3.743e-06 3.743e-06 3.743e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.224e-06 3.224e-06 3.224e-06 0.00% Castro::computeInitialDt() 2 2.819e-06 2.819e-06 2.819e-06 0.00% Castro::FluxRegCrseInit 10 2.686e-06 2.686e-06 2.686e-06 0.00% Amr::init() 1 2.572e-06 2.572e-06 2.572e-06 0.00% Castro::FluxRegFineAdd() 10 2.3e-06 2.3e-06 2.3e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.052e-06 2.052e-06 2.052e-06 0.00% AmrLevel::checkPointPre() 3 1.952e-06 1.952e-06 1.952e-06 0.00% Castro::post_regrid() 1 1.564e-06 1.564e-06 1.564e-06 0.00% Amr::initialInit() 1 1.091e-06 1.091e-06 1.091e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8822 0.8822 0.8822 100.00% Amr::coarseTimeStep() 10 0.7056 0.7056 0.7056 79.98% Amr::timeStep() 10 0.5869 0.5869 0.5869 66.52% Castro::advance() 10 0.5788 0.5788 0.5788 65.61% Castro::subcycle_advance_ctu() 10 0.5664 0.5664 0.5664 64.20% Castro::do_advance_ctu() 10 0.5656 0.5656 0.5656 64.12% Gravity::solve_phi_with_mlmg() 11 0.2848 0.2848 0.2848 32.28% Gravity::actual_solve_with_mlmg() 11 0.2776 0.2776 0.2776 31.47% Castro::construct_new_gravity() 10 0.2595 0.2595 0.2595 29.42% MLMG::solve() 11 0.2573 0.2573 0.2573 29.16% Gravity::solve_for_phi() 10 0.2444 0.2444 0.2444 27.71% MLMG::oneIter() 82 0.2429 0.2429 0.2429 27.54% MLMG::mgVcycle() 82 0.2393 0.2393 0.2393 27.13% VisMF::Write(FabArray) 11 0.2359 0.2359 0.2359 26.74% Castro::construct_ctu_hydro_source() 10 0.2182 0.2182 0.2182 24.73% Amr::checkPoint() 3 0.175 0.175 0.175 19.84% AmrLevel::checkPoint() 3 0.171 0.171 0.171 19.38% StateData::checkPoint() 12 0.1709 0.1709 0.1709 19.37% Amr::init() 1 0.142 0.142 0.142 16.09% MLCellLinOp::smooth() 1640 0.1182 0.1182 0.1182 13.40% MLCellLinOp::applyBC() 4433 0.1105 0.1105 0.1105 12.53% MLMG::mgVcycle_bottom 82 0.07388 0.07388 0.07388 8.37% MLMG::actualBottomSolve() 82 0.07385 0.07385 0.07385 8.37% MLCGSolver::bicgstab 82 0.07315 0.07315 0.07315 8.29% Amr::writePlotFile() 2 0.06797 0.06797 0.06797 7.70% Castro::clean_state() 62 0.0516 0.0516 0.0516 5.85% Amr::initialInit() 1 0.04831 0.04831 0.04831 5.48% FillPatchIterator::Initialize 41 0.04718 0.04718 0.04718 5.35% FillPatchSingleLevel 41 0.04568 0.04568 0.04568 5.18% Amr::FinalizeInit() 1 0.04404 0.04404 0.04404 4.99% Castro::post_init() 1 0.04262 0.04262 0.04262 4.83% StateDataPhysBCFunct::() 41 0.0417 0.0417 0.0417 4.73% Gravity::multilevel_solve_for_new_phi() 1 0.0408 0.0408 0.0408 4.62% Gravity::actual_multilevel_solve() 1 0.04078 0.04078 0.04078 4.62% MLCellLinOp::apply() 1142 0.03656 0.03656 0.03656 4.14% MLMG::mgVcycle_down::0 82 0.03424 0.03424 0.03424 3.88% FabArray::FillBoundary() 4023 0.03337 0.03337 0.03337 3.78% MLPoisson::Fsmooth() 3280 0.03268 0.03268 0.03268 3.70% FillBoundary_nowait() 4023 0.03257 0.03257 0.03257 3.69% Castro::initialize_do_advance() 10 0.02624 0.02624 0.02624 2.97% MLMG::mgVcycle_up::0 82 0.02592 0.02592 0.02592 2.94% StateData::FillBoundary(geom) 328 0.02378 0.02378 0.02378 2.70% MLCellLinOp::correctionResidual() 492 0.02241 0.02241 0.02241 2.54% Castro::computeTemp() 63 0.02149 0.02149 0.02149 2.44% amrex::Dot() 1114 0.02058 0.02058 0.02058 2.33% MLMG:computeResOfCorrection() 410 0.01974 0.01974 0.01974 2.24% Gravity::get_new_grav_vector() 11 0.01662 0.01662 0.01662 1.88% Castro::expand_state() 10 0.01641 0.01641 0.01641 1.86% MLPoisson::define() 11 0.01612 0.01612 0.01612 1.83% Castro::normalize_species() 62 0.01586 0.01586 0.01586 1.80% MLMG::mgVcycle_down::1 82 0.01584 0.01584 0.01584 1.80% Castro::construct_old_gravity() 10 0.01495 0.01495 0.01495 1.69% Gravity::get_old_grav_vector() 10 0.01494 0.01494 0.01494 1.69% amrex::Copy() 1029 0.01484 0.01484 0.01484 1.68% MLMG::mgVcycle_down::2 82 0.01472 0.01472 0.01472 1.67% MLMG::mgVcycle_down::3 82 0.0145 0.0145 0.0145 1.64% FabArray::norminf() 743 0.01445 0.01445 0.01445 1.64% FabArray::ParallelCopy() 861 0.01432 0.01432 0.01432 1.62% MLMG::mgVcycle_down::4 82 0.01429 0.01429 0.01429 1.62% Castro::enforce_min_density() 62 0.01412 0.01412 0.01412 1.60% FabArray::ParallelCopy_nowait() 861 0.01408 0.01408 0.01408 1.60% Castro::do_new_sources() 10 0.01321 0.01321 0.01321 1.50% FabArray::setVal() 1144 0.01319 0.01319 0.01319 1.49% MLCGSolver::ParallelAllReduce 1514 0.01235 0.01235 0.01235 1.40% Castro::do_old_sources() 10 0.01206 0.01206 0.01206 1.37% MLMG::addInterpCorrection() 410 0.01185 0.01185 0.01185 1.34% Castro::initialize_advance() 10 0.01173 0.01173 0.01173 1.33% MLMG::mgVcycle_up::4 82 0.01163 0.01163 0.01163 1.32% MLMG::mgVcycle_up::1 82 0.01154 0.01154 0.01154 1.31% MLMG::mgVcycle_up::2 82 0.01126 0.01126 0.01126 1.28% MLMG::mgVcycle_up::3 82 0.01109 0.01109 0.01109 1.26% MLCellLinOp::defineAuxData() 11 0.01102 0.01102 0.01102 1.25% amrex::average_down 410 0.011 0.011 0.011 1.25% MLPoisson::Fapply() 1142 0.01047 0.01047 0.01047 1.19% FabArray::Saxpy() 813 0.008146 0.008146 0.008146 0.92% FabArray::Xpay() 821 0.008146 0.008146 0.008146 0.92% Castro::post_timestep() 10 0.007892 0.007892 0.007892 0.89% MLCellLinOp::solutionResidual() 93 0.007214 0.007214 0.007214 0.82% Gravity::fill_multipole_BCs() 11 0.006935 0.006935 0.006935 0.79% MLMG::computeResidual() 82 0.006229 0.006229 0.006229 0.71% Castro::reset_internal_energy(MultiFab) 63 0.006045 0.006045 0.006045 0.69% MLCellLinOp::defineBC() 11 0.004853 0.004853 0.004853 0.55% MLMG::prepareForSolve() 11 0.004662 0.004662 0.004662 0.53% BndryData::define() 11 0.004652 0.004652 0.004652 0.53% FabArray::LinComb() 557 0.004593 0.004593 0.004593 0.52% Castro::estTimeStep() 21 0.004591 0.004591 0.004591 0.52% amrex::Add() 164 0.004325 0.004325 0.004325 0.49% Amr::InitializeInit() 1 0.004269 0.004269 0.004269 0.48% Amr::defBaseLevel() 1 0.004223 0.004223 0.004223 0.48% Castro::initData() 1 0.003741 0.003741 0.003741 0.42% Castro::construct_new_source() 50 0.003544 0.003544 0.003544 0.40% Castro::construct_new_gravity_source() 10 0.003295 0.003295 0.003295 0.37% Castro::construct_old_source() 50 0.002784 0.002784 0.002784 0.32% Castro::construct_old_gravity_source() 10 0.002766 0.002766 0.002766 0.31% Castro::computeNewDt() 9 0.002191 0.002191 0.002191 0.25% MLMG::ResNormInf() 93 0.002137 0.002137 0.002137 0.24% Castro::apply_source_to_state() 20 0.001827 0.001827 0.001827 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001677 0.001677 0.001677 0.19% Castro::reset_internal_energy(Fab) 504 0.001549 0.001549 0.001549 0.18% FabArrayBase::getCPC() 1323 0.001421 0.001421 0.001421 0.16% MLMG::getGradSolution() 11 0.0014 0.0014 0.0014 0.16% MLCellLinOp::compGrad() 11 0.001394 0.001394 0.001394 0.16% MLCellLinOp::setLevelBC() 11 0.001392 0.001392 0.001392 0.16% FabArray::mult() 43 0.001339 0.001339 0.001339 0.15% FabArray::setDomainBndry() 41 0.001309 0.001309 0.001309 0.15% Castro::check_for_nan() 20 0.001187 0.001187 0.001187 0.13% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.13% Castro::post_regrid() 1 0.001152 0.001152 0.001152 0.13% MLPoisson::prepareForSolve() 11 0.001135 0.001135 0.001135 0.13% MLCellLinOp::prepareForSolve() 11 0.001126 0.001126 0.001126 0.13% MLMG::computeMLResidual() 11 0.001021 0.001021 0.001021 0.12% Castro::enforce_speed_limit() 62 0.000983 0.000983 0.000983 0.11% Castro::computeInitialDt() 2 0.0009522 0.0009522 0.0009522 0.11% Gravity::update_max_rhs() 11 0.0008178 0.0008178 0.0008178 0.09% FabArrayBase::CPC::define() 454 0.0006963 0.0006963 0.0006963 0.08% FabArrayBase::getFB() 4023 0.0006895 0.0006895 0.0006895 0.08% Castro::finalize_advance() 10 0.0006004 0.0006004 0.0006004 0.07% Amr::InitAmr() 1 0.0004864 0.0004864 0.0004864 0.06% Gravity::swapTimeLevels() 10 0.0004375 0.0004375 0.0004375 0.05% Castro::Castro() 1 0.0004196 0.0004196 0.0004196 0.05% MLMG::MLResNormInf() 11 0.0002858 0.0002858 0.0002858 0.03% MultiFab::max() 11 0.0002597 0.0002597 0.0002597 0.03% MLLinOp::define() 11 0.0002321 0.0002321 0.0002321 0.03% MLLinOp::defineGrids() 11 0.0002225 0.0002225 0.0002225 0.03% MLMG::MLRhsNormInf() 11 0.0002199 0.0002199 0.0002199 0.02% Castro::buildMetrics() 1 0.0001581 0.0001581 0.0001581 0.02% FabArrayBase::FB::FB() 56 8.189e-05 8.189e-05 8.189e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.254e-05 5.254e-05 5.254e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.075e-05 4.075e-05 4.075e-05 0.00% makeSFC 55 3.919e-05 3.919e-05 3.919e-05 0.00% Castro::swap_state_time_levels() 10 3.385e-05 3.385e-05 3.385e-05 0.00% Castro::enforce_consistent_e() 1 3.379e-05 3.379e-05 3.379e-05 0.00% Castro::finalize_do_advance() 10 3.27e-05 3.27e-05 3.27e-05 0.00% Amr::writeSmallPlotFile() 1 2.978e-05 2.978e-05 2.978e-05 0.00% StateData::define() 4 2.651e-05 2.651e-05 2.651e-05 0.00% Castro::initMFs() 1 1.889e-05 1.889e-05 1.889e-05 0.00% DistributionMapping::Distribute() 56 1.552e-05 1.552e-05 1.552e-05 0.00% Castro::create_source_corrector() 10 9.875e-06 9.875e-06 9.875e-06 0.00% Amr::initSubcycle() 1 8.647e-06 8.647e-06 8.647e-06 0.00% Castro::retry_advance_ctu() 10 4.851e-06 4.851e-06 4.851e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.388e-06 4.388e-06 4.388e-06 0.00% AmrLevel::checkPointPost() 3 4.17e-06 4.17e-06 4.17e-06 0.00% Gravity::set_mass_offset() 11 3.969e-06 3.969e-06 3.969e-06 0.00% Castro::FluxRegCrseInit 10 2.686e-06 2.686e-06 2.686e-06 0.00% Castro::FluxRegFineAdd() 10 2.3e-06 2.3e-06 2.3e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.052e-06 2.052e-06 2.052e-06 0.00% AmrLevel::checkPointPre() 3 1.952e-06 1.952e-06 1.952e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-18-gd6f8b2cce0fb) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-18-gd6f8b2cce0fb) initialized Starting run at 10:13:36 UTC on 2023-01-16. Successfully read inputs file ... Castro git describe: 23.01-14-g78178b78e AMReX git describe: 23.01-18-gd6f8b2cce Microphysics git describe: 23.01-3-g1e475055 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.480274703 Restart time = 0.069577258 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.04816049 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048330343 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047842333 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057903552 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061773788 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033874616 seconds Ending run at 10:13:36 UTC on 2023-01-16. Run time = 0.3685058 Run time without initialization = 0.29826244 Average number of zones advanced per microsecond: 4.395 Average number of zones advanced per microsecond per rank: 4.395 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3685 ... 0.3685 ... 0.3685 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0825 0.0825 0.0825 22.39% VisMF::Read() 3 0.04087 0.04087 0.04087 11.09% MLCellLinOp::applyBC() 1946 0.03283 0.03283 0.03283 8.91% VisMF::Write(FabArray) 1 0.03236 0.03236 0.03236 8.78% Amr::restart() 1 0.02488 0.02488 0.02488 6.75% MLPoisson::Fsmooth() 1440 0.0141 0.0141 0.0141 3.82% FillBoundary_nowait() 1766 0.01266 0.01266 0.01266 3.43% StateData::FillBoundary(geom) 160 0.01192 0.01192 0.01192 3.23% amrex::Dot() 484 0.008713 0.008713 0.008713 2.36% StateDataPhysBCFunct::() 20 0.007435 0.007435 0.007435 2.02% amrex::Copy() 463 0.006941 0.006941 0.006941 1.88% FabArray::setVal() 537 0.00636 0.00636 0.00636 1.73% Castro::enforce_min_density() 30 0.006311 0.006311 0.006311 1.71% FabArray::norminf() 326 0.006237 0.006237 0.006237 1.69% Castro::normalize_species() 30 0.006202 0.006202 0.006202 1.68% FabArray::ParallelCopy_nowait() 380 0.005982 0.005982 0.005982 1.62% Castro::computeTemp() 30 0.005546 0.005546 0.005546 1.50% MLCellLinOp::defineAuxData() 6 0.005262 0.005262 0.005262 1.43% MLPoisson::Fapply() 500 0.00449 0.00449 0.00449 1.22% FabArray::Saxpy() 355 0.003621 0.003621 0.003621 0.98% FabArray::Xpay() 361 0.003515 0.003515 0.003515 0.95% Gravity::fill_multipole_BCs() 6 0.003508 0.003508 0.003508 0.95% Castro::estTimeStep() 10 0.002973 0.002973 0.002973 0.81% MLMG::addInterpCorrection() 180 0.002873 0.002873 0.002873 0.78% amrex::average_down 180 0.002554 0.002554 0.002554 0.69% BndryData::define() 6 0.002008 0.002008 0.002008 0.54% FabArray::LinComb() 242 0.001921 0.001921 0.001921 0.52% Castro::reset_internal_energy(MultiFab) 30 0.00191 0.00191 0.00191 0.52% amrex::Add() 72 0.001837 0.001837 0.001837 0.50% Castro::construct_new_gravity_source() 5 0.001696 0.001696 0.001696 0.46% Castro::do_advance_ctu() 5 0.001597 0.001597 0.001597 0.43% Castro::construct_old_gravity_source() 5 0.001442 0.001442 0.001442 0.39% Amr::writePlotFile() 1 0.001346 0.001346 0.001346 0.37% MLCGSolver::bicgstab 36 0.000933 0.000933 0.000933 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009046 0.0009046 0.0009046 0.25% Castro::reset_internal_energy(Fab) 240 0.0007558 0.0007558 0.0007558 0.21% MLCellLinOp::setLevelBC() 6 0.000742 0.000742 0.000742 0.20% Gravity::actual_solve_with_mlmg() 6 0.000714 0.000714 0.000714 0.19% FabArray::mult() 22 0.0006573 0.0006573 0.0006573 0.18% FabArray::setDomainBndry() 20 0.0006549 0.0006549 0.0006549 0.18% MLCellLinOp::prepareForSolve() 6 0.0006005 0.0006005 0.0006005 0.16% MultiFab::contains_nan() 10 0.0005867 0.0005867 0.0005867 0.16% Castro::enforce_speed_limit() 30 0.0005478 0.0005478 0.0005478 0.15% MLCellLinOp::compGrad() 6 0.0004908 0.0004908 0.0004908 0.13% MLCellLinOp::smooth() 720 0.000486 0.000486 0.000486 0.13% Amr::InitAmr() 1 0.0004592 0.0004592 0.0004592 0.12% MLMG::prepareForSolve() 6 0.0004572 0.0004572 0.0004572 0.12% FabArrayBase::CPC::define() 244 0.0004025 0.0004025 0.0004025 0.11% FabArray::FillBoundary() 1766 0.0003796 0.0003796 0.0003796 0.10% FabArrayBase::getCPC() 632 0.0003345 0.0003345 0.0003345 0.09% Gravity::get_old_grav_vector() 5 0.0002894 0.0002894 0.0002894 0.08% Gravity::get_new_grav_vector() 5 0.0002686 0.0002686 0.0002686 0.07% main() 1 0.0002646 0.0002646 0.0002646 0.07% FabArrayBase::getFB() 1766 0.0002575 0.0002575 0.0002575 0.07% MLCellLinOp::apply() 500 0.0001949 0.0001949 0.0001949 0.05% MLMG::mgVcycle() 36 0.0001822 0.0001822 0.0001822 0.05% Amr::coarseTimeStep() 5 0.000154 0.000154 0.000154 0.04% Castro::construct_new_source() 25 0.0001365 0.0001365 0.0001365 0.04% MultiFab::max() 6 0.000134 0.000134 0.000134 0.04% MLCGSolver::ParallelAllReduce 659 0.0001244 0.0001244 0.0001244 0.03% MLCellLinOp::defineBC() 6 0.000111 0.000111 0.000111 0.03% FabArray::ParallelCopy() 380 0.0001108 0.0001108 0.0001108 0.03% MLLinOp::defineGrids() 6 0.0001019 0.0001019 0.0001019 0.03% FillPatchIterator::Initialize 20 9.86e-05 9.86e-05 9.86e-05 0.03% MLCellLinOp::correctionResidual() 216 9.359e-05 9.359e-05 9.359e-05 0.03% Castro::subcycle_advance_ctu() 5 7.768e-05 7.768e-05 7.768e-05 0.02% AmrLevel::restart() 1 7.735e-05 7.735e-05 7.735e-05 0.02% Amr::timeStep() 5 7.681e-05 7.681e-05 7.681e-05 0.02% Gravity::solve_for_phi() 5 6.282e-05 6.282e-05 6.282e-05 0.02% FabArrayBase::FB::FB() 26 6.087e-05 6.087e-05 6.087e-05 0.02% StateData::restartDoit() 4 6.012e-05 6.012e-05 6.012e-05 0.02% Gravity::update_max_rhs() 6 5.437e-05 5.437e-05 5.437e-05 0.01% MLMG:computeResOfCorrection() 180 4.721e-05 4.721e-05 4.721e-05 0.01% Castro::clean_state() 30 4.267e-05 4.267e-05 4.267e-05 0.01% MLMG::mgVcycle_down::0 36 3.993e-05 3.993e-05 3.993e-05 0.01% MLMG::actualBottomSolve() 36 3.97e-05 3.97e-05 3.97e-05 0.01% Castro::expand_state() 5 3.779e-05 3.779e-05 3.779e-05 0.01% MLMG::mgVcycle_down::1 36 3.461e-05 3.461e-05 3.461e-05 0.01% MLMG::solve() 6 3.254e-05 3.254e-05 3.254e-05 0.01% MLMG::mgVcycle_down::2 36 3.185e-05 3.185e-05 3.185e-05 0.01% MLMG::mgVcycle_down::4 36 3.18e-05 3.18e-05 3.18e-05 0.01% MLMG::mgVcycle_down::3 36 3.088e-05 3.088e-05 3.088e-05 0.01% Castro::initialize_advance() 5 3.016e-05 3.016e-05 3.016e-05 0.01% MLMG::mgVcycle_up::4 36 2.832e-05 2.832e-05 2.832e-05 0.01% Castro::buildMetrics() 1 2.637e-05 2.637e-05 2.637e-05 0.01% Amr::writeSmallPlotFile() 1 2.604e-05 2.604e-05 2.604e-05 0.01% Castro::finalize_advance() 5 2.586e-05 2.586e-05 2.586e-05 0.01% MLMG::oneIter() 36 2.374e-05 2.374e-05 2.374e-05 0.01% MLMG::mgVcycle_up::0 36 2.321e-05 2.321e-05 2.321e-05 0.01% Castro::initialize_do_advance() 5 2.223e-05 2.223e-05 2.223e-05 0.01% Castro::swap_state_time_levels() 5 2.201e-05 2.201e-05 2.201e-05 0.01% MLCellLinOp::solutionResidual() 42 2.16e-05 2.16e-05 2.16e-05 0.01% MLMG::mgVcycle_up::3 36 2.148e-05 2.148e-05 2.148e-05 0.01% Castro::post_restart() 1 2.098e-05 2.098e-05 2.098e-05 0.01% MLMG::mgVcycle_up::2 36 2.091e-05 2.091e-05 2.091e-05 0.01% MLMG::mgVcycle_up::1 36 2.051e-05 2.051e-05 2.051e-05 0.01% Castro::initMFs() 1 1.972e-05 1.972e-05 1.972e-05 0.01% Castro::finalize_do_advance() 5 1.794e-05 1.794e-05 1.794e-05 0.00% MLMG::ResNormInf() 42 1.729e-05 1.729e-05 1.729e-05 0.00% Castro::construct_new_gravity() 5 1.623e-05 1.623e-05 1.623e-05 0.00% MLPoisson::define() 6 1.474e-05 1.474e-05 1.474e-05 0.00% MLMG::mgVcycle_bottom 36 1.45e-05 1.45e-05 1.45e-05 0.00% Castro::advance() 5 1.361e-05 1.361e-05 1.361e-05 0.00% FillPatchSingleLevel 20 1.356e-05 1.356e-05 1.356e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.335e-05 1.335e-05 1.335e-05 0.00% makeSFC 30 1.304e-05 1.304e-05 1.304e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.237e-05 1.237e-05 1.237e-05 0.00% MLMG::computeResidual() 36 1.214e-05 1.214e-05 1.214e-05 0.00% Castro::construct_old_source() 25 1.019e-05 1.019e-05 1.019e-05 0.00% Castro::do_new_sources() 5 9.195e-06 9.195e-06 9.195e-06 0.00% DistributionMapping::Distribute() 31 9.118e-06 9.118e-06 9.118e-06 0.00% Amr::initSubcycle() 1 8.862e-06 8.862e-06 8.862e-06 0.00% Castro::do_old_sources() 5 7.978e-06 7.978e-06 7.978e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.843e-06 6.843e-06 6.843e-06 0.00% Gravity::actual_multilevel_solve() 1 6.571e-06 6.571e-06 6.571e-06 0.00% Castro::check_for_nan() 10 6.486e-06 6.486e-06 6.486e-06 0.00% Castro::apply_source_to_state() 10 5.604e-06 5.604e-06 5.604e-06 0.00% Castro::post_timestep() 5 5.007e-06 5.007e-06 5.007e-06 0.00% Castro::construct_old_gravity() 5 4.912e-06 4.912e-06 4.912e-06 0.00% MLLinOp::define() 6 4.551e-06 4.551e-06 4.551e-06 0.00% MLPoisson::prepareForSolve() 6 4.238e-06 4.238e-06 4.238e-06 0.00% Gravity::swapTimeLevels() 5 4.115e-06 4.115e-06 4.115e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.46e-06 3.46e-06 3.46e-06 0.00% MLMG::computeMLResidual() 6 3.28e-06 3.28e-06 3.28e-06 0.00% MLMG::getGradSolution() 6 3.037e-06 3.037e-06 3.037e-06 0.00% Castro::computeNewDt() 5 3.033e-06 3.033e-06 3.033e-06 0.00% Castro::create_source_corrector() 5 2.756e-06 2.756e-06 2.756e-06 0.00% MLMG::MLRhsNormInf() 6 2.092e-06 2.092e-06 2.092e-06 0.00% MLMG::MLResNormInf() 6 2.089e-06 2.089e-06 2.089e-06 0.00% Castro::retry_advance_ctu() 5 2.08e-06 2.08e-06 2.08e-06 0.00% Gravity::set_mass_offset() 6 1.786e-06 1.786e-06 1.786e-06 0.00% AmrLevel::AmrLevel() 1 1.178e-06 1.178e-06 1.178e-06 0.00% Castro::FluxRegCrseInit 5 1.175e-06 1.175e-06 1.175e-06 0.00% Castro::FluxRegFineAdd() 5 1.129e-06 1.129e-06 1.129e-06 0.00% Amr::init() 1 1.129e-06 1.129e-06 1.129e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.116e-06 1.116e-06 1.116e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3685 0.3685 0.3685 100.00% Amr::coarseTimeStep() 5 0.2642 0.2642 0.2642 71.68% Amr::timeStep() 5 0.2618 0.2618 0.2618 71.04% Castro::advance() 5 0.2583 0.2583 0.2583 70.10% Castro::subcycle_advance_ctu() 5 0.2519 0.2519 0.2519 68.36% Castro::do_advance_ctu() 5 0.2519 0.2519 0.2519 68.34% Castro::construct_new_gravity() 5 0.1297 0.1297 0.1297 35.20% Gravity::solve_phi_with_mlmg() 6 0.1245 0.1245 0.1245 33.79% Gravity::solve_for_phi() 5 0.1214 0.1214 0.1214 32.95% Gravity::actual_solve_with_mlmg() 6 0.1208 0.1208 0.1208 32.77% MLMG::solve() 6 0.1097 0.1097 0.1097 29.78% MLMG::oneIter() 36 0.1029 0.1029 0.1029 27.93% MLMG::mgVcycle() 36 0.1014 0.1014 0.1014 27.52% Castro::construct_ctu_hydro_source() 5 0.08253 0.08253 0.08253 22.39% Amr::init() 1 0.06964 0.06964 0.06964 18.90% Amr::restart() 1 0.06964 0.06964 0.06964 18.90% MLCellLinOp::smooth() 720 0.04966 0.04966 0.04966 13.47% MLCellLinOp::applyBC() 1946 0.04619 0.04619 0.04619 12.53% AmrLevel::restart() 1 0.04115 0.04115 0.04115 11.17% StateData::restartDoit() 4 0.04107 0.04107 0.04107 11.14% VisMF::Read() 3 0.04087 0.04087 0.04087 11.09% Amr::writePlotFile() 1 0.03396 0.03396 0.03396 9.21% VisMF::Write(FabArray) 1 0.03236 0.03236 0.03236 8.78% MLMG::mgVcycle_bottom 36 0.03148 0.03148 0.03148 8.54% MLMG::actualBottomSolve() 36 0.03146 0.03146 0.03146 8.54% MLCGSolver::bicgstab 36 0.03116 0.03116 0.03116 8.45% FillPatchIterator::Initialize 20 0.0221 0.0221 0.0221 6.00% FillPatchSingleLevel 20 0.02135 0.02135 0.02135 5.79% Castro::clean_state() 30 0.02131 0.02131 0.02131 5.78% StateDataPhysBCFunct::() 20 0.01935 0.01935 0.01935 5.25% MLCellLinOp::apply() 500 0.01554 0.01554 0.01554 4.22% MLMG::mgVcycle_down::0 36 0.01416 0.01416 0.01416 3.84% MLPoisson::Fsmooth() 1440 0.0141 0.0141 0.0141 3.82% FabArray::FillBoundary() 1766 0.01335 0.01335 0.01335 3.62% FillBoundary_nowait() 1766 0.01297 0.01297 0.01297 3.52% StateData::FillBoundary(geom) 160 0.01192 0.01192 0.01192 3.23% MLMG::mgVcycle_up::0 36 0.01067 0.01067 0.01067 2.90% Castro::initialize_do_advance() 5 0.01042 0.01042 0.01042 2.83% MLCellLinOp::correctionResidual() 216 0.009461 0.009461 0.009461 2.57% MLPoisson::define() 6 0.00881 0.00881 0.00881 2.39% amrex::Dot() 484 0.008713 0.008713 0.008713 2.36% MLMG:computeResOfCorrection() 180 0.00831 0.00831 0.00831 2.25% Castro::computeTemp() 30 0.008211 0.008211 0.008211 2.23% Gravity::get_new_grav_vector() 5 0.008173 0.008173 0.008173 2.22% Castro::construct_old_gravity() 5 0.007908 0.007908 0.007908 2.15% Gravity::get_old_grav_vector() 5 0.007903 0.007903 0.007903 2.14% amrex::Copy() 463 0.006941 0.006941 0.006941 1.88% MLMG::mgVcycle_down::1 36 0.006781 0.006781 0.006781 1.84% FabArray::ParallelCopy() 380 0.006455 0.006455 0.006455 1.75% Castro::do_new_sources() 5 0.006433 0.006433 0.006433 1.75% FabArray::setVal() 537 0.00636 0.00636 0.00636 1.73% FabArray::ParallelCopy_nowait() 380 0.006344 0.006344 0.006344 1.72% MLMG::mgVcycle_down::2 36 0.006318 0.006318 0.006318 1.71% Castro::enforce_min_density() 30 0.006311 0.006311 0.006311 1.71% FabArray::norminf() 326 0.006237 0.006237 0.006237 1.69% Castro::expand_state() 5 0.006208 0.006208 0.006208 1.68% Castro::normalize_species() 30 0.006202 0.006202 0.006202 1.68% MLMG::mgVcycle_down::3 36 0.006151 0.006151 0.006151 1.67% MLMG::mgVcycle_down::4 36 0.006137 0.006137 0.006137 1.67% Castro::initialize_advance() 5 0.006111 0.006111 0.006111 1.66% MLCellLinOp::defineAuxData() 6 0.005996 0.005996 0.005996 1.63% MLCGSolver::ParallelAllReduce 659 0.005253 0.005253 0.005253 1.43% MLMG::addInterpCorrection() 180 0.005124 0.005124 0.005124 1.39% MLMG::mgVcycle_up::1 36 0.00499 0.00499 0.00499 1.35% MLMG::mgVcycle_up::4 36 0.004957 0.004957 0.004957 1.35% MLMG::mgVcycle_up::2 36 0.004844 0.004844 0.004844 1.31% amrex::average_down 180 0.004775 0.004775 0.004775 1.30% MLMG::mgVcycle_up::3 36 0.00474 0.00474 0.00474 1.29% Castro::do_old_sources() 5 0.004732 0.004732 0.004732 1.28% MLPoisson::Fapply() 500 0.00449 0.00449 0.00449 1.22% Gravity::fill_multipole_BCs() 6 0.003643 0.003643 0.003643 0.99% FabArray::Saxpy() 355 0.003621 0.003621 0.003621 0.98% FabArray::Xpay() 361 0.003515 0.003515 0.003515 0.95% Castro::post_restart() 1 0.003443 0.003443 0.003443 0.93% Castro::post_timestep() 5 0.003383 0.003383 0.003383 0.92% Gravity::multilevel_solve_for_new_phi() 1 0.003329 0.003329 0.003329 0.90% Gravity::actual_multilevel_solve() 1 0.003316 0.003316 0.003316 0.90% MLCellLinOp::solutionResidual() 42 0.0032 0.0032 0.0032 0.87% Castro::estTimeStep() 10 0.002973 0.002973 0.002973 0.81% Castro::reset_internal_energy(MultiFab) 30 0.002665 0.002665 0.002665 0.72% MLCellLinOp::defineBC() 6 0.002664 0.002664 0.002664 0.72% MLMG::computeResidual() 36 0.002649 0.002649 0.002649 0.72% BndryData::define() 6 0.002553 0.002553 0.002553 0.69% MLMG::prepareForSolve() 6 0.002489 0.002489 0.002489 0.68% Castro::computeNewDt() 5 0.002203 0.002203 0.002203 0.60% FabArray::LinComb() 242 0.001921 0.001921 0.001921 0.52% amrex::Add() 72 0.001837 0.001837 0.001837 0.50% Castro::construct_new_source() 25 0.001833 0.001833 0.001833 0.50% Castro::construct_new_gravity_source() 5 0.001696 0.001696 0.001696 0.46% Castro::construct_old_source() 25 0.001452 0.001452 0.001452 0.39% Castro::construct_old_gravity_source() 5 0.001442 0.001442 0.001442 0.39% MLMG::ResNormInf() 42 0.0009445 0.0009445 0.0009445 0.26% Castro::apply_source_to_state() 10 0.0009213 0.0009213 0.0009213 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009046 0.0009046 0.0009046 0.25% Castro::reset_internal_energy(Fab) 240 0.0007558 0.0007558 0.0007558 0.21% MLMG::getGradSolution() 6 0.0007513 0.0007513 0.0007513 0.20% MLCellLinOp::compGrad() 6 0.0007483 0.0007483 0.0007483 0.20% MLCellLinOp::setLevelBC() 6 0.000742 0.000742 0.000742 0.20% FabArrayBase::getCPC() 632 0.000737 0.000737 0.000737 0.20% FabArray::mult() 22 0.0006573 0.0006573 0.0006573 0.18% FabArray::setDomainBndry() 20 0.0006549 0.0006549 0.0006549 0.18% MLPoisson::prepareForSolve() 6 0.0006047 0.0006047 0.0006047 0.16% MLCellLinOp::prepareForSolve() 6 0.0006005 0.0006005 0.0006005 0.16% Castro::check_for_nan() 10 0.0005932 0.0005932 0.0005932 0.16% MultiFab::contains_nan() 10 0.0005867 0.0005867 0.0005867 0.16% MLMG::computeMLResidual() 6 0.0005666 0.0005666 0.0005666 0.15% Castro::enforce_speed_limit() 30 0.0005478 0.0005478 0.0005478 0.15% Amr::InitAmr() 1 0.000468 0.000468 0.000468 0.13% Gravity::update_max_rhs() 6 0.0004382 0.0004382 0.0004382 0.12% FabArrayBase::CPC::define() 244 0.0004025 0.0004025 0.0004025 0.11% FabArrayBase::getFB() 1766 0.0003184 0.0003184 0.0003184 0.09% Castro::finalize_advance() 5 0.0002836 0.0002836 0.0002836 0.08% Gravity::swapTimeLevels() 5 0.0002251 0.0002251 0.0002251 0.06% MLMG::MLResNormInf() 6 0.0001535 0.0001535 0.0001535 0.04% Castro::buildMetrics() 1 0.0001452 0.0001452 0.0001452 0.04% MLLinOp::define() 6 0.0001348 0.0001348 0.0001348 0.04% MultiFab::max() 6 0.000134 0.000134 0.000134 0.04% MLLinOp::defineGrids() 6 0.0001302 0.0001302 0.0001302 0.04% MLMG::MLRhsNormInf() 6 0.0001161 0.0001161 0.0001161 0.03% FabArrayBase::FB::FB() 26 6.087e-05 6.087e-05 6.087e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.719e-05 2.719e-05 2.719e-05 0.01% Amr::writeSmallPlotFile() 1 2.604e-05 2.604e-05 2.604e-05 0.01% Castro::swap_state_time_levels() 5 2.201e-05 2.201e-05 2.201e-05 0.01% makeSFC 30 2.035e-05 2.035e-05 2.035e-05 0.01% Castro::initMFs() 1 1.972e-05 1.972e-05 1.972e-05 0.01% Castro::finalize_do_advance() 5 1.794e-05 1.794e-05 1.794e-05 0.00% DistributionMapping::Distribute() 31 9.118e-06 9.118e-06 9.118e-06 0.00% Amr::initSubcycle() 1 8.862e-06 8.862e-06 8.862e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.266e-06 5.266e-06 5.266e-06 0.00% Castro::create_source_corrector() 5 2.756e-06 2.756e-06 2.756e-06 0.00% Castro::retry_advance_ctu() 5 2.08e-06 2.08e-06 2.08e-06 0.00% Gravity::set_mass_offset() 6 1.786e-06 1.786e-06 1.786e-06 0.00% AmrLevel::AmrLevel() 1 1.178e-06 1.178e-06 1.178e-06 0.00% Castro::FluxRegCrseInit 5 1.175e-06 1.175e-06 1.175e-06 0.00% Castro::FluxRegFineAdd() 5 1.129e-06 1.129e-06 1.129e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.116e-06 1.116e-06 1.116e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-18-gd6f8b2cce0fb) finalized