Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-10-g76d08651adb9) initialized Starting run at 08:54:09 UTC on 2022-02-15. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.02-10-g76d08651a Microphysics git describe: 22.02-1-g11641bf5 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.037823461 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022802813 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.112109938 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048351316 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048207306 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.048610729 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.071717547 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.037130718 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.060296113 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047842613 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049365832 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.047612478 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061164472 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.034886297 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.021808144 seconds Ending run at 08:54:10 UTC on 2022-02-15. Run time = 0.811470401 Run time without initialization = 0.689646205 Average number of zones advanced per microsecond: 3.801 Average number of zones advanced per microsecond per rank: 3.801 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8115 ... 0.8115 ... 0.8115 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2004 0.2004 0.2004 24.69% VisMF::Write(FabArray) 11 0.1468 0.1468 0.1468 18.09% MLCellLinOp::applyBC() 4433 0.09197 0.09197 0.09197 11.33% MLPoisson::Fsmooth() 3280 0.05938 0.05938 0.05938 7.32% FabArray::setVal() 1144 0.0258 0.0258 0.0258 3.18% MLCGSolver::bicgstab 82 0.0248 0.0248 0.0248 3.06% StateData::FillBoundary(geom) 328 0.02323 0.02323 0.02323 2.86% MultiFab::Dot() 1114 0.02108 0.02108 0.02108 2.60% StateDataPhysBCFunct::() 41 0.01756 0.01756 0.01756 2.16% FillBoundary_nowait() 4023 0.01706 0.01706 0.01706 2.10% FabArray::ParallelCopy_nowait() 861 0.01595 0.01595 0.01595 1.97% MultiFab::LinComb() 1586 0.01224 0.01224 0.01224 1.51% Castro::computeTemp() 63 0.01212 0.01212 0.01212 1.49% Gravity::fill_multipole_BCs() 11 0.01174 0.01174 0.01174 1.45% MLPoisson::Fapply() 1142 0.01053 0.01053 0.01053 1.30% MLCellLinOp::defineAuxData() 11 0.009923 0.009923 0.009923 1.22% Castro::enforce_min_density() 62 0.008577 0.008577 0.008577 1.06% MLMG::addInterpCorrection() 410 0.00703 0.00703 0.00703 0.87% amrex::average_down 410 0.006453 0.006453 0.006453 0.80% MultiFab::Xpay() 585 0.006007 0.006007 0.006007 0.74% FabArray::setDomainBndry() 41 0.005566 0.005566 0.005566 0.69% Castro::reset_internal_energy() 63 0.005182 0.005182 0.005182 0.64% Castro::estTimeStep() 21 0.004659 0.004659 0.004659 0.57% Castro::expand_state() 10 0.004592 0.004592 0.004592 0.57% Amr::checkPoint() 3 0.004585 0.004585 0.004585 0.57% Castro::enforce_speed_limit() 62 0.004306 0.004306 0.004306 0.53% BndryData::define() 11 0.00355 0.00355 0.00355 0.44% Castro::do_advance_ctu() 10 0.00335 0.00335 0.00335 0.41% Castro::construct_new_gravity_source() 10 0.003112 0.003112 0.003112 0.38% Amr::writePlotFile() 2 0.003106 0.003106 0.003106 0.38% Gravity::get_new_grav_vector() 11 0.002759 0.002759 0.002759 0.34% Castro::normalize_species() 62 0.002622 0.002622 0.002622 0.32% Castro::construct_old_gravity_source() 10 0.00258 0.00258 0.00258 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002212 0.002212 0.002212 0.27% MLCellLinOp::compGrad() 11 0.002012 0.002012 0.002012 0.25% MLMG::ResNormInf() 93 0.00189 0.00189 0.00189 0.23% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.22% Gravity::get_old_grav_vector() 10 0.001755 0.001755 0.001755 0.22% MLMG::oneIter() 82 0.001635 0.001635 0.001635 0.20% MLCellLinOp::setLevelBC() 11 0.001516 0.001516 0.001516 0.19% Gravity::actual_solve_with_mlmg() 11 0.001487 0.001487 0.001487 0.18% FabArray::mult() 43 0.001327 0.001327 0.001327 0.16% MLCellLinOp::prepareForSolve() 11 0.001323 0.001323 0.001323 0.16% Castro::initData() 1 0.001227 0.001227 0.001227 0.15% MLCellLinOp::smooth() 1640 0.001091 0.001091 0.001091 0.13% FabArrayBase::getCPC() 1323 0.0009018 0.0009018 0.0009018 0.11% FabArray::FillBoundary() 4023 0.0008444 0.0008444 0.0008444 0.10% FabArrayBase::getFB() 4023 0.000789 0.000789 0.000789 0.10% MLMG::prepareForSolve() 11 0.000774 0.000774 0.000774 0.10% FabArrayBase::CPC::define() 454 0.0007335 0.0007335 0.0007335 0.09% Gravity::update_max_rhs() 11 0.000535 0.000535 0.000535 0.07% MLCellLinOp::apply() 1142 0.0005185 0.0005185 0.0005185 0.06% MultiFab::Copy() 11 0.0005037 0.0005037 0.0005037 0.06% CGSolver::sxay() 1586 0.0004995 0.0004995 0.0004995 0.06% Amr::InitAmr() 1 0.000443 0.000443 0.000443 0.05% Gravity::solve_for_phi() 10 0.0004423 0.0004423 0.0004423 0.05% MLMG::mgVcycle() 82 0.0003825 0.0003825 0.0003825 0.05% main() 1 0.0003334 0.0003334 0.0003334 0.04% MultiFab::min() 10 0.0003276 0.0003276 0.0003276 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003232 0.0003232 0.0003232 0.04% FabArray::ParallelCopy() 861 0.0003048 0.0003048 0.0003048 0.04% Castro::construct_new_gravity() 10 0.0002605 0.0002605 0.0002605 0.03% MultiFab::max() 11 0.0002593 0.0002593 0.0002593 0.03% FillPatchIterator::Initialize 41 0.0002573 0.0002573 0.0002573 0.03% MLCellLinOp::correctionResidual() 492 0.0002266 0.0002266 0.0002266 0.03% Amr::coarseTimeStep() 10 0.0002201 0.0002201 0.0002201 0.03% Amr::timeStep() 10 0.0002169 0.0002169 0.0002169 0.03% Gravity::actual_multilevel_solve() 1 0.000212 0.000212 0.000212 0.03% MLMG::MLRhsNormInf() 11 0.0001987 0.0001987 0.0001987 0.02% MLCellLinOp::defineBC() 11 0.0001961 0.0001961 0.0001961 0.02% MLLinOp::defineGrids() 11 0.00019 0.00019 0.00019 0.02% Castro::subcycle_advance_ctu() 10 0.0001861 0.0001861 0.0001861 0.02% StateData::checkPoint() 12 0.0001448 0.0001448 0.0001448 0.02% Amr::defBaseLevel() 1 0.0001446 0.0001446 0.0001446 0.02% MLMG:computeResOfCorrection() 410 0.0001393 0.0001393 0.0001393 0.02% MLMG::actualBottomSolve() 82 0.0001095 0.0001095 0.0001095 0.01% MLMG::mgVcycle_down::0 82 9.363e-05 9.363e-05 9.363e-05 0.01% Castro::advance() 10 9.159e-05 9.159e-05 9.159e-05 0.01% FabArrayBase::FB::FB() 56 9.129e-05 9.129e-05 9.129e-05 0.01% MLMG::mgVcycle_down::1 82 7.938e-05 7.938e-05 7.938e-05 0.01% Castro::initialize_advance() 10 7.838e-05 7.838e-05 7.838e-05 0.01% MLMG::mgVcycle_down::3 82 7.82e-05 7.82e-05 7.82e-05 0.01% AmrLevel::checkPoint() 3 7.558e-05 7.558e-05 7.558e-05 0.01% MLMG::mgVcycle_down::2 82 7.335e-05 7.335e-05 7.335e-05 0.01% MLMG::mgVcycle_down::4 82 7.14e-05 7.14e-05 7.14e-05 0.01% Castro::clean_state() 62 7.061e-05 7.061e-05 7.061e-05 0.01% MLMG::solve() 11 6.728e-05 6.728e-05 6.728e-05 0.01% MLMG::mgVcycle_up::4 82 5.831e-05 5.831e-05 5.831e-05 0.01% Castro::initialize_do_advance() 10 5.633e-05 5.633e-05 5.633e-05 0.01% Castro::finalize_advance() 10 5.569e-05 5.569e-05 5.569e-05 0.01% Castro::post_timestep() 10 5.412e-05 5.412e-05 5.412e-05 0.01% MLMG::mgVcycle_up::0 82 5.113e-05 5.113e-05 5.113e-05 0.01% MLCellLinOp::solutionResidual() 93 4.972e-05 4.972e-05 4.972e-05 0.01% MLMG::mgVcycle_up::1 82 4.903e-05 4.903e-05 4.903e-05 0.01% MLMG::mgVcycle_up::2 82 4.86e-05 4.86e-05 4.86e-05 0.01% MLMG::mgVcycle_up::3 82 4.837e-05 4.837e-05 4.837e-05 0.01% Castro::construct_new_source() 50 4.11e-05 4.11e-05 4.11e-05 0.01% StateData::define() 4 4.029e-05 4.029e-05 4.029e-05 0.00% Castro::finalize_do_advance() 10 3.993e-05 3.993e-05 3.993e-05 0.00% Castro::swap_state_time_levels() 10 3.784e-05 3.784e-05 3.784e-05 0.00% MLMG::computeResidual() 82 3.568e-05 3.568e-05 3.568e-05 0.00% Castro::enforce_consistent_e() 1 3.348e-05 3.348e-05 3.348e-05 0.00% MLMG::mgVcycle_bottom 82 3.294e-05 3.294e-05 3.294e-05 0.00% FillPatchSingleLevel 41 2.921e-05 2.921e-05 2.921e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.754e-05 2.754e-05 2.754e-05 0.00% MLLinOp::define() 11 2.642e-05 2.642e-05 2.642e-05 0.00% makeSFC 55 2.622e-05 2.622e-05 2.622e-05 0.00% MLPoisson::define() 11 2.351e-05 2.351e-05 2.351e-05 0.00% Amr::writeSmallPlotFile() 1 2.276e-05 2.276e-05 2.276e-05 0.00% Amr::FinalizeInit() 1 2.011e-05 2.011e-05 2.011e-05 0.00% Castro::construct_old_source() 50 1.967e-05 1.967e-05 1.967e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.619e-05 1.619e-05 1.619e-05 0.00% Castro::do_new_sources() 10 1.598e-05 1.598e-05 1.598e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.587e-05 1.587e-05 1.587e-05 0.00% DistributionMapping::Distribute() 56 1.47e-05 1.47e-05 1.47e-05 0.00% Castro::do_old_sources() 10 1.441e-05 1.441e-05 1.441e-05 0.00% Castro::apply_source_to_state() 20 1.246e-05 1.246e-05 1.246e-05 0.00% Castro::construct_old_gravity() 10 1.046e-05 1.046e-05 1.046e-05 0.00% Amr::initSubcycle() 1 9.847e-06 9.847e-06 9.847e-06 0.00% AmrLevel::AmrLevel(dm) 1 9.376e-06 9.376e-06 9.376e-06 0.00% Gravity::swapTimeLevels() 10 8.879e-06 8.879e-06 8.879e-06 0.00% MLPoisson::prepareForSolve() 11 8.843e-06 8.843e-06 8.843e-06 0.00% MLMG::buildFineMask() 11 7.289e-06 7.289e-06 7.289e-06 0.00% MLMG::computeMLResidual() 11 6.927e-06 6.927e-06 6.927e-06 0.00% MLMG::getGradSolution() 11 6.069e-06 6.069e-06 6.069e-06 0.00% Amr::InitializeInit() 1 5.894e-06 5.894e-06 5.894e-06 0.00% Castro::create_source_corrector() 10 5.733e-06 5.733e-06 5.733e-06 0.00% Castro::computeNewDt() 9 5.729e-06 5.729e-06 5.729e-06 0.00% AmrLevel::checkPointPost() 3 5.683e-06 5.683e-06 5.683e-06 0.00% MLMG::MLResNormInf() 11 5.48e-06 5.48e-06 5.48e-06 0.00% Gravity::set_mass_offset() 11 4.499e-06 4.499e-06 4.499e-06 0.00% Castro::retry_advance_ctu() 10 4.149e-06 4.149e-06 4.149e-06 0.00% Castro::post_init() 1 3.683e-06 3.683e-06 3.683e-06 0.00% Amr::init() 1 3.172e-06 3.172e-06 3.172e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.018e-06 3.018e-06 3.018e-06 0.00% Castro::FluxRegFineAdd() 10 2.975e-06 2.975e-06 2.975e-06 0.00% Castro::computeInitialDt() 2 2.36e-06 2.36e-06 2.36e-06 0.00% AmrLevel::checkPointPre() 3 2.153e-06 2.153e-06 2.153e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.139e-06 2.139e-06 2.139e-06 0.00% Castro::post_regrid() 1 1.147e-06 1.147e-06 1.147e-06 0.00% Amr::initialInit() 1 8.7e-07 8.7e-07 8.7e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8115 0.8115 0.8115 100.00% Amr::coarseTimeStep() 10 0.6676 0.6676 0.6676 82.27% Amr::timeStep() 10 0.5928 0.5928 0.5928 73.05% Castro::advance() 10 0.5858 0.5858 0.5858 72.19% Castro::subcycle_advance_ctu() 10 0.5728 0.5728 0.5728 70.58% Castro::do_advance_ctu() 10 0.5726 0.5726 0.5726 70.56% Gravity::solve_phi_with_mlmg() 11 0.3247 0.3247 0.3247 40.01% Gravity::actual_solve_with_mlmg() 11 0.3127 0.3127 0.3127 38.54% Castro::construct_new_gravity() 10 0.2976 0.2976 0.2976 36.67% MLMG::solve() 11 0.2903 0.2903 0.2903 35.77% Gravity::solve_for_phi() 10 0.2771 0.2771 0.2771 34.15% MLMG::oneIter() 82 0.2747 0.2747 0.2747 33.86% MLMG::mgVcycle() 82 0.2731 0.2731 0.2731 33.65% Castro::construct_ctu_hydro_source() 10 0.2004 0.2004 0.2004 24.69% VisMF::Write(FabArray) 11 0.1468 0.1468 0.1468 18.09% MLCellLinOp::smooth() 1640 0.1426 0.1426 0.1426 17.57% Amr::init() 1 0.1212 0.1212 0.1212 14.94% MLCellLinOp::applyBC() 4433 0.1108 0.1108 0.1108 13.65% Amr::checkPoint() 3 0.11 0.11 0.11 13.55% AmrLevel::checkPoint() 3 0.1054 0.1054 0.1054 12.99% StateData::checkPoint() 12 0.1053 0.1053 0.1053 12.98% MLMG::mgVcycle_bottom 82 0.08042 0.08042 0.08042 9.91% MLMG::actualBottomSolve() 82 0.08039 0.08039 0.08039 9.91% MLCGSolver::bicgstab 82 0.07966 0.07966 0.07966 9.82% Amr::initialInit() 1 0.06047 0.06047 0.06047 7.45% MLPoisson::Fsmooth() 3280 0.05938 0.05938 0.05938 7.32% Amr::FinalizeInit() 1 0.05138 0.05138 0.05138 6.33% FillPatchIterator::Initialize 41 0.0508 0.0508 0.0508 6.26% Castro::post_init() 1 0.05067 0.05067 0.05067 6.24% Gravity::multilevel_solve_for_new_phi() 1 0.04825 0.04825 0.04825 5.95% Gravity::actual_multilevel_solve() 1 0.04824 0.04824 0.04824 5.94% FillPatchSingleLevel 41 0.04498 0.04498 0.04498 5.54% Amr::writePlotFile() 2 0.04473 0.04473 0.04473 5.51% StateDataPhysBCFunct::() 41 0.04078 0.04078 0.04078 5.03% MLCellLinOp::apply() 1142 0.03919 0.03919 0.03919 4.83% MLMG::mgVcycle_down::0 82 0.03745 0.03745 0.03745 4.61% MLMG::mgVcycle_up::0 82 0.03222 0.03222 0.03222 3.97% Castro::clean_state() 62 0.03217 0.03217 0.03217 3.96% FabArray::setVal() 1144 0.0258 0.0258 0.0258 3.18% StateData::FillBoundary(geom) 328 0.02323 0.02323 0.02323 2.86% Castro::initialize_do_advance() 10 0.0227 0.0227 0.0227 2.80% Gravity::get_new_grav_vector() 11 0.02246 0.02246 0.02246 2.77% MLCellLinOp::correctionResidual() 492 0.02204 0.02204 0.02204 2.72% MultiFab::Dot() 1114 0.02108 0.02108 0.02108 2.60% MLMG:computeResOfCorrection() 410 0.01896 0.01896 0.01896 2.34% FabArray::FillBoundary() 4023 0.01879 0.01879 0.01879 2.32% MLMG::mgVcycle_down::1 82 0.01829 0.01829 0.01829 2.25% FillBoundary_nowait() 4023 0.01794 0.01794 0.01794 2.21% Castro::expand_state() 10 0.01784 0.01784 0.01784 2.20% MLMG::mgVcycle_down::2 82 0.01773 0.01773 0.01773 2.19% Castro::computeTemp() 63 0.0173 0.0173 0.0173 2.13% FabArray::ParallelCopy() 861 0.01717 0.01717 0.01717 2.12% MLMG::mgVcycle_down::3 82 0.01694 0.01694 0.01694 2.09% MLPoisson::define() 11 0.01689 0.01689 0.01689 2.08% FabArray::ParallelCopy_nowait() 861 0.01687 0.01687 0.01687 2.08% MLMG::mgVcycle_down::4 82 0.01613 0.01613 0.01613 1.99% Castro::construct_old_gravity() 10 0.01548 0.01548 0.01548 1.91% Gravity::get_old_grav_vector() 10 0.01547 0.01547 0.01547 1.91% MLMG::mgVcycle_up::2 82 0.01383 0.01383 0.01383 1.70% MLMG::mgVcycle_up::1 82 0.01358 0.01358 0.01358 1.67% MLMG::addInterpCorrection() 410 0.01353 0.01353 0.01353 1.67% MLMG::mgVcycle_up::3 82 0.01316 0.01316 0.01316 1.62% MLMG::mgVcycle_up::4 82 0.01296 0.01296 0.01296 1.60% amrex::average_down 410 0.01296 0.01296 0.01296 1.60% Castro::initialize_advance() 10 0.01287 0.01287 0.01287 1.59% CGSolver::sxay() 1586 0.01274 0.01274 0.01274 1.57% MLCGSolver::ParallelAllReduce 1514 0.0127 0.0127 0.0127 1.57% MultiFab::LinComb() 1586 0.01224 0.01224 0.01224 1.51% Gravity::fill_multipole_BCs() 11 0.01174 0.01174 0.01174 1.45% MLCellLinOp::defineAuxData() 11 0.01159 0.01159 0.01159 1.43% Castro::do_new_sources() 10 0.01102 0.01102 0.01102 1.36% MLPoisson::Fapply() 1142 0.01053 0.01053 0.01053 1.30% Amr::InitializeInit() 1 0.009086 0.009086 0.009086 1.12% Amr::defBaseLevel() 1 0.00908 0.00908 0.00908 1.12% Castro::enforce_min_density() 62 0.008577 0.008577 0.008577 1.06% Castro::do_old_sources() 10 0.007637 0.007637 0.007637 0.94% MLCellLinOp::solutionResidual() 93 0.007613 0.007613 0.007613 0.94% Castro::post_timestep() 10 0.006748 0.006748 0.006748 0.83% MLMG::computeResidual() 82 0.006588 0.006588 0.006588 0.81% MultiFab::Xpay() 585 0.006007 0.006007 0.006007 0.74% MLMG::prepareForSolve() 11 0.005735 0.005735 0.005735 0.71% FabArray::setDomainBndry() 41 0.005566 0.005566 0.005566 0.69% Castro::reset_internal_energy() 63 0.005182 0.005182 0.005182 0.64% MLCellLinOp::defineBC() 11 0.005005 0.005005 0.005005 0.62% BndryData::define() 11 0.004809 0.004809 0.004809 0.59% Castro::estTimeStep() 21 0.004659 0.004659 0.004659 0.57% Castro::enforce_speed_limit() 62 0.004306 0.004306 0.004306 0.53% Castro::initData() 1 0.003287 0.003287 0.003287 0.41% Castro::construct_new_source() 50 0.003153 0.003153 0.003153 0.39% Castro::construct_new_gravity_source() 10 0.003112 0.003112 0.003112 0.38% Castro::normalize_species() 62 0.002622 0.002622 0.002622 0.32% Castro::construct_old_source() 50 0.0026 0.0026 0.0026 0.32% Castro::construct_old_gravity_source() 10 0.00258 0.00258 0.00258 0.32% MLMG::getGradSolution() 11 0.002553 0.002553 0.002553 0.31% MLCellLinOp::compGrad() 11 0.002547 0.002547 0.002547 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002212 0.002212 0.002212 0.27% Castro::computeNewDt() 9 0.002008 0.002008 0.002008 0.25% MLMG::ResNormInf() 93 0.00189 0.00189 0.00189 0.23% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.22% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.22% FabArrayBase::getCPC() 1323 0.001635 0.001635 0.001635 0.20% MLCellLinOp::setLevelBC() 11 0.001516 0.001516 0.001516 0.19% MLPoisson::prepareForSolve() 11 0.001331 0.001331 0.001331 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.16% MLCellLinOp::prepareForSolve() 11 0.001323 0.001323 0.001323 0.16% MLMG::computeMLResidual() 11 0.001068 0.001068 0.001068 0.13% Gravity::swapTimeLevels() 10 0.001023 0.001023 0.001023 0.13% Gravity::update_max_rhs() 11 0.0009384 0.0009384 0.0009384 0.12% FabArrayBase::getFB() 4023 0.0008803 0.0008803 0.0008803 0.11% FabArrayBase::CPC::define() 454 0.0007335 0.0007335 0.0007335 0.09% Castro::computeInitialDt() 2 0.0007028 0.0007028 0.0007028 0.09% MultiFab::Copy() 11 0.0005037 0.0005037 0.0005037 0.06% Castro::post_regrid() 1 0.0004728 0.0004728 0.0004728 0.06% Amr::InitAmr() 1 0.0004529 0.0004529 0.0004529 0.06% MultiFab::min() 10 0.0003276 0.0003276 0.0003276 0.04% MLLinOp::define() 11 0.0002748 0.0002748 0.0002748 0.03% MultiFab::max() 11 0.0002593 0.0002593 0.0002593 0.03% MLMG::MLResNormInf() 11 0.0002506 0.0002506 0.0002506 0.03% MLLinOp::defineGrids() 11 0.0002484 0.0002484 0.0002484 0.03% MLMG::MLRhsNormInf() 11 0.0001987 0.0001987 0.0001987 0.02% FabArrayBase::FB::FB() 56 9.129e-05 9.129e-05 9.129e-05 0.01% Castro::finalize_advance() 10 5.866e-05 5.866e-05 5.866e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.621e-05 5.621e-05 5.621e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.967e-05 4.967e-05 4.967e-05 0.01% StateData::define() 4 4.029e-05 4.029e-05 4.029e-05 0.00% makeSFC 55 4.001e-05 4.001e-05 4.001e-05 0.00% Castro::finalize_do_advance() 10 3.993e-05 3.993e-05 3.993e-05 0.00% Castro::swap_state_time_levels() 10 3.784e-05 3.784e-05 3.784e-05 0.00% Castro::enforce_consistent_e() 1 3.348e-05 3.348e-05 3.348e-05 0.00% Amr::writeSmallPlotFile() 1 2.276e-05 2.276e-05 2.276e-05 0.00% DistributionMapping::Distribute() 56 1.47e-05 1.47e-05 1.47e-05 0.00% Amr::initSubcycle() 1 9.847e-06 9.847e-06 9.847e-06 0.00% MLMG::buildFineMask() 11 7.289e-06 7.289e-06 7.289e-06 0.00% Castro::create_source_corrector() 10 5.733e-06 5.733e-06 5.733e-06 0.00% AmrLevel::checkPointPost() 3 5.683e-06 5.683e-06 5.683e-06 0.00% Gravity::set_mass_offset() 11 4.499e-06 4.499e-06 4.499e-06 0.00% Castro::retry_advance_ctu() 10 4.149e-06 4.149e-06 4.149e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.935e-06 3.935e-06 3.935e-06 0.00% Castro::FluxRegFineAdd() 10 2.975e-06 2.975e-06 2.975e-06 0.00% AmrLevel::checkPointPre() 3 2.153e-06 2.153e-06 2.153e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.139e-06 2.139e-06 2.139e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-10-g76d08651adb9) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-10-g76d08651adb9) initialized Starting run at 08:54:11 UTC on 2022-02-15. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.02-10-g76d08651a Microphysics git describe: 22.02-1-g11641bf5 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.487873765 Restart time = 0.054661901 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.11336709 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048210309 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050808172 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060849334 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062559934 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023118274 seconds Ending run at 08:54:11 UTC on 2022-02-15. Run time = 0.41444882 Run time without initialization = 0.359250123 Average number of zones advanced per microsecond: 3.648 Average number of zones advanced per microsecond per rank: 3.648 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4145 ... 0.4145 ... 0.4145 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1285 0.1285 0.1285 30.99% VisMF::Read() 3 0.04288 0.04288 0.04288 10.35% MLCellLinOp::applyBC() 1946 0.03758 0.03758 0.03758 9.07% MLPoisson::Fsmooth() 1440 0.02545 0.02545 0.02545 6.14% VisMF::Write(FabArray) 1 0.02187 0.02187 0.02187 5.28% FabArray::setVal() 537 0.01601 0.01601 0.01601 3.86% StateData::FillBoundary(geom) 160 0.01091 0.01091 0.01091 2.63% MLCGSolver::bicgstab 36 0.009216 0.009216 0.009216 2.22% MultiFab::Dot() 484 0.00912 0.00912 0.00912 2.20% StateDataPhysBCFunct::() 20 0.008521 0.008521 0.008521 2.06% Castro::computeTemp() 30 0.007159 0.007159 0.007159 1.73% FillBoundary_nowait() 1766 0.007113 0.007113 0.007113 1.72% FabArray::ParallelCopy_nowait() 380 0.00697 0.00697 0.00697 1.68% Gravity::fill_multipole_BCs() 6 0.006423 0.006423 0.006423 1.55% Castro::enforce_min_density() 30 0.006265 0.006265 0.006265 1.51% MLCellLinOp::defineAuxData() 6 0.005446 0.005446 0.005446 1.31% MultiFab::LinComb() 690 0.005245 0.005245 0.005245 1.27% MLPoisson::Fapply() 500 0.004575 0.004575 0.004575 1.10% Castro::expand_state() 5 0.004163 0.004163 0.004163 1.00% FabArray::setDomainBndry() 20 0.004125 0.004125 0.004125 1.00% MLMG::addInterpCorrection() 180 0.003016 0.003016 0.003016 0.73% Amr::restart() 1 0.003013 0.003013 0.003013 0.73% amrex::average_down 180 0.00277 0.00277 0.00277 0.67% Castro::estTimeStep() 10 0.002704 0.002704 0.002704 0.65% MultiFab::Xpay() 258 0.002621 0.002621 0.002621 0.63% Castro::do_advance_ctu() 5 0.002506 0.002506 0.002506 0.60% Castro::reset_internal_energy() 30 0.002374 0.002374 0.002374 0.57% BndryData::define() 6 0.001924 0.001924 0.001924 0.46% Castro::enforce_speed_limit() 30 0.00174 0.00174 0.00174 0.42% Gravity::get_new_grav_vector() 5 0.001715 0.001715 0.001715 0.41% Castro::construct_new_gravity_source() 5 0.001546 0.001546 0.001546 0.37% Castro::construct_old_gravity_source() 5 0.001418 0.001418 0.001418 0.34% MLCellLinOp::compGrad() 6 0.001382 0.001382 0.001382 0.33% Amr::writePlotFile() 1 0.001336 0.001336 0.001336 0.32% Castro::normalize_species() 30 0.001301 0.001301 0.001301 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001191 0.001191 0.001191 0.29% MultiFab::Saxpy() 10 0.0009179 0.0009179 0.0009179 0.22% Gravity::get_old_grav_vector() 5 0.0009033 0.0009033 0.0009033 0.22% MLMG::ResNormInf() 42 0.0008352 0.0008352 0.0008352 0.20% MLCellLinOp::setLevelBC() 6 0.000819 0.000819 0.000819 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007298 0.0007298 0.0007298 0.18% MLMG::oneIter() 36 0.0007255 0.0007255 0.0007255 0.18% MLCellLinOp::prepareForSolve() 6 0.0006936 0.0006936 0.0006936 0.17% FabArray::mult() 22 0.0006521 0.0006521 0.0006521 0.16% Gravity::update_max_rhs() 6 0.0005226 0.0005226 0.0005226 0.13% MLCellLinOp::smooth() 720 0.0004856 0.0004856 0.0004856 0.12% MLMG::prepareForSolve() 6 0.0004381 0.0004381 0.0004381 0.11% FabArrayBase::CPC::define() 244 0.000435 0.000435 0.000435 0.10% FabArrayBase::getCPC() 632 0.0004328 0.0004328 0.0004328 0.10% FabArray::FillBoundary() 1766 0.0004137 0.0004137 0.0004137 0.10% Amr::InitAmr() 1 0.0003693 0.0003693 0.0003693 0.09% MultiFab::Copy() 6 0.0003523 0.0003523 0.0003523 0.08% FabArrayBase::getFB() 1766 0.000331 0.000331 0.000331 0.08% main() 1 0.0002637 0.0002637 0.0002637 0.06% Gravity::solve_for_phi() 5 0.0002349 0.0002349 0.0002349 0.06% Gravity::actual_multilevel_solve() 1 0.0002152 0.0002152 0.0002152 0.05% CGSolver::sxay() 690 0.0002145 0.0002145 0.0002145 0.05% MLCellLinOp::apply() 500 0.0002125 0.0002125 0.0002125 0.05% MLMG::mgVcycle() 36 0.0001871 0.0001871 0.0001871 0.05% Castro::construct_new_gravity() 5 0.0001783 0.0001783 0.0001783 0.04% MultiFab::min() 5 0.0001634 0.0001634 0.0001634 0.04% Castro::construct_new_source() 25 0.0001461 0.0001461 0.0001461 0.04% FabArray::ParallelCopy() 380 0.0001382 0.0001382 0.0001382 0.03% MLCGSolver::ParallelAllReduce 659 0.0001356 0.0001356 0.0001356 0.03% MultiFab::max() 6 0.0001349 0.0001349 0.0001349 0.03% FillPatchIterator::Initialize 20 0.0001225 0.0001225 0.0001225 0.03% Amr::coarseTimeStep() 5 0.00011 0.00011 0.00011 0.03% MLCellLinOp::defineBC() 6 0.0001065 0.0001065 0.0001065 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% Amr::timeStep() 5 0.0001005 0.0001005 0.0001005 0.02% MLCellLinOp::correctionResidual() 216 9.894e-05 9.894e-05 9.894e-05 0.02% MLLinOp::defineGrids() 6 9.453e-05 9.453e-05 9.453e-05 0.02% Castro::advance() 5 7.713e-05 7.713e-05 7.713e-05 0.02% AmrLevel::restart() 1 7.347e-05 7.347e-05 7.347e-05 0.02% StateData::restartDoit() 4 7.318e-05 7.318e-05 7.318e-05 0.02% Castro::subcycle_advance_ctu() 5 6.564e-05 6.564e-05 6.564e-05 0.02% MLMG:computeResOfCorrection() 180 6.468e-05 6.468e-05 6.468e-05 0.02% FabArrayBase::FB::FB() 26 6.253e-05 6.253e-05 6.253e-05 0.02% MLMG::actualBottomSolve() 36 4.498e-05 4.498e-05 4.498e-05 0.01% Castro::create_source_corrector() 5 4.291e-05 4.291e-05 4.291e-05 0.01% Castro::post_timestep() 5 3.929e-05 3.929e-05 3.929e-05 0.01% MLMG::mgVcycle_down::0 36 3.845e-05 3.845e-05 3.845e-05 0.01% Castro::do_new_sources() 5 3.761e-05 3.761e-05 3.761e-05 0.01% Castro::initialize_advance() 5 3.749e-05 3.749e-05 3.749e-05 0.01% Castro::construct_old_source() 25 3.538e-05 3.538e-05 3.538e-05 0.01% MLMG::mgVcycle_down::1 36 3.53e-05 3.53e-05 3.53e-05 0.01% Castro::clean_state() 30 3.391e-05 3.391e-05 3.391e-05 0.01% MLMG::mgVcycle_down::2 36 3.235e-05 3.235e-05 3.235e-05 0.01% MLMG::solve() 6 3.212e-05 3.212e-05 3.212e-05 0.01% MLMG::mgVcycle_down::4 36 3e-05 3e-05 3e-05 0.01% MLMG::mgVcycle_down::3 36 2.966e-05 2.966e-05 2.966e-05 0.01% Castro::initialize_do_advance() 5 2.953e-05 2.953e-05 2.953e-05 0.01% MLLinOp::define() 6 2.737e-05 2.737e-05 2.737e-05 0.01% Castro::post_restart() 1 2.705e-05 2.705e-05 2.705e-05 0.01% Castro::swap_state_time_levels() 5 2.676e-05 2.676e-05 2.676e-05 0.01% MLMG::mgVcycle_up::4 36 2.66e-05 2.66e-05 2.66e-05 0.01% Castro::finalize_advance() 5 2.583e-05 2.583e-05 2.583e-05 0.01% MLCellLinOp::solutionResidual() 42 2.466e-05 2.466e-05 2.466e-05 0.01% Amr::writeSmallPlotFile() 1 2.378e-05 2.378e-05 2.378e-05 0.01% MLMG::mgVcycle_up::0 36 2.301e-05 2.301e-05 2.301e-05 0.01% MLMG::mgVcycle_up::3 36 2.14e-05 2.14e-05 2.14e-05 0.01% MLMG::mgVcycle_up::2 36 2.135e-05 2.135e-05 2.135e-05 0.01% Castro::finalize_do_advance() 5 2.106e-05 2.106e-05 2.106e-05 0.01% MLMG::mgVcycle_up::1 36 2.099e-05 2.099e-05 2.099e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.632e-05 1.632e-05 1.632e-05 0.00% FillPatchSingleLevel 20 1.624e-05 1.624e-05 1.624e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.588e-05 1.588e-05 1.588e-05 0.00% MLMG::computeResidual() 36 1.523e-05 1.523e-05 1.523e-05 0.00% MLPoisson::define() 6 1.487e-05 1.487e-05 1.487e-05 0.00% MLMG::mgVcycle_bottom 36 1.475e-05 1.475e-05 1.475e-05 0.00% makeSFC 30 1.318e-05 1.318e-05 1.318e-05 0.00% DistributionMapping::Distribute() 31 8.751e-06 8.751e-06 8.751e-06 0.00% Amr::initSubcycle() 1 8.747e-06 8.747e-06 8.747e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.805e-06 7.805e-06 7.805e-06 0.00% Castro::do_old_sources() 5 7.641e-06 7.641e-06 7.641e-06 0.00% Castro::apply_source_to_state() 10 5.953e-06 5.953e-06 5.953e-06 0.00% Castro::construct_old_gravity() 5 5.244e-06 5.244e-06 5.244e-06 0.00% MLPoisson::prepareForSolve() 6 4.644e-06 4.644e-06 4.644e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.19e-06 4.19e-06 4.19e-06 0.00% Gravity::swapTimeLevels() 5 4.115e-06 4.115e-06 4.115e-06 0.00% MLMG::buildFineMask() 6 3.6e-06 3.6e-06 3.6e-06 0.00% Castro::computeNewDt() 5 3.251e-06 3.251e-06 3.251e-06 0.00% MLMG::getGradSolution() 6 3.074e-06 3.074e-06 3.074e-06 0.00% MLMG::computeMLResidual() 6 2.988e-06 2.988e-06 2.988e-06 0.00% MLMG::MLResNormInf() 6 2.327e-06 2.327e-06 2.327e-06 0.00% Gravity::set_mass_offset() 6 2.318e-06 2.318e-06 2.318e-06 0.00% Castro::retry_advance_ctu() 5 1.762e-06 1.762e-06 1.762e-06 0.00% Castro::FluxRegFineAdd() 5 1.408e-06 1.408e-06 1.408e-06 0.00% Amr::init() 1 1.174e-06 1.174e-06 1.174e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.027e-06 1.027e-06 1.027e-06 0.00% AmrLevel::AmrLevel() 1 9.47e-07 9.47e-07 9.47e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4145 0.4145 0.4145 100.00% Amr::coarseTimeStep() 5 0.3359 0.3359 0.3359 81.04% Amr::timeStep() 5 0.3344 0.3344 0.3344 80.67% Castro::advance() 5 0.3296 0.3296 0.3296 79.52% Castro::subcycle_advance_ctu() 5 0.3193 0.3193 0.3193 77.04% Castro::do_advance_ctu() 5 0.3192 0.3192 0.3192 77.02% Castro::construct_new_gravity() 5 0.1455 0.1455 0.1455 35.12% Gravity::solve_phi_with_mlmg() 6 0.1413 0.1413 0.1413 34.08% Gravity::solve_for_phi() 5 0.1355 0.1355 0.1355 32.70% Gravity::actual_solve_with_mlmg() 6 0.1347 0.1347 0.1347 32.50% Castro::construct_ctu_hydro_source() 5 0.1285 0.1285 0.1285 30.99% MLMG::solve() 6 0.1222 0.1222 0.1222 29.49% MLMG::oneIter() 36 0.1144 0.1144 0.1144 27.60% MLMG::mgVcycle() 36 0.1137 0.1137 0.1137 27.43% MLCellLinOp::smooth() 720 0.05951 0.05951 0.05951 14.36% Amr::init() 1 0.0547 0.0547 0.0547 13.20% Amr::restart() 1 0.0547 0.0547 0.0547 13.20% MLCellLinOp::applyBC() 1946 0.0455 0.0455 0.0455 10.98% AmrLevel::restart() 1 0.04381 0.04381 0.04381 10.57% StateData::restartDoit() 4 0.04373 0.04373 0.04373 10.55% VisMF::Read() 3 0.04288 0.04288 0.04288 10.35% MLMG::mgVcycle_bottom 36 0.03284 0.03284 0.03284 7.92% MLMG::actualBottomSolve() 36 0.03282 0.03282 0.03282 7.92% MLCGSolver::bicgstab 36 0.03251 0.03251 0.03251 7.84% FillPatchIterator::Initialize 20 0.02576 0.02576 0.02576 6.21% MLPoisson::Fsmooth() 1440 0.02545 0.02545 0.02545 6.14% Amr::writePlotFile() 1 0.0232 0.0232 0.0232 5.60% VisMF::Write(FabArray) 1 0.02187 0.02187 0.02187 5.28% FillPatchSingleLevel 20 0.02151 0.02151 0.02151 5.19% StateDataPhysBCFunct::() 20 0.01943 0.01943 0.01943 4.69% Castro::clean_state() 30 0.01887 0.01887 0.01887 4.55% MLCellLinOp::apply() 500 0.01642 0.01642 0.01642 3.96% FabArray::setVal() 537 0.01601 0.01601 0.01601 3.86% Castro::initialize_do_advance() 5 0.01589 0.01589 0.01589 3.83% MLMG::mgVcycle_down::0 36 0.01582 0.01582 0.01582 3.82% MLMG::mgVcycle_up::0 36 0.01355 0.01355 0.01355 3.27% Castro::expand_state() 5 0.01266 0.01266 0.01266 3.06% StateData::FillBoundary(geom) 160 0.01091 0.01091 0.01091 2.63% Castro::initialize_advance() 5 0.01018 0.01018 0.01018 2.46% Gravity::get_new_grav_vector() 5 0.009824 0.009824 0.009824 2.37% Castro::computeTemp() 30 0.009534 0.009534 0.009534 2.30% MLPoisson::define() 6 0.009263 0.009263 0.009263 2.23% MLCellLinOp::correctionResidual() 216 0.00924 0.00924 0.00924 2.23% MultiFab::Dot() 484 0.00912 0.00912 0.00912 2.20% MLMG:computeResOfCorrection() 180 0.007995 0.007995 0.007995 1.93% FabArray::FillBoundary() 1766 0.00792 0.00792 0.00792 1.91% Castro::construct_old_gravity() 5 0.00789 0.00789 0.00789 1.90% Gravity::get_old_grav_vector() 5 0.007885 0.007885 0.007885 1.90% MLMG::mgVcycle_down::1 36 0.007691 0.007691 0.007691 1.86% FabArray::ParallelCopy() 380 0.007546 0.007546 0.007546 1.82% FillBoundary_nowait() 1766 0.007506 0.007506 0.007506 1.81% FabArray::ParallelCopy_nowait() 380 0.007408 0.007408 0.007408 1.79% MLMG::mgVcycle_down::2 36 0.007403 0.007403 0.007403 1.79% MLMG::mgVcycle_down::3 36 0.007063 0.007063 0.007063 1.70% MLMG::mgVcycle_down::4 36 0.00676 0.00676 0.00676 1.63% Castro::post_restart() 1 0.006574 0.006574 0.006574 1.59% Castro::do_new_sources() 5 0.006533 0.006533 0.006533 1.58% Gravity::fill_multipole_BCs() 6 0.006423 0.006423 0.006423 1.55% MLCellLinOp::defineAuxData() 6 0.006362 0.006362 0.006362 1.53% Castro::enforce_min_density() 30 0.006265 0.006265 0.006265 1.51% Gravity::multilevel_solve_for_new_phi() 1 0.006182 0.006182 0.006182 1.49% Gravity::actual_multilevel_solve() 1 0.006165 0.006165 0.006165 1.49% MLMG::mgVcycle_up::2 36 0.005768 0.005768 0.005768 1.39% MLMG::addInterpCorrection() 180 0.005751 0.005751 0.005751 1.39% MLMG::mgVcycle_up::1 36 0.005697 0.005697 0.005697 1.37% MLCGSolver::ParallelAllReduce 659 0.00552 0.00552 0.00552 1.33% amrex::average_down 180 0.005518 0.005518 0.005518 1.33% MLMG::mgVcycle_up::3 36 0.00547 0.00547 0.00547 1.32% CGSolver::sxay() 690 0.005459 0.005459 0.005459 1.32% MLMG::mgVcycle_up::4 36 0.005429 0.005429 0.005429 1.31% MultiFab::LinComb() 690 0.005245 0.005245 0.005245 1.27% Castro::do_old_sources() 5 0.005055 0.005055 0.005055 1.22% Castro::post_timestep() 5 0.004669 0.004669 0.004669 1.13% MLPoisson::Fapply() 500 0.004575 0.004575 0.004575 1.10% FabArray::setDomainBndry() 20 0.004125 0.004125 0.004125 1.00% MLMG::prepareForSolve() 6 0.003447 0.003447 0.003447 0.83% MLCellLinOp::solutionResidual() 42 0.003383 0.003383 0.003383 0.82% MLMG::computeResidual() 36 0.002805 0.002805 0.002805 0.68% MLCellLinOp::defineBC() 6 0.002736 0.002736 0.002736 0.66% Castro::estTimeStep() 10 0.002704 0.002704 0.002704 0.65% BndryData::define() 6 0.002629 0.002629 0.002629 0.63% MultiFab::Xpay() 258 0.002621 0.002621 0.002621 0.63% Castro::reset_internal_energy() 30 0.002374 0.002374 0.002374 0.57% Castro::enforce_speed_limit() 30 0.00174 0.00174 0.00174 0.42% Castro::construct_new_source() 25 0.001692 0.001692 0.001692 0.41% MLMG::getGradSolution() 6 0.001676 0.001676 0.001676 0.40% MLCellLinOp::compGrad() 6 0.001673 0.001673 0.001673 0.40% Castro::construct_new_gravity_source() 5 0.001546 0.001546 0.001546 0.37% Castro::construct_old_source() 25 0.001454 0.001454 0.001454 0.35% Castro::construct_old_gravity_source() 5 0.001418 0.001418 0.001418 0.34% Castro::computeNewDt() 5 0.001417 0.001417 0.001417 0.34% Castro::normalize_species() 30 0.001301 0.001301 0.001301 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001191 0.001191 0.001191 0.29% Castro::apply_source_to_state() 10 0.0009239 0.0009239 0.0009239 0.22% MultiFab::Saxpy() 10 0.0009179 0.0009179 0.0009179 0.22% Gravity::swapTimeLevels() 5 0.0008847 0.0008847 0.0008847 0.21% FabArrayBase::getCPC() 632 0.0008679 0.0008679 0.0008679 0.21% MLMG::ResNormInf() 42 0.0008352 0.0008352 0.0008352 0.20% MLCellLinOp::setLevelBC() 6 0.000819 0.000819 0.000819 0.20% Gravity::update_max_rhs() 6 0.0007345 0.0007345 0.0007345 0.18% MLPoisson::prepareForSolve() 6 0.0006983 0.0006983 0.0006983 0.17% MLCellLinOp::prepareForSolve() 6 0.0006936 0.0006936 0.0006936 0.17% FabArray::mult() 22 0.0006521 0.0006521 0.0006521 0.16% MLMG::computeMLResidual() 6 0.000596 0.000596 0.000596 0.14% FabArrayBase::CPC::define() 244 0.000435 0.000435 0.000435 0.10% FabArrayBase::getFB() 1766 0.0003936 0.0003936 0.0003936 0.09% Amr::InitAmr() 1 0.0003781 0.0003781 0.0003781 0.09% MultiFab::Copy() 6 0.0003523 0.0003523 0.0003523 0.08% MultiFab::min() 5 0.0001634 0.0001634 0.0001634 0.04% MLLinOp::define() 6 0.000151 0.000151 0.000151 0.04% MultiFab::max() 6 0.0001349 0.0001349 0.0001349 0.03% MLMG::MLResNormInf() 6 0.0001336 0.0001336 0.0001336 0.03% MLLinOp::defineGrids() 6 0.0001236 0.0001236 0.0001236 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% FabArrayBase::FB::FB() 26 6.253e-05 6.253e-05 6.253e-05 0.02% Castro::create_source_corrector() 5 4.291e-05 4.291e-05 4.291e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.805e-05 2.805e-05 2.805e-05 0.01% Castro::finalize_advance() 5 2.724e-05 2.724e-05 2.724e-05 0.01% Castro::swap_state_time_levels() 5 2.676e-05 2.676e-05 2.676e-05 0.01% Amr::writeSmallPlotFile() 1 2.378e-05 2.378e-05 2.378e-05 0.01% Castro::finalize_do_advance() 5 2.106e-05 2.106e-05 2.106e-05 0.01% makeSFC 30 2.025e-05 2.025e-05 2.025e-05 0.00% DistributionMapping::Distribute() 31 8.751e-06 8.751e-06 8.751e-06 0.00% Amr::initSubcycle() 1 8.747e-06 8.747e-06 8.747e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.879e-06 5.879e-06 5.879e-06 0.00% MLMG::buildFineMask() 6 3.6e-06 3.6e-06 3.6e-06 0.00% Gravity::set_mass_offset() 6 2.318e-06 2.318e-06 2.318e-06 0.00% Castro::retry_advance_ctu() 5 1.762e-06 1.762e-06 1.762e-06 0.00% Castro::FluxRegFineAdd() 5 1.408e-06 1.408e-06 1.408e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.027e-06 1.027e-06 1.027e-06 0.00% AmrLevel::AmrLevel() 1 9.47e-07 9.47e-07 9.47e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-10-g76d08651adb9) finalized