Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-17-gd8bc97f92a1a) initialized Starting run at 09:44:53 UTC on 2022-11-16. Successfully read inputs file ... Castro git describe: 22.11-12-g3b5735e85 AMReX git describe: 22.11-17-gd8bc97f92 Microphysics git describe: 22.11-22-gb5f64698 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.054264607 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.031253979 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.051291565 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.053180306 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.062834917 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.063544149 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.06732892 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.050195899 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.057647985 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.054687163 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.075428402 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.069179984 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059541124 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.081972785 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030827999 seconds Ending run at 09:44:54 UTC on 2022-11-16. Run time = 0.917020244 Run time without initialization = 0.778321074 Average number of zones advanced per microsecond: 3.368 Average number of zones advanced per microsecond per rank: 3.368 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.917 ... 0.917 ... 0.917 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2157 0.2157 0.2157 23.52% VisMF::Write(FabArray) 11 0.2093 0.2093 0.2093 22.83% MLCellLinOp::applyBC() 4433 0.08202 0.08202 0.08202 8.94% MLPoisson::Fsmooth() 3280 0.06535 0.06535 0.06535 7.13% Amr::checkPoint() 3 0.03631 0.03631 0.03631 3.96% MLCGSolver::bicgstab 82 0.02456 0.02456 0.02456 2.68% StateData::FillBoundary(geom) 328 0.02363 0.02363 0.02363 2.58% MultiFab::Dot() 1114 0.02298 0.02298 0.02298 2.51% StateDataPhysBCFunct::() 41 0.01784 0.01784 0.01784 1.95% Castro::normalize_species() 62 0.01547 0.01547 0.01547 1.69% MultiFab::LinComb() 1586 0.0148 0.0148 0.0148 1.61% Castro::computeTemp() 63 0.01466 0.01466 0.01466 1.60% FabArray::setVal() 1144 0.01455 0.01455 0.01455 1.59% FillBoundary_nowait() 4023 0.0144 0.0144 0.0144 1.57% FabArray::ParallelCopy_nowait() 861 0.01342 0.01342 0.01342 1.46% MLPoisson::Fapply() 1142 0.01213 0.01213 0.01213 1.32% MLCellLinOp::defineAuxData() 11 0.01196 0.01196 0.01196 1.30% Gravity::fill_multipole_BCs() 11 0.009115 0.009115 0.009115 0.99% MLMG::addInterpCorrection() 410 0.007981 0.007981 0.007981 0.87% Castro::enforce_min_density() 62 0.007954 0.007954 0.007954 0.87% amrex::average_down 410 0.006986 0.006986 0.006986 0.76% MultiFab::Xpay() 585 0.006761 0.006761 0.006761 0.74% Castro::estTimeStep() 21 0.0058 0.0058 0.0058 0.63% Castro::do_advance_ctu() 10 0.005078 0.005078 0.005078 0.55% Castro::reset_internal_energy(MultiFab) 63 0.004628 0.004628 0.004628 0.50% BndryData::define() 11 0.004101 0.004101 0.004101 0.45% Castro::construct_new_gravity_source() 10 0.003309 0.003309 0.003309 0.36% Amr::writePlotFile() 2 0.002844 0.002844 0.002844 0.31% Castro::construct_old_gravity_source() 10 0.002599 0.002599 0.002599 0.28% MLMG::ResNormInf() 93 0.002112 0.002112 0.002112 0.23% Gravity::get_new_grav_vector() 11 0.001941 0.001941 0.001941 0.21% MultiFab::Saxpy() 20 0.001801 0.001801 0.001801 0.20% Gravity::get_old_grav_vector() 10 0.001754 0.001754 0.001754 0.19% Castro::expand_state() 10 0.001734 0.001734 0.001734 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001725 0.001725 0.001725 0.19% MultiFab::Add() 82 0.001704 0.001704 0.001704 0.19% Castro::reset_internal_energy(Fab) 504 0.001693 0.001693 0.001693 0.18% MLCellLinOp::setLevelBC() 11 0.001578 0.001578 0.001578 0.17% Gravity::actual_solve_with_mlmg() 11 0.001457 0.001457 0.001457 0.16% FabArray::mult() 43 0.001334 0.001334 0.001334 0.15% FabArray::setDomainBndry() 41 0.001324 0.001324 0.001324 0.14% Castro::initData() 1 0.001323 0.001323 0.001323 0.14% MLMG::prepareForSolve() 11 0.001246 0.001246 0.001246 0.14% MLCellLinOp::prepareForSolve() 11 0.001187 0.001187 0.001187 0.13% MultiFab::contains_nan() 20 0.001184 0.001184 0.001184 0.13% MLCellLinOp::smooth() 1640 0.001072 0.001072 0.001072 0.12% MLCellLinOp::compGrad() 11 0.0009353 0.0009353 0.0009353 0.10% Castro::enforce_speed_limit() 62 0.0008859 0.0008859 0.0008859 0.10% FabArray::FillBoundary() 4023 0.0008414 0.0008414 0.0008414 0.09% Castro::subcycle_advance_ctu() 10 0.0007855 0.0007855 0.0007855 0.09% FabArrayBase::getCPC() 1323 0.0007701 0.0007701 0.0007701 0.08% FabArrayBase::CPC::define() 454 0.0006367 0.0006367 0.0006367 0.07% Castro::finalize_advance() 10 0.0006167 0.0006167 0.0006167 0.07% FabArrayBase::getFB() 4023 0.0006119 0.0006119 0.0006119 0.07% Amr::InitAmr() 1 0.0004728 0.0004728 0.0004728 0.05% MLCellLinOp::apply() 1142 0.000444 0.000444 0.000444 0.05% Gravity::solve_for_phi() 10 0.0004221 0.0004221 0.0004221 0.05% Gravity::update_max_rhs() 11 0.0004194 0.0004194 0.0004194 0.05% CGSolver::sxay() 1586 0.0003931 0.0003931 0.0003931 0.04% Amr::coarseTimeStep() 10 0.0003901 0.0003901 0.0003901 0.04% MultiFab::Copy() 11 0.0003376 0.0003376 0.0003376 0.04% FillPatchIterator::Initialize 41 0.0002974 0.0002974 0.0002974 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002899 0.0002899 0.0002899 0.03% MLCellLinOp::defineBC() 11 0.0002792 0.0002792 0.0002792 0.03% MultiFab::max() 11 0.0002596 0.0002596 0.0002596 0.03% main() 1 0.0002517 0.0002517 0.0002517 0.03% FabArray::ParallelCopy() 861 0.000248 0.000248 0.000248 0.03% MLCellLinOp::correctionResidual() 492 0.0002374 0.0002374 0.0002374 0.03% MLMG::MLRhsNormInf() 11 0.0002198 0.0002198 0.0002198 0.02% MLMG::mgVcycle() 82 0.0002097 0.0002097 0.0002097 0.02% MLLinOp::defineGrids() 11 0.000209 0.000209 0.000209 0.02% Castro::construct_new_gravity() 10 0.000205 0.000205 0.000205 0.02% MLMG:computeResOfCorrection() 410 0.0001544 0.0001544 0.0001544 0.02% StateData::checkPoint() 12 0.0001535 0.0001535 0.0001535 0.02% Amr::timeStep() 10 0.0001502 0.0001502 0.0001502 0.02% MLMG::mgVcycle_down::0 82 0.0001204 0.0001204 0.0001204 0.01% MLMG::mgVcycle_down::1 82 0.0001093 0.0001093 0.0001093 0.01% AmrLevel::checkPoint() 3 0.0001084 0.0001084 0.0001084 0.01% MLMG::mgVcycle_down::2 82 0.0001038 0.0001038 0.0001038 0.01% MLMG::mgVcycle_down::3 82 9.724e-05 9.724e-05 9.724e-05 0.01% MLMG::mgVcycle_down::4 82 9.497e-05 9.497e-05 9.497e-05 0.01% Castro::Castro() 1 8.405e-05 8.405e-05 8.405e-05 0.01% FabArrayBase::FB::FB() 56 8.388e-05 8.388e-05 8.388e-05 0.01% MLMG::actualBottomSolve() 82 8.293e-05 8.293e-05 8.293e-05 0.01% Castro::initialize_advance() 10 8.28e-05 8.28e-05 8.28e-05 0.01% Castro::advance() 10 8.164e-05 8.164e-05 8.164e-05 0.01% Castro::clean_state() 62 7.98e-05 7.98e-05 7.98e-05 0.01% MLMG::mgVcycle_up::4 82 7.663e-05 7.663e-05 7.663e-05 0.01% MLMG::solve() 11 6.778e-05 6.778e-05 6.778e-05 0.01% MLMG::oneIter() 82 6.752e-05 6.752e-05 6.752e-05 0.01% MLMG::mgVcycle_up::1 82 6.54e-05 6.54e-05 6.54e-05 0.01% Castro::initialize_do_advance() 10 6.43e-05 6.43e-05 6.43e-05 0.01% MLMG::mgVcycle_up::0 82 6.194e-05 6.194e-05 6.194e-05 0.01% MLMG::mgVcycle_up::3 82 6.111e-05 6.111e-05 6.111e-05 0.01% MLMG::mgVcycle_up::2 82 5.93e-05 5.93e-05 5.93e-05 0.01% MLCellLinOp::solutionResidual() 93 4.746e-05 4.746e-05 4.746e-05 0.01% MLMG::computeResidual() 82 4.221e-05 4.221e-05 4.221e-05 0.00% StateData::define() 4 4.027e-05 4.027e-05 4.027e-05 0.00% Castro::swap_state_time_levels() 10 3.812e-05 3.812e-05 3.812e-05 0.00% MLMG::mgVcycle_bottom 82 3.63e-05 3.63e-05 3.63e-05 0.00% Castro::enforce_consistent_e() 1 3.478e-05 3.478e-05 3.478e-05 0.00% Gravity::actual_multilevel_solve() 1 3.25e-05 3.25e-05 3.25e-05 0.00% Castro::finalize_do_advance() 10 3.15e-05 3.15e-05 3.15e-05 0.00% FillPatchSingleLevel 41 2.694e-05 2.694e-05 2.694e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.572e-05 2.572e-05 2.572e-05 0.00% Castro::initMFs() 1 2.568e-05 2.568e-05 2.568e-05 0.00% Amr::writeSmallPlotFile() 1 2.453e-05 2.453e-05 2.453e-05 0.00% makeSFC 55 2.448e-05 2.448e-05 2.448e-05 0.00% MLPoisson::define() 11 2.401e-05 2.401e-05 2.401e-05 0.00% MLLinOp::define() 11 2.398e-05 2.398e-05 2.398e-05 0.00% Amr::defBaseLevel() 1 2.383e-05 2.383e-05 2.383e-05 0.00% Castro::buildMetrics() 1 2.316e-05 2.316e-05 2.316e-05 0.00% Amr::FinalizeInit() 1 1.936e-05 1.936e-05 1.936e-05 0.00% Castro::construct_old_source() 50 1.882e-05 1.882e-05 1.882e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.745e-05 1.745e-05 1.745e-05 0.00% Castro::construct_new_source() 50 1.685e-05 1.685e-05 1.685e-05 0.00% Castro::do_new_sources() 10 1.607e-05 1.607e-05 1.607e-05 0.00% Castro::do_old_sources() 10 1.565e-05 1.565e-05 1.565e-05 0.00% DistributionMapping::Distribute() 56 1.447e-05 1.447e-05 1.447e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.289e-05 1.289e-05 1.289e-05 0.00% Castro::construct_old_gravity() 10 1.11e-05 1.11e-05 1.11e-05 0.00% Castro::check_for_nan() 20 1.105e-05 1.105e-05 1.105e-05 0.00% Castro::apply_source_to_state() 20 1.089e-05 1.089e-05 1.089e-05 0.00% MLMG::computeMLResidual() 11 9.842e-06 9.842e-06 9.842e-06 0.00% Castro::post_timestep() 10 9.451e-06 9.451e-06 9.451e-06 0.00% MLPoisson::prepareForSolve() 11 8.937e-06 8.937e-06 8.937e-06 0.00% Gravity::swapTimeLevels() 10 8.668e-06 8.668e-06 8.668e-06 0.00% Amr::initSubcycle() 1 8.423e-06 8.423e-06 8.423e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.37e-06 7.37e-06 7.37e-06 0.00% Castro::computeNewDt() 9 6.929e-06 6.929e-06 6.929e-06 0.00% MLMG::getGradSolution() 11 6.583e-06 6.583e-06 6.583e-06 0.00% Amr::InitializeInit() 1 4.944e-06 4.944e-06 4.944e-06 0.00% AmrLevel::checkPointPost() 3 4.674e-06 4.674e-06 4.674e-06 0.00% Gravity::set_mass_offset() 11 4.077e-06 4.077e-06 4.077e-06 0.00% AmrLevel::checkPointPre() 3 3.996e-06 3.996e-06 3.996e-06 0.00% Castro::retry_advance_ctu() 10 3.903e-06 3.903e-06 3.903e-06 0.00% Castro::create_source_corrector() 10 3.8e-06 3.8e-06 3.8e-06 0.00% MLMG::MLResNormInf() 11 3.426e-06 3.426e-06 3.426e-06 0.00% Castro::post_init() 1 3.319e-06 3.319e-06 3.319e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.89e-06 2.89e-06 2.89e-06 0.00% Castro::FluxRegCrseInit 10 2.842e-06 2.842e-06 2.842e-06 0.00% Amr::init() 1 2.502e-06 2.502e-06 2.502e-06 0.00% Castro::computeInitialDt() 2 2.344e-06 2.344e-06 2.344e-06 0.00% Castro::FluxRegFineAdd() 10 2.032e-06 2.032e-06 2.032e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.716e-06 1.716e-06 1.716e-06 0.00% Castro::post_regrid() 1 1.108e-06 1.108e-06 1.108e-06 0.00% Amr::initialInit() 1 9.43e-07 9.43e-07 9.43e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.917 0.917 0.917 100.00% Amr::coarseTimeStep() 10 0.7473 0.7473 0.7473 81.49% Amr::timeStep() 10 0.6105 0.6105 0.6105 66.57% Castro::advance() 10 0.6035 0.6035 0.6035 65.81% Castro::subcycle_advance_ctu() 10 0.5921 0.5921 0.5921 64.57% Castro::do_advance_ctu() 10 0.5913 0.5913 0.5913 64.48% Gravity::solve_phi_with_mlmg() 11 0.3229 0.3229 0.3229 35.21% Gravity::actual_solve_with_mlmg() 11 0.3136 0.3136 0.3136 34.19% Castro::construct_new_gravity() 10 0.2942 0.2942 0.2942 32.08% MLMG::solve() 11 0.2901 0.2901 0.2901 31.63% Gravity::solve_for_phi() 10 0.2784 0.2784 0.2784 30.36% MLMG::oneIter() 82 0.2749 0.2749 0.2749 29.97% MLMG::mgVcycle() 82 0.2731 0.2731 0.2731 29.78% Castro::construct_ctu_hydro_source() 10 0.2157 0.2157 0.2157 23.52% VisMF::Write(FabArray) 11 0.2093 0.2093 0.2093 22.83% Amr::checkPoint() 3 0.1866 0.1866 0.1866 20.34% AmrLevel::checkPoint() 3 0.1502 0.1502 0.1502 16.38% StateData::checkPoint() 12 0.1501 0.1501 0.1501 16.37% MLCellLinOp::smooth() 1640 0.1393 0.1393 0.1393 15.19% Amr::init() 1 0.1381 0.1381 0.1381 15.06% MLCellLinOp::applyBC() 4433 0.09796 0.09796 0.09796 10.68% MLMG::mgVcycle_bottom 82 0.0842 0.0842 0.0842 9.18% MLMG::actualBottomSolve() 82 0.08417 0.08417 0.08417 9.18% MLCGSolver::bicgstab 82 0.08334 0.08334 0.08334 9.09% MLPoisson::Fsmooth() 3280 0.06535 0.06535 0.06535 7.13% Amr::writePlotFile() 2 0.06221 0.06221 0.06221 6.78% Amr::initialInit() 1 0.05247 0.05247 0.05247 5.72% Amr::FinalizeInit() 1 0.04816 0.04816 0.04816 5.25% FillPatchIterator::Initialize 41 0.04707 0.04707 0.04707 5.13% Castro::post_init() 1 0.04686 0.04686 0.04686 5.11% FillPatchSingleLevel 41 0.04545 0.04545 0.04545 4.96% Gravity::multilevel_solve_for_new_phi() 1 0.04496 0.04496 0.04496 4.90% Gravity::actual_multilevel_solve() 1 0.04495 0.04495 0.04495 4.90% Castro::clean_state() 62 0.04444 0.04444 0.04444 4.85% StateDataPhysBCFunct::() 41 0.04147 0.04147 0.04147 4.52% MLCellLinOp::apply() 1142 0.03715 0.03715 0.03715 4.05% MLMG::mgVcycle_down::0 82 0.036 0.036 0.036 3.93% MLMG::mgVcycle_up::0 82 0.03099 0.03099 0.03099 3.38% StateData::FillBoundary(geom) 328 0.02363 0.02363 0.02363 2.58% Castro::initialize_do_advance() 10 0.02317 0.02317 0.02317 2.53% MultiFab::Dot() 1114 0.02298 0.02298 0.02298 2.51% MLCellLinOp::correctionResidual() 492 0.02184 0.02184 0.02184 2.38% Castro::computeTemp() 63 0.02098 0.02098 0.02098 2.29% MLPoisson::define() 11 0.01897 0.01897 0.01897 2.07% MLMG:computeResOfCorrection() 410 0.01886 0.01886 0.01886 2.06% MLMG::mgVcycle_down::1 82 0.01821 0.01821 0.01821 1.99% MLMG::mgVcycle_down::2 82 0.01772 0.01772 0.01772 1.93% Gravity::get_new_grav_vector() 11 0.0174 0.0174 0.0174 1.90% MLMG::mgVcycle_down::3 82 0.01686 0.01686 0.01686 1.84% MLMG::mgVcycle_down::4 82 0.01604 0.01604 0.01604 1.75% FabArray::FillBoundary() 4023 0.01594 0.01594 0.01594 1.74% Castro::expand_state() 10 0.0156 0.0156 0.0156 1.70% Castro::normalize_species() 62 0.01547 0.01547 0.01547 1.69% CGSolver::sxay() 1586 0.01519 0.01519 0.01519 1.66% FillBoundary_nowait() 4023 0.0151 0.0151 0.0151 1.65% Castro::construct_old_gravity() 10 0.01482 0.01482 0.01482 1.62% Gravity::get_old_grav_vector() 10 0.01481 0.01481 0.01481 1.61% MultiFab::LinComb() 1586 0.0148 0.0148 0.0148 1.61% FabArray::setVal() 1144 0.01455 0.01455 0.01455 1.59% FabArray::ParallelCopy() 861 0.01448 0.01448 0.01448 1.58% FabArray::ParallelCopy_nowait() 861 0.01423 0.01423 0.01423 1.55% MLCGSolver::ParallelAllReduce 1514 0.01367 0.01367 0.01367 1.49% MLMG::mgVcycle_up::2 82 0.01364 0.01364 0.01364 1.49% MLMG::mgVcycle_up::1 82 0.01345 0.01345 0.01345 1.47% MLCellLinOp::defineAuxData() 11 0.0133 0.0133 0.0133 1.45% MLMG::addInterpCorrection() 410 0.01326 0.01326 0.01326 1.45% Castro::do_new_sources() 10 0.01309 0.01309 0.01309 1.43% MLMG::mgVcycle_up::3 82 0.01298 0.01298 0.01298 1.42% MLMG::mgVcycle_up::4 82 0.01281 0.01281 0.01281 1.40% amrex::average_down 410 0.01223 0.01223 0.01223 1.33% MLPoisson::Fapply() 1142 0.01213 0.01213 0.01213 1.32% Castro::initialize_advance() 10 0.01071 0.01071 0.01071 1.17% Castro::do_old_sources() 10 0.01046 0.01046 0.01046 1.14% Gravity::fill_multipole_BCs() 11 0.009115 0.009115 0.009115 0.99% Castro::enforce_min_density() 62 0.007954 0.007954 0.007954 0.87% MLCellLinOp::solutionResidual() 93 0.007268 0.007268 0.007268 0.79% Castro::post_timestep() 10 0.006839 0.006839 0.006839 0.75% MultiFab::Xpay() 585 0.006761 0.006761 0.006761 0.74% Castro::reset_internal_energy(MultiFab) 63 0.006321 0.006321 0.006321 0.69% MLMG::computeResidual() 82 0.006288 0.006288 0.006288 0.69% Castro::estTimeStep() 21 0.0058 0.0058 0.0058 0.63% MLMG::prepareForSolve() 11 0.005496 0.005496 0.005496 0.60% MLCellLinOp::defineBC() 11 0.005356 0.005356 0.005356 0.58% BndryData::define() 11 0.005077 0.005077 0.005077 0.55% Amr::InitializeInit() 1 0.004314 0.004314 0.004314 0.47% Amr::defBaseLevel() 1 0.004309 0.004309 0.004309 0.47% Castro::initData() 1 0.003792 0.003792 0.003792 0.41% Castro::computeNewDt() 9 0.003452 0.003452 0.003452 0.38% Castro::construct_new_source() 50 0.003326 0.003326 0.003326 0.36% Castro::construct_new_gravity_source() 10 0.003309 0.003309 0.003309 0.36% Castro::construct_old_source() 50 0.002618 0.002618 0.002618 0.29% Castro::construct_old_gravity_source() 10 0.002599 0.002599 0.002599 0.28% MLMG::ResNormInf() 93 0.002112 0.002112 0.002112 0.23% Castro::apply_source_to_state() 20 0.001812 0.001812 0.001812 0.20% MultiFab::Saxpy() 20 0.001801 0.001801 0.001801 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001725 0.001725 0.001725 0.19% MultiFab::Add() 82 0.001704 0.001704 0.001704 0.19% Castro::reset_internal_energy(Fab) 504 0.001693 0.001693 0.001693 0.18% MLCellLinOp::setLevelBC() 11 0.001578 0.001578 0.001578 0.17% MLMG::getGradSolution() 11 0.001449 0.001449 0.001449 0.16% MLCellLinOp::compGrad() 11 0.001442 0.001442 0.001442 0.16% FabArrayBase::getCPC() 1323 0.001407 0.001407 0.001407 0.15% FabArray::mult() 43 0.001334 0.001334 0.001334 0.15% FabArray::setDomainBndry() 41 0.001324 0.001324 0.001324 0.14% MLPoisson::prepareForSolve() 11 0.001196 0.001196 0.001196 0.13% Castro::check_for_nan() 20 0.001195 0.001195 0.001195 0.13% MLCellLinOp::prepareForSolve() 11 0.001187 0.001187 0.001187 0.13% MultiFab::contains_nan() 20 0.001184 0.001184 0.001184 0.13% Castro::post_regrid() 1 0.001121 0.001121 0.001121 0.12% MLMG::computeMLResidual() 11 0.001033 0.001033 0.001033 0.11% Castro::enforce_speed_limit() 62 0.0008859 0.0008859 0.0008859 0.10% Castro::computeInitialDt() 2 0.0008526 0.0008526 0.0008526 0.09% Gravity::update_max_rhs() 11 0.0008272 0.0008272 0.0008272 0.09% FabArrayBase::getFB() 4023 0.0006958 0.0006958 0.0006958 0.08% FabArrayBase::CPC::define() 454 0.0006367 0.0006367 0.0006367 0.07% Castro::finalize_advance() 10 0.0006216 0.0006216 0.0006216 0.07% Amr::InitAmr() 1 0.0004813 0.0004813 0.0004813 0.05% Gravity::swapTimeLevels() 10 0.0004686 0.0004686 0.0004686 0.05% Castro::Castro() 1 0.0004406 0.0004406 0.0004406 0.05% MultiFab::Copy() 11 0.0003376 0.0003376 0.0003376 0.04% MLLinOp::define() 11 0.0002845 0.0002845 0.0002845 0.03% MLMG::MLResNormInf() 11 0.000283 0.000283 0.000283 0.03% MLLinOp::defineGrids() 11 0.0002606 0.0002606 0.0002606 0.03% MultiFab::max() 11 0.0002596 0.0002596 0.0002596 0.03% MLMG::MLRhsNormInf() 11 0.0002198 0.0002198 0.0002198 0.02% Castro::buildMetrics() 1 0.0001651 0.0001651 0.0001651 0.02% FabArrayBase::FB::FB() 56 8.388e-05 8.388e-05 8.388e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 4.988e-05 4.988e-05 4.988e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.764e-05 4.764e-05 4.764e-05 0.01% StateData::define() 4 4.027e-05 4.027e-05 4.027e-05 0.00% Castro::swap_state_time_levels() 10 3.812e-05 3.812e-05 3.812e-05 0.00% makeSFC 55 3.699e-05 3.699e-05 3.699e-05 0.00% Castro::enforce_consistent_e() 1 3.478e-05 3.478e-05 3.478e-05 0.00% Castro::finalize_do_advance() 10 3.15e-05 3.15e-05 3.15e-05 0.00% Castro::initMFs() 1 2.568e-05 2.568e-05 2.568e-05 0.00% Amr::writeSmallPlotFile() 1 2.453e-05 2.453e-05 2.453e-05 0.00% DistributionMapping::Distribute() 56 1.447e-05 1.447e-05 1.447e-05 0.00% Amr::initSubcycle() 1 8.423e-06 8.423e-06 8.423e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.853e-06 4.853e-06 4.853e-06 0.00% AmrLevel::checkPointPost() 3 4.674e-06 4.674e-06 4.674e-06 0.00% Gravity::set_mass_offset() 11 4.077e-06 4.077e-06 4.077e-06 0.00% AmrLevel::checkPointPre() 3 3.996e-06 3.996e-06 3.996e-06 0.00% Castro::retry_advance_ctu() 10 3.903e-06 3.903e-06 3.903e-06 0.00% Castro::create_source_corrector() 10 3.8e-06 3.8e-06 3.8e-06 0.00% Castro::FluxRegCrseInit 10 2.842e-06 2.842e-06 2.842e-06 0.00% Castro::FluxRegFineAdd() 10 2.032e-06 2.032e-06 2.032e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.716e-06 1.716e-06 1.716e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-17-gd8bc97f92a1a) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-17-gd8bc97f92a1a) initialized Starting run at 09:44:54 UTC on 2022-11-16. Successfully read inputs file ... Castro git describe: 22.11-12-g3b5735e85 AMReX git describe: 22.11-17-gd8bc97f92 Microphysics git describe: 22.11-22-gb5f64698 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.487093353 Restart time = 0.048696654 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.054350678 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05225419 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061272166 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062802074 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.080879565 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032588321 seconds Ending run at 09:44:55 UTC on 2022-11-16. Run time = 0.39387724 Run time without initialization = 0.344597625 Average number of zones advanced per microsecond: 3.804 Average number of zones advanced per microsecond per rank: 3.804 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3939 ... 0.3939 ... 0.3939 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1093 0.1093 0.1093 27.75% VisMF::Read() 3 0.04076 0.04076 0.04076 10.35% MLCellLinOp::applyBC() 1946 0.03517 0.03517 0.03517 8.93% VisMF::Write(FabArray) 1 0.03109 0.03109 0.03109 7.89% MLPoisson::Fsmooth() 1440 0.02756 0.02756 0.02756 7.00% StateData::FillBoundary(geom) 160 0.01177 0.01177 0.01177 2.99% MLCGSolver::bicgstab 36 0.01039 0.01039 0.01039 2.64% MultiFab::Dot() 484 0.009678 0.009678 0.009678 2.46% Castro::normalize_species() 30 0.008786 0.008786 0.008786 2.23% Castro::computeTemp() 30 0.008068 0.008068 0.008068 2.05% FabArray::setVal() 537 0.00687 0.00687 0.00687 1.74% MLCellLinOp::defineAuxData() 6 0.006279 0.006279 0.006279 1.59% FillBoundary_nowait() 1766 0.006277 0.006277 0.006277 1.59% MultiFab::LinComb() 690 0.006197 0.006197 0.006197 1.57% FabArray::ParallelCopy_nowait() 380 0.006042 0.006042 0.006042 1.53% Castro::enforce_min_density() 30 0.005667 0.005667 0.005667 1.44% StateDataPhysBCFunct::() 20 0.005525 0.005525 0.005525 1.40% MLPoisson::Fapply() 500 0.005161 0.005161 0.005161 1.31% Gravity::fill_multipole_BCs() 6 0.004673 0.004673 0.004673 1.19% Amr::restart() 1 0.00365 0.00365 0.00365 0.93% MLMG::addInterpCorrection() 180 0.003401 0.003401 0.003401 0.86% amrex::average_down 180 0.002977 0.002977 0.002977 0.76% MultiFab::Xpay() 258 0.002911 0.002911 0.002911 0.74% Castro::do_advance_ctu() 5 0.002485 0.002485 0.002485 0.63% Castro::estTimeStep() 10 0.00247 0.00247 0.00247 0.63% BndryData::define() 6 0.002158 0.002158 0.002158 0.55% Castro::construct_new_gravity_source() 5 0.001782 0.001782 0.001782 0.45% Castro::reset_internal_energy(Fab) 240 0.001777 0.001777 0.001777 0.45% Castro::reset_internal_energy(MultiFab) 30 0.001654 0.001654 0.001654 0.42% Amr::writePlotFile() 1 0.001592 0.001592 0.001592 0.40% Castro::construct_old_gravity_source() 5 0.001501 0.001501 0.001501 0.38% Castro::enforce_speed_limit() 30 0.001282 0.001282 0.001282 0.33% Gravity::get_old_grav_vector() 5 0.0009821 0.0009821 0.0009821 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009301 0.0009301 0.0009301 0.24% MLMG::ResNormInf() 42 0.0009221 0.0009221 0.0009221 0.23% MultiFab::Saxpy() 10 0.0009185 0.0009185 0.0009185 0.23% Gravity::get_new_grav_vector() 5 0.0008923 0.0008923 0.0008923 0.23% Castro::expand_state() 5 0.0008699 0.0008699 0.0008699 0.22% MLCellLinOp::setLevelBC() 6 0.000824 0.000824 0.000824 0.21% MultiFab::Add() 36 0.0007319 0.0007319 0.0007319 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007256 0.0007256 0.0007256 0.18% FabArray::setDomainBndry() 20 0.0006585 0.0006585 0.0006585 0.17% MLMG::prepareForSolve() 6 0.0006562 0.0006562 0.0006562 0.17% FabArray::mult() 22 0.0006533 0.0006533 0.0006533 0.17% MLCellLinOp::prepareForSolve() 6 0.0006459 0.0006459 0.0006459 0.16% MultiFab::contains_nan() 10 0.0005959 0.0005959 0.0005959 0.15% MLCellLinOp::compGrad() 6 0.0004968 0.0004968 0.0004968 0.13% MLCellLinOp::smooth() 720 0.0004661 0.0004661 0.0004661 0.12% Amr::InitAmr() 1 0.0004054 0.0004054 0.0004054 0.10% FabArray::FillBoundary() 1766 0.0003918 0.0003918 0.0003918 0.10% FabArrayBase::CPC::define() 244 0.0003742 0.0003742 0.0003742 0.09% FabArrayBase::getCPC() 632 0.0003698 0.0003698 0.0003698 0.09% Castro::finalize_advance() 5 0.0003372 0.0003372 0.0003372 0.09% main() 1 0.0002693 0.0002693 0.0002693 0.07% FabArrayBase::getFB() 1766 0.0002617 0.0002617 0.0002617 0.07% Gravity::update_max_rhs() 6 0.0002373 0.0002373 0.0002373 0.06% Gravity::solve_for_phi() 5 0.0002089 0.0002089 0.0002089 0.05% Amr::coarseTimeStep() 5 0.0002033 0.0002033 0.0002033 0.05% Castro::subcycle_advance_ctu() 5 0.0002027 0.0002027 0.0002027 0.05% MLCellLinOp::apply() 500 0.0001996 0.0001996 0.0001996 0.05% CGSolver::sxay() 690 0.0001824 0.0001824 0.0001824 0.05% MultiFab::Copy() 6 0.0001788 0.0001788 0.0001788 0.05% Castro::construct_new_gravity() 5 0.0001681 0.0001681 0.0001681 0.04% MultiFab::max() 6 0.0001465 0.0001465 0.0001465 0.04% MLCellLinOp::defineBC() 6 0.0001446 0.0001446 0.0001446 0.04% FillPatchIterator::Initialize 20 0.0001394 0.0001394 0.0001394 0.04% Castro::advance() 5 0.0001374 0.0001374 0.0001374 0.03% MLCGSolver::ParallelAllReduce 659 0.0001243 0.0001243 0.0001243 0.03% Castro::create_source_corrector() 5 0.0001183 0.0001183 0.0001183 0.03% FabArray::ParallelCopy() 380 0.0001169 0.0001169 0.0001169 0.03% MLMG::MLRhsNormInf() 6 0.0001147 0.0001147 0.0001147 0.03% Castro::construct_new_source() 25 0.0001084 0.0001084 0.0001084 0.03% MLCellLinOp::correctionResidual() 216 9.715e-05 9.715e-05 9.715e-05 0.02% MLMG::mgVcycle() 36 9.083e-05 9.083e-05 9.083e-05 0.02% MLLinOp::defineGrids() 6 8.596e-05 8.596e-05 8.596e-05 0.02% AmrLevel::restart() 1 8.367e-05 8.367e-05 8.367e-05 0.02% Amr::timeStep() 5 7.651e-05 7.651e-05 7.651e-05 0.02% StateData::restartDoit() 4 7.191e-05 7.191e-05 7.191e-05 0.02% MLMG:computeResOfCorrection() 180 7.047e-05 7.047e-05 7.047e-05 0.02% Castro::post_timestep() 5 6.251e-05 6.251e-05 6.251e-05 0.02% Castro::construct_old_source() 25 5.843e-05 5.843e-05 5.843e-05 0.01% FabArrayBase::FB::FB() 26 5.518e-05 5.518e-05 5.518e-05 0.01% MLMG::mgVcycle_down::0 36 5.12e-05 5.12e-05 5.12e-05 0.01% MLMG::mgVcycle_down::1 36 4.767e-05 4.767e-05 4.767e-05 0.01% Castro::initialize_do_advance() 5 4.744e-05 4.744e-05 4.744e-05 0.01% MLMG::mgVcycle_down::2 36 4.518e-05 4.518e-05 4.518e-05 0.01% Castro::computeNewDt() 5 4.352e-05 4.352e-05 4.352e-05 0.01% Castro::clean_state() 30 4.317e-05 4.317e-05 4.317e-05 0.01% MLMG::mgVcycle_down::4 36 4.204e-05 4.204e-05 4.204e-05 0.01% MLMG::mgVcycle_down::3 36 4.166e-05 4.166e-05 4.166e-05 0.01% Castro::initialize_advance() 5 4.157e-05 4.157e-05 4.157e-05 0.01% Castro::construct_old_gravity() 5 3.831e-05 3.831e-05 3.831e-05 0.01% MLMG::actualBottomSolve() 36 3.786e-05 3.786e-05 3.786e-05 0.01% Castro::buildMetrics() 1 3.769e-05 3.769e-05 3.769e-05 0.01% MLMG::mgVcycle_up::4 36 3.528e-05 3.528e-05 3.528e-05 0.01% MLMG::mgVcycle_up::3 36 3.129e-05 3.129e-05 3.129e-05 0.01% MLMG::solve() 6 3.071e-05 3.071e-05 3.071e-05 0.01% Gravity::actual_multilevel_solve() 1 3.031e-05 3.031e-05 3.031e-05 0.01% MLMG::oneIter() 36 2.917e-05 2.917e-05 2.917e-05 0.01% MLMG::mgVcycle_up::0 36 2.824e-05 2.824e-05 2.824e-05 0.01% Castro::post_restart() 1 2.809e-05 2.809e-05 2.809e-05 0.01% Amr::writeSmallPlotFile() 1 2.69e-05 2.69e-05 2.69e-05 0.01% MLMG::mgVcycle_up::2 36 2.64e-05 2.64e-05 2.64e-05 0.01% Castro::swap_state_time_levels() 5 2.533e-05 2.533e-05 2.533e-05 0.01% Castro::initMFs() 1 2.524e-05 2.524e-05 2.524e-05 0.01% MLMG::mgVcycle_up::1 36 2.505e-05 2.505e-05 2.505e-05 0.01% MLLinOp::define() 6 2.102e-05 2.102e-05 2.102e-05 0.01% MLCellLinOp::solutionResidual() 42 2.088e-05 2.088e-05 2.088e-05 0.01% MLMG::computeResidual() 36 1.953e-05 1.953e-05 1.953e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.795e-05 1.795e-05 1.795e-05 0.00% Castro::finalize_do_advance() 5 1.776e-05 1.776e-05 1.776e-05 0.00% MLMG::mgVcycle_bottom 36 1.606e-05 1.606e-05 1.606e-05 0.00% MLPoisson::define() 6 1.475e-05 1.475e-05 1.475e-05 0.00% makeSFC 30 1.405e-05 1.405e-05 1.405e-05 0.00% FillPatchSingleLevel 20 1.33e-05 1.33e-05 1.33e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.26e-05 1.26e-05 1.26e-05 0.00% Castro::do_new_sources() 5 1.006e-05 1.006e-05 1.006e-05 0.00% Castro::do_old_sources() 5 8.694e-06 8.694e-06 8.694e-06 0.00% DistributionMapping::Distribute() 31 8.443e-06 8.443e-06 8.443e-06 0.00% Castro::check_for_nan() 10 8.428e-06 8.428e-06 8.428e-06 0.00% Amr::initSubcycle() 1 8.125e-06 8.125e-06 8.125e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.735e-06 6.735e-06 6.735e-06 0.00% Castro::apply_source_to_state() 10 5.381e-06 5.381e-06 5.381e-06 0.00% MLMG::computeMLResidual() 6 5.144e-06 5.144e-06 5.144e-06 0.00% MLPoisson::prepareForSolve() 6 5.135e-06 5.135e-06 5.135e-06 0.00% Gravity::swapTimeLevels() 5 4.349e-06 4.349e-06 4.349e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.657e-06 3.657e-06 3.657e-06 0.00% MLMG::getGradSolution() 6 3.049e-06 3.049e-06 3.049e-06 0.00% Gravity::set_mass_offset() 6 2.035e-06 2.035e-06 2.035e-06 0.00% MLMG::MLResNormInf() 6 2.034e-06 2.034e-06 2.034e-06 0.00% Castro::retry_advance_ctu() 5 1.789e-06 1.789e-06 1.789e-06 0.00% Castro::FluxRegCrseInit 5 1.681e-06 1.681e-06 1.681e-06 0.00% Amr::init() 1 1.139e-06 1.139e-06 1.139e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.016e-06 1.016e-06 1.016e-06 0.00% Castro::FluxRegFineAdd() 5 1.009e-06 1.009e-06 1.009e-06 0.00% AmrLevel::AmrLevel() 1 8.29e-07 8.29e-07 8.29e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3939 0.3939 0.3939 100.00% Amr::coarseTimeStep() 5 0.3118 0.3118 0.3118 79.14% Amr::timeStep() 5 0.3103 0.3103 0.3103 78.76% Castro::advance() 5 0.3039 0.3039 0.3039 77.15% Castro::subcycle_advance_ctu() 5 0.298 0.298 0.298 75.66% Castro::do_advance_ctu() 5 0.2978 0.2978 0.2978 75.61% Castro::construct_new_gravity() 5 0.1456 0.1456 0.1456 36.96% Gravity::solve_phi_with_mlmg() 6 0.1413 0.1413 0.1413 35.88% Gravity::solve_for_phi() 5 0.1378 0.1378 0.1378 34.97% Gravity::actual_solve_with_mlmg() 6 0.1365 0.1365 0.1365 34.66% MLMG::solve() 6 0.1242 0.1242 0.1242 31.53% MLMG::oneIter() 36 0.1169 0.1169 0.1169 29.68% MLMG::mgVcycle() 36 0.1162 0.1162 0.1162 29.49% Castro::construct_ctu_hydro_source() 5 0.1093 0.1093 0.1093 27.75% MLCellLinOp::smooth() 720 0.05928 0.05928 0.05928 15.05% Amr::init() 1 0.04875 0.04875 0.04875 12.38% Amr::restart() 1 0.04875 0.04875 0.04875 12.38% MLCellLinOp::applyBC() 1946 0.04215 0.04215 0.04215 10.70% AmrLevel::restart() 1 0.04098 0.04098 0.04098 10.40% StateData::restartDoit() 4 0.04089 0.04089 0.04089 10.38% VisMF::Read() 3 0.04076 0.04076 0.04076 10.35% MLMG::mgVcycle_bottom 36 0.03559 0.03559 0.03559 9.04% MLMG::actualBottomSolve() 36 0.03558 0.03558 0.03558 9.03% MLCGSolver::bicgstab 36 0.03522 0.03522 0.03522 8.94% Amr::writePlotFile() 1 0.03269 0.03269 0.03269 8.30% VisMF::Write(FabArray) 1 0.03109 0.03109 0.03109 7.89% MLPoisson::Fsmooth() 1440 0.02756 0.02756 0.02756 7.00% Castro::clean_state() 30 0.02728 0.02728 0.02728 6.92% FillPatchIterator::Initialize 20 0.02009 0.02009 0.02009 5.10% FillPatchSingleLevel 20 0.0193 0.0193 0.0193 4.90% StateDataPhysBCFunct::() 20 0.0173 0.0173 0.0173 4.39% MLCellLinOp::apply() 500 0.01598 0.01598 0.01598 4.06% MLMG::mgVcycle_down::0 36 0.0155 0.0155 0.0155 3.94% MLMG::mgVcycle_up::0 36 0.01328 0.01328 0.01328 3.37% StateData::FillBoundary(geom) 160 0.01177 0.01177 0.01177 2.99% Castro::computeTemp() 30 0.0115 0.0115 0.0115 2.92% Castro::initialize_do_advance() 5 0.01044 0.01044 0.01044 2.65% MLPoisson::define() 6 0.01003 0.01003 0.01003 2.55% MultiFab::Dot() 484 0.009678 0.009678 0.009678 2.46% MLCellLinOp::correctionResidual() 216 0.009363 0.009363 0.009363 2.38% Castro::normalize_species() 30 0.008786 0.008786 0.008786 2.23% MLMG:computeResOfCorrection() 180 0.008085 0.008085 0.008085 2.05% MLMG::mgVcycle_down::1 36 0.00775 0.00775 0.00775 1.97% Castro::do_new_sources() 5 0.007746 0.007746 0.007746 1.97% Gravity::get_new_grav_vector() 5 0.007644 0.007644 0.007644 1.94% MLMG::mgVcycle_down::2 36 0.00753 0.00753 0.00753 1.91% Castro::construct_old_gravity() 5 0.007389 0.007389 0.007389 1.88% Gravity::get_old_grav_vector() 5 0.007351 0.007351 0.007351 1.87% MLMG::mgVcycle_down::3 36 0.007133 0.007133 0.007133 1.81% MLCellLinOp::defineAuxData() 6 0.007016 0.007016 0.007016 1.78% FabArray::FillBoundary() 1766 0.006986 0.006986 0.006986 1.77% FabArray::setVal() 537 0.00687 0.00687 0.00687 1.74% MLMG::mgVcycle_down::4 36 0.006833 0.006833 0.006833 1.73% FillBoundary_nowait() 1766 0.006594 0.006594 0.006594 1.67% FabArray::ParallelCopy() 380 0.006533 0.006533 0.006533 1.66% FabArray::ParallelCopy_nowait() 380 0.006417 0.006417 0.006417 1.63% Castro::do_old_sources() 5 0.006404 0.006404 0.006404 1.63% CGSolver::sxay() 690 0.006379 0.006379 0.006379 1.62% Castro::post_timestep() 5 0.006274 0.006274 0.006274 1.59% MultiFab::LinComb() 690 0.006197 0.006197 0.006197 1.57% MLMG::mgVcycle_up::2 36 0.005805 0.005805 0.005805 1.47% MLCGSolver::ParallelAllReduce 659 0.005792 0.005792 0.005792 1.47% Castro::expand_state() 5 0.005764 0.005764 0.005764 1.46% MLMG::mgVcycle_up::1 36 0.005706 0.005706 0.005706 1.45% Castro::enforce_min_density() 30 0.005667 0.005667 0.005667 1.44% MLMG::addInterpCorrection() 180 0.005662 0.005662 0.005662 1.44% MLMG::mgVcycle_up::3 36 0.005493 0.005493 0.005493 1.39% MLMG::mgVcycle_up::4 36 0.005464 0.005464 0.005464 1.39% Castro::initialize_advance() 5 0.0054 0.0054 0.0054 1.37% amrex::average_down 180 0.005266 0.005266 0.005266 1.34% MLPoisson::Fapply() 500 0.005161 0.005161 0.005161 1.31% Gravity::fill_multipole_BCs() 6 0.004673 0.004673 0.004673 1.19% Castro::post_restart() 1 0.003943 0.003943 0.003943 1.00% Gravity::multilevel_solve_for_new_phi() 1 0.003823 0.003823 0.003823 0.97% Gravity::actual_multilevel_solve() 1 0.003805 0.003805 0.003805 0.97% Castro::reset_internal_energy(MultiFab) 30 0.003431 0.003431 0.003431 0.87% MLCellLinOp::solutionResidual() 42 0.003234 0.003234 0.003234 0.82% MLMG::prepareForSolve() 6 0.002923 0.002923 0.002923 0.74% MultiFab::Xpay() 258 0.002911 0.002911 0.002911 0.74% MLCellLinOp::defineBC() 6 0.002865 0.002865 0.002865 0.73% BndryData::define() 6 0.00272 0.00272 0.00272 0.69% MLMG::computeResidual() 36 0.002694 0.002694 0.002694 0.68% Castro::estTimeStep() 10 0.00247 0.00247 0.00247 0.63% Castro::construct_new_source() 25 0.00189 0.00189 0.00189 0.48% Castro::construct_new_gravity_source() 5 0.001782 0.001782 0.001782 0.45% Castro::reset_internal_energy(Fab) 240 0.001777 0.001777 0.001777 0.45% Castro::construct_old_source() 25 0.001559 0.001559 0.001559 0.40% Castro::construct_old_gravity_source() 5 0.001501 0.001501 0.001501 0.38% Castro::computeNewDt() 5 0.001284 0.001284 0.001284 0.33% Castro::enforce_speed_limit() 30 0.001282 0.001282 0.001282 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009301 0.0009301 0.0009301 0.24% Castro::apply_source_to_state() 10 0.0009239 0.0009239 0.0009239 0.23% MLMG::ResNormInf() 42 0.0009221 0.0009221 0.0009221 0.23% MultiFab::Saxpy() 10 0.0009185 0.0009185 0.0009185 0.23% MLCellLinOp::setLevelBC() 6 0.000824 0.000824 0.000824 0.21% MLMG::getGradSolution() 6 0.0007794 0.0007794 0.0007794 0.20% MLCellLinOp::compGrad() 6 0.0007764 0.0007764 0.0007764 0.20% FabArrayBase::getCPC() 632 0.000744 0.000744 0.000744 0.19% MultiFab::Add() 36 0.0007319 0.0007319 0.0007319 0.19% FabArray::setDomainBndry() 20 0.0006585 0.0006585 0.0006585 0.17% FabArray::mult() 22 0.0006533 0.0006533 0.0006533 0.17% MLPoisson::prepareForSolve() 6 0.0006511 0.0006511 0.0006511 0.17% MLCellLinOp::prepareForSolve() 6 0.0006459 0.0006459 0.0006459 0.16% Castro::check_for_nan() 10 0.0006043 0.0006043 0.0006043 0.15% MultiFab::contains_nan() 10 0.0005959 0.0005959 0.0005959 0.15% MLMG::computeMLResidual() 6 0.000565 0.000565 0.000565 0.14% Gravity::update_max_rhs() 6 0.0004642 0.0004642 0.0004642 0.12% Amr::InitAmr() 1 0.0004135 0.0004135 0.0004135 0.10% FabArrayBase::CPC::define() 244 0.0003742 0.0003742 0.0003742 0.09% Castro::finalize_advance() 5 0.0003399 0.0003399 0.0003399 0.09% FabArrayBase::getFB() 1766 0.0003168 0.0003168 0.0003168 0.08% Gravity::swapTimeLevels() 5 0.0002255 0.0002255 0.0002255 0.06% MultiFab::Copy() 6 0.0001788 0.0001788 0.0001788 0.05% Castro::buildMetrics() 1 0.0001576 0.0001576 0.0001576 0.04% MLMG::MLResNormInf() 6 0.0001491 0.0001491 0.0001491 0.04% MultiFab::max() 6 0.0001465 0.0001465 0.0001465 0.04% MLLinOp::define() 6 0.0001357 0.0001357 0.0001357 0.03% Castro::create_source_corrector() 5 0.0001183 0.0001183 0.0001183 0.03% MLLinOp::defineGrids() 6 0.0001147 0.0001147 0.0001147 0.03% MLMG::MLRhsNormInf() 6 0.0001147 0.0001147 0.0001147 0.03% FabArrayBase::FB::FB() 26 5.518e-05 5.518e-05 5.518e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.768e-05 2.768e-05 2.768e-05 0.01% Amr::writeSmallPlotFile() 1 2.69e-05 2.69e-05 2.69e-05 0.01% Castro::swap_state_time_levels() 5 2.533e-05 2.533e-05 2.533e-05 0.01% Castro::initMFs() 1 2.524e-05 2.524e-05 2.524e-05 0.01% makeSFC 30 2.094e-05 2.094e-05 2.094e-05 0.01% Castro::finalize_do_advance() 5 1.776e-05 1.776e-05 1.776e-05 0.00% DistributionMapping::Distribute() 31 8.443e-06 8.443e-06 8.443e-06 0.00% Amr::initSubcycle() 1 8.125e-06 8.125e-06 8.125e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.208e-06 5.208e-06 5.208e-06 0.00% Gravity::set_mass_offset() 6 2.035e-06 2.035e-06 2.035e-06 0.00% Castro::retry_advance_ctu() 5 1.789e-06 1.789e-06 1.789e-06 0.00% Castro::FluxRegCrseInit 5 1.681e-06 1.681e-06 1.681e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.016e-06 1.016e-06 1.016e-06 0.00% Castro::FluxRegFineAdd() 5 1.009e-06 1.009e-06 1.009e-06 0.00% AmrLevel::AmrLevel() 1 8.29e-07 8.29e-07 8.29e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-17-gd8bc97f92a1a) finalized