Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-36-g806108c5b892) initialized Starting run at 08:28:12 UTC on 2022-05-27. Successfully read inputs file ... Castro git describe: 22.05-33-g9203058b8 AMReX git describe: 22.05-36-g806108c5b Microphysics git describe: 22.05-1-g39742967 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.04217942 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024169766 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048711134 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.0580143 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.062383226 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059835188 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.048311099 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.043034765 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053183521 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.061448685 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061098709 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.055365276 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063979076 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040277625 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024474396 seconds Ending run at 08:28:13 UTC on 2022-05-27. Run time = 0.799039489 Run time without initialization = 0.680735988 Average number of zones advanced per microsecond: 3.851 Average number of zones advanced per microsecond per rank: 3.851 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7991 ... 0.7991 ... 0.7991 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1798 0.1798 0.1798 22.50% VisMF::Write(FabArray) 11 0.1666 0.1666 0.1666 20.85% MLCellLinOp::applyBC() 4433 0.08024 0.08024 0.08024 10.04% MLPoisson::Fsmooth() 3280 0.06343 0.06343 0.06343 7.94% MLCGSolver::bicgstab 82 0.02357 0.02357 0.02357 2.95% StateData::FillBoundary(geom) 328 0.02314 0.02314 0.02314 2.90% MultiFab::Dot() 1114 0.02184 0.02184 0.02184 2.73% Castro::normalize_species() 62 0.01861 0.01861 0.01861 2.33% Castro::computeTemp() 63 0.01612 0.01612 0.01612 2.02% MultiFab::LinComb() 1586 0.01416 0.01416 0.01416 1.77% FabArray::setVal() 1144 0.01409 0.01409 0.01409 1.76% FillBoundary_nowait() 4023 0.01408 0.01408 0.01408 1.76% FabArray::ParallelCopy_nowait() 861 0.01304 0.01304 0.01304 1.63% StateDataPhysBCFunct::() 41 0.01221 0.01221 0.01221 1.53% Castro::enforce_min_density() 62 0.01188 0.01188 0.01188 1.49% MLPoisson::Fapply() 1142 0.01163 0.01163 0.01163 1.46% MLCellLinOp::defineAuxData() 11 0.01126 0.01126 0.01126 1.41% Gravity::fill_multipole_BCs() 11 0.008668 0.008668 0.008668 1.08% Castro::estTimeStep() 21 0.007579 0.007579 0.007579 0.95% MLMG::addInterpCorrection() 410 0.007427 0.007427 0.007427 0.93% amrex::average_down 410 0.006761 0.006761 0.006761 0.85% MultiFab::Xpay() 585 0.006496 0.006496 0.006496 0.81% Amr::checkPoint() 3 0.005081 0.005081 0.005081 0.64% Castro::do_advance_ctu() 10 0.005074 0.005074 0.005074 0.63% Castro::reset_internal_energy(MultiFab) 63 0.00493 0.00493 0.00493 0.62% BndryData::define() 11 0.003744 0.003744 0.003744 0.47% Castro::construct_new_gravity_source() 10 0.003226 0.003226 0.003226 0.40% Castro::construct_old_gravity_source() 10 0.002706 0.002706 0.002706 0.34% Amr::writePlotFile() 2 0.002475 0.002475 0.002475 0.31% Castro::enforce_speed_limit() 62 0.002123 0.002123 0.002123 0.27% Gravity::get_new_grav_vector() 11 0.001921 0.001921 0.001921 0.24% MLMG::ResNormInf() 93 0.001875 0.001875 0.001875 0.23% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.23% Gravity::get_old_grav_vector() 10 0.001734 0.001734 0.001734 0.22% Castro::expand_state() 10 0.001732 0.001732 0.001732 0.22% MLMG::oneIter() 82 0.001644 0.001644 0.001644 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00161 0.00161 0.00161 0.20% Castro::reset_internal_energy(Fab) 504 0.001524 0.001524 0.001524 0.19% MLCellLinOp::setLevelBC() 11 0.001502 0.001502 0.001502 0.19% FabArray::mult() 43 0.001314 0.001314 0.001314 0.16% Gravity::actual_solve_with_mlmg() 11 0.001311 0.001311 0.001311 0.16% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.16% Castro::initData() 1 0.001209 0.001209 0.001209 0.15% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.15% MLCellLinOp::prepareForSolve() 11 0.001164 0.001164 0.001164 0.15% MLCellLinOp::smooth() 1640 0.001116 0.001116 0.001116 0.14% MLMG::prepareForSolve() 11 0.001029 0.001029 0.001029 0.13% MLCellLinOp::compGrad() 11 0.0009282 0.0009282 0.0009282 0.12% FabArray::FillBoundary() 4023 0.0007736 0.0007736 0.0007736 0.10% FabArrayBase::getCPC() 1323 0.0007546 0.0007546 0.0007546 0.09% FabArrayBase::CPC::define() 454 0.0006694 0.0006694 0.0006694 0.08% FabArrayBase::getFB() 4023 0.0005944 0.0005944 0.0005944 0.07% Gravity::update_max_rhs() 11 0.0004564 0.0004564 0.0004564 0.06% Amr::InitAmr() 1 0.0004499 0.0004499 0.0004499 0.06% MLCellLinOp::apply() 1142 0.0004259 0.0004259 0.0004259 0.05% Gravity::solve_for_phi() 10 0.0004109 0.0004109 0.0004109 0.05% CGSolver::sxay() 1586 0.0003515 0.0003515 0.0003515 0.04% Amr::coarseTimeStep() 10 0.0003103 0.0003103 0.0003103 0.04% FillPatchIterator::Initialize 41 0.000293 0.000293 0.000293 0.04% MLCellLinOp::defineBC() 11 0.0002857 0.0002857 0.0002857 0.04% main() 1 0.0002829 0.0002829 0.0002829 0.04% MLCGSolver::ParallelAllReduce 1514 0.00028 0.00028 0.00028 0.04% MultiFab::max() 11 0.0002672 0.0002672 0.0002672 0.03% FabArray::ParallelCopy() 861 0.000265 0.000265 0.000265 0.03% MultiFab::Copy() 11 0.0002555 0.0002555 0.0002555 0.03% MLCellLinOp::correctionResidual() 492 0.0002215 0.0002215 0.0002215 0.03% Castro::construct_new_gravity() 10 0.0002097 0.0002097 0.0002097 0.03% MLMG::mgVcycle() 82 0.0002095 0.0002095 0.0002095 0.03% Amr::timeStep() 10 0.0002087 0.0002087 0.0002087 0.03% MLMG::MLRhsNormInf() 11 0.0002008 0.0002008 0.0002008 0.03% Castro::subcycle_advance_ctu() 10 0.0001958 0.0001958 0.0001958 0.02% MLLinOp::defineGrids() 11 0.0001815 0.0001815 0.0001815 0.02% StateData::checkPoint() 12 0.0001348 0.0001348 0.0001348 0.02% MLMG:computeResOfCorrection() 410 0.0001212 0.0001212 0.0001212 0.02% Castro::advance() 10 9.708e-05 9.708e-05 9.708e-05 0.01% MLMG::mgVcycle_down::0 82 9.372e-05 9.372e-05 9.372e-05 0.01% MLMG::actualBottomSolve() 82 9.335e-05 9.335e-05 9.335e-05 0.01% Castro::finalize_advance() 10 8.947e-05 8.947e-05 8.947e-05 0.01% Castro::initialize_advance() 10 8.341e-05 8.341e-05 8.341e-05 0.01% FabArrayBase::FB::FB() 56 8.337e-05 8.337e-05 8.337e-05 0.01% Castro::Castro() 1 7.907e-05 7.907e-05 7.907e-05 0.01% MLMG::mgVcycle_down::1 82 7.665e-05 7.665e-05 7.665e-05 0.01% AmrLevel::checkPoint() 3 7.547e-05 7.547e-05 7.547e-05 0.01% Castro::clean_state() 62 7.509e-05 7.509e-05 7.509e-05 0.01% MLMG::solve() 11 7.384e-05 7.384e-05 7.384e-05 0.01% MLMG::mgVcycle_down::2 82 7.342e-05 7.342e-05 7.342e-05 0.01% MLMG::mgVcycle_down::3 82 7.003e-05 7.003e-05 7.003e-05 0.01% MLMG::mgVcycle_down::4 82 6.928e-05 6.928e-05 6.928e-05 0.01% Castro::construct_new_source() 50 6.499e-05 6.499e-05 6.499e-05 0.01% Castro::initialize_do_advance() 10 6.482e-05 6.482e-05 6.482e-05 0.01% MLMG::mgVcycle_up::4 82 5.553e-05 5.553e-05 5.553e-05 0.01% MLCellLinOp::solutionResidual() 93 5.189e-05 5.189e-05 5.189e-05 0.01% MLMG::mgVcycle_up::0 82 4.801e-05 4.801e-05 4.801e-05 0.01% MLMG::mgVcycle_up::1 82 4.755e-05 4.755e-05 4.755e-05 0.01% MLMG::mgVcycle_up::3 82 4.69e-05 4.69e-05 4.69e-05 0.01% MLMG::mgVcycle_up::2 82 4.505e-05 4.505e-05 4.505e-05 0.01% Castro::swap_state_time_levels() 10 4.112e-05 4.112e-05 4.112e-05 0.01% StateData::define() 4 3.589e-05 3.589e-05 3.589e-05 0.00% Castro::finalize_do_advance() 10 3.546e-05 3.546e-05 3.546e-05 0.00% MLMG::computeResidual() 82 3.435e-05 3.435e-05 3.435e-05 0.00% Castro::enforce_consistent_e() 1 3.198e-05 3.198e-05 3.198e-05 0.00% Castro::initMFs() 1 3.191e-05 3.191e-05 3.191e-05 0.00% Gravity::actual_multilevel_solve() 1 3.168e-05 3.168e-05 3.168e-05 0.00% MLMG::mgVcycle_bottom 82 3.165e-05 3.165e-05 3.165e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.944e-05 2.944e-05 2.944e-05 0.00% Amr::FinalizeInit() 1 2.864e-05 2.864e-05 2.864e-05 0.00% FillPatchSingleLevel 41 2.66e-05 2.66e-05 2.66e-05 0.00% Castro::post_timestep() 10 2.604e-05 2.604e-05 2.604e-05 0.00% makeSFC 55 2.577e-05 2.577e-05 2.577e-05 0.00% Amr::writeSmallPlotFile() 1 2.508e-05 2.508e-05 2.508e-05 0.00% MLLinOp::define() 11 2.415e-05 2.415e-05 2.415e-05 0.00% MLPoisson::define() 11 2.397e-05 2.397e-05 2.397e-05 0.00% Castro::buildMetrics() 1 2.314e-05 2.314e-05 2.314e-05 0.00% Amr::defBaseLevel() 1 2.141e-05 2.141e-05 2.141e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.875e-05 1.875e-05 1.875e-05 0.00% Castro::construct_old_source() 50 1.739e-05 1.739e-05 1.739e-05 0.00% Castro::do_new_sources() 10 1.619e-05 1.619e-05 1.619e-05 0.00% Castro::do_old_sources() 10 1.517e-05 1.517e-05 1.517e-05 0.00% DistributionMapping::Distribute() 56 1.454e-05 1.454e-05 1.454e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.316e-05 1.316e-05 1.316e-05 0.00% Castro::apply_source_to_state() 20 1.163e-05 1.163e-05 1.163e-05 0.00% Castro::check_for_nan() 20 1.137e-05 1.137e-05 1.137e-05 0.00% Gravity::swapTimeLevels() 10 1.047e-05 1.047e-05 1.047e-05 0.00% Castro::construct_old_gravity() 10 1.029e-05 1.029e-05 1.029e-05 0.00% Amr::initSubcycle() 1 8.595e-06 8.595e-06 8.595e-06 0.00% Castro::computeNewDt() 9 8.545e-06 8.545e-06 8.545e-06 0.00% Castro::post_init() 1 8.454e-06 8.454e-06 8.454e-06 0.00% MLPoisson::prepareForSolve() 11 8.429e-06 8.429e-06 8.429e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.576e-06 7.576e-06 7.576e-06 0.00% MLMG::computeMLResidual() 11 6.87e-06 6.87e-06 6.87e-06 0.00% AmrLevel::checkPointPost() 3 6.7e-06 6.7e-06 6.7e-06 0.00% Amr::InitializeInit() 1 6.336e-06 6.336e-06 6.336e-06 0.00% MLMG::getGradSolution() 11 6.025e-06 6.025e-06 6.025e-06 0.00% MLMG::buildFineMask() 11 5.013e-06 5.013e-06 5.013e-06 0.00% MLMG::MLResNormInf() 11 4.549e-06 4.549e-06 4.549e-06 0.00% Castro::create_source_corrector() 10 4.41e-06 4.41e-06 4.41e-06 0.00% Castro::retry_advance_ctu() 10 4.053e-06 4.053e-06 4.053e-06 0.00% Gravity::set_mass_offset() 11 3.745e-06 3.745e-06 3.745e-06 0.00% Castro::FluxRegCrseInit 10 3.356e-06 3.356e-06 3.356e-06 0.00% Amr::init() 1 2.925e-06 2.925e-06 2.925e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.877e-06 2.877e-06 2.877e-06 0.00% Castro::FluxRegFineAdd() 10 2.56e-06 2.56e-06 2.56e-06 0.00% Castro::computeInitialDt() 2 2.354e-06 2.354e-06 2.354e-06 0.00% AmrLevel::checkPointPre() 3 2.065e-06 2.065e-06 2.065e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.925e-06 1.925e-06 1.925e-06 0.00% Castro::post_regrid() 1 1.19e-06 1.19e-06 1.19e-06 0.00% Amr::initialInit() 1 1.169e-06 1.169e-06 1.169e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7991 0.7991 0.7991 100.00% Amr::coarseTimeStep() 10 0.656 0.656 0.656 82.10% Amr::timeStep() 10 0.5684 0.5684 0.5684 71.13% Castro::advance() 10 0.559 0.559 0.559 69.96% Castro::subcycle_advance_ctu() 10 0.5461 0.5461 0.5461 68.34% Castro::do_advance_ctu() 10 0.5459 0.5459 0.5459 68.31% Gravity::solve_phi_with_mlmg() 11 0.3109 0.3109 0.3109 38.90% Gravity::actual_solve_with_mlmg() 11 0.302 0.302 0.302 37.79% Castro::construct_new_gravity() 10 0.2823 0.2823 0.2823 35.32% MLMG::solve() 11 0.2799 0.2799 0.2799 35.03% Gravity::solve_for_phi() 10 0.2674 0.2674 0.2674 33.46% MLMG::oneIter() 82 0.2655 0.2655 0.2655 33.22% MLMG::mgVcycle() 82 0.2638 0.2638 0.2638 33.02% Castro::construct_ctu_hydro_source() 10 0.1798 0.1798 0.1798 22.50% VisMF::Write(FabArray) 11 0.1666 0.1666 0.1666 20.85% MLCellLinOp::smooth() 1640 0.1359 0.1359 0.1359 17.01% Amr::checkPoint() 3 0.1256 0.1256 0.1256 15.72% AmrLevel::checkPoint() 3 0.1205 0.1205 0.1205 15.08% StateData::checkPoint() 12 0.1205 0.1205 0.1205 15.07% Amr::init() 1 0.1177 0.1177 0.1177 14.73% MLCellLinOp::applyBC() 4433 0.09578 0.09578 0.09578 11.99% MLMG::mgVcycle_bottom 82 0.08056 0.08056 0.08056 10.08% MLMG::actualBottomSolve() 82 0.08053 0.08053 0.08053 10.08% MLCGSolver::bicgstab 82 0.07972 0.07972 0.07972 9.98% MLPoisson::Fsmooth() 3280 0.06343 0.06343 0.06343 7.94% Castro::clean_state() 62 0.05442 0.05442 0.05442 6.81% Amr::initialInit() 1 0.05127 0.05127 0.05127 6.42% Amr::writePlotFile() 2 0.04876 0.04876 0.04876 6.10% Amr::FinalizeInit() 1 0.04721 0.04721 0.04721 5.91% Castro::post_init() 1 0.04583 0.04583 0.04583 5.73% Gravity::multilevel_solve_for_new_phi() 1 0.04395 0.04395 0.04395 5.50% Gravity::actual_multilevel_solve() 1 0.04393 0.04393 0.04393 5.50% FillPatchIterator::Initialize 41 0.04099 0.04099 0.04099 5.13% FillPatchSingleLevel 41 0.03941 0.03941 0.03941 4.93% MLCellLinOp::apply() 1142 0.03601 0.03601 0.03601 4.51% MLMG::mgVcycle_down::0 82 0.03544 0.03544 0.03544 4.44% StateDataPhysBCFunct::() 41 0.03535 0.03535 0.03535 4.42% MLMG::mgVcycle_up::0 82 0.03048 0.03048 0.03048 3.81% StateData::FillBoundary(geom) 328 0.02314 0.02314 0.02314 2.90% Castro::computeTemp() 63 0.02258 0.02258 0.02258 2.83% MultiFab::Dot() 1114 0.02184 0.02184 0.02184 2.73% MLCellLinOp::correctionResidual() 492 0.02109 0.02109 0.02109 2.64% Castro::initialize_do_advance() 10 0.02004 0.02004 0.02004 2.51% Castro::normalize_species() 62 0.01861 0.01861 0.01861 2.33% MLMG:computeResOfCorrection() 410 0.0182 0.0182 0.0182 2.28% MLPoisson::define() 11 0.01779 0.01779 0.01779 2.23% MLMG::mgVcycle_down::1 82 0.01765 0.01765 0.01765 2.21% MLMG::mgVcycle_down::2 82 0.01708 0.01708 0.01708 2.14% Gravity::get_new_grav_vector() 11 0.01647 0.01647 0.01647 2.06% MLMG::mgVcycle_down::3 82 0.01627 0.01627 0.01627 2.04% FabArray::FillBoundary() 4023 0.01553 0.01553 0.01553 1.94% MLMG::mgVcycle_down::4 82 0.01538 0.01538 0.01538 1.92% FillBoundary_nowait() 4023 0.01476 0.01476 0.01476 1.85% Castro::construct_old_gravity() 10 0.01452 0.01452 0.01452 1.82% CGSolver::sxay() 1586 0.01452 0.01452 0.01452 1.82% Gravity::get_old_grav_vector() 10 0.01451 0.01451 0.01451 1.82% MultiFab::LinComb() 1586 0.01416 0.01416 0.01416 1.77% FabArray::ParallelCopy() 861 0.01412 0.01412 0.01412 1.77% FabArray::setVal() 1144 0.01409 0.01409 0.01409 1.76% FabArray::ParallelCopy_nowait() 861 0.01386 0.01386 0.01386 1.73% Castro::do_new_sources() 10 0.01377 0.01377 0.01377 1.72% MLMG::mgVcycle_up::2 82 0.01316 0.01316 0.01316 1.65% MLCGSolver::ParallelAllReduce 1514 0.01297 0.01297 0.01297 1.62% MLMG::mgVcycle_up::1 82 0.01294 0.01294 0.01294 1.62% Castro::initialize_advance() 10 0.01277 0.01277 0.01277 1.60% MLCellLinOp::defineAuxData() 11 0.01256 0.01256 0.01256 1.57% MLMG::addInterpCorrection() 410 0.01246 0.01246 0.01246 1.56% MLMG::mgVcycle_up::3 82 0.01242 0.01242 0.01242 1.55% MLMG::mgVcycle_up::4 82 0.01224 0.01224 0.01224 1.53% Castro::enforce_min_density() 62 0.01188 0.01188 0.01188 1.49% amrex::average_down 410 0.01182 0.01182 0.01182 1.48% MLPoisson::Fapply() 1142 0.01163 0.01163 0.01163 1.46% Castro::expand_state() 10 0.01148 0.01148 0.01148 1.44% Castro::do_old_sources() 10 0.01124 0.01124 0.01124 1.41% Castro::post_timestep() 10 0.009129 0.009129 0.009129 1.14% Gravity::fill_multipole_BCs() 11 0.008668 0.008668 0.008668 1.08% Castro::estTimeStep() 21 0.007579 0.007579 0.007579 0.95% MLCellLinOp::solutionResidual() 93 0.007181 0.007181 0.007181 0.90% MultiFab::Xpay() 585 0.006496 0.006496 0.006496 0.81% Castro::reset_internal_energy(MultiFab) 63 0.006454 0.006454 0.006454 0.81% MLMG::computeResidual() 82 0.006192 0.006192 0.006192 0.77% MLMG::prepareForSolve() 11 0.005097 0.005097 0.005097 0.64% MLCellLinOp::defineBC() 11 0.004944 0.004944 0.004944 0.62% BndryData::define() 11 0.004659 0.004659 0.004659 0.58% Amr::InitializeInit() 1 0.004062 0.004062 0.004062 0.51% Amr::defBaseLevel() 1 0.004056 0.004056 0.004056 0.51% Castro::initData() 1 0.003564 0.003564 0.003564 0.45% Castro::computeNewDt() 9 0.003468 0.003468 0.003468 0.43% Castro::construct_new_source() 50 0.003291 0.003291 0.003291 0.41% Castro::construct_new_gravity_source() 10 0.003226 0.003226 0.003226 0.40% Castro::construct_old_source() 50 0.002723 0.002723 0.002723 0.34% Castro::construct_old_gravity_source() 10 0.002706 0.002706 0.002706 0.34% Castro::enforce_speed_limit() 62 0.002123 0.002123 0.002123 0.27% MLMG::ResNormInf() 93 0.001875 0.001875 0.001875 0.23% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.23% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00161 0.00161 0.00161 0.20% Castro::reset_internal_energy(Fab) 504 0.001524 0.001524 0.001524 0.19% MLCellLinOp::setLevelBC() 11 0.001502 0.001502 0.001502 0.19% FabArrayBase::getCPC() 1323 0.001424 0.001424 0.001424 0.18% MLMG::getGradSolution() 11 0.001407 0.001407 0.001407 0.18% MLCellLinOp::compGrad() 11 0.001401 0.001401 0.001401 0.18% FabArray::mult() 43 0.001314 0.001314 0.001314 0.16% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.16% Castro::check_for_nan() 20 0.001186 0.001186 0.001186 0.15% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.15% MLPoisson::prepareForSolve() 11 0.001172 0.001172 0.001172 0.15% MLCellLinOp::prepareForSolve() 11 0.001164 0.001164 0.001164 0.15% Castro::post_regrid() 1 0.001138 0.001138 0.001138 0.14% MLMG::computeMLResidual() 11 0.00103 0.00103 0.00103 0.13% Gravity::update_max_rhs() 11 0.0008745 0.0008745 0.0008745 0.11% Castro::computeInitialDt() 2 0.0006855 0.0006855 0.0006855 0.09% FabArrayBase::getFB() 4023 0.0006778 0.0006778 0.0006778 0.08% FabArrayBase::CPC::define() 454 0.0006694 0.0006694 0.0006694 0.08% Amr::InitAmr() 1 0.0004585 0.0004585 0.0004585 0.06% Gravity::swapTimeLevels() 10 0.0004382 0.0004382 0.0004382 0.05% Castro::Castro() 1 0.0004223 0.0004223 0.0004223 0.05% MultiFab::max() 11 0.0002672 0.0002672 0.0002672 0.03% MLLinOp::define() 11 0.0002599 0.0002599 0.0002599 0.03% MLMG::MLResNormInf() 11 0.0002563 0.0002563 0.0002563 0.03% MultiFab::Copy() 11 0.0002555 0.0002555 0.0002555 0.03% MLLinOp::defineGrids() 11 0.0002357 0.0002357 0.0002357 0.03% MLMG::MLRhsNormInf() 11 0.0002008 0.0002008 0.0002008 0.03% Castro::buildMetrics() 1 0.0001469 0.0001469 0.0001469 0.02% Castro::finalize_advance() 10 9.539e-05 9.539e-05 9.539e-05 0.01% FabArrayBase::FB::FB() 56 8.337e-05 8.337e-05 8.337e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.228e-05 5.228e-05 5.228e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.347e-05 4.347e-05 4.347e-05 0.01% Castro::swap_state_time_levels() 10 4.112e-05 4.112e-05 4.112e-05 0.01% makeSFC 55 3.912e-05 3.912e-05 3.912e-05 0.00% StateData::define() 4 3.589e-05 3.589e-05 3.589e-05 0.00% Castro::finalize_do_advance() 10 3.546e-05 3.546e-05 3.546e-05 0.00% Castro::enforce_consistent_e() 1 3.198e-05 3.198e-05 3.198e-05 0.00% Castro::initMFs() 1 3.191e-05 3.191e-05 3.191e-05 0.00% Amr::writeSmallPlotFile() 1 2.508e-05 2.508e-05 2.508e-05 0.00% DistributionMapping::Distribute() 56 1.454e-05 1.454e-05 1.454e-05 0.00% Amr::initSubcycle() 1 8.595e-06 8.595e-06 8.595e-06 0.00% AmrLevel::checkPointPost() 3 6.7e-06 6.7e-06 6.7e-06 0.00% MLMG::buildFineMask() 11 5.013e-06 5.013e-06 5.013e-06 0.00% Castro::create_source_corrector() 10 4.41e-06 4.41e-06 4.41e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.067e-06 4.067e-06 4.067e-06 0.00% Castro::retry_advance_ctu() 10 4.053e-06 4.053e-06 4.053e-06 0.00% Gravity::set_mass_offset() 11 3.745e-06 3.745e-06 3.745e-06 0.00% Castro::FluxRegCrseInit 10 3.356e-06 3.356e-06 3.356e-06 0.00% Castro::FluxRegFineAdd() 10 2.56e-06 2.56e-06 2.56e-06 0.00% AmrLevel::checkPointPre() 3 2.065e-06 2.065e-06 2.065e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.925e-06 1.925e-06 1.925e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-36-g806108c5b892) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-36-g806108c5b892) initialized Starting run at 08:28:14 UTC on 2022-05-27. Successfully read inputs file ... Castro git describe: 22.05-33-g9203058b8 AMReX git describe: 22.05-36-g806108c5b Microphysics git describe: 22.05-1-g39742967 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.43861609 Restart time = 0.074808613 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052561424 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048623287 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062773746 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062002658 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064079974 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02554351 seconds Ending run at 08:28:14 UTC on 2022-05-27. Run time = 0.391339742 Run time without initialization = 0.315968682 Average number of zones advanced per microsecond: 4.148 Average number of zones advanced per microsecond per rank: 4.148 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3914 ... 0.3914 ... 0.3914 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0945 0.0945 0.0945 24.16% VisMF::Read() 3 0.03926 0.03926 0.03926 10.03% MLCellLinOp::applyBC() 1946 0.03388 0.03388 0.03388 8.66% Amr::restart() 1 0.03137 0.03137 0.03137 8.02% MLPoisson::Fsmooth() 1440 0.0266 0.0266 0.0266 6.80% VisMF::Write(FabArray) 1 0.02418 0.02418 0.02418 6.18% StateData::FillBoundary(geom) 160 0.01144 0.01144 0.01144 2.92% MLCGSolver::bicgstab 36 0.00994 0.00994 0.00994 2.54% Castro::normalize_species() 30 0.009279 0.009279 0.009279 2.37% MultiFab::Dot() 484 0.009159 0.009159 0.009159 2.34% Castro::computeTemp() 30 0.007879 0.007879 0.007879 2.01% FabArray::setVal() 537 0.006539 0.006539 0.006539 1.67% Castro::enforce_min_density() 30 0.006172 0.006172 0.006172 1.58% FillBoundary_nowait() 1766 0.006063 0.006063 0.006063 1.55% MLCellLinOp::defineAuxData() 6 0.005992 0.005992 0.005992 1.53% MultiFab::LinComb() 690 0.005889 0.005889 0.005889 1.50% FabArray::ParallelCopy_nowait() 380 0.00569 0.00569 0.00569 1.45% StateDataPhysBCFunct::() 20 0.005664 0.005664 0.005664 1.45% MLPoisson::Fapply() 500 0.004889 0.004889 0.004889 1.25% Gravity::fill_multipole_BCs() 6 0.004753 0.004753 0.004753 1.21% Castro::estTimeStep() 10 0.003731 0.003731 0.003731 0.95% MLMG::addInterpCorrection() 180 0.003151 0.003151 0.003151 0.81% amrex::average_down 180 0.002886 0.002886 0.002886 0.74% MultiFab::Xpay() 258 0.002776 0.002776 0.002776 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002149 0.002149 0.002149 0.55% Castro::do_advance_ctu() 5 0.002102 0.002102 0.002102 0.54% BndryData::define() 6 0.001993 0.001993 0.001993 0.51% Castro::construct_new_gravity_source() 5 0.001617 0.001617 0.001617 0.41% Amr::writePlotFile() 1 0.001442 0.001442 0.001442 0.37% Castro::construct_old_gravity_source() 5 0.001334 0.001334 0.001334 0.34% Castro::enforce_speed_limit() 30 0.001021 0.001021 0.001021 0.26% MultiFab::Saxpy() 10 0.0009152 0.0009152 0.0009152 0.23% Gravity::get_old_grav_vector() 5 0.0008692 0.0008692 0.0008692 0.22% Castro::expand_state() 5 0.000864 0.000864 0.000864 0.22% Gravity::get_new_grav_vector() 5 0.000858 0.000858 0.000858 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008427 0.0008427 0.0008427 0.22% MLMG::ResNormInf() 42 0.0008216 0.0008216 0.0008216 0.21% MLCellLinOp::setLevelBC() 6 0.0007953 0.0007953 0.0007953 0.20% MLMG::oneIter() 36 0.0007185 0.0007185 0.0007185 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007045 0.0007045 0.0007045 0.18% Castro::reset_internal_energy(Fab) 240 0.000667 0.000667 0.000667 0.17% FabArray::mult() 22 0.0006361 0.0006361 0.0006361 0.16% FabArray::setDomainBndry() 20 0.0006194 0.0006194 0.0006194 0.16% MLCellLinOp::prepareForSolve() 6 0.0006084 0.0006084 0.0006084 0.16% MultiFab::contains_nan() 10 0.0005799 0.0005799 0.0005799 0.15% MLMG::prepareForSolve() 6 0.0005516 0.0005516 0.0005516 0.14% MLCellLinOp::compGrad() 6 0.0004868 0.0004868 0.0004868 0.12% MLCellLinOp::smooth() 720 0.0004765 0.0004765 0.0004765 0.12% Amr::InitAmr() 1 0.0003876 0.0003876 0.0003876 0.10% FabArrayBase::CPC::define() 244 0.0003839 0.0003839 0.0003839 0.10% FabArrayBase::getCPC() 632 0.0003563 0.0003563 0.0003563 0.09% FabArray::FillBoundary() 1766 0.0003498 0.0003498 0.0003498 0.09% main() 1 0.000261 0.000261 0.000261 0.07% FabArrayBase::getFB() 1766 0.0002518 0.0002518 0.0002518 0.06% Gravity::update_max_rhs() 6 0.0002284 0.0002284 0.0002284 0.06% Gravity::solve_for_phi() 5 0.0002099 0.0002099 0.0002099 0.05% MLCellLinOp::apply() 500 0.0001847 0.0001847 0.0001847 0.05% CGSolver::sxay() 690 0.0001692 0.0001692 0.0001692 0.04% Castro::subcycle_advance_ctu() 5 0.0001599 0.0001599 0.0001599 0.04% Amr::coarseTimeStep() 5 0.0001554 0.0001554 0.0001554 0.04% MLCellLinOp::defineBC() 6 0.0001505 0.0001505 0.0001505 0.04% MultiFab::Copy() 6 0.0001373 0.0001373 0.0001373 0.04% FillPatchIterator::Initialize 20 0.0001368 0.0001368 0.0001368 0.03% MultiFab::max() 6 0.0001341 0.0001341 0.0001341 0.03% FabArray::ParallelCopy() 380 0.0001298 0.0001298 0.0001298 0.03% MLCGSolver::ParallelAllReduce 659 0.0001189 0.0001189 0.0001189 0.03% Castro::construct_new_gravity() 5 0.0001074 0.0001074 0.0001074 0.03% Amr::timeStep() 5 0.0001068 0.0001068 0.0001068 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% MLLinOp::defineGrids() 6 9.22e-05 9.22e-05 9.22e-05 0.02% MLCellLinOp::correctionResidual() 216 8.962e-05 8.962e-05 8.962e-05 0.02% MLMG::mgVcycle() 36 8.769e-05 8.769e-05 8.769e-05 0.02% Castro::advance() 5 8.643e-05 8.643e-05 8.643e-05 0.02% AmrLevel::restart() 1 7.773e-05 7.773e-05 7.773e-05 0.02% Castro::construct_new_source() 25 7.604e-05 7.604e-05 7.604e-05 0.02% StateData::restartDoit() 4 7.4e-05 7.4e-05 7.4e-05 0.02% FabArrayBase::FB::FB() 26 5.591e-05 5.591e-05 5.591e-05 0.01% MLMG:computeResOfCorrection() 180 5.224e-05 5.224e-05 5.224e-05 0.01% Castro::initialize_do_advance() 5 4.352e-05 4.352e-05 4.352e-05 0.01% MLMG::actualBottomSolve() 36 4.284e-05 4.284e-05 4.284e-05 0.01% Castro::create_source_corrector() 5 4.199e-05 4.199e-05 4.199e-05 0.01% MLMG::mgVcycle_down::0 36 3.848e-05 3.848e-05 3.848e-05 0.01% Castro::initialize_advance() 5 3.831e-05 3.831e-05 3.831e-05 0.01% Castro::finalize_advance() 5 3.801e-05 3.801e-05 3.801e-05 0.01% Castro::clean_state() 30 3.779e-05 3.779e-05 3.779e-05 0.01% MLMG::mgVcycle_down::1 36 3.611e-05 3.611e-05 3.611e-05 0.01% Castro::construct_old_source() 25 3.604e-05 3.604e-05 3.604e-05 0.01% MLMG::mgVcycle_down::2 36 3.538e-05 3.538e-05 3.538e-05 0.01% MLMG::solve() 6 3.461e-05 3.461e-05 3.461e-05 0.01% MLMG::mgVcycle_down::4 36 3.333e-05 3.333e-05 3.333e-05 0.01% Castro::post_restart() 1 3.253e-05 3.253e-05 3.253e-05 0.01% MLMG::mgVcycle_down::3 36 3.122e-05 3.122e-05 3.122e-05 0.01% Castro::buildMetrics() 1 3.019e-05 3.019e-05 3.019e-05 0.01% Gravity::actual_multilevel_solve() 1 2.9e-05 2.9e-05 2.9e-05 0.01% Castro::swap_state_time_levels() 5 2.747e-05 2.747e-05 2.747e-05 0.01% Castro::initMFs() 1 2.671e-05 2.671e-05 2.671e-05 0.01% MLMG::mgVcycle_up::4 36 2.647e-05 2.647e-05 2.647e-05 0.01% Amr::writeSmallPlotFile() 1 2.557e-05 2.557e-05 2.557e-05 0.01% MLCellLinOp::solutionResidual() 42 2.37e-05 2.37e-05 2.37e-05 0.01% MLMG::mgVcycle_up::0 36 2.358e-05 2.358e-05 2.358e-05 0.01% MLMG::mgVcycle_up::2 36 2.268e-05 2.268e-05 2.268e-05 0.01% MLMG::mgVcycle_up::3 36 2.152e-05 2.152e-05 2.152e-05 0.01% MLMG::mgVcycle_up::1 36 2.059e-05 2.059e-05 2.059e-05 0.01% MLLinOp::define() 6 2.011e-05 2.011e-05 2.011e-05 0.01% Castro::finalize_do_advance() 5 1.785e-05 1.785e-05 1.785e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.716e-05 1.716e-05 1.716e-05 0.00% MLPoisson::define() 6 1.52e-05 1.52e-05 1.52e-05 0.00% MLMG::computeResidual() 36 1.499e-05 1.499e-05 1.499e-05 0.00% MLMG::mgVcycle_bottom 36 1.45e-05 1.45e-05 1.45e-05 0.00% makeSFC 30 1.41e-05 1.41e-05 1.41e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.345e-05 1.345e-05 1.345e-05 0.00% FillPatchSingleLevel 20 1.321e-05 1.321e-05 1.321e-05 0.00% Amr::initSubcycle() 1 8.894e-06 8.894e-06 8.894e-06 0.00% Castro::do_new_sources() 5 8.87e-06 8.87e-06 8.87e-06 0.00% Castro::do_old_sources() 5 8.065e-06 8.065e-06 8.065e-06 0.00% DistributionMapping::Distribute() 31 8.023e-06 8.023e-06 8.023e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.008e-06 7.008e-06 7.008e-06 0.00% Castro::check_for_nan() 10 6.634e-06 6.634e-06 6.634e-06 0.00% Castro::apply_source_to_state() 10 5.553e-06 5.553e-06 5.553e-06 0.00% Castro::construct_old_gravity() 5 5.396e-06 5.396e-06 5.396e-06 0.00% MLPoisson::prepareForSolve() 6 4.765e-06 4.765e-06 4.765e-06 0.00% Castro::post_timestep() 5 4.538e-06 4.538e-06 4.538e-06 0.00% Gravity::swapTimeLevels() 5 4.531e-06 4.531e-06 4.531e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.32e-06 3.32e-06 3.32e-06 0.00% MLMG::buildFineMask() 6 3.214e-06 3.214e-06 3.214e-06 0.00% Castro::computeNewDt() 5 2.975e-06 2.975e-06 2.975e-06 0.00% MLMG::computeMLResidual() 6 2.892e-06 2.892e-06 2.892e-06 0.00% MLMG::getGradSolution() 6 2.73e-06 2.73e-06 2.73e-06 0.00% MLMG::MLResNormInf() 6 2.265e-06 2.265e-06 2.265e-06 0.00% Castro::retry_advance_ctu() 5 2.117e-06 2.117e-06 2.117e-06 0.00% Gravity::set_mass_offset() 6 2.057e-06 2.057e-06 2.057e-06 0.00% Castro::FluxRegCrseInit 5 1.784e-06 1.784e-06 1.784e-06 0.00% AmrLevel::AmrLevel() 1 1.292e-06 1.292e-06 1.292e-06 0.00% Castro::FluxRegFineAdd() 5 1.232e-06 1.232e-06 1.232e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.05e-06 1.05e-06 1.05e-06 0.00% Amr::init() 1 1.005e-06 1.005e-06 1.005e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3914 0.3914 0.3914 100.00% Amr::coarseTimeStep() 5 0.2902 0.2902 0.2902 74.15% Amr::timeStep() 5 0.2878 0.2878 0.2878 73.53% Castro::advance() 5 0.2826 0.2826 0.2826 72.21% Castro::subcycle_advance_ctu() 5 0.2761 0.2761 0.2761 70.54% Castro::do_advance_ctu() 5 0.2759 0.2759 0.2759 70.50% Castro::construct_new_gravity() 5 0.1393 0.1393 0.1393 35.61% Gravity::solve_phi_with_mlmg() 6 0.1353 0.1353 0.1353 34.57% Gravity::solve_for_phi() 5 0.1318 0.1318 0.1318 33.68% Gravity::actual_solve_with_mlmg() 6 0.1304 0.1304 0.1304 33.32% MLMG::solve() 6 0.1187 0.1187 0.1187 30.32% MLMG::oneIter() 36 0.1118 0.1118 0.1118 28.57% MLMG::mgVcycle() 36 0.1111 0.1111 0.1111 28.39% Castro::construct_ctu_hydro_source() 5 0.09455 0.09455 0.09455 24.16% Amr::init() 1 0.07485 0.07485 0.07485 19.13% Amr::restart() 1 0.07485 0.07485 0.07485 19.13% MLCellLinOp::smooth() 720 0.0572 0.0572 0.0572 14.61% MLCellLinOp::applyBC() 1946 0.0406 0.0406 0.0406 10.37% AmrLevel::restart() 1 0.03946 0.03946 0.03946 10.08% StateData::restartDoit() 4 0.03938 0.03938 0.03938 10.06% VisMF::Read() 3 0.03926 0.03926 0.03926 10.03% MLMG::mgVcycle_bottom 36 0.03384 0.03384 0.03384 8.65% MLMG::actualBottomSolve() 36 0.03383 0.03383 0.03383 8.64% MLCGSolver::bicgstab 36 0.03348 0.03348 0.03348 8.56% Castro::clean_state() 30 0.0272 0.0272 0.0272 6.95% MLPoisson::Fsmooth() 1440 0.0266 0.0266 0.0266 6.80% Amr::writePlotFile() 1 0.02563 0.02563 0.02563 6.55% VisMF::Write(FabArray) 1 0.02418 0.02418 0.02418 6.18% FillPatchIterator::Initialize 20 0.01984 0.01984 0.01984 5.07% FillPatchSingleLevel 20 0.01908 0.01908 0.01908 4.88% StateDataPhysBCFunct::() 20 0.0171 0.0171 0.0171 4.37% MLCellLinOp::apply() 500 0.01529 0.01529 0.01529 3.91% MLMG::mgVcycle_down::0 36 0.01503 0.01503 0.01503 3.84% MLMG::mgVcycle_up::0 36 0.01287 0.01287 0.01287 3.29% StateData::FillBoundary(geom) 160 0.01144 0.01144 0.01144 2.92% Castro::initialize_do_advance() 5 0.0113 0.0113 0.0113 2.89% Castro::computeTemp() 30 0.01069 0.01069 0.01069 2.73% MLPoisson::define() 6 0.009506 0.009506 0.009506 2.43% Castro::normalize_species() 30 0.009279 0.009279 0.009279 2.37% MultiFab::Dot() 484 0.009159 0.009159 0.009159 2.34% MLCellLinOp::correctionResidual() 216 0.008927 0.008927 0.008927 2.28% MLMG:computeResOfCorrection() 180 0.007706 0.007706 0.007706 1.97% MLMG::mgVcycle_down::1 36 0.007461 0.007461 0.007461 1.91% Gravity::get_new_grav_vector() 5 0.007428 0.007428 0.007428 1.90% Castro::construct_old_gravity() 5 0.007214 0.007214 0.007214 1.84% Gravity::get_old_grav_vector() 5 0.007209 0.007209 0.007209 1.84% MLMG::mgVcycle_down::2 36 0.007174 0.007174 0.007174 1.83% Castro::do_new_sources() 5 0.007144 0.007144 0.007144 1.83% MLMG::mgVcycle_down::3 36 0.006827 0.006827 0.006827 1.74% FabArray::FillBoundary() 1766 0.00672 0.00672 0.00672 1.72% MLCellLinOp::defineAuxData() 6 0.006695 0.006695 0.006695 1.71% FabArray::setVal() 537 0.006539 0.006539 0.006539 1.67% MLMG::mgVcycle_down::4 36 0.006488 0.006488 0.006488 1.66% Castro::initialize_advance() 5 0.006419 0.006419 0.006419 1.64% FillBoundary_nowait() 1766 0.006371 0.006371 0.006371 1.63% FabArray::ParallelCopy() 380 0.006189 0.006189 0.006189 1.58% Castro::enforce_min_density() 30 0.006172 0.006172 0.006172 1.58% FabArray::ParallelCopy_nowait() 380 0.006059 0.006059 0.006059 1.55% CGSolver::sxay() 690 0.006058 0.006058 0.006058 1.55% MultiFab::LinComb() 690 0.005889 0.005889 0.005889 1.50% Castro::do_old_sources() 5 0.005875 0.005875 0.005875 1.50% Castro::expand_state() 5 0.005852 0.005852 0.005852 1.50% MLMG::mgVcycle_up::2 36 0.005531 0.005531 0.005531 1.41% MLCGSolver::ParallelAllReduce 659 0.005487 0.005487 0.005487 1.40% MLMG::mgVcycle_up::1 36 0.005424 0.005424 0.005424 1.39% MLMG::addInterpCorrection() 180 0.005254 0.005254 0.005254 1.34% MLMG::mgVcycle_up::3 36 0.005217 0.005217 0.005217 1.33% MLMG::mgVcycle_up::4 36 0.005149 0.005149 0.005149 1.32% Castro::post_timestep() 5 0.005062 0.005062 0.005062 1.29% amrex::average_down 180 0.005004 0.005004 0.005004 1.28% MLPoisson::Fapply() 500 0.004889 0.004889 0.004889 1.25% Gravity::fill_multipole_BCs() 6 0.004753 0.004753 0.004753 1.21% Castro::post_restart() 1 0.003851 0.003851 0.003851 0.98% Castro::estTimeStep() 10 0.003731 0.003731 0.003731 0.95% Gravity::multilevel_solve_for_new_phi() 1 0.003722 0.003722 0.003722 0.95% Gravity::actual_multilevel_solve() 1 0.003705 0.003705 0.003705 0.95% MLCellLinOp::solutionResidual() 42 0.003161 0.003161 0.003161 0.81% Castro::reset_internal_energy(MultiFab) 30 0.002816 0.002816 0.002816 0.72% MultiFab::Xpay() 258 0.002776 0.002776 0.002776 0.71% MLMG::prepareForSolve() 6 0.002687 0.002687 0.002687 0.69% MLCellLinOp::defineBC() 6 0.002654 0.002654 0.002654 0.68% MLMG::computeResidual() 36 0.002628 0.002628 0.002628 0.67% BndryData::define() 6 0.002504 0.002504 0.002504 0.64% Castro::computeNewDt() 5 0.002249 0.002249 0.002249 0.57% Castro::construct_new_source() 25 0.001693 0.001693 0.001693 0.43% Castro::construct_new_gravity_source() 5 0.001617 0.001617 0.001617 0.41% Castro::construct_old_source() 25 0.00137 0.00137 0.00137 0.35% Castro::construct_old_gravity_source() 5 0.001334 0.001334 0.001334 0.34% Castro::enforce_speed_limit() 30 0.001021 0.001021 0.001021 0.26% Castro::apply_source_to_state() 10 0.0009208 0.0009208 0.0009208 0.24% MultiFab::Saxpy() 10 0.0009152 0.0009152 0.0009152 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008427 0.0008427 0.0008427 0.22% MLMG::ResNormInf() 42 0.0008216 0.0008216 0.0008216 0.21% MLCellLinOp::setLevelBC() 6 0.0007953 0.0007953 0.0007953 0.20% MLMG::getGradSolution() 6 0.0007499 0.0007499 0.0007499 0.19% MLCellLinOp::compGrad() 6 0.0007472 0.0007472 0.0007472 0.19% FabArrayBase::getCPC() 632 0.0007401 0.0007401 0.0007401 0.19% Castro::reset_internal_energy(Fab) 240 0.000667 0.000667 0.000667 0.17% FabArray::mult() 22 0.0006361 0.0006361 0.0006361 0.16% FabArray::setDomainBndry() 20 0.0006194 0.0006194 0.0006194 0.16% MLPoisson::prepareForSolve() 6 0.0006132 0.0006132 0.0006132 0.16% MLCellLinOp::prepareForSolve() 6 0.0006084 0.0006084 0.0006084 0.16% Castro::check_for_nan() 10 0.0005865 0.0005865 0.0005865 0.15% MultiFab::contains_nan() 10 0.0005799 0.0005799 0.0005799 0.15% MLMG::computeMLResidual() 6 0.0005507 0.0005507 0.0005507 0.14% Gravity::update_max_rhs() 6 0.0004374 0.0004374 0.0004374 0.11% Amr::InitAmr() 1 0.0003965 0.0003965 0.0003965 0.10% FabArrayBase::CPC::define() 244 0.0003839 0.0003839 0.0003839 0.10% FabArrayBase::getFB() 1766 0.0003077 0.0003077 0.0003077 0.08% Gravity::swapTimeLevels() 5 0.0002218 0.0002218 0.0002218 0.06% Castro::buildMetrics() 1 0.0001449 0.0001449 0.0001449 0.04% MLLinOp::define() 6 0.0001413 0.0001413 0.0001413 0.04% MultiFab::Copy() 6 0.0001373 0.0001373 0.0001373 0.04% MultiFab::max() 6 0.0001341 0.0001341 0.0001341 0.03% MLMG::MLResNormInf() 6 0.0001327 0.0001327 0.0001327 0.03% MLLinOp::defineGrids() 6 0.0001212 0.0001212 0.0001212 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% FabArrayBase::FB::FB() 26 5.591e-05 5.591e-05 5.591e-05 0.01% Castro::create_source_corrector() 5 4.199e-05 4.199e-05 4.199e-05 0.01% Castro::finalize_advance() 5 4.103e-05 4.103e-05 4.103e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.797e-05 2.797e-05 2.797e-05 0.01% Castro::swap_state_time_levels() 5 2.747e-05 2.747e-05 2.747e-05 0.01% Castro::initMFs() 1 2.671e-05 2.671e-05 2.671e-05 0.01% Amr::writeSmallPlotFile() 1 2.557e-05 2.557e-05 2.557e-05 0.01% makeSFC 30 2.097e-05 2.097e-05 2.097e-05 0.01% Castro::finalize_do_advance() 5 1.785e-05 1.785e-05 1.785e-05 0.00% Amr::initSubcycle() 1 8.894e-06 8.894e-06 8.894e-06 0.00% DistributionMapping::Distribute() 31 8.023e-06 8.023e-06 8.023e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.477e-06 4.477e-06 4.477e-06 0.00% MLMG::buildFineMask() 6 3.214e-06 3.214e-06 3.214e-06 0.00% Castro::retry_advance_ctu() 5 2.117e-06 2.117e-06 2.117e-06 0.00% Gravity::set_mass_offset() 6 2.057e-06 2.057e-06 2.057e-06 0.00% Castro::FluxRegCrseInit 5 1.784e-06 1.784e-06 1.784e-06 0.00% AmrLevel::AmrLevel() 1 1.292e-06 1.292e-06 1.292e-06 0.00% Castro::FluxRegFineAdd() 5 1.232e-06 1.232e-06 1.232e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.05e-06 1.05e-06 1.05e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-36-g806108c5b892) finalized