Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-1-g4a53367b19e1) initialized Starting run at 09:55:24 UTC on 2022-12-05. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-1-g4a53367b1 Microphysics git describe: 22.12-10-g85f387df reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052398905 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029978415 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047896247 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051322461 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050838671 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059544468 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.062084295 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.048073041 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.079772127 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057809534 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050242863 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057625832 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06438805 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048059029 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029750228 seconds Ending run at 09:55:25 UTC on 2022-12-05. Run time = 0.842795705 Run time without initialization = 0.708015764 Average number of zones advanced per microsecond: 3.703 Average number of zones advanced per microsecond per rank: 3.703 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8428 ... 0.8428 ... 0.8428 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2002 0.2002 0.2002 23.75% Castro::construct_ctu_hydro_source() 10 0.1941 0.1941 0.1941 23.03% MLCellLinOp::applyBC() 4433 0.07868 0.07868 0.07868 9.34% MLPoisson::Fsmooth() 3280 0.06285 0.06285 0.06285 7.46% StateData::FillBoundary(geom) 328 0.02313 0.02313 0.02313 2.74% amrex::Dot() 1114 0.0219 0.0219 0.0219 2.60% StateDataPhysBCFunct::() 41 0.0199 0.0199 0.0199 2.36% amrex::Copy() 1029 0.01559 0.01559 0.01559 1.85% Castro::computeTemp() 63 0.01459 0.01459 0.01459 1.73% FabArray::LinComb() 1586 0.01419 0.01419 0.01419 1.68% FillBoundary_nowait() 4023 0.01409 0.01409 0.01409 1.67% Castro::normalize_species() 62 0.01405 0.01405 0.01405 1.67% FabArray::setVal() 1144 0.01403 0.01403 0.01403 1.66% FabArray::ParallelCopy_nowait() 861 0.01284 0.01284 0.01284 1.52% FabArray::norminf() 639 0.01266 0.01266 0.01266 1.50% MLPoisson::Fapply() 1142 0.01158 0.01158 0.01158 1.37% MLCellLinOp::defineAuxData() 11 0.01119 0.01119 0.01119 1.33% Castro::enforce_min_density() 62 0.01099 0.01099 0.01099 1.30% Gravity::fill_multipole_BCs() 11 0.008122 0.008122 0.008122 0.96% MLMG::addInterpCorrection() 410 0.00761 0.00761 0.00761 0.90% amrex::average_down 410 0.006798 0.006798 0.006798 0.81% FabArray::Xpay() 585 0.00649 0.00649 0.00649 0.77% Amr::checkPoint() 3 0.005186 0.005186 0.005186 0.62% Castro::estTimeStep() 21 0.004753 0.004753 0.004753 0.56% Castro::reset_internal_energy(MultiFab) 63 0.004385 0.004385 0.004385 0.52% BndryData::define() 11 0.00373 0.00373 0.00373 0.44% Castro::construct_new_gravity_source() 10 0.003294 0.003294 0.003294 0.39% MLCGSolver::bicgstab 82 0.003235 0.003235 0.003235 0.38% Castro::do_advance_ctu() 10 0.002823 0.002823 0.002823 0.33% Castro::construct_old_gravity_source() 10 0.002628 0.002628 0.002628 0.31% amrex::Add() 164 0.002494 0.002494 0.002494 0.30% Amr::writePlotFile() 2 0.002418 0.002418 0.002418 0.29% MLMG::ResNormInf() 93 0.00206 0.00206 0.00206 0.24% FabArray::Saxpy() 20 0.001803 0.001803 0.001803 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001597 0.001597 0.001597 0.19% MLCellLinOp::setLevelBC() 11 0.00151 0.00151 0.00151 0.18% Castro::reset_internal_energy(Fab) 504 0.001435 0.001435 0.001435 0.17% Castro::enforce_speed_limit() 62 0.00136 0.00136 0.00136 0.16% Gravity::actual_solve_with_mlmg() 11 0.001358 0.001358 0.001358 0.16% FabArray::mult() 43 0.001309 0.001309 0.001309 0.16% FabArray::setDomainBndry() 41 0.001284 0.001284 0.001284 0.15% Castro::initData() 1 0.001281 0.001281 0.001281 0.15% MLMG::prepareForSolve() 11 0.001208 0.001208 0.001208 0.14% MultiFab::contains_nan() 20 0.001191 0.001191 0.001191 0.14% MLCellLinOp::prepareForSolve() 11 0.001145 0.001145 0.001145 0.14% MLCellLinOp::smooth() 1640 0.001012 0.001012 0.001012 0.12% MLCellLinOp::compGrad() 11 0.0009083 0.0009083 0.0009083 0.11% FabArray::FillBoundary() 4023 0.0007803 0.0007803 0.0007803 0.09% FabArrayBase::getCPC() 1323 0.0007647 0.0007647 0.0007647 0.09% FabArrayBase::CPC::define() 454 0.0006801 0.0006801 0.0006801 0.08% Gravity::get_new_grav_vector() 11 0.0006261 0.0006261 0.0006261 0.07% FabArrayBase::getFB() 4023 0.0006242 0.0006242 0.0006242 0.07% Gravity::get_old_grav_vector() 10 0.0005549 0.0005549 0.0005549 0.07% Amr::InitAmr() 1 0.0004731 0.0004731 0.0004731 0.06% MLCellLinOp::apply() 1142 0.0004531 0.0004531 0.0004531 0.05% MLLinOp::defineGrids() 11 0.0003764 0.0003764 0.0003764 0.04% MLMG::mgVcycle() 82 0.0003729 0.0003729 0.0003729 0.04% CGSolver::sxay() 1586 0.0003415 0.0003415 0.0003415 0.04% Amr::coarseTimeStep() 10 0.0003359 0.0003359 0.0003359 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003041 0.0003041 0.0003041 0.04% FillPatchIterator::Initialize 41 0.000282 0.000282 0.000282 0.03% MLCellLinOp::defineBC() 11 0.0002744 0.0002744 0.0002744 0.03% main() 1 0.0002678 0.0002678 0.0002678 0.03% FabArray::ParallelCopy() 861 0.0002553 0.0002553 0.0002553 0.03% MultiFab::max() 11 0.0002537 0.0002537 0.0002537 0.03% Castro::subcycle_advance_ctu() 10 0.000242 0.000242 0.000242 0.03% MLMG::MLRhsNormInf() 11 0.0002137 0.0002137 0.0002137 0.03% MLCellLinOp::correctionResidual() 492 0.0002082 0.0002082 0.0002082 0.02% Amr::timeStep() 10 0.0001595 0.0001595 0.0001595 0.02% MLMG:computeResOfCorrection() 410 0.0001421 0.0001421 0.0001421 0.02% Gravity::update_max_rhs() 11 0.0001373 0.0001373 0.0001373 0.02% StateData::checkPoint() 12 0.0001338 0.0001338 0.0001338 0.02% Gravity::solve_for_phi() 10 0.0001107 0.0001107 0.0001107 0.01% MLMG::mgVcycle_down::0 82 0.0001056 0.0001056 0.0001056 0.01% MLMG::mgVcycle_down::1 82 9.378e-05 9.378e-05 9.378e-05 0.01% MLMG::mgVcycle_down::2 82 8.55e-05 8.55e-05 8.55e-05 0.01% FabArrayBase::FB::FB() 56 8.475e-05 8.475e-05 8.475e-05 0.01% Castro::initialize_advance() 10 8.3e-05 8.3e-05 8.3e-05 0.01% MLMG::mgVcycle_down::3 82 8.278e-05 8.278e-05 8.278e-05 0.01% MLMG::mgVcycle_down::4 82 8.218e-05 8.218e-05 8.218e-05 0.01% MLMG::actualBottomSolve() 82 7.652e-05 7.652e-05 7.652e-05 0.01% Castro::Castro() 1 7.58e-05 7.58e-05 7.58e-05 0.01% Castro::expand_state() 10 7.488e-05 7.488e-05 7.488e-05 0.01% Castro::clean_state() 62 7.384e-05 7.384e-05 7.384e-05 0.01% AmrLevel::checkPoint() 3 7.179e-05 7.179e-05 7.179e-05 0.01% MLMG::solve() 11 7.091e-05 7.091e-05 7.091e-05 0.01% MLMG::mgVcycle_up::4 82 7.061e-05 7.061e-05 7.061e-05 0.01% Castro::advance() 10 6.996e-05 6.996e-05 6.996e-05 0.01% Castro::initialize_do_advance() 10 6.344e-05 6.344e-05 6.344e-05 0.01% Castro::construct_new_source() 50 6.18e-05 6.18e-05 6.18e-05 0.01% MLMG::oneIter() 82 6.037e-05 6.037e-05 6.037e-05 0.01% Castro::finalize_advance() 10 5.723e-05 5.723e-05 5.723e-05 0.01% MLMG::mgVcycle_up::0 82 5.429e-05 5.429e-05 5.429e-05 0.01% MLMG::mgVcycle_up::1 82 5.344e-05 5.344e-05 5.344e-05 0.01% MLMG::mgVcycle_up::3 82 5.178e-05 5.178e-05 5.178e-05 0.01% MLMG::mgVcycle_up::2 82 5.152e-05 5.152e-05 5.152e-05 0.01% MLCellLinOp::solutionResidual() 93 4.55e-05 4.55e-05 4.55e-05 0.01% MLMG::computeResidual() 82 4.498e-05 4.498e-05 4.498e-05 0.01% StateData::define() 4 4.296e-05 4.296e-05 4.296e-05 0.01% Castro::swap_state_time_levels() 10 4.271e-05 4.271e-05 4.271e-05 0.01% MLMG::mgVcycle_bottom 82 3.413e-05 3.413e-05 3.413e-05 0.00% MLPoisson::define() 11 3.405e-05 3.405e-05 3.405e-05 0.00% Castro::finalize_do_advance() 10 3.355e-05 3.355e-05 3.355e-05 0.00% Castro::enforce_consistent_e() 1 3.219e-05 3.219e-05 3.219e-05 0.00% FillPatchSingleLevel 41 3.053e-05 3.053e-05 3.053e-05 0.00% MultiFab::Add() 82 3.052e-05 3.052e-05 3.052e-05 0.00% Castro::initMFs() 1 2.764e-05 2.764e-05 2.764e-05 0.00% Amr::writeSmallPlotFile() 1 2.494e-05 2.494e-05 2.494e-05 0.00% MLLinOp::define() 11 2.474e-05 2.474e-05 2.474e-05 0.00% makeSFC 55 2.466e-05 2.466e-05 2.466e-05 0.00% Castro::construct_new_gravity() 10 2.35e-05 2.35e-05 2.35e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.267e-05 2.267e-05 2.267e-05 0.00% Castro::buildMetrics() 1 2.095e-05 2.095e-05 2.095e-05 0.00% Amr::FinalizeInit() 1 2.052e-05 2.052e-05 2.052e-05 0.00% Amr::defBaseLevel() 1 1.93e-05 1.93e-05 1.93e-05 0.00% Castro::construct_old_source() 50 1.806e-05 1.806e-05 1.806e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.798e-05 1.798e-05 1.798e-05 0.00% Castro::do_new_sources() 10 1.576e-05 1.576e-05 1.576e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.456e-05 1.456e-05 1.456e-05 0.00% Castro::do_old_sources() 10 1.454e-05 1.454e-05 1.454e-05 0.00% DistributionMapping::Distribute() 56 1.443e-05 1.443e-05 1.443e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.417e-05 1.417e-05 1.417e-05 0.00% Castro::check_for_nan() 20 1.09e-05 1.09e-05 1.09e-05 0.00% Castro::apply_source_to_state() 20 1.035e-05 1.035e-05 1.035e-05 0.00% Castro::construct_old_gravity() 10 9.468e-06 9.468e-06 9.468e-06 0.00% MLMG::computeMLResidual() 11 9.217e-06 9.217e-06 9.217e-06 0.00% Castro::post_timestep() 10 9.045e-06 9.045e-06 9.045e-06 0.00% Gravity::swapTimeLevels() 10 8.922e-06 8.922e-06 8.922e-06 0.00% MLPoisson::prepareForSolve() 11 8.849e-06 8.849e-06 8.849e-06 0.00% Amr::initSubcycle() 1 8.669e-06 8.669e-06 8.669e-06 0.00% Castro::post_init() 1 8.635e-06 8.635e-06 8.635e-06 0.00% Gravity::actual_multilevel_solve() 1 8.385e-06 8.385e-06 8.385e-06 0.00% MLMG::getGradSolution() 11 6.773e-06 6.773e-06 6.773e-06 0.00% Castro::computeNewDt() 9 6.302e-06 6.302e-06 6.302e-06 0.00% AmrLevel::checkPointPost() 3 5.914e-06 5.914e-06 5.914e-06 0.00% MultiFab::Copy() 11 5.28e-06 5.28e-06 5.28e-06 0.00% Amr::InitializeInit() 1 4.829e-06 4.829e-06 4.829e-06 0.00% Castro::retry_advance_ctu() 10 4.713e-06 4.713e-06 4.713e-06 0.00% Gravity::set_mass_offset() 11 3.926e-06 3.926e-06 3.926e-06 0.00% Castro::create_source_corrector() 10 3.751e-06 3.751e-06 3.751e-06 0.00% MLMG::MLResNormInf() 11 3.347e-06 3.347e-06 3.347e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.965e-06 2.965e-06 2.965e-06 0.00% Amr::init() 1 2.907e-06 2.907e-06 2.907e-06 0.00% Castro::FluxRegCrseInit 10 2.745e-06 2.745e-06 2.745e-06 0.00% Castro::FluxRegFineAdd() 10 2.392e-06 2.392e-06 2.392e-06 0.00% Castro::computeInitialDt() 2 2.204e-06 2.204e-06 2.204e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.985e-06 1.985e-06 1.985e-06 0.00% AmrLevel::checkPointPre() 3 1.908e-06 1.908e-06 1.908e-06 0.00% Castro::post_regrid() 1 1.184e-06 1.184e-06 1.184e-06 0.00% Amr::initialInit() 1 9.21e-07 9.21e-07 9.21e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8428 0.8428 0.8428 100.00% Amr::coarseTimeStep() 10 0.6781 0.6781 0.6781 80.45% Amr::timeStep() 10 0.5789 0.5789 0.5789 68.69% Castro::advance() 10 0.5709 0.5709 0.5709 67.74% Castro::subcycle_advance_ctu() 10 0.5579 0.5579 0.5579 66.19% Castro::do_advance_ctu() 10 0.5576 0.5576 0.5576 66.16% Gravity::solve_phi_with_mlmg() 11 0.3097 0.3097 0.3097 36.74% Gravity::actual_solve_with_mlmg() 11 0.3011 0.3011 0.3011 35.72% Castro::construct_new_gravity() 10 0.281 0.281 0.281 33.34% MLMG::solve() 11 0.2789 0.2789 0.2789 33.10% Gravity::solve_for_phi() 10 0.2658 0.2658 0.2658 31.54% MLMG::oneIter() 82 0.2641 0.2641 0.2641 31.34% MLMG::mgVcycle() 82 0.2624 0.2624 0.2624 31.13% VisMF::Write(FabArray) 11 0.2002 0.2002 0.2002 23.75% Castro::construct_ctu_hydro_source() 10 0.1941 0.1941 0.1941 23.03% Amr::checkPoint() 3 0.1487 0.1487 0.1487 17.64% AmrLevel::checkPoint() 3 0.1435 0.1435 0.1435 17.02% StateData::checkPoint() 12 0.1434 0.1434 0.1434 17.01% Amr::init() 1 0.1342 0.1342 0.1342 15.92% MLCellLinOp::smooth() 1640 0.1339 0.1339 0.1339 15.88% MLCellLinOp::applyBC() 4433 0.09426 0.09426 0.09426 11.18% MLMG::mgVcycle_bottom 82 0.08095 0.08095 0.08095 9.61% MLMG::actualBottomSolve() 82 0.08092 0.08092 0.08092 9.60% MLCGSolver::bicgstab 82 0.08013 0.08013 0.08013 9.51% MLPoisson::Fsmooth() 3280 0.06285 0.06285 0.06285 7.46% Amr::writePlotFile() 2 0.05985 0.05985 0.05985 7.10% Amr::initialInit() 1 0.05167 0.05167 0.05167 6.13% FillPatchIterator::Initialize 41 0.04863 0.04863 0.04863 5.77% Amr::FinalizeInit() 1 0.04748 0.04748 0.04748 5.63% FillPatchSingleLevel 41 0.04707 0.04707 0.04707 5.58% Castro::post_init() 1 0.04617 0.04617 0.04617 5.48% Castro::clean_state() 62 0.04605 0.04605 0.04605 5.46% Gravity::multilevel_solve_for_new_phi() 1 0.04424 0.04424 0.04424 5.25% Gravity::actual_multilevel_solve() 1 0.04422 0.04422 0.04422 5.25% StateDataPhysBCFunct::() 41 0.04303 0.04303 0.04303 5.11% MLCellLinOp::apply() 1142 0.03579 0.03579 0.03579 4.25% MLMG::mgVcycle_down::0 82 0.03477 0.03477 0.03477 4.13% MLMG::mgVcycle_up::0 82 0.02979 0.02979 0.02979 3.53% Castro::initialize_do_advance() 10 0.02647 0.02647 0.02647 3.14% StateData::FillBoundary(geom) 328 0.02313 0.02313 0.02313 2.74% amrex::Dot() 1114 0.0219 0.0219 0.0219 2.60% MLCellLinOp::correctionResidual() 492 0.02092 0.02092 0.02092 2.48% Castro::computeTemp() 63 0.02041 0.02041 0.02041 2.42% Castro::expand_state() 10 0.01873 0.01873 0.01873 2.22% MLMG:computeResOfCorrection() 410 0.01809 0.01809 0.01809 2.15% MLPoisson::define() 11 0.01788 0.01788 0.01788 2.12% MLMG::mgVcycle_down::1 82 0.01744 0.01744 0.01744 2.07% MLMG::mgVcycle_down::2 82 0.01697 0.01697 0.01697 2.01% Gravity::get_new_grav_vector() 11 0.01676 0.01676 0.01676 1.99% MLMG::mgVcycle_down::3 82 0.01609 0.01609 0.01609 1.91% amrex::Copy() 1029 0.01559 0.01559 0.01559 1.85% FabArray::FillBoundary() 4023 0.01558 0.01558 0.01558 1.85% MLMG::mgVcycle_down::4 82 0.01535 0.01535 0.01535 1.82% FillBoundary_nowait() 4023 0.0148 0.0148 0.0148 1.76% CGSolver::sxay() 1586 0.01453 0.01453 0.01453 1.72% Castro::construct_old_gravity() 10 0.01445 0.01445 0.01445 1.71% Gravity::get_old_grav_vector() 10 0.01444 0.01444 0.01444 1.71% FabArray::LinComb() 1586 0.01419 0.01419 0.01419 1.68% Castro::normalize_species() 62 0.01405 0.01405 0.01405 1.67% FabArray::setVal() 1144 0.01403 0.01403 0.01403 1.66% FabArray::ParallelCopy() 861 0.01393 0.01393 0.01393 1.65% FabArray::ParallelCopy_nowait() 861 0.01367 0.01367 0.01367 1.62% MLCGSolver::ParallelAllReduce 1514 0.01307 0.01307 0.01307 1.55% MLMG::mgVcycle_up::2 82 0.01307 0.01307 0.01307 1.55% MLMG::mgVcycle_up::1 82 0.01288 0.01288 0.01288 1.53% FabArray::norminf() 639 0.01266 0.01266 0.01266 1.50% MLMG::addInterpCorrection() 410 0.0126 0.0126 0.0126 1.49% MLCellLinOp::defineAuxData() 11 0.01248 0.01248 0.01248 1.48% MLMG::mgVcycle_up::3 82 0.01242 0.01242 0.01242 1.47% Castro::initialize_advance() 10 0.01239 0.01239 0.01239 1.47% MLMG::mgVcycle_up::4 82 0.0123 0.0123 0.0123 1.46% Castro::do_new_sources() 10 0.01197 0.01197 0.01197 1.42% amrex::average_down 410 0.01173 0.01173 0.01173 1.39% MLPoisson::Fapply() 1142 0.01158 0.01158 0.01158 1.37% Castro::enforce_min_density() 62 0.01099 0.01099 0.01099 1.30% Castro::do_old_sources() 10 0.01037 0.01037 0.01037 1.23% Gravity::fill_multipole_BCs() 11 0.008348 0.008348 0.008348 0.99% Castro::post_timestep() 10 0.007847 0.007847 0.007847 0.93% MLCellLinOp::solutionResidual() 93 0.007111 0.007111 0.007111 0.84% FabArray::Xpay() 585 0.00649 0.00649 0.00649 0.77% MLMG::computeResidual() 82 0.006149 0.006149 0.006149 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005819 0.005819 0.005819 0.69% MLMG::prepareForSolve() 11 0.005282 0.005282 0.005282 0.63% MLCellLinOp::defineBC() 11 0.004917 0.004917 0.004917 0.58% Castro::estTimeStep() 21 0.004753 0.004753 0.004753 0.56% BndryData::define() 11 0.004642 0.004642 0.004642 0.55% Amr::InitializeInit() 1 0.004192 0.004192 0.004192 0.50% Amr::defBaseLevel() 1 0.004187 0.004187 0.004187 0.50% Castro::initData() 1 0.003688 0.003688 0.003688 0.44% Castro::construct_new_source() 50 0.003356 0.003356 0.003356 0.40% Castro::construct_new_gravity_source() 10 0.003294 0.003294 0.003294 0.39% Castro::construct_old_source() 50 0.002646 0.002646 0.002646 0.31% Castro::construct_old_gravity_source() 10 0.002628 0.002628 0.002628 0.31% amrex::Add() 164 0.002494 0.002494 0.002494 0.30% MLMG::ResNormInf() 93 0.00206 0.00206 0.00206 0.24% Castro::computeNewDt() 9 0.001919 0.001919 0.001919 0.23% Castro::apply_source_to_state() 20 0.001813 0.001813 0.001813 0.22% FabArray::Saxpy() 20 0.001803 0.001803 0.001803 0.21% MultiFab::Add() 82 0.001686 0.001686 0.001686 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001597 0.001597 0.001597 0.19% MLCellLinOp::setLevelBC() 11 0.00151 0.00151 0.00151 0.18% FabArrayBase::getCPC() 1323 0.001445 0.001445 0.001445 0.17% Castro::reset_internal_energy(Fab) 504 0.001435 0.001435 0.001435 0.17% MLMG::getGradSolution() 11 0.001405 0.001405 0.001405 0.17% MLCellLinOp::compGrad() 11 0.001398 0.001398 0.001398 0.17% Castro::enforce_speed_limit() 62 0.00136 0.00136 0.00136 0.16% FabArray::mult() 43 0.001309 0.001309 0.001309 0.16% FabArray::setDomainBndry() 41 0.001284 0.001284 0.001284 0.15% Castro::check_for_nan() 20 0.001202 0.001202 0.001202 0.14% MultiFab::contains_nan() 20 0.001191 0.001191 0.001191 0.14% MLPoisson::prepareForSolve() 11 0.001154 0.001154 0.001154 0.14% MLCellLinOp::prepareForSolve() 11 0.001145 0.001145 0.001145 0.14% Castro::post_regrid() 1 0.00113 0.00113 0.00113 0.13% MLMG::computeMLResidual() 11 0.001016 0.001016 0.001016 0.12% Castro::computeInitialDt() 2 0.0008382 0.0008382 0.0008382 0.10% Gravity::update_max_rhs() 11 0.0008215 0.0008215 0.0008215 0.10% FabArrayBase::getFB() 4023 0.0007089 0.0007089 0.0007089 0.08% FabArrayBase::CPC::define() 454 0.0006801 0.0006801 0.0006801 0.08% Castro::finalize_advance() 10 0.0005696 0.0005696 0.0005696 0.07% Amr::InitAmr() 1 0.0004817 0.0004817 0.0004817 0.06% MLLinOp::define() 11 0.0004557 0.0004557 0.0004557 0.05% MLLinOp::defineGrids() 11 0.0004309 0.0004309 0.0004309 0.05% Gravity::swapTimeLevels() 10 0.0004304 0.0004304 0.0004304 0.05% Castro::Castro() 1 0.000419 0.000419 0.000419 0.05% MultiFab::Copy() 11 0.0003237 0.0003237 0.0003237 0.04% MLMG::MLResNormInf() 11 0.0002782 0.0002782 0.0002782 0.03% MultiFab::max() 11 0.0002537 0.0002537 0.0002537 0.03% MLMG::MLRhsNormInf() 11 0.0002137 0.0002137 0.0002137 0.03% Castro::buildMetrics() 1 0.0001566 0.0001566 0.0001566 0.02% FabArrayBase::FB::FB() 56 8.475e-05 8.475e-05 8.475e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.713e-05 5.713e-05 5.713e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.258e-05 5.258e-05 5.258e-05 0.01% StateData::define() 4 4.296e-05 4.296e-05 4.296e-05 0.01% Castro::swap_state_time_levels() 10 4.271e-05 4.271e-05 4.271e-05 0.01% makeSFC 55 3.802e-05 3.802e-05 3.802e-05 0.00% Castro::finalize_do_advance() 10 3.355e-05 3.355e-05 3.355e-05 0.00% Castro::enforce_consistent_e() 1 3.219e-05 3.219e-05 3.219e-05 0.00% Castro::initMFs() 1 2.764e-05 2.764e-05 2.764e-05 0.00% Amr::writeSmallPlotFile() 1 2.494e-05 2.494e-05 2.494e-05 0.00% DistributionMapping::Distribute() 56 1.443e-05 1.443e-05 1.443e-05 0.00% Amr::initSubcycle() 1 8.669e-06 8.669e-06 8.669e-06 0.00% AmrLevel::checkPointPost() 3 5.914e-06 5.914e-06 5.914e-06 0.00% Castro::retry_advance_ctu() 10 4.713e-06 4.713e-06 4.713e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.032e-06 4.032e-06 4.032e-06 0.00% Gravity::set_mass_offset() 11 3.926e-06 3.926e-06 3.926e-06 0.00% Castro::create_source_corrector() 10 3.751e-06 3.751e-06 3.751e-06 0.00% Castro::FluxRegCrseInit 10 2.745e-06 2.745e-06 2.745e-06 0.00% Castro::FluxRegFineAdd() 10 2.392e-06 2.392e-06 2.392e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.985e-06 1.985e-06 1.985e-06 0.00% AmrLevel::checkPointPre() 3 1.908e-06 1.908e-06 1.908e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-1-g4a53367b19e1) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-1-g4a53367b19e1) initialized Starting run at 09:55:26 UTC on 2022-12-05. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-1-g4a53367b1 Microphysics git describe: 22.12-10-g85f387df reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.454544663 Restart time = 0.048073513 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051458167 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051315471 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050996783 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058729369 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.079237851 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031941639 seconds Ending run at 09:55:26 UTC on 2022-12-05. Run time = 0.372699618 Run time without initialization = 0.324071007 Average number of zones advanced per microsecond: 4.045 Average number of zones advanced per microsecond per rank: 4.045 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3727 ... 0.3727 ... 0.3727 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0939 0.0939 0.0939 25.20% VisMF::Read() 3 0.04 0.04 0.04 10.73% MLCellLinOp::applyBC() 1946 0.03409 0.03409 0.03409 9.15% VisMF::Write(FabArray) 1 0.03032 0.03032 0.03032 8.13% MLPoisson::Fsmooth() 1440 0.02659 0.02659 0.02659 7.13% StateData::FillBoundary(geom) 160 0.01113 0.01113 0.01113 2.99% amrex::Dot() 484 0.009348 0.009348 0.009348 2.51% Castro::computeTemp() 30 0.009223 0.009223 0.009223 2.47% Castro::normalize_species() 30 0.008072 0.008072 0.008072 2.17% amrex::Copy() 463 0.007264 0.007264 0.007264 1.95% FabArray::setVal() 537 0.006624 0.006624 0.006624 1.78% FillBoundary_nowait() 1766 0.006223 0.006223 0.006223 1.67% MLCellLinOp::defineAuxData() 6 0.006021 0.006021 0.006021 1.62% FabArray::LinComb() 690 0.005983 0.005983 0.005983 1.61% FabArray::ParallelCopy_nowait() 380 0.00583 0.00583 0.00583 1.56% StateDataPhysBCFunct::() 20 0.005793 0.005793 0.005793 1.55% FabArray::norminf() 278 0.005415 0.005415 0.005415 1.45% MLPoisson::Fapply() 500 0.004955 0.004955 0.004955 1.33% Castro::enforce_min_density() 30 0.00452 0.00452 0.00452 1.21% Gravity::fill_multipole_BCs() 6 0.004307 0.004307 0.004307 1.16% Amr::restart() 1 0.00377 0.00377 0.00377 1.01% MLMG::addInterpCorrection() 180 0.003278 0.003278 0.003278 0.88% amrex::average_down 180 0.002934 0.002934 0.002934 0.79% FabArray::Xpay() 258 0.002813 0.002813 0.002813 0.75% Castro::estTimeStep() 10 0.00269 0.00269 0.00269 0.72% BndryData::define() 6 0.002023 0.002023 0.002023 0.54% Castro::enforce_speed_limit() 30 0.002011 0.002011 0.002011 0.54% Castro::reset_internal_energy(MultiFab) 30 0.001773 0.001773 0.001773 0.48% Castro::construct_new_gravity_source() 5 0.00177 0.00177 0.00177 0.47% Castro::construct_old_gravity_source() 5 0.00158 0.00158 0.00158 0.42% Amr::writePlotFile() 1 0.001457 0.001457 0.001457 0.39% MLCGSolver::bicgstab 36 0.001394 0.001394 0.001394 0.37% Castro::do_advance_ctu() 5 0.001309 0.001309 0.001309 0.35% amrex::Add() 72 0.001071 0.001071 0.001071 0.29% Castro::subcycle_advance_ctu() 5 0.001016 0.001016 0.001016 0.27% Castro::reset_internal_energy(Fab) 240 0.0009435 0.0009435 0.0009435 0.25% FabArray::Saxpy() 10 0.0009203 0.0009203 0.0009203 0.25% MLMG::ResNormInf() 42 0.0009151 0.0009151 0.0009151 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008855 0.0008855 0.0008855 0.24% MLCellLinOp::setLevelBC() 6 0.0008011 0.0008011 0.0008011 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007898 0.0007898 0.0007898 0.21% FabArray::setDomainBndry() 20 0.0006424 0.0006424 0.0006424 0.17% FabArray::mult() 22 0.0006407 0.0006407 0.0006407 0.17% MLMG::prepareForSolve() 6 0.0006356 0.0006356 0.0006356 0.17% MLCellLinOp::prepareForSolve() 6 0.0006169 0.0006169 0.0006169 0.17% MultiFab::contains_nan() 10 0.0005897 0.0005897 0.0005897 0.16% MLCellLinOp::compGrad() 6 0.000486 0.000486 0.000486 0.13% MLCellLinOp::smooth() 720 0.0004498 0.0004498 0.0004498 0.12% FabArrayBase::CPC::define() 244 0.000394 0.000394 0.000394 0.11% Amr::InitAmr() 1 0.0003777 0.0003777 0.0003777 0.10% FabArrayBase::getCPC() 632 0.0003644 0.0003644 0.0003644 0.10% Gravity::get_old_grav_vector() 5 0.0003621 0.0003621 0.0003621 0.10% FabArray::FillBoundary() 1766 0.0003316 0.0003316 0.0003316 0.09% Gravity::get_new_grav_vector() 5 0.0002751 0.0002751 0.0002751 0.07% main() 1 0.0002727 0.0002727 0.0002727 0.07% FabArrayBase::getFB() 1766 0.0002619 0.0002619 0.0002619 0.07% Castro::create_source_corrector() 5 0.0002123 0.0002123 0.0002123 0.06% MLCellLinOp::apply() 500 0.0002004 0.0002004 0.0002004 0.05% MLMG::mgVcycle() 36 0.0001637 0.0001637 0.0001637 0.04% CGSolver::sxay() 690 0.0001618 0.0001618 0.0001618 0.04% Amr::coarseTimeStep() 5 0.0001567 0.0001567 0.0001567 0.04% Castro::construct_new_source() 25 0.0001564 0.0001564 0.0001564 0.04% MLCellLinOp::defineBC() 6 0.000141 0.000141 0.000141 0.04% FillPatchIterator::Initialize 20 0.000136 0.000136 0.000136 0.04% MLLinOp::defineGrids() 6 0.0001345 0.0001345 0.0001345 0.04% MultiFab::max() 6 0.0001326 0.0001326 0.0001326 0.04% MLCGSolver::ParallelAllReduce 659 0.0001283 0.0001283 0.0001283 0.03% MLMG::MLRhsNormInf() 6 0.0001135 0.0001135 0.0001135 0.03% FabArray::ParallelCopy() 380 0.0001131 0.0001131 0.0001131 0.03% Castro::construct_old_source() 25 9.894e-05 9.894e-05 9.894e-05 0.03% Amr::timeStep() 5 9.605e-05 9.605e-05 9.605e-05 0.03% Castro::post_timestep() 5 9.109e-05 9.109e-05 9.109e-05 0.02% Castro::initialize_do_advance() 5 8.841e-05 8.841e-05 8.841e-05 0.02% MLCellLinOp::correctionResidual() 216 8.744e-05 8.744e-05 8.744e-05 0.02% Castro::advance() 5 8.595e-05 8.595e-05 8.595e-05 0.02% AmrLevel::restart() 1 8.218e-05 8.218e-05 8.218e-05 0.02% Castro::computeNewDt() 5 7.954e-05 7.954e-05 7.954e-05 0.02% Gravity::update_max_rhs() 6 7.799e-05 7.799e-05 7.799e-05 0.02% StateData::restartDoit() 4 7.539e-05 7.539e-05 7.539e-05 0.02% Castro::finalize_advance() 5 7.487e-05 7.487e-05 7.487e-05 0.02% Castro::initialize_advance() 5 6.909e-05 6.909e-05 6.909e-05 0.02% Gravity::solve_for_phi() 5 6.625e-05 6.625e-05 6.625e-05 0.02% MLMG:computeResOfCorrection() 180 6.182e-05 6.182e-05 6.182e-05 0.02% Castro::construct_old_gravity() 5 6.11e-05 6.11e-05 6.11e-05 0.02% Castro::construct_new_gravity() 5 5.837e-05 5.837e-05 5.837e-05 0.02% FabArrayBase::FB::FB() 26 5.764e-05 5.764e-05 5.764e-05 0.02% MLMG::mgVcycle_down::0 36 4.76e-05 4.76e-05 4.76e-05 0.01% Castro::post_restart() 1 4.177e-05 4.177e-05 4.177e-05 0.01% MLMG::mgVcycle_down::1 36 4.103e-05 4.103e-05 4.103e-05 0.01% Castro::clean_state() 30 3.936e-05 3.936e-05 3.936e-05 0.01% Castro::expand_state() 5 3.925e-05 3.925e-05 3.925e-05 0.01% MLMG::mgVcycle_down::2 36 3.696e-05 3.696e-05 3.696e-05 0.01% MLMG::mgVcycle_down::4 36 3.61e-05 3.61e-05 3.61e-05 0.01% MLMG::mgVcycle_down::3 36 3.602e-05 3.602e-05 3.602e-05 0.01% Castro::buildMetrics() 1 3.48e-05 3.48e-05 3.48e-05 0.01% MLMG::solve() 6 3.33e-05 3.33e-05 3.33e-05 0.01% MLMG::actualBottomSolve() 36 3.284e-05 3.284e-05 3.284e-05 0.01% MLMG::mgVcycle_up::4 36 3.131e-05 3.131e-05 3.131e-05 0.01% Castro::initMFs() 1 3.079e-05 3.079e-05 3.079e-05 0.01% MLMG::oneIter() 36 2.969e-05 2.969e-05 2.969e-05 0.01% Castro::swap_state_time_levels() 5 2.803e-05 2.803e-05 2.803e-05 0.01% MLPoisson::define() 6 2.617e-05 2.617e-05 2.617e-05 0.01% Amr::writeSmallPlotFile() 1 2.558e-05 2.558e-05 2.558e-05 0.01% MLMG::mgVcycle_up::0 36 2.495e-05 2.495e-05 2.495e-05 0.01% MLMG::mgVcycle_up::3 36 2.188e-05 2.188e-05 2.188e-05 0.01% MLMG::mgVcycle_up::2 36 2.174e-05 2.174e-05 2.174e-05 0.01% MLLinOp::define() 6 2.147e-05 2.147e-05 2.147e-05 0.01% MLMG::mgVcycle_up::1 36 2.07e-05 2.07e-05 2.07e-05 0.01% MLCellLinOp::solutionResidual() 42 2.021e-05 2.021e-05 2.021e-05 0.01% MLMG::computeResidual() 36 1.967e-05 1.967e-05 1.967e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.905e-05 1.905e-05 1.905e-05 0.01% Castro::finalize_do_advance() 5 1.74e-05 1.74e-05 1.74e-05 0.00% MLMG::mgVcycle_bottom 36 1.633e-05 1.633e-05 1.633e-05 0.00% FillPatchSingleLevel 20 1.477e-05 1.477e-05 1.477e-05 0.00% makeSFC 30 1.404e-05 1.404e-05 1.404e-05 0.00% MultiFab::Add() 36 1.269e-05 1.269e-05 1.269e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.245e-05 1.245e-05 1.245e-05 0.00% Castro::do_new_sources() 5 1.015e-05 1.015e-05 1.015e-05 0.00% Amr::initSubcycle() 1 8.72e-06 8.72e-06 8.72e-06 0.00% Gravity::actual_multilevel_solve() 1 8.297e-06 8.297e-06 8.297e-06 0.00% DistributionMapping::Distribute() 31 8.293e-06 8.293e-06 8.293e-06 0.00% Castro::do_old_sources() 5 8.184e-06 8.184e-06 8.184e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.81e-06 7.81e-06 7.81e-06 0.00% Castro::check_for_nan() 10 6.292e-06 6.292e-06 6.292e-06 0.00% Castro::apply_source_to_state() 10 5.49e-06 5.49e-06 5.49e-06 0.00% MLPoisson::prepareForSolve() 6 4.847e-06 4.847e-06 4.847e-06 0.00% Gravity::swapTimeLevels() 5 4.559e-06 4.559e-06 4.559e-06 0.00% MLMG::computeMLResidual() 6 4.217e-06 4.217e-06 4.217e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.355e-06 3.355e-06 3.355e-06 0.00% MLMG::getGradSolution() 6 3.165e-06 3.165e-06 3.165e-06 0.00% MultiFab::Copy() 6 2.837e-06 2.837e-06 2.837e-06 0.00% Castro::retry_advance_ctu() 5 2.642e-06 2.642e-06 2.642e-06 0.00% Castro::FluxRegCrseInit 5 2.219e-06 2.219e-06 2.219e-06 0.00% Gravity::set_mass_offset() 6 2.153e-06 2.153e-06 2.153e-06 0.00% MLMG::MLResNormInf() 6 2.127e-06 2.127e-06 2.127e-06 0.00% Castro::FluxRegFineAdd() 5 1.324e-06 1.324e-06 1.324e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.034e-06 1.034e-06 1.034e-06 0.00% Amr::init() 1 1.023e-06 1.023e-06 1.023e-06 0.00% AmrLevel::AmrLevel() 1 8.8e-07 8.8e-07 8.8e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3727 0.3727 0.3727 100.00% Amr::coarseTimeStep() 5 0.2919 0.2919 0.2919 78.31% Amr::timeStep() 5 0.2901 0.2901 0.2901 77.83% Castro::advance() 5 0.2862 0.2862 0.2862 76.78% Castro::subcycle_advance_ctu() 5 0.2788 0.2788 0.2788 74.80% Castro::do_advance_ctu() 5 0.2778 0.2778 0.2778 74.52% Castro::construct_new_gravity() 5 0.1409 0.1409 0.1409 37.81% Gravity::solve_phi_with_mlmg() 6 0.1369 0.1369 0.1369 36.74% Gravity::solve_for_phi() 5 0.1334 0.1334 0.1334 35.78% Gravity::actual_solve_with_mlmg() 6 0.1324 0.1324 0.1324 35.52% MLMG::solve() 6 0.1204 0.1204 0.1204 32.30% MLMG::oneIter() 36 0.1133 0.1133 0.1133 30.39% MLMG::mgVcycle() 36 0.1125 0.1125 0.1125 30.19% Castro::construct_ctu_hydro_source() 5 0.09395 0.09395 0.09395 25.20% MLCellLinOp::smooth() 720 0.05737 0.05737 0.05737 15.39% Amr::init() 1 0.04812 0.04812 0.04812 12.91% Amr::restart() 1 0.04811 0.04811 0.04811 12.91% MLCellLinOp::applyBC() 1946 0.04097 0.04097 0.04097 10.99% AmrLevel::restart() 1 0.04021 0.04021 0.04021 10.79% StateData::restartDoit() 4 0.04012 0.04012 0.04012 10.76% VisMF::Read() 3 0.04 0.04 0.04 10.73% MLMG::mgVcycle_bottom 36 0.03456 0.03456 0.03456 9.27% MLMG::actualBottomSolve() 36 0.03455 0.03455 0.03455 9.27% MLCGSolver::bicgstab 36 0.03421 0.03421 0.03421 9.18% Amr::writePlotFile() 1 0.03202 0.03202 0.03202 8.59% VisMF::Write(FabArray) 1 0.03032 0.03032 0.03032 8.13% MLPoisson::Fsmooth() 1440 0.02659 0.02659 0.02659 7.13% Castro::clean_state() 30 0.02658 0.02658 0.02658 7.13% FillPatchIterator::Initialize 20 0.01969 0.01969 0.01969 5.28% FillPatchSingleLevel 20 0.01891 0.01891 0.01891 5.07% StateDataPhysBCFunct::() 20 0.01692 0.01692 0.01692 4.54% MLCellLinOp::apply() 500 0.01552 0.01552 0.01552 4.16% MLMG::mgVcycle_down::0 36 0.01505 0.01505 0.01505 4.04% MLMG::mgVcycle_up::0 36 0.01284 0.01284 0.01284 3.44% Castro::computeTemp() 30 0.01194 0.01194 0.01194 3.20% Castro::initialize_do_advance() 5 0.01161 0.01161 0.01161 3.11% StateData::FillBoundary(geom) 160 0.01113 0.01113 0.01113 2.99% MLPoisson::define() 6 0.009659 0.009659 0.009659 2.59% amrex::Dot() 484 0.009348 0.009348 0.009348 2.51% MLCellLinOp::correctionResidual() 216 0.009024 0.009024 0.009024 2.42% Castro::normalize_species() 30 0.008072 0.008072 0.008072 2.17% MLMG:computeResOfCorrection() 180 0.007791 0.007791 0.007791 2.09% MLMG::mgVcycle_down::1 36 0.007501 0.007501 0.007501 2.01% Castro::do_new_sources() 5 0.007473 0.007473 0.007473 2.00% Gravity::get_new_grav_vector() 5 0.007389 0.007389 0.007389 1.98% amrex::Copy() 463 0.007264 0.007264 0.007264 1.95% MLMG::mgVcycle_down::2 36 0.007254 0.007254 0.007254 1.95% Castro::construct_old_gravity() 5 0.007218 0.007218 0.007218 1.94% Gravity::get_old_grav_vector() 5 0.007157 0.007157 0.007157 1.92% Castro::initialize_advance() 5 0.006943 0.006943 0.006943 1.86% MLMG::mgVcycle_down::3 36 0.00688 0.00688 0.00688 1.85% FabArray::FillBoundary() 1766 0.006874 0.006874 0.006874 1.84% MLCellLinOp::defineAuxData() 6 0.006759 0.006759 0.006759 1.81% FabArray::setVal() 537 0.006624 0.006624 0.006624 1.78% MLMG::mgVcycle_down::4 36 0.006598 0.006598 0.006598 1.77% FillBoundary_nowait() 1766 0.006542 0.006542 0.006542 1.76% FabArray::ParallelCopy() 380 0.006326 0.006326 0.006326 1.70% FabArray::ParallelCopy_nowait() 380 0.006213 0.006213 0.006213 1.67% CGSolver::sxay() 690 0.006145 0.006145 0.006145 1.65% Castro::do_old_sources() 5 0.006028 0.006028 0.006028 1.62% FabArray::LinComb() 690 0.005983 0.005983 0.005983 1.61% MLCGSolver::ParallelAllReduce 659 0.005606 0.005606 0.005606 1.50% Castro::expand_state() 5 0.005581 0.005581 0.005581 1.50% MLMG::mgVcycle_up::2 36 0.005572 0.005572 0.005572 1.49% MLMG::mgVcycle_up::1 36 0.005524 0.005524 0.005524 1.48% MLMG::addInterpCorrection() 180 0.005441 0.005441 0.005441 1.46% FabArray::norminf() 278 0.005415 0.005415 0.005415 1.45% MLMG::mgVcycle_up::3 36 0.005291 0.005291 0.005291 1.42% MLMG::mgVcycle_up::4 36 0.005285 0.005285 0.005285 1.42% amrex::average_down 180 0.005121 0.005121 0.005121 1.37% MLPoisson::Fapply() 500 0.004955 0.004955 0.004955 1.33% Castro::enforce_min_density() 30 0.00452 0.00452 0.00452 1.21% Gravity::fill_multipole_BCs() 6 0.004427 0.004427 0.004427 1.19% Castro::post_restart() 1 0.003949 0.003949 0.003949 1.06% Castro::post_timestep() 5 0.003838 0.003838 0.003838 1.03% Gravity::multilevel_solve_for_new_phi() 1 0.003809 0.003809 0.003809 1.02% Gravity::actual_multilevel_solve() 1 0.00379 0.00379 0.00379 1.02% MLCellLinOp::solutionResidual() 42 0.003202 0.003202 0.003202 0.86% MLMG::prepareForSolve() 6 0.002814 0.002814 0.002814 0.75% FabArray::Xpay() 258 0.002813 0.002813 0.002813 0.75% Castro::reset_internal_energy(MultiFab) 30 0.002716 0.002716 0.002716 0.73% Castro::estTimeStep() 10 0.00269 0.00269 0.00269 0.72% MLCellLinOp::defineBC() 6 0.002688 0.002688 0.002688 0.72% MLMG::computeResidual() 36 0.002664 0.002664 0.002664 0.71% BndryData::define() 6 0.002547 0.002547 0.002547 0.68% Castro::enforce_speed_limit() 30 0.002011 0.002011 0.002011 0.54% Castro::construct_new_source() 25 0.001927 0.001927 0.001927 0.52% Castro::construct_new_gravity_source() 5 0.00177 0.00177 0.00177 0.47% Castro::construct_old_source() 25 0.001678 0.001678 0.001678 0.45% Castro::computeNewDt() 5 0.001627 0.001627 0.001627 0.44% Castro::construct_old_gravity_source() 5 0.00158 0.00158 0.00158 0.42% amrex::Add() 72 0.001071 0.001071 0.001071 0.29% Castro::reset_internal_energy(Fab) 240 0.0009435 0.0009435 0.0009435 0.25% Castro::apply_source_to_state() 10 0.0009257 0.0009257 0.0009257 0.25% FabArray::Saxpy() 10 0.0009203 0.0009203 0.0009203 0.25% MLMG::ResNormInf() 42 0.0009151 0.0009151 0.0009151 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008855 0.0008855 0.0008855 0.24% MLCellLinOp::setLevelBC() 6 0.0008011 0.0008011 0.0008011 0.21% MLMG::getGradSolution() 6 0.0007609 0.0007609 0.0007609 0.20% FabArrayBase::getCPC() 632 0.0007585 0.0007585 0.0007585 0.20% MLCellLinOp::compGrad() 6 0.0007577 0.0007577 0.0007577 0.20% MultiFab::Add() 36 0.0007232 0.0007232 0.0007232 0.19% FabArray::setDomainBndry() 20 0.0006424 0.0006424 0.0006424 0.17% FabArray::mult() 22 0.0006407 0.0006407 0.0006407 0.17% MLPoisson::prepareForSolve() 6 0.0006218 0.0006218 0.0006218 0.17% MLCellLinOp::prepareForSolve() 6 0.0006169 0.0006169 0.0006169 0.17% Castro::check_for_nan() 10 0.000596 0.000596 0.000596 0.16% MultiFab::contains_nan() 10 0.0005897 0.0005897 0.0005897 0.16% MLMG::computeMLResidual() 6 0.000562 0.000562 0.000562 0.15% Gravity::update_max_rhs() 6 0.0004524 0.0004524 0.0004524 0.12% FabArrayBase::CPC::define() 244 0.000394 0.000394 0.000394 0.11% Amr::InitAmr() 1 0.0003864 0.0003864 0.0003864 0.10% Castro::finalize_advance() 5 0.0003447 0.0003447 0.0003447 0.09% FabArrayBase::getFB() 1766 0.0003195 0.0003195 0.0003195 0.09% Gravity::swapTimeLevels() 5 0.000222 0.000222 0.000222 0.06% Castro::create_source_corrector() 5 0.0002123 0.0002123 0.0002123 0.06% MLLinOp::define() 6 0.0001859 0.0001859 0.0001859 0.05% MultiFab::Copy() 6 0.0001744 0.0001744 0.0001744 0.05% MLLinOp::defineGrids() 6 0.0001644 0.0001644 0.0001644 0.04% Castro::buildMetrics() 1 0.0001547 0.0001547 0.0001547 0.04% MLMG::MLResNormInf() 6 0.0001495 0.0001495 0.0001495 0.04% MultiFab::max() 6 0.0001326 0.0001326 0.0001326 0.04% MLMG::MLRhsNormInf() 6 0.0001135 0.0001135 0.0001135 0.03% FabArrayBase::FB::FB() 26 5.764e-05 5.764e-05 5.764e-05 0.02% Castro::initMFs() 1 3.079e-05 3.079e-05 3.079e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.888e-05 2.888e-05 2.888e-05 0.01% Castro::swap_state_time_levels() 5 2.803e-05 2.803e-05 2.803e-05 0.01% Amr::writeSmallPlotFile() 1 2.558e-05 2.558e-05 2.558e-05 0.01% makeSFC 30 2.107e-05 2.107e-05 2.107e-05 0.01% Castro::finalize_do_advance() 5 1.74e-05 1.74e-05 1.74e-05 0.00% Amr::initSubcycle() 1 8.72e-06 8.72e-06 8.72e-06 0.00% DistributionMapping::Distribute() 31 8.293e-06 8.293e-06 8.293e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.615e-06 4.615e-06 4.615e-06 0.00% Castro::retry_advance_ctu() 5 2.642e-06 2.642e-06 2.642e-06 0.00% Castro::FluxRegCrseInit 5 2.219e-06 2.219e-06 2.219e-06 0.00% Gravity::set_mass_offset() 6 2.153e-06 2.153e-06 2.153e-06 0.00% Castro::FluxRegFineAdd() 5 1.324e-06 1.324e-06 1.324e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.034e-06 1.034e-06 1.034e-06 0.00% AmrLevel::AmrLevel() 1 8.8e-07 8.8e-07 8.8e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-1-g4a53367b19e1) finalized