Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-11-g0f4f9877c81e) initialized Starting run at 09:08:34 UTC on 2023-03-16. Successfully read inputs file ... Castro git describe: 23.03-8-g17ee5df0a AMReX git describe: 23.03-11-g0f4f9877c Microphysics git describe: 23.03-13-ga480b6a9 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058611671 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.03437754 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.046889475 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049994953 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.05741368 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.058823053 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.072253068 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057840331 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.059440362 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047395437 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056517082 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058643593 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061743545 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057813947 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033609646 seconds Ending run at 09:08:35 UTC on 2023-03-16. Run time = 0.861047223 Run time without initialization = 0.718996467 Average number of zones advanced per microsecond: 3.646 Average number of zones advanced per microsecond per rank: 3.646 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8611 ... 0.8611 ... 0.8611 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2354 0.2354 0.2354 27.33% Castro::construct_ctu_hydro_source() 10 0.2119 0.2119 0.2119 24.61% MLCellLinOp::applyBC() 4433 0.07701 0.07701 0.07701 8.94% MLPoisson::Fsmooth() 3280 0.03234 0.03234 0.03234 3.76% FillBoundary_nowait() 4023 0.03142 0.03142 0.03142 3.65% StateData::FillBoundary(geom) 328 0.02519 0.02519 0.02519 2.92% amrex::Dot() 1114 0.02062 0.02062 0.02062 2.39% Castro::normalize_species() 62 0.01584 0.01584 0.01584 1.84% amrex::Copy() 1029 0.01487 0.01487 0.01487 1.73% FabArray::norminf() 743 0.01435 0.01435 0.01435 1.67% Castro::computeTemp() 63 0.01399 0.01399 0.01399 1.62% FabArray::ParallelCopy_nowait() 861 0.01344 0.01344 0.01344 1.56% FabArray::setVal() 1144 0.01315 0.01315 0.01315 1.53% StateDataPhysBCFunct::() 41 0.01194 0.01194 0.01194 1.39% MLPoisson::Fapply() 1142 0.01046 0.01046 0.01046 1.21% MLCellLinOp::defineAuxData() 11 0.009737 0.009737 0.009737 1.13% FabArray::Saxpy() 813 0.008152 0.008152 0.008152 0.95% FabArray::Xpay() 821 0.008116 0.008116 0.008116 0.94% Castro::enforce_min_density() 62 0.007732 0.007732 0.007732 0.90% MLMG::addInterpCorrection() 410 0.006784 0.006784 0.006784 0.79% Gravity::fill_multipole_BCs() 11 0.006463 0.006463 0.006463 0.75% amrex::average_down 410 0.005965 0.005965 0.005965 0.69% FabArray::LinComb() 557 0.004537 0.004537 0.004537 0.53% amrex::Add() 164 0.004372 0.004372 0.004372 0.51% Castro::reset_internal_energy(MultiFab) 63 0.004075 0.004075 0.004075 0.47% Amr::checkPoint() 3 0.003986 0.003986 0.003986 0.46% Castro::estTimeStep() 21 0.00392 0.00392 0.00392 0.46% BndryData::define() 11 0.003794 0.003794 0.003794 0.44% Castro::construct_new_gravity_source() 10 0.003244 0.003244 0.003244 0.38% Castro::do_advance_ctu() 10 0.002879 0.002879 0.002879 0.33% Amr::writePlotFile() 2 0.002427 0.002427 0.002427 0.28% Castro::construct_old_gravity_source() 10 0.002267 0.002267 0.002267 0.26% MLCGSolver::bicgstab 82 0.002205 0.002205 0.002205 0.26% Castro::reset_internal_energy(Fab) 504 0.002164 0.002164 0.002164 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001701 0.001701 0.001701 0.20% Gravity::actual_solve_with_mlmg() 11 0.001483 0.001483 0.001483 0.17% MLCellLinOp::setLevelBC() 11 0.001402 0.001402 0.001402 0.16% FabArray::mult() 43 0.001317 0.001317 0.001317 0.15% Castro::enforce_speed_limit() 62 0.001317 0.001317 0.001317 0.15% Castro::initData() 1 0.001295 0.001295 0.001295 0.15% FabArray::setDomainBndry() 41 0.001293 0.001293 0.001293 0.15% MultiFab::contains_nan() 20 0.001211 0.001211 0.001211 0.14% MLCellLinOp::smooth() 1640 0.001173 0.001173 0.001173 0.14% MLCellLinOp::prepareForSolve() 11 0.001146 0.001146 0.001146 0.13% MLCellLinOp::compGrad() 11 0.0009215 0.0009215 0.0009215 0.11% MLMG::prepareForSolve() 11 0.0008791 0.0008791 0.0008791 0.10% FabArray::FillBoundary() 4023 0.0008634 0.0008634 0.0008634 0.10% FabArrayBase::getCPC() 1323 0.0007446 0.0007446 0.0007446 0.09% FabArrayBase::CPC::define() 454 0.0006724 0.0006724 0.0006724 0.08% Gravity::get_new_grav_vector() 11 0.0006398 0.0006398 0.0006398 0.07% FabArrayBase::getFB() 4023 0.0006157 0.0006157 0.0006157 0.07% Gravity::get_old_grav_vector() 10 0.0005453 0.0005453 0.0005453 0.06% Amr::InitAmr() 1 0.0004942 0.0004942 0.0004942 0.06% MLCellLinOp::apply() 1142 0.000482 0.000482 0.000482 0.06% MLMG::mgVcycle() 82 0.0003717 0.0003717 0.0003717 0.04% MLLinOp::defineGrids() 11 0.0003699 0.0003699 0.0003699 0.04% Amr::coarseTimeStep() 10 0.0003329 0.0003329 0.0003329 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003178 0.0003178 0.0003178 0.04% main() 1 0.0003028 0.0003028 0.0003028 0.04% MultiFab::max() 11 0.0002591 0.0002591 0.0002591 0.03% FabArray::ParallelCopy() 861 0.0002492 0.0002492 0.0002492 0.03% FillPatchIterator::Initialize 41 0.0002305 0.0002305 0.0002305 0.03% MLCellLinOp::correctionResidual() 492 0.0002216 0.0002216 0.0002216 0.03% MLCellLinOp::defineBC() 11 0.0002125 0.0002125 0.0002125 0.02% Amr::timeStep() 10 0.000192 0.000192 0.000192 0.02% Castro::subcycle_advance_ctu() 10 0.0001528 0.0001528 0.0001528 0.02% StateData::checkPoint() 12 0.0001336 0.0001336 0.0001336 0.02% Gravity::update_max_rhs() 11 0.0001213 0.0001213 0.0001213 0.01% MLMG:computeResOfCorrection() 410 0.000112 0.000112 0.000112 0.01% Gravity::solve_for_phi() 10 0.0001113 0.0001113 0.0001113 0.01% Castro::advance() 10 0.0001058 0.0001058 0.0001058 0.01% MLMG::mgVcycle_down::0 82 9.106e-05 9.106e-05 9.106e-05 0.01% MLMG::actualBottomSolve() 82 9.011e-05 9.011e-05 9.011e-05 0.01% MLMG::mgVcycle_down::2 82 8.802e-05 8.802e-05 8.802e-05 0.01% FabArrayBase::FB::FB() 56 8.764e-05 8.764e-05 8.764e-05 0.01% Castro::Castro() 1 8.711e-05 8.711e-05 8.711e-05 0.01% MLMG::mgVcycle_down::1 82 8.698e-05 8.698e-05 8.698e-05 0.01% Castro::finalize_advance() 10 8.105e-05 8.105e-05 8.105e-05 0.01% MLMG::mgVcycle_down::3 82 7.99e-05 7.99e-05 7.99e-05 0.01% Castro::expand_state() 10 7.983e-05 7.983e-05 7.983e-05 0.01% MLMG::mgVcycle_down::4 82 7.784e-05 7.784e-05 7.784e-05 0.01% Castro::clean_state() 62 7.66e-05 7.66e-05 7.66e-05 0.01% MLMG::solve() 11 7.627e-05 7.627e-05 7.627e-05 0.01% AmrLevel::checkPoint() 3 7.453e-05 7.453e-05 7.453e-05 0.01% Castro::initialize_advance() 10 7.158e-05 7.158e-05 7.158e-05 0.01% MLMG::mgVcycle_up::4 82 6.577e-05 6.577e-05 6.577e-05 0.01% MLMG::mgVcycle_up::0 82 5.858e-05 5.858e-05 5.858e-05 0.01% MLMG::mgVcycle_up::2 82 5.62e-05 5.62e-05 5.62e-05 0.01% MLMG::oneIter() 82 5.452e-05 5.452e-05 5.452e-05 0.01% MLMG::mgVcycle_up::1 82 5.426e-05 5.426e-05 5.426e-05 0.01% MLMG::mgVcycle_up::3 82 5.415e-05 5.415e-05 5.415e-05 0.01% Castro::initialize_do_advance() 10 5.173e-05 5.173e-05 5.173e-05 0.01% MLCellLinOp::solutionResidual() 93 5.163e-05 5.163e-05 5.163e-05 0.01% StateData::define() 4 5.092e-05 5.092e-05 5.092e-05 0.01% Castro::swap_state_time_levels() 10 4.007e-05 4.007e-05 4.007e-05 0.00% Castro::finalize_do_advance() 10 3.8e-05 3.8e-05 3.8e-05 0.00% Castro::enforce_consistent_e() 1 3.506e-05 3.506e-05 3.506e-05 0.00% MLMG::ResNormInf() 93 3.382e-05 3.382e-05 3.382e-05 0.00% MLMG::computeResidual() 82 3.166e-05 3.166e-05 3.166e-05 0.00% FillPatchSingleLevel 41 3.085e-05 3.085e-05 3.085e-05 0.00% MLMG::mgVcycle_bottom 82 3.031e-05 3.031e-05 3.031e-05 0.00% Amr::writeSmallPlotFile() 1 2.51e-05 2.51e-05 2.51e-05 0.00% Castro::construct_new_gravity() 10 2.473e-05 2.473e-05 2.473e-05 0.00% makeSFC 55 2.396e-05 2.396e-05 2.396e-05 0.00% Castro::initMFs() 1 2.382e-05 2.382e-05 2.382e-05 0.00% MLPoisson::define() 11 2.21e-05 2.21e-05 2.21e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.184e-05 2.184e-05 2.184e-05 0.00% Castro::construct_old_source() 50 1.977e-05 1.977e-05 1.977e-05 0.00% Amr::defBaseLevel() 1 1.963e-05 1.963e-05 1.963e-05 0.00% Amr::FinalizeInit() 1 1.931e-05 1.931e-05 1.931e-05 0.00% Castro::buildMetrics() 1 1.862e-05 1.862e-05 1.862e-05 0.00% Castro::construct_new_source() 50 1.783e-05 1.783e-05 1.783e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.737e-05 1.737e-05 1.737e-05 0.00% Castro::do_new_sources() 10 1.723e-05 1.723e-05 1.723e-05 0.00% Castro::do_old_sources() 10 1.609e-05 1.609e-05 1.609e-05 0.00% DistributionMapping::Distribute() 56 1.517e-05 1.517e-05 1.517e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.46e-05 1.46e-05 1.46e-05 0.00% MLPoisson::prepareForSolve() 11 1.44e-05 1.44e-05 1.44e-05 0.00% Castro::check_for_nan() 20 1.155e-05 1.155e-05 1.155e-05 0.00% MLLinOp::define() 11 1.112e-05 1.112e-05 1.112e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.008e-05 1.008e-05 1.008e-05 0.00% Castro::apply_source_to_state() 20 9.875e-06 9.875e-06 9.875e-06 0.00% Castro::post_timestep() 10 9.51e-06 9.51e-06 9.51e-06 0.00% Gravity::swapTimeLevels() 10 9.312e-06 9.312e-06 9.312e-06 0.00% Castro::construct_old_gravity() 10 8.807e-06 8.807e-06 8.807e-06 0.00% Amr::initSubcycle() 1 8.399e-06 8.399e-06 8.399e-06 0.00% Gravity::actual_multilevel_solve() 1 7.904e-06 7.904e-06 7.904e-06 0.00% MLMG::computeMLResidual() 11 7.3e-06 7.3e-06 7.3e-06 0.00% Castro::computeNewDt() 9 7.233e-06 7.233e-06 7.233e-06 0.00% MLMG::getGradSolution() 11 5.733e-06 5.733e-06 5.733e-06 0.00% Castro::create_source_corrector() 10 5.513e-06 5.513e-06 5.513e-06 0.00% Amr::InitializeInit() 1 4.567e-06 4.567e-06 4.567e-06 0.00% Gravity::set_mass_offset() 11 4.142e-06 4.142e-06 4.142e-06 0.00% AmrLevel::checkPointPost() 3 3.831e-06 3.831e-06 3.831e-06 0.00% Castro::post_init() 1 3.747e-06 3.747e-06 3.747e-06 0.00% Castro::retry_advance_ctu() 10 3.717e-06 3.717e-06 3.717e-06 0.00% MLMG::MLRhsNormInf() 11 3.705e-06 3.705e-06 3.705e-06 0.00% MLMG::MLResNormInf() 11 3.588e-06 3.588e-06 3.588e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.971e-06 2.971e-06 2.971e-06 0.00% Castro::FluxRegCrseInit 10 2.91e-06 2.91e-06 2.91e-06 0.00% Castro::computeInitialDt() 2 2.884e-06 2.884e-06 2.884e-06 0.00% Amr::init() 1 2.674e-06 2.674e-06 2.674e-06 0.00% Castro::FluxRegFineAdd() 10 2.116e-06 2.116e-06 2.116e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.908e-06 1.908e-06 1.908e-06 0.00% AmrLevel::checkPointPre() 3 1.742e-06 1.742e-06 1.742e-06 0.00% Castro::post_regrid() 1 1.194e-06 1.194e-06 1.194e-06 0.00% Amr::initialInit() 1 1.053e-06 1.053e-06 1.053e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8611 0.8611 0.8611 100.00% Amr::coarseTimeStep() 10 0.6852 0.6852 0.6852 79.57% Amr::timeStep() 10 0.5668 0.5668 0.5668 65.83% Castro::advance() 10 0.5595 0.5595 0.5595 64.98% Castro::subcycle_advance_ctu() 10 0.5469 0.5469 0.5469 63.51% Castro::do_advance_ctu() 10 0.5467 0.5467 0.5467 63.49% Gravity::solve_phi_with_mlmg() 11 0.2846 0.2846 0.2846 33.06% Gravity::actual_solve_with_mlmg() 11 0.2777 0.2777 0.2777 32.25% Castro::construct_new_gravity() 10 0.2602 0.2602 0.2602 30.22% MLMG::solve() 11 0.2569 0.2569 0.2569 29.83% Gravity::solve_for_phi() 10 0.2441 0.2441 0.2441 28.35% MLMG::oneIter() 82 0.2425 0.2425 0.2425 28.16% MLMG::mgVcycle() 82 0.2388 0.2388 0.2388 27.74% VisMF::Write(FabArray) 11 0.2354 0.2354 0.2354 27.33% Castro::construct_ctu_hydro_source() 10 0.2119 0.2119 0.2119 24.61% Amr::checkPoint() 3 0.1744 0.1744 0.1744 20.25% AmrLevel::checkPoint() 3 0.1704 0.1704 0.1704 19.79% StateData::checkPoint() 12 0.1703 0.1703 0.1703 19.78% Amr::init() 1 0.1414 0.1414 0.1414 16.42% MLCellLinOp::smooth() 1640 0.1175 0.1175 0.1175 13.64% MLCellLinOp::applyBC() 4433 0.11 0.11 0.11 12.77% MLMG::mgVcycle_bottom 82 0.07378 0.07378 0.07378 8.57% MLMG::actualBottomSolve() 82 0.07375 0.07375 0.07375 8.56% MLCGSolver::bicgstab 82 0.07304 0.07304 0.07304 8.48% Amr::writePlotFile() 2 0.06812 0.06812 0.06812 7.91% Amr::initialInit() 1 0.04829 0.04829 0.04829 5.61% Castro::clean_state() 62 0.04439 0.04439 0.04439 5.16% Amr::FinalizeInit() 1 0.04412 0.04412 0.04412 5.12% Castro::post_init() 1 0.04283 0.04283 0.04283 4.97% FillPatchIterator::Initialize 41 0.04274 0.04274 0.04274 4.96% FillPatchSingleLevel 41 0.04122 0.04122 0.04122 4.79% Gravity::multilevel_solve_for_new_phi() 1 0.04096 0.04096 0.04096 4.76% Gravity::actual_multilevel_solve() 1 0.04094 0.04094 0.04094 4.75% StateDataPhysBCFunct::() 41 0.03713 0.03713 0.03713 4.31% MLCellLinOp::apply() 1142 0.0365 0.0365 0.0365 4.24% MLMG::mgVcycle_down::0 82 0.03404 0.03404 0.03404 3.95% FabArray::FillBoundary() 4023 0.03298 0.03298 0.03298 3.83% MLPoisson::Fsmooth() 3280 0.03234 0.03234 0.03234 3.76% FillBoundary_nowait() 4023 0.03212 0.03212 0.03212 3.73% MLMG::mgVcycle_up::0 82 0.02585 0.02585 0.02585 3.00% StateData::FillBoundary(geom) 328 0.02519 0.02519 0.02519 2.92% MLCellLinOp::correctionResidual() 492 0.02238 0.02238 0.02238 2.60% amrex::Dot() 1114 0.02062 0.02062 0.02062 2.39% Castro::computeTemp() 63 0.02023 0.02023 0.02023 2.35% MLMG:computeResOfCorrection() 410 0.01973 0.01973 0.01973 2.29% Castro::initialize_do_advance() 10 0.01852 0.01852 0.01852 2.15% Gravity::get_new_grav_vector() 11 0.01766 0.01766 0.01766 2.05% MLPoisson::define() 11 0.01652 0.01652 0.01652 1.92% MLMG::mgVcycle_down::1 82 0.01585 0.01585 0.01585 1.84% Castro::normalize_species() 62 0.01584 0.01584 0.01584 1.84% Castro::construct_old_gravity() 10 0.01517 0.01517 0.01517 1.76% Gravity::get_old_grav_vector() 10 0.01516 0.01516 0.01516 1.76% amrex::Copy() 1029 0.01487 0.01487 0.01487 1.73% MLMG::mgVcycle_down::2 82 0.01485 0.01485 0.01485 1.73% FabArray::ParallelCopy() 861 0.01448 0.01448 0.01448 1.68% MLMG::mgVcycle_down::3 82 0.01447 0.01447 0.01447 1.68% FabArray::norminf() 743 0.01435 0.01435 0.01435 1.67% MLMG::mgVcycle_down::4 82 0.01424 0.01424 0.01424 1.65% FabArray::ParallelCopy_nowait() 861 0.01423 0.01423 0.01423 1.65% FabArray::setVal() 1144 0.01315 0.01315 0.01315 1.53% Castro::do_new_sources() 10 0.01297 0.01297 0.01297 1.51% MLCGSolver::ParallelAllReduce 1514 0.0124 0.0124 0.0124 1.44% MLMG::addInterpCorrection() 410 0.01199 0.01199 0.01199 1.39% Castro::initialize_advance() 10 0.0119 0.0119 0.0119 1.38% MLMG::mgVcycle_up::4 82 0.01159 0.01159 0.01159 1.35% MLMG::mgVcycle_up::1 82 0.01152 0.01152 0.01152 1.34% Castro::expand_state() 10 0.01148 0.01148 0.01148 1.33% MLMG::mgVcycle_up::2 82 0.01126 0.01126 0.01126 1.31% amrex::average_down 410 0.01117 0.01117 0.01117 1.30% MLCellLinOp::defineAuxData() 11 0.01108 0.01108 0.01108 1.29% MLMG::mgVcycle_up::3 82 0.01102 0.01102 0.01102 1.28% MLPoisson::Fapply() 1142 0.01046 0.01046 0.01046 1.21% Castro::do_old_sources() 10 0.009445 0.009445 0.009445 1.10% FabArray::Saxpy() 813 0.008152 0.008152 0.008152 0.95% FabArray::Xpay() 821 0.008116 0.008116 0.008116 0.94% Castro::enforce_min_density() 62 0.007732 0.007732 0.007732 0.90% MLCellLinOp::solutionResidual() 93 0.007222 0.007222 0.007222 0.84% Castro::post_timestep() 10 0.007112 0.007112 0.007112 0.83% Gravity::fill_multipole_BCs() 11 0.006717 0.006717 0.006717 0.78% Castro::reset_internal_energy(MultiFab) 63 0.006239 0.006239 0.006239 0.72% MLMG::computeResidual() 82 0.006232 0.006232 0.006232 0.72% MLCellLinOp::defineBC() 11 0.004988 0.004988 0.004988 0.58% BndryData::define() 11 0.004775 0.004775 0.004775 0.55% MLMG::prepareForSolve() 11 0.004732 0.004732 0.004732 0.55% FabArray::LinComb() 557 0.004537 0.004537 0.004537 0.53% amrex::Add() 164 0.004372 0.004372 0.004372 0.51% Amr::InitializeInit() 1 0.004165 0.004165 0.004165 0.48% Amr::defBaseLevel() 1 0.004161 0.004161 0.004161 0.48% Castro::estTimeStep() 21 0.00392 0.00392 0.00392 0.46% Castro::initData() 1 0.003637 0.003637 0.003637 0.42% Castro::construct_new_source() 50 0.003261 0.003261 0.003261 0.38% Castro::construct_new_gravity_source() 10 0.003244 0.003244 0.003244 0.38% Castro::construct_old_source() 50 0.002287 0.002287 0.002287 0.27% Castro::construct_old_gravity_source() 10 0.002267 0.002267 0.002267 0.26% Castro::reset_internal_energy(Fab) 504 0.002164 0.002164 0.002164 0.25% MLMG::ResNormInf() 93 0.002086 0.002086 0.002086 0.24% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001701 0.001701 0.001701 0.20% Castro::computeNewDt() 9 0.001551 0.001551 0.001551 0.18% FabArrayBase::getCPC() 1323 0.001417 0.001417 0.001417 0.16% MLMG::getGradSolution() 11 0.001404 0.001404 0.001404 0.16% MLCellLinOp::setLevelBC() 11 0.001402 0.001402 0.001402 0.16% MLCellLinOp::compGrad() 11 0.001398 0.001398 0.001398 0.16% FabArray::mult() 43 0.001317 0.001317 0.001317 0.15% Castro::enforce_speed_limit() 62 0.001317 0.001317 0.001317 0.15% FabArray::setDomainBndry() 41 0.001293 0.001293 0.001293 0.15% Castro::check_for_nan() 20 0.001222 0.001222 0.001222 0.14% MultiFab::contains_nan() 20 0.001211 0.001211 0.001211 0.14% MLPoisson::prepareForSolve() 11 0.00116 0.00116 0.00116 0.13% MLCellLinOp::prepareForSolve() 11 0.001146 0.001146 0.001146 0.13% Castro::post_regrid() 1 0.001117 0.001117 0.001117 0.13% MLMG::computeMLResidual() 11 0.001029 0.001029 0.001029 0.12% Castro::computeInitialDt() 2 0.0008814 0.0008814 0.0008814 0.10% Gravity::update_max_rhs() 11 0.0008315 0.0008315 0.0008315 0.10% FabArrayBase::getFB() 4023 0.0007033 0.0007033 0.0007033 0.08% FabArrayBase::CPC::define() 454 0.0006724 0.0006724 0.0006724 0.08% Castro::finalize_advance() 10 0.0006036 0.0006036 0.0006036 0.07% Amr::InitAmr() 1 0.0005026 0.0005026 0.0005026 0.06% Gravity::swapTimeLevels() 10 0.0004415 0.0004415 0.0004415 0.05% Castro::Castro() 1 0.000438 0.000438 0.000438 0.05% MLLinOp::define() 11 0.0004349 0.0004349 0.0004349 0.05% MLLinOp::defineGrids() 11 0.0004238 0.0004238 0.0004238 0.05% MLMG::MLResNormInf() 11 0.0002859 0.0002859 0.0002859 0.03% MultiFab::max() 11 0.0002591 0.0002591 0.0002591 0.03% MLMG::MLRhsNormInf() 11 0.000222 0.000222 0.000222 0.03% Castro::buildMetrics() 1 0.000165 0.000165 0.000165 0.02% FabArrayBase::FB::FB() 56 8.764e-05 8.764e-05 8.764e-05 0.01% AmrLevel::AmrLevel(dm) 1 6.1e-05 6.1e-05 6.1e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.2e-05 5.2e-05 5.2e-05 0.01% StateData::define() 4 5.092e-05 5.092e-05 5.092e-05 0.01% Castro::swap_state_time_levels() 10 4.007e-05 4.007e-05 4.007e-05 0.00% Castro::finalize_do_advance() 10 3.8e-05 3.8e-05 3.8e-05 0.00% makeSFC 55 3.74e-05 3.74e-05 3.74e-05 0.00% Castro::enforce_consistent_e() 1 3.506e-05 3.506e-05 3.506e-05 0.00% Amr::writeSmallPlotFile() 1 2.51e-05 2.51e-05 2.51e-05 0.00% Castro::initMFs() 1 2.382e-05 2.382e-05 2.382e-05 0.00% DistributionMapping::Distribute() 56 1.517e-05 1.517e-05 1.517e-05 0.00% Amr::initSubcycle() 1 8.399e-06 8.399e-06 8.399e-06 0.00% Castro::create_source_corrector() 10 5.513e-06 5.513e-06 5.513e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.709e-06 4.709e-06 4.709e-06 0.00% Gravity::set_mass_offset() 11 4.142e-06 4.142e-06 4.142e-06 0.00% AmrLevel::checkPointPost() 3 3.831e-06 3.831e-06 3.831e-06 0.00% Castro::retry_advance_ctu() 10 3.717e-06 3.717e-06 3.717e-06 0.00% Castro::FluxRegCrseInit 10 2.91e-06 2.91e-06 2.91e-06 0.00% Castro::FluxRegFineAdd() 10 2.116e-06 2.116e-06 2.116e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.908e-06 1.908e-06 1.908e-06 0.00% AmrLevel::checkPointPre() 3 1.742e-06 1.742e-06 1.742e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 101 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 114 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1054 KiB 39 MiB Castro::initialize_do_advance() 80 80 24 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2222 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7612 KiB 14 MiB MLMG::prepareForSolve() 660 660 3670 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 212 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 180 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7523 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1851 B 2048 KiB Gravity::solve_for_phi() 80 80 579 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 97 KiB 2048 KiB BndryData::define() 1056 1056 335 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 213 KiB 671 KiB Castro::estTimeStep() 21 21 2203 B 480 KiB VisMF::Write(FabArray) 656 656 3522 B 320 KiB Castro::normalize_species() 62 62 5991 B 320 KiB amrex::average_down 1067 1067 1291 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1199 B 257 KiB amrex::Dot() 1360 1360 3531 B 160 KiB FabArray::norminf() 907 907 2463 B 160 KiB Castro::do_advance_ctu() 10 10 517 B 160 KiB MultiFab::max() 11 11 46 B 160 KiB Castro::initData() 1 1 28 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 28 B 20 KiB MLPoisson::Fsmooth() 132 132 3588 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 319 B 9648 B MLCellLinOp::applyBC() 8866 8866 230 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3853 B 6144 B StateData::FillBoundary(geom) 1992 1992 49 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 123 B 1472 B MLCellLinOp::defineBC() 66 66 377 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 608 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 53 KiB 8192 KiB VisMF::Write(FabArray) 744 744 641 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3588 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 319 B 9648 B MLCellLinOp::applyBC() 4433 4433 228 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3853 B 6144 B Gravity::get_new_grav_vector() 3 3 2900 B 3072 B StateData::FillBoundary(geom) 1992 1992 49 B 2880 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 305 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 907 907 18 B 272 B Castro::estTimeStep() 21 21 0 B 32 B MultiFab::contains_nan() 20 20 0 B 16 B MultiFab::max() 11 11 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B Castro::do_advance_ctu() 10 10 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-11-g0f4f9877c81e) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-11-g0f4f9877c81e) initialized Starting run at 09:08:36 UTC on 2023-03-16. Successfully read inputs file ... Castro git describe: 23.03-8-g17ee5df0a AMReX git describe: 23.03-11-g0f4f9877c Microphysics git describe: 23.03-13-ga480b6a9 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.485249378 Restart time = 0.04798981 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050540388 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047326471 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047276951 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059836971 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062464483 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03321426 seconds Ending run at 09:08:36 UTC on 2023-03-16. Run time = 0.349667432 Run time without initialization = 0.30105295 Average number of zones advanced per microsecond: 4.354 Average number of zones advanced per microsecond per rank: 4.354 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3497 ... 0.3497 ... 0.3497 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0878 0.0878 0.0878 25.10% VisMF::Read() 3 0.04161 0.04161 0.04161 11.90% MLCellLinOp::applyBC() 1946 0.03303 0.03303 0.03303 9.45% VisMF::Write(FabArray) 1 0.03167 0.03167 0.03167 9.05% MLPoisson::Fsmooth() 1440 0.01392 0.01392 0.01392 3.98% FillBoundary_nowait() 1766 0.01277 0.01277 0.01277 3.65% StateData::FillBoundary(geom) 160 0.01218 0.01218 0.01218 3.48% amrex::Dot() 484 0.008842 0.008842 0.008842 2.53% amrex::Copy() 463 0.006955 0.006955 0.006955 1.99% FabArray::setVal() 537 0.006248 0.006248 0.006248 1.79% FabArray::norminf() 326 0.006209 0.006209 0.006209 1.78% Castro::normalize_species() 30 0.006181 0.006181 0.006181 1.77% FabArray::ParallelCopy_nowait() 380 0.006094 0.006094 0.006094 1.74% Castro::computeTemp() 30 0.005872 0.005872 0.005872 1.68% StateDataPhysBCFunct::() 20 0.005708 0.005708 0.005708 1.63% Castro::enforce_min_density() 30 0.005469 0.005469 0.005469 1.56% MLCellLinOp::defineAuxData() 6 0.005264 0.005264 0.005264 1.51% MLPoisson::Fapply() 500 0.004457 0.004457 0.004457 1.27% FabArray::Saxpy() 355 0.003641 0.003641 0.003641 1.04% FabArray::Xpay() 361 0.003501 0.003501 0.003501 1.00% Gravity::fill_multipole_BCs() 6 0.003145 0.003145 0.003145 0.90% Castro::estTimeStep() 10 0.003026 0.003026 0.003026 0.87% MLMG::addInterpCorrection() 180 0.002954 0.002954 0.002954 0.84% Amr::restart() 1 0.002642 0.002642 0.002642 0.76% amrex::average_down 180 0.002619 0.002619 0.002619 0.75% BndryData::define() 6 0.002061 0.002061 0.002061 0.59% FabArray::LinComb() 242 0.001946 0.001946 0.001946 0.56% amrex::Add() 72 0.001849 0.001849 0.001849 0.53% Castro::reset_internal_energy(MultiFab) 30 0.001814 0.001814 0.001814 0.52% Castro::construct_new_gravity_source() 5 0.001616 0.001616 0.001616 0.46% Castro::do_advance_ctu() 5 0.001581 0.001581 0.001581 0.45% Amr::writePlotFile() 1 0.001381 0.001381 0.001381 0.39% Castro::construct_old_gravity_source() 5 0.001357 0.001357 0.001357 0.39% MLCGSolver::bicgstab 36 0.0009749 0.0009749 0.0009749 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009061 0.0009061 0.0009061 0.26% Castro::reset_internal_energy(Fab) 240 0.0007848 0.0007848 0.0007848 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007709 0.0007709 0.0007709 0.22% MLCellLinOp::setLevelBC() 6 0.0007647 0.0007647 0.0007647 0.22% FabArray::mult() 22 0.0006495 0.0006495 0.0006495 0.19% FabArray::setDomainBndry() 20 0.0006433 0.0006433 0.0006433 0.18% MLCellLinOp::prepareForSolve() 6 0.0006149 0.0006149 0.0006149 0.18% MultiFab::contains_nan() 10 0.0005932 0.0005932 0.0005932 0.17% MLCellLinOp::smooth() 720 0.0005192 0.0005192 0.0005192 0.15% MLCellLinOp::compGrad() 6 0.0004914 0.0004914 0.0004914 0.14% MLMG::prepareForSolve() 6 0.000489 0.000489 0.000489 0.14% Amr::InitAmr() 1 0.0004178 0.0004178 0.0004178 0.12% Castro::enforce_speed_limit() 30 0.0004149 0.0004149 0.0004149 0.12% FabArrayBase::CPC::define() 244 0.000413 0.000413 0.000413 0.12% FabArray::FillBoundary() 1766 0.0003607 0.0003607 0.0003607 0.10% FabArrayBase::getCPC() 632 0.0003463 0.0003463 0.0003463 0.10% Gravity::get_old_grav_vector() 5 0.0003029 0.0003029 0.0003029 0.09% main() 1 0.0003017 0.0003017 0.0003017 0.09% Gravity::get_new_grav_vector() 5 0.0002716 0.0002716 0.0002716 0.08% FabArrayBase::getFB() 1766 0.0002504 0.0002504 0.0002504 0.07% MLCellLinOp::apply() 500 0.0002034 0.0002034 0.0002034 0.06% MLMG::mgVcycle() 36 0.0001684 0.0001684 0.0001684 0.05% Amr::coarseTimeStep() 5 0.0001554 0.0001554 0.0001554 0.04% MultiFab::max() 6 0.0001352 0.0001352 0.0001352 0.04% MLCGSolver::ParallelAllReduce 659 0.0001322 0.0001322 0.0001322 0.04% MLLinOp::defineGrids() 6 0.00013 0.00013 0.00013 0.04% MLCellLinOp::defineBC() 6 0.0001175 0.0001175 0.0001175 0.03% FillPatchIterator::Initialize 20 0.0001169 0.0001169 0.0001169 0.03% FabArray::ParallelCopy() 380 0.0001115 0.0001115 0.0001115 0.03% MLCellLinOp::correctionResidual() 216 9.345e-05 9.345e-05 9.345e-05 0.03% Amr::timeStep() 5 8.137e-05 8.137e-05 8.137e-05 0.02% AmrLevel::restart() 1 8.091e-05 8.091e-05 8.091e-05 0.02% Castro::subcycle_advance_ctu() 5 8.067e-05 8.067e-05 8.067e-05 0.02% StateData::restartDoit() 4 7.012e-05 7.012e-05 7.012e-05 0.02% Gravity::solve_for_phi() 5 6.665e-05 6.665e-05 6.665e-05 0.02% Gravity::update_max_rhs() 6 6.016e-05 6.016e-05 6.016e-05 0.02% FabArrayBase::FB::FB() 26 5.844e-05 5.844e-05 5.844e-05 0.02% Castro::advance() 5 5.498e-05 5.498e-05 5.498e-05 0.02% MLMG:computeResOfCorrection() 180 5.25e-05 5.25e-05 5.25e-05 0.02% Castro::create_source_corrector() 5 4.449e-05 4.449e-05 4.449e-05 0.01% MLMG::mgVcycle_down::0 36 4.158e-05 4.158e-05 4.158e-05 0.01% Castro::expand_state() 5 3.994e-05 3.994e-05 3.994e-05 0.01% MLMG::mgVcycle_down::2 36 3.992e-05 3.992e-05 3.992e-05 0.01% MLMG::mgVcycle_down::1 36 3.919e-05 3.919e-05 3.919e-05 0.01% MLMG::actualBottomSolve() 36 3.912e-05 3.912e-05 3.912e-05 0.01% Castro::clean_state() 30 3.783e-05 3.783e-05 3.783e-05 0.01% Castro::initialize_advance() 5 3.609e-05 3.609e-05 3.609e-05 0.01% MLMG::mgVcycle_down::4 36 3.564e-05 3.564e-05 3.564e-05 0.01% MLMG::solve() 6 3.539e-05 3.539e-05 3.539e-05 0.01% MLMG::mgVcycle_down::3 36 3.47e-05 3.47e-05 3.47e-05 0.01% MLMG::mgVcycle_up::4 36 3.046e-05 3.046e-05 3.046e-05 0.01% Castro::finalize_advance() 5 2.826e-05 2.826e-05 2.826e-05 0.01% Castro::initMFs() 1 2.761e-05 2.761e-05 2.761e-05 0.01% Castro::buildMetrics() 1 2.757e-05 2.757e-05 2.757e-05 0.01% Castro::initialize_do_advance() 5 2.593e-05 2.593e-05 2.593e-05 0.01% MLMG::mgVcycle_up::0 36 2.531e-05 2.531e-05 2.531e-05 0.01% Amr::writeSmallPlotFile() 1 2.491e-05 2.491e-05 2.491e-05 0.01% Castro::swap_state_time_levels() 5 2.41e-05 2.41e-05 2.41e-05 0.01% MLMG::mgVcycle_up::3 36 2.37e-05 2.37e-05 2.37e-05 0.01% MLMG::oneIter() 36 2.334e-05 2.334e-05 2.334e-05 0.01% MLCellLinOp::solutionResidual() 42 2.298e-05 2.298e-05 2.298e-05 0.01% MLMG::mgVcycle_up::2 36 2.288e-05 2.288e-05 2.288e-05 0.01% MLMG::mgVcycle_up::1 36 2.279e-05 2.279e-05 2.279e-05 0.01% Castro::post_restart() 1 1.976e-05 1.976e-05 1.976e-05 0.01% Castro::finalize_do_advance() 5 1.944e-05 1.944e-05 1.944e-05 0.01% Gravity::solve_phi_with_mlmg() 6 1.792e-05 1.792e-05 1.792e-05 0.01% MLMG::ResNormInf() 42 1.724e-05 1.724e-05 1.724e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.533e-05 1.533e-05 1.533e-05 0.00% MLPoisson::define() 6 1.456e-05 1.456e-05 1.456e-05 0.00% MLMG::mgVcycle_bottom 36 1.435e-05 1.435e-05 1.435e-05 0.00% FillPatchSingleLevel 20 1.421e-05 1.421e-05 1.421e-05 0.00% MLMG::computeResidual() 36 1.382e-05 1.382e-05 1.382e-05 0.00% makeSFC 30 1.327e-05 1.327e-05 1.327e-05 0.00% Castro::construct_new_gravity() 5 1.29e-05 1.29e-05 1.29e-05 0.00% Castro::construct_old_source() 25 1.092e-05 1.092e-05 1.092e-05 0.00% DistributionMapping::Distribute() 31 1.057e-05 1.057e-05 1.057e-05 0.00% MLPoisson::prepareForSolve() 6 9.568e-06 9.568e-06 9.568e-06 0.00% Castro::construct_new_source() 25 9.525e-06 9.525e-06 9.525e-06 0.00% Amr::initSubcycle() 1 9.117e-06 9.117e-06 9.117e-06 0.00% Castro::do_new_sources() 5 8.739e-06 8.739e-06 8.739e-06 0.00% Castro::do_old_sources() 5 8.169e-06 8.169e-06 8.169e-06 0.00% Gravity::actual_multilevel_solve() 1 7.838e-06 7.838e-06 7.838e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.792e-06 6.792e-06 6.792e-06 0.00% Castro::construct_old_gravity() 5 5.719e-06 5.719e-06 5.719e-06 0.00% Castro::check_for_nan() 10 5.578e-06 5.578e-06 5.578e-06 0.00% MLLinOp::define() 6 5.266e-06 5.266e-06 5.266e-06 0.00% Castro::apply_source_to_state() 10 5.138e-06 5.138e-06 5.138e-06 0.00% Castro::post_timestep() 5 5.071e-06 5.071e-06 5.071e-06 0.00% Gravity::swapTimeLevels() 5 4.06e-06 4.06e-06 4.06e-06 0.00% Castro::computeNewDt() 5 3.635e-06 3.635e-06 3.635e-06 0.00% MLMG::computeMLResidual() 6 3.55e-06 3.55e-06 3.55e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.367e-06 3.367e-06 3.367e-06 0.00% MLMG::getGradSolution() 6 3.171e-06 3.171e-06 3.171e-06 0.00% MLMG::MLResNormInf() 6 2.326e-06 2.326e-06 2.326e-06 0.00% Gravity::set_mass_offset() 6 1.99e-06 1.99e-06 1.99e-06 0.00% MLMG::MLRhsNormInf() 6 1.961e-06 1.961e-06 1.961e-06 0.00% Castro::retry_advance_ctu() 5 1.829e-06 1.829e-06 1.829e-06 0.00% Castro::FluxRegCrseInit 5 1.477e-06 1.477e-06 1.477e-06 0.00% Amr::init() 1 1.274e-06 1.274e-06 1.274e-06 0.00% Castro::FluxRegFineAdd() 5 1.218e-06 1.218e-06 1.218e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.108e-06 1.108e-06 1.108e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3497 0.3497 0.3497 99.99% Amr::coarseTimeStep() 5 0.2676 0.2676 0.2676 76.52% Amr::timeStep() 5 0.2652 0.2652 0.2652 75.82% Castro::advance() 5 0.2621 0.2621 0.2621 74.96% Castro::subcycle_advance_ctu() 5 0.2561 0.2561 0.2561 73.23% Castro::do_advance_ctu() 5 0.256 0.256 0.256 73.21% Castro::construct_new_gravity() 5 0.1297 0.1297 0.1297 37.09% Gravity::solve_phi_with_mlmg() 6 0.1249 0.1249 0.1249 35.71% Gravity::solve_for_phi() 5 0.1218 0.1218 0.1218 34.84% Gravity::actual_solve_with_mlmg() 6 0.1215 0.1215 0.1215 34.74% MLMG::solve() 6 0.1103 0.1103 0.1103 31.54% MLMG::oneIter() 36 0.1034 0.1034 0.1034 29.57% MLMG::mgVcycle() 36 0.1019 0.1019 0.1019 29.13% Castro::construct_ctu_hydro_source() 5 0.08777 0.08777 0.08777 25.10% MLCellLinOp::smooth() 720 0.04967 0.04967 0.04967 14.20% Amr::init() 1 0.04805 0.04805 0.04805 13.74% Amr::restart() 1 0.04805 0.04805 0.04805 13.74% MLCellLinOp::applyBC() 1946 0.04647 0.04647 0.04647 13.29% AmrLevel::restart() 1 0.04182 0.04182 0.04182 11.96% StateData::restartDoit() 4 0.04173 0.04173 0.04173 11.93% VisMF::Read() 3 0.04161 0.04161 0.04161 11.90% Amr::writePlotFile() 1 0.0333 0.0333 0.0333 9.52% MLMG::mgVcycle_bottom 36 0.0317 0.0317 0.0317 9.06% MLMG::actualBottomSolve() 36 0.03168 0.03168 0.03168 9.06% VisMF::Write(FabArray) 1 0.03167 0.03167 0.03167 9.05% MLCGSolver::bicgstab 36 0.03138 0.03138 0.03138 8.97% FillPatchIterator::Initialize 20 0.02069 0.02069 0.02069 5.92% Castro::clean_state() 30 0.02057 0.02057 0.02057 5.88% FillPatchSingleLevel 20 0.01993 0.01993 0.01993 5.70% StateDataPhysBCFunct::() 20 0.01788 0.01788 0.01788 5.11% MLCellLinOp::apply() 500 0.01564 0.01564 0.01564 4.47% MLMG::mgVcycle_down::0 36 0.01427 0.01427 0.01427 4.08% MLPoisson::Fsmooth() 1440 0.01392 0.01392 0.01392 3.98% FabArray::FillBoundary() 1766 0.01344 0.01344 0.01344 3.84% FillBoundary_nowait() 1766 0.01308 0.01308 0.01308 3.74% StateData::FillBoundary(geom) 160 0.01218 0.01218 0.01218 3.48% MLMG::mgVcycle_up::0 36 0.01077 0.01077 0.01077 3.08% Castro::initialize_do_advance() 5 0.009565 0.009565 0.009565 2.74% MLCellLinOp::correctionResidual() 216 0.0095 0.0095 0.0095 2.72% MLPoisson::define() 6 0.008909 0.008909 0.008909 2.55% amrex::Dot() 484 0.008842 0.008842 0.008842 2.53% Castro::computeTemp() 30 0.00847 0.00847 0.00847 2.42% MLMG:computeResOfCorrection() 180 0.008343 0.008343 0.008343 2.39% Gravity::get_new_grav_vector() 5 0.007736 0.007736 0.007736 2.21% Castro::construct_old_gravity() 5 0.007533 0.007533 0.007533 2.15% Gravity::get_old_grav_vector() 5 0.007528 0.007528 0.007528 2.15% Castro::do_new_sources() 5 0.007405 0.007405 0.007405 2.12% amrex::Copy() 463 0.006955 0.006955 0.006955 1.99% MLMG::mgVcycle_down::1 36 0.006811 0.006811 0.006811 1.95% FabArray::ParallelCopy() 380 0.006584 0.006584 0.006584 1.88% FabArray::ParallelCopy_nowait() 380 0.006473 0.006473 0.006473 1.85% MLMG::mgVcycle_down::2 36 0.006344 0.006344 0.006344 1.81% FabArray::setVal() 537 0.006248 0.006248 0.006248 1.79% FabArray::norminf() 326 0.006209 0.006209 0.006209 1.78% Castro::normalize_species() 30 0.006181 0.006181 0.006181 1.77% MLMG::mgVcycle_down::3 36 0.006175 0.006175 0.006175 1.77% MLMG::mgVcycle_down::4 36 0.006115 0.006115 0.006115 1.75% Castro::expand_state() 5 0.006094 0.006094 0.006094 1.74% MLCellLinOp::defineAuxData() 6 0.005993 0.005993 0.005993 1.71% Castro::initialize_advance() 5 0.005678 0.005678 0.005678 1.62% Castro::enforce_min_density() 30 0.005469 0.005469 0.005469 1.56% MLCGSolver::ParallelAllReduce 659 0.00533 0.00533 0.00533 1.52% MLMG::addInterpCorrection() 180 0.005221 0.005221 0.005221 1.49% MLMG::mgVcycle_up::4 36 0.004992 0.004992 0.004992 1.43% MLMG::mgVcycle_up::1 36 0.004957 0.004957 0.004957 1.42% amrex::average_down 180 0.004902 0.004902 0.004902 1.40% MLMG::mgVcycle_up::2 36 0.004838 0.004838 0.004838 1.38% MLMG::mgVcycle_up::3 36 0.004732 0.004732 0.004732 1.35% MLPoisson::Fapply() 500 0.004457 0.004457 0.004457 1.27% Castro::do_old_sources() 5 0.00443 0.00443 0.00443 1.27% FabArray::Saxpy() 355 0.003641 0.003641 0.003641 1.04% FabArray::Xpay() 361 0.003501 0.003501 0.003501 1.00% Castro::post_restart() 1 0.003408 0.003408 0.003408 0.97% Gravity::multilevel_solve_for_new_phi() 1 0.0033 0.0033 0.0033 0.94% Gravity::actual_multilevel_solve() 1 0.003284 0.003284 0.003284 0.94% Gravity::fill_multipole_BCs() 6 0.00328 0.00328 0.00328 0.94% MLCellLinOp::solutionResidual() 42 0.003224 0.003224 0.003224 0.92% Castro::estTimeStep() 10 0.003026 0.003026 0.003026 0.87% Castro::post_timestep() 5 0.002948 0.002948 0.002948 0.84% MLCellLinOp::defineBC() 6 0.002737 0.002737 0.002737 0.78% MLMG::computeResidual() 36 0.002671 0.002671 0.002671 0.76% BndryData::define() 6 0.002619 0.002619 0.002619 0.75% Castro::reset_internal_energy(MultiFab) 30 0.002599 0.002599 0.002599 0.74% MLMG::prepareForSolve() 6 0.002558 0.002558 0.002558 0.73% Castro::computeNewDt() 5 0.002273 0.002273 0.002273 0.65% FabArray::LinComb() 242 0.001946 0.001946 0.001946 0.56% amrex::Add() 72 0.001849 0.001849 0.001849 0.53% Castro::construct_new_source() 25 0.001626 0.001626 0.001626 0.46% Castro::construct_new_gravity_source() 5 0.001616 0.001616 0.001616 0.46% Castro::construct_old_source() 25 0.001368 0.001368 0.001368 0.39% Castro::construct_old_gravity_source() 5 0.001357 0.001357 0.001357 0.39% Castro::apply_source_to_state() 10 0.0009182 0.0009182 0.0009182 0.26% MLMG::ResNormInf() 42 0.000917 0.000917 0.000917 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009061 0.0009061 0.0009061 0.26% Castro::reset_internal_energy(Fab) 240 0.0007848 0.0007848 0.0007848 0.22% MLCellLinOp::setLevelBC() 6 0.0007647 0.0007647 0.0007647 0.22% FabArrayBase::getCPC() 632 0.0007593 0.0007593 0.0007593 0.22% MLMG::getGradSolution() 6 0.0007558 0.0007558 0.0007558 0.22% MLCellLinOp::compGrad() 6 0.0007526 0.0007526 0.0007526 0.22% FabArray::mult() 22 0.0006495 0.0006495 0.0006495 0.19% FabArray::setDomainBndry() 20 0.0006433 0.0006433 0.0006433 0.18% MLPoisson::prepareForSolve() 6 0.0006245 0.0006245 0.0006245 0.18% MLCellLinOp::prepareForSolve() 6 0.0006149 0.0006149 0.0006149 0.18% Castro::check_for_nan() 10 0.0005988 0.0005988 0.0005988 0.17% MultiFab::contains_nan() 10 0.0005932 0.0005932 0.0005932 0.17% MLMG::computeMLResidual() 6 0.0005699 0.0005699 0.0005699 0.16% Gravity::update_max_rhs() 6 0.0004429 0.0004429 0.0004429 0.13% Amr::InitAmr() 1 0.0004269 0.0004269 0.0004269 0.12% Castro::enforce_speed_limit() 30 0.0004149 0.0004149 0.0004149 0.12% FabArrayBase::CPC::define() 244 0.000413 0.000413 0.000413 0.12% FabArrayBase::getFB() 1766 0.0003089 0.0003089 0.0003089 0.09% Castro::finalize_advance() 5 0.0002942 0.0002942 0.0002942 0.08% Gravity::swapTimeLevels() 5 0.000224 0.000224 0.000224 0.06% MLLinOp::define() 6 0.0001649 0.0001649 0.0001649 0.05% MLLinOp::defineGrids() 6 0.0001597 0.0001597 0.0001597 0.05% MLMG::MLResNormInf() 6 0.0001517 0.0001517 0.0001517 0.04% Castro::buildMetrics() 1 0.0001489 0.0001489 0.0001489 0.04% MultiFab::max() 6 0.0001352 0.0001352 0.0001352 0.04% MLMG::MLRhsNormInf() 6 0.000117 0.000117 0.000117 0.03% FabArrayBase::FB::FB() 26 5.844e-05 5.844e-05 5.844e-05 0.02% Castro::create_source_corrector() 5 4.449e-05 4.449e-05 4.449e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.853e-05 2.853e-05 2.853e-05 0.01% Castro::initMFs() 1 2.761e-05 2.761e-05 2.761e-05 0.01% Amr::writeSmallPlotFile() 1 2.491e-05 2.491e-05 2.491e-05 0.01% Castro::swap_state_time_levels() 5 2.41e-05 2.41e-05 2.41e-05 0.01% makeSFC 30 2.174e-05 2.174e-05 2.174e-05 0.01% Castro::finalize_do_advance() 5 1.944e-05 1.944e-05 1.944e-05 0.01% DistributionMapping::Distribute() 31 1.057e-05 1.057e-05 1.057e-05 0.00% Amr::initSubcycle() 1 9.117e-06 9.117e-06 9.117e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.467e-06 5.467e-06 5.467e-06 0.00% Gravity::set_mass_offset() 6 1.99e-06 1.99e-06 1.99e-06 0.00% Castro::retry_advance_ctu() 5 1.829e-06 1.829e-06 1.829e-06 0.00% Castro::FluxRegCrseInit 5 1.477e-06 1.477e-06 1.477e-06 0.00% Castro::FluxRegFineAdd() 5 1.218e-06 1.218e-06 1.218e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.108e-06 1.108e-06 1.108e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 250 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 115 MiB 692 MiB Castro::initMFs() 48 48 59 MiB 68 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB FillPatchIterator::Initialize 160 160 1305 KiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2662 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6592 KiB 14 MiB MLMG::prepareForSolve() 361 361 3870 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 222 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 226 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6587 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 14 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2450 B 2048 KiB Gravity::solve_for_phi() 40 40 711 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 19 KiB 2048 KiB BndryData::define() 576 576 356 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 228 KiB 671 KiB Castro::estTimeStep() 10 10 4057 B 480 KiB VisMF::Write(FabArray) 112 112 1424 B 320 KiB Castro::normalize_species() 30 30 5738 B 320 KiB amrex::average_down 469 469 1336 B 257 KiB MLMG::addInterpCorrection() 468 468 1278 B 257 KiB amrex::Dot() 592 592 3721 B 160 KiB FabArray::norminf() 398 398 2620 B 160 KiB Castro::do_advance_ctu() 5 5 703 B 160 KiB MultiFab::max() 6 6 59 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 33 B 20 KiB MLPoisson::Fsmooth() 60 60 3764 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 55 B 10 KiB FillBoundary_nowait() 336 336 317 B 9648 B MLCellLinOp::applyBC() 3892 3892 244 B 9344 B amrex::Copy() 56 56 5886 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 56 B 2880 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCGSolver::bicgstab 324 324 130 B 1472 B MLCellLinOp::defineBC() 36 36 400 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1493 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 129 KiB 8192 KiB VisMF::Write(FabArray) 120 120 280 KiB 3584 KiB VisMF::Read() 24 24 223 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3764 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 55 B 10 KiB FillBoundary_nowait() 336 336 317 B 9648 B MLCellLinOp::applyBC() 1946 1946 242 B 9328 B amrex::Copy() 56 56 5886 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2589 B 3072 B StateData::FillBoundary(geom) 960 960 57 B 2880 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 557 B 1296 B amrex::average_down 37 37 217 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 27 B 400 B FabArray::norminf() 398 398 19 B 272 B Castro::estTimeStep() 10 10 0 B 32 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::do_advance_ctu() 5 5 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-11-g0f4f9877c81e) finalized