Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-16-g9473062293af) initialized Starting run at 08:24:02 UTC on 2022-05-13. Successfully read inputs file ... Castro git describe: 22.05-16-gcd83f56f0 AMReX git describe: 22.05-16-g947306229 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.042919804 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024657341 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048235337 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.0603529 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.068700904 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.061443721 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.051818217 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040425673 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052811445 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.063307145 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062911104 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062231593 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063581171 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040024136 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024821813 seconds Ending run at 08:24:03 UTC on 2022-05-13. Run time = 0.81806359 Run time without initialization = 0.701139436 Average number of zones advanced per microsecond: 3.739 Average number of zones advanced per microsecond per rank: 3.739 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8181 ... 0.8181 ... 0.8181 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1967 0.1967 0.1967 24.04% VisMF::Write(FabArray) 11 0.1661 0.1661 0.1661 20.31% MLCellLinOp::applyBC() 4379 0.08036 0.08036 0.08036 9.82% MLPoisson::Fsmooth() 3240 0.06323 0.06323 0.06323 7.73% StateData::FillBoundary(geom) 328 0.02416 0.02416 0.02416 2.95% MLCGSolver::bicgstab 81 0.02386 0.02386 0.02386 2.92% MultiFab::Dot() 1100 0.02229 0.02229 0.02229 2.72% Castro::normalize_species() 62 0.01852 0.01852 0.01852 2.26% Castro::computeTemp() 63 0.01555 0.01555 0.01555 1.90% MultiFab::LinComb() 1566 0.01439 0.01439 0.01439 1.76% FabArray::setVal() 1135 0.01426 0.01426 0.01426 1.74% FillBoundary_nowait() 3974 0.01397 0.01397 0.01397 1.71% Castro::enforce_min_density() 62 0.01365 0.01365 0.01365 1.67% FabArray::ParallelCopy_nowait() 851 0.01306 0.01306 0.01306 1.60% MLCellLinOp::defineAuxData() 11 0.01167 0.01167 0.01167 1.43% MLPoisson::Fapply() 1128 0.01167 0.01167 0.01167 1.43% StateDataPhysBCFunct::() 41 0.01132 0.01132 0.01132 1.38% Gravity::fill_multipole_BCs() 11 0.008469 0.008469 0.008469 1.04% MLMG::addInterpCorrection() 405 0.007414 0.007414 0.007414 0.91% Castro::estTimeStep() 21 0.007327 0.007327 0.007327 0.90% amrex::average_down 405 0.006878 0.006878 0.006878 0.84% MultiFab::Xpay() 578 0.006609 0.006609 0.006609 0.81% Castro::do_advance_ctu() 10 0.005415 0.005415 0.005415 0.66% Castro::reset_internal_energy(MultiFab) 63 0.005362 0.005362 0.005362 0.66% Amr::checkPoint() 3 0.004238 0.004238 0.004238 0.52% BndryData::define() 11 0.003927 0.003927 0.003927 0.48% Castro::enforce_speed_limit() 62 0.003435 0.003435 0.003435 0.42% Castro::construct_new_gravity_source() 10 0.002569 0.002569 0.002569 0.31% Amr::writePlotFile() 2 0.00249 0.00249 0.00249 0.30% Castro::construct_old_gravity_source() 10 0.00215 0.00215 0.00215 0.26% Gravity::get_new_grav_vector() 11 0.001925 0.001925 0.001925 0.24% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.22% MLMG::ResNormInf() 92 0.001809 0.001809 0.001809 0.22% Gravity::get_old_grav_vector() 10 0.001735 0.001735 0.001735 0.21% Castro::expand_state() 10 0.00173 0.00173 0.00173 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001663 0.001663 0.001663 0.20% Castro::reset_internal_energy(Fab) 504 0.001606 0.001606 0.001606 0.20% MLMG::oneIter() 81 0.001598 0.001598 0.001598 0.20% MLCellLinOp::setLevelBC() 11 0.00156 0.00156 0.00156 0.19% Gravity::actual_solve_with_mlmg() 11 0.001363 0.001363 0.001363 0.17% FabArray::setDomainBndry() 41 0.001314 0.001314 0.001314 0.16% FabArray::mult() 43 0.001313 0.001313 0.001313 0.16% Castro::initData() 1 0.001232 0.001232 0.001232 0.15% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% MLCellLinOp::prepareForSolve() 11 0.00118 0.00118 0.00118 0.14% MLCellLinOp::smooth() 1620 0.001072 0.001072 0.001072 0.13% MLMG::prepareForSolve() 11 0.001023 0.001023 0.001023 0.13% MLCellLinOp::compGrad() 11 0.0009216 0.0009216 0.0009216 0.11% FabArray::FillBoundary() 3974 0.0007714 0.0007714 0.0007714 0.09% FabArrayBase::getCPC() 1313 0.0007442 0.0007442 0.0007442 0.09% FabArrayBase::CPC::define() 454 0.0006695 0.0006695 0.0006695 0.08% FabArrayBase::getFB() 3974 0.0005918 0.0005918 0.0005918 0.07% Amr::InitAmr() 1 0.0004584 0.0004584 0.0004584 0.06% MLCellLinOp::apply() 1128 0.0004345 0.0004345 0.0004345 0.05% Gravity::solve_for_phi() 10 0.0004327 0.0004327 0.0004327 0.05% Gravity::update_max_rhs() 11 0.0004107 0.0004107 0.0004107 0.05% CGSolver::sxay() 1566 0.000357 0.000357 0.000357 0.04% FillPatchIterator::Initialize 41 0.0002895 0.0002895 0.0002895 0.04% MLCellLinOp::defineBC() 11 0.0002785 0.0002785 0.0002785 0.03% main() 1 0.0002766 0.0002766 0.0002766 0.03% MLCGSolver::ParallelAllReduce 1495 0.0002734 0.0002734 0.0002734 0.03% MultiFab::Copy() 11 0.0002624 0.0002624 0.0002624 0.03% FabArray::ParallelCopy() 851 0.0002605 0.0002605 0.0002605 0.03% MultiFab::max() 11 0.0002517 0.0002517 0.0002517 0.03% MLCellLinOp::correctionResidual() 486 0.0002292 0.0002292 0.0002292 0.03% Castro::subcycle_advance_ctu() 10 0.0002288 0.0002288 0.0002288 0.03% Castro::construct_new_gravity() 10 0.000202 0.000202 0.000202 0.02% MLMG::MLRhsNormInf() 11 0.0002011 0.0002011 0.0002011 0.02% Amr::coarseTimeStep() 10 0.0001977 0.0001977 0.0001977 0.02% MLMG::mgVcycle() 81 0.0001954 0.0001954 0.0001954 0.02% Amr::timeStep() 10 0.0001815 0.0001815 0.0001815 0.02% MLLinOp::defineGrids() 11 0.0001681 0.0001681 0.0001681 0.02% StateData::checkPoint() 12 0.0001359 0.0001359 0.0001359 0.02% MLMG:computeResOfCorrection() 405 0.0001208 0.0001208 0.0001208 0.01% Castro::advance() 10 0.0001202 0.0001202 0.0001202 0.01% MLMG::actualBottomSolve() 81 9.696e-05 9.696e-05 9.696e-05 0.01% MLMG::mgVcycle_down::0 81 9.365e-05 9.365e-05 9.365e-05 0.01% MLMG::mgVcycle_down::1 81 9.333e-05 9.333e-05 9.333e-05 0.01% Castro::Castro() 1 9.171e-05 9.171e-05 9.171e-05 0.01% FabArrayBase::FB::FB() 56 8.557e-05 8.557e-05 8.557e-05 0.01% Castro::initialize_advance() 10 8.171e-05 8.171e-05 8.171e-05 0.01% MLMG::mgVcycle_down::2 81 8.111e-05 8.111e-05 8.111e-05 0.01% MLMG::mgVcycle_down::4 81 7.378e-05 7.378e-05 7.378e-05 0.01% MLMG::mgVcycle_down::3 81 7.341e-05 7.341e-05 7.341e-05 0.01% Castro::clean_state() 62 7.27e-05 7.27e-05 7.27e-05 0.01% AmrLevel::checkPoint() 3 7.204e-05 7.204e-05 7.204e-05 0.01% MLMG::solve() 11 7.052e-05 7.052e-05 7.052e-05 0.01% Castro::initialize_do_advance() 10 6.331e-05 6.331e-05 6.331e-05 0.01% MLMG::mgVcycle_up::4 81 5.746e-05 5.746e-05 5.746e-05 0.01% MLMG::mgVcycle_up::0 81 5.486e-05 5.486e-05 5.486e-05 0.01% Castro::finalize_advance() 10 5.365e-05 5.365e-05 5.365e-05 0.01% MLCellLinOp::solutionResidual() 92 5.257e-05 5.257e-05 5.257e-05 0.01% MLMG::mgVcycle_up::1 81 4.948e-05 4.948e-05 4.948e-05 0.01% MLMG::mgVcycle_up::3 81 4.824e-05 4.824e-05 4.824e-05 0.01% MLMG::mgVcycle_up::2 81 4.649e-05 4.649e-05 4.649e-05 0.01% Castro::construct_new_source() 50 4.562e-05 4.562e-05 4.562e-05 0.01% Castro::swap_state_time_levels() 10 3.832e-05 3.832e-05 3.832e-05 0.00% StateData::define() 4 3.809e-05 3.809e-05 3.809e-05 0.00% Castro::finalize_do_advance() 10 3.72e-05 3.72e-05 3.72e-05 0.00% Castro::enforce_consistent_e() 1 3.418e-05 3.418e-05 3.418e-05 0.00% Amr::defBaseLevel() 1 3.163e-05 3.163e-05 3.163e-05 0.00% MLMG::computeResidual() 81 3.15e-05 3.15e-05 3.15e-05 0.00% MLMG::mgVcycle_bottom 81 3.132e-05 3.132e-05 3.132e-05 0.00% Gravity::actual_multilevel_solve() 1 3.113e-05 3.113e-05 3.113e-05 0.00% Castro::initMFs() 1 3.045e-05 3.045e-05 3.045e-05 0.00% makeSFC 55 2.679e-05 2.679e-05 2.679e-05 0.00% FillPatchSingleLevel 41 2.674e-05 2.674e-05 2.674e-05 0.00% Amr::writeSmallPlotFile() 1 2.462e-05 2.462e-05 2.462e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.399e-05 2.399e-05 2.399e-05 0.00% Castro::buildMetrics() 1 2.33e-05 2.33e-05 2.33e-05 0.00% MLPoisson::define() 11 2.329e-05 2.329e-05 2.329e-05 0.00% MLLinOp::define() 11 2.18e-05 2.18e-05 2.18e-05 0.00% Amr::FinalizeInit() 1 2.073e-05 2.073e-05 2.073e-05 0.00% Castro::construct_old_source() 50 1.776e-05 1.776e-05 1.776e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.75e-05 1.75e-05 1.75e-05 0.00% Castro::do_new_sources() 10 1.727e-05 1.727e-05 1.727e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.631e-05 1.631e-05 1.631e-05 0.00% Castro::do_old_sources() 10 1.588e-05 1.588e-05 1.588e-05 0.00% DistributionMapping::Distribute() 56 1.433e-05 1.433e-05 1.433e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.242e-05 1.242e-05 1.242e-05 0.00% Castro::apply_source_to_state() 20 1.083e-05 1.083e-05 1.083e-05 0.00% Castro::check_for_nan() 20 1.028e-05 1.028e-05 1.028e-05 0.00% Amr::initSubcycle() 1 9.736e-06 9.736e-06 9.736e-06 0.00% Castro::construct_old_gravity() 10 9.709e-06 9.709e-06 9.709e-06 0.00% Gravity::swapTimeLevels() 10 8.889e-06 8.889e-06 8.889e-06 0.00% MLPoisson::prepareForSolve() 11 8.336e-06 8.336e-06 8.336e-06 0.00% Castro::post_timestep() 10 7.634e-06 7.634e-06 7.634e-06 0.00% MLMG::computeMLResidual() 11 6.614e-06 6.614e-06 6.614e-06 0.00% Amr::InitializeInit() 1 5.822e-06 5.822e-06 5.822e-06 0.00% MLMG::buildFineMask() 11 5.581e-06 5.581e-06 5.581e-06 0.00% MLMG::getGradSolution() 11 5.492e-06 5.492e-06 5.492e-06 0.00% Castro::computeNewDt() 9 5.251e-06 5.251e-06 5.251e-06 0.00% AmrLevel::checkPointPost() 3 4.876e-06 4.876e-06 4.876e-06 0.00% Castro::retry_advance_ctu() 10 4.597e-06 4.597e-06 4.597e-06 0.00% MLMG::MLResNormInf() 11 4.348e-06 4.348e-06 4.348e-06 0.00% Castro::post_init() 1 4.11e-06 4.11e-06 4.11e-06 0.00% Castro::create_source_corrector() 10 3.891e-06 3.891e-06 3.891e-06 0.00% Gravity::set_mass_offset() 11 3.805e-06 3.805e-06 3.805e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.557e-06 3.557e-06 3.557e-06 0.00% Castro::FluxRegCrseInit 10 2.845e-06 2.845e-06 2.845e-06 0.00% Amr::init() 1 2.708e-06 2.708e-06 2.708e-06 0.00% Castro::computeInitialDt() 2 2.63e-06 2.63e-06 2.63e-06 0.00% Castro::FluxRegFineAdd() 10 2.205e-06 2.205e-06 2.205e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.837e-06 1.837e-06 1.837e-06 0.00% AmrLevel::checkPointPre() 3 1.677e-06 1.677e-06 1.677e-06 0.00% Castro::post_regrid() 1 1.432e-06 1.432e-06 1.432e-06 0.00% Amr::initialInit() 1 9.84e-07 9.84e-07 9.84e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8181 0.8181 0.8181 100.00% Amr::coarseTimeStep() 10 0.6761 0.6761 0.6761 82.64% Amr::timeStep() 10 0.5914 0.5914 0.5914 72.29% Castro::advance() 10 0.5802 0.5802 0.5802 70.92% Castro::subcycle_advance_ctu() 10 0.5665 0.5665 0.5665 69.25% Castro::do_advance_ctu() 10 0.5663 0.5663 0.5663 69.22% Gravity::solve_phi_with_mlmg() 11 0.3125 0.3125 0.3125 38.20% Gravity::actual_solve_with_mlmg() 11 0.3038 0.3038 0.3038 37.13% Castro::construct_new_gravity() 10 0.287 0.287 0.287 35.09% MLMG::solve() 11 0.281 0.281 0.281 34.35% Gravity::solve_for_phi() 10 0.2717 0.2717 0.2717 33.21% MLMG::oneIter() 81 0.2666 0.2666 0.2666 32.59% MLMG::mgVcycle() 81 0.265 0.265 0.265 32.40% Castro::construct_ctu_hydro_source() 10 0.1967 0.1967 0.1967 24.04% VisMF::Write(FabArray) 11 0.1661 0.1661 0.1661 20.31% MLCellLinOp::smooth() 1620 0.1356 0.1356 0.1356 16.58% Amr::checkPoint() 3 0.1235 0.1235 0.1235 15.09% AmrLevel::checkPoint() 3 0.1192 0.1192 0.1192 14.58% StateData::checkPoint() 12 0.1192 0.1192 0.1192 14.57% Amr::init() 1 0.1163 0.1163 0.1163 14.22% MLCellLinOp::applyBC() 4379 0.09579 0.09579 0.09579 11.71% MLMG::mgVcycle_bottom 81 0.08174 0.08174 0.08174 9.99% MLMG::actualBottomSolve() 81 0.08171 0.08171 0.08171 9.99% MLCGSolver::bicgstab 81 0.08089 0.08089 0.08089 9.89% MLPoisson::Fsmooth() 3240 0.06323 0.06323 0.06323 7.73% Castro::clean_state() 62 0.05744 0.05744 0.05744 7.02% Amr::writePlotFile() 2 0.04959 0.04959 0.04959 6.06% Amr::initialInit() 1 0.04865 0.04865 0.04865 5.95% Amr::FinalizeInit() 1 0.04458 0.04458 0.04458 5.45% Castro::post_init() 1 0.04324 0.04324 0.04324 5.29% Gravity::multilevel_solve_for_new_phi() 1 0.04131 0.04131 0.04131 5.05% Gravity::actual_multilevel_solve() 1 0.0413 0.0413 0.0413 5.05% FillPatchIterator::Initialize 41 0.04106 0.04106 0.04106 5.02% FillPatchSingleLevel 41 0.03946 0.03946 0.03946 4.82% MLCellLinOp::apply() 1128 0.03607 0.03607 0.03607 4.41% StateDataPhysBCFunct::() 41 0.03549 0.03549 0.03549 4.34% MLMG::mgVcycle_down::0 81 0.03523 0.03523 0.03523 4.31% MLMG::mgVcycle_up::0 81 0.0302 0.0302 0.0302 3.69% StateData::FillBoundary(geom) 328 0.02416 0.02416 0.02416 2.95% Castro::computeTemp() 63 0.02252 0.02252 0.02252 2.75% MultiFab::Dot() 1100 0.02229 0.02229 0.02229 2.72% MLCellLinOp::correctionResidual() 486 0.02122 0.02122 0.02122 2.59% Castro::initialize_do_advance() 10 0.02069 0.02069 0.02069 2.53% Castro::normalize_species() 62 0.01852 0.01852 0.01852 2.26% MLPoisson::define() 11 0.01843 0.01843 0.01843 2.25% MLMG:computeResOfCorrection() 405 0.01827 0.01827 0.01827 2.23% MLMG::mgVcycle_down::1 81 0.01767 0.01767 0.01767 2.16% MLMG::mgVcycle_down::2 81 0.01717 0.01717 0.01717 2.10% Gravity::get_new_grav_vector() 11 0.01703 0.01703 0.01703 2.08% MLMG::mgVcycle_down::3 81 0.01632 0.01632 0.01632 2.00% MLMG::mgVcycle_down::4 81 0.01554 0.01554 0.01554 1.90% FabArray::FillBoundary() 3974 0.01542 0.01542 0.01542 1.89% CGSolver::sxay() 1566 0.01475 0.01475 0.01475 1.80% Castro::construct_old_gravity() 10 0.0147 0.0147 0.0147 1.80% Gravity::get_old_grav_vector() 10 0.01469 0.01469 0.01469 1.80% FillBoundary_nowait() 3974 0.01465 0.01465 0.01465 1.79% MultiFab::LinComb() 1566 0.01439 0.01439 0.01439 1.76% FabArray::setVal() 1135 0.01426 0.01426 0.01426 1.74% FabArray::ParallelCopy() 851 0.01411 0.01411 0.01411 1.72% FabArray::ParallelCopy_nowait() 851 0.01385 0.01385 0.01385 1.69% Castro::enforce_min_density() 62 0.01365 0.01365 0.01365 1.67% Castro::initialize_advance() 10 0.01348 0.01348 0.01348 1.65% MLCGSolver::ParallelAllReduce 1495 0.01322 0.01322 0.01322 1.62% MLMG::mgVcycle_up::2 81 0.01322 0.01322 0.01322 1.62% MLMG::mgVcycle_up::1 81 0.01302 0.01302 0.01302 1.59% MLCellLinOp::defineAuxData() 11 0.01301 0.01301 0.01301 1.59% MLMG::addInterpCorrection() 405 0.0125 0.0125 0.0125 1.53% MLMG::mgVcycle_up::3 81 0.0125 0.0125 0.0125 1.53% MLMG::mgVcycle_up::4 81 0.01223 0.01223 0.01223 1.49% amrex::average_down 405 0.01196 0.01196 0.01196 1.46% Castro::do_old_sources() 10 0.01181 0.01181 0.01181 1.44% MLPoisson::Fapply() 1128 0.01167 0.01167 0.01167 1.43% Castro::do_new_sources() 10 0.01148 0.01148 0.01148 1.40% Castro::post_timestep() 10 0.01101 0.01101 0.01101 1.35% Castro::expand_state() 10 0.01082 0.01082 0.01082 1.32% Gravity::fill_multipole_BCs() 11 0.008469 0.008469 0.008469 1.04% Castro::estTimeStep() 21 0.007327 0.007327 0.007327 0.90% MLCellLinOp::solutionResidual() 92 0.007129 0.007129 0.007129 0.87% Castro::reset_internal_energy(MultiFab) 63 0.006968 0.006968 0.006968 0.85% MultiFab::Xpay() 578 0.006609 0.006609 0.006609 0.81% MLMG::computeResidual() 81 0.006145 0.006145 0.006145 0.75% MLCellLinOp::defineBC() 11 0.005149 0.005149 0.005149 0.63% MLMG::prepareForSolve() 11 0.005143 0.005143 0.005143 0.63% BndryData::define() 11 0.004871 0.004871 0.004871 0.60% Amr::InitializeInit() 1 0.004068 0.004068 0.004068 0.50% Amr::defBaseLevel() 1 0.004062 0.004062 0.004062 0.50% Castro::computeNewDt() 9 0.003527 0.003527 0.003527 0.43% Castro::initData() 1 0.003522 0.003522 0.003522 0.43% Castro::enforce_speed_limit() 62 0.003435 0.003435 0.003435 0.42% Castro::construct_new_source() 50 0.002614 0.002614 0.002614 0.32% Castro::construct_new_gravity_source() 10 0.002569 0.002569 0.002569 0.31% Castro::construct_old_source() 50 0.002168 0.002168 0.002168 0.27% Castro::construct_old_gravity_source() 10 0.00215 0.00215 0.00215 0.26% Castro::apply_source_to_state() 20 0.001826 0.001826 0.001826 0.22% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.22% MLMG::ResNormInf() 92 0.001809 0.001809 0.001809 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001663 0.001663 0.001663 0.20% Castro::reset_internal_energy(Fab) 504 0.001606 0.001606 0.001606 0.20% MLCellLinOp::setLevelBC() 11 0.00156 0.00156 0.00156 0.19% FabArrayBase::getCPC() 1313 0.001414 0.001414 0.001414 0.17% MLMG::getGradSolution() 11 0.001411 0.001411 0.001411 0.17% MLCellLinOp::compGrad() 11 0.001405 0.001405 0.001405 0.17% FabArray::setDomainBndry() 41 0.001314 0.001314 0.001314 0.16% FabArray::mult() 43 0.001313 0.001313 0.001313 0.16% Castro::check_for_nan() 20 0.001192 0.001192 0.001192 0.15% MLPoisson::prepareForSolve() 11 0.001189 0.001189 0.001189 0.15% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% MLCellLinOp::prepareForSolve() 11 0.00118 0.00118 0.00118 0.14% Castro::post_regrid() 1 0.001103 0.001103 0.001103 0.13% MLMG::computeMLResidual() 11 0.001022 0.001022 0.001022 0.12% Gravity::update_max_rhs() 11 0.0008088 0.0008088 0.0008088 0.10% FabArrayBase::getFB() 3974 0.0006773 0.0006773 0.0006773 0.08% Castro::computeInitialDt() 2 0.0006753 0.0006753 0.0006753 0.08% FabArrayBase::CPC::define() 454 0.0006695 0.0006695 0.0006695 0.08% Amr::InitAmr() 1 0.0004681 0.0004681 0.0004681 0.06% Gravity::swapTimeLevels() 10 0.0004595 0.0004595 0.0004595 0.06% Castro::Castro() 1 0.0004492 0.0004492 0.0004492 0.05% MultiFab::Copy() 11 0.0002624 0.0002624 0.0002624 0.03% MLMG::MLResNormInf() 11 0.0002547 0.0002547 0.0002547 0.03% MultiFab::max() 11 0.0002517 0.0002517 0.0002517 0.03% MLLinOp::define() 11 0.0002441 0.0002441 0.0002441 0.03% MLLinOp::defineGrids() 11 0.0002224 0.0002224 0.0002224 0.03% MLMG::MLRhsNormInf() 11 0.0002011 0.0002011 0.0002011 0.02% Castro::buildMetrics() 1 0.0001601 0.0001601 0.0001601 0.02% FabArrayBase::FB::FB() 56 8.557e-05 8.557e-05 8.557e-05 0.01% Castro::finalize_advance() 10 5.87e-05 5.87e-05 5.87e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.439e-05 5.439e-05 5.439e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.24e-05 5.24e-05 5.24e-05 0.01% makeSFC 55 3.998e-05 3.998e-05 3.998e-05 0.00% Castro::swap_state_time_levels() 10 3.832e-05 3.832e-05 3.832e-05 0.00% StateData::define() 4 3.809e-05 3.809e-05 3.809e-05 0.00% Castro::finalize_do_advance() 10 3.72e-05 3.72e-05 3.72e-05 0.00% Castro::enforce_consistent_e() 1 3.418e-05 3.418e-05 3.418e-05 0.00% Castro::initMFs() 1 3.045e-05 3.045e-05 3.045e-05 0.00% Amr::writeSmallPlotFile() 1 2.462e-05 2.462e-05 2.462e-05 0.00% DistributionMapping::Distribute() 56 1.433e-05 1.433e-05 1.433e-05 0.00% Amr::initSubcycle() 1 9.736e-06 9.736e-06 9.736e-06 0.00% MLMG::buildFineMask() 11 5.581e-06 5.581e-06 5.581e-06 0.00% AmrLevel::checkPointPost() 3 4.876e-06 4.876e-06 4.876e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.707e-06 4.707e-06 4.707e-06 0.00% Castro::retry_advance_ctu() 10 4.597e-06 4.597e-06 4.597e-06 0.00% Castro::create_source_corrector() 10 3.891e-06 3.891e-06 3.891e-06 0.00% Gravity::set_mass_offset() 11 3.805e-06 3.805e-06 3.805e-06 0.00% Castro::FluxRegCrseInit 10 2.845e-06 2.845e-06 2.845e-06 0.00% Castro::FluxRegFineAdd() 10 2.205e-06 2.205e-06 2.205e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.837e-06 1.837e-06 1.837e-06 0.00% AmrLevel::checkPointPre() 3 1.677e-06 1.677e-06 1.677e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-16-g9473062293af) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-16-g9473062293af) initialized Starting run at 08:24:03 UTC on 2022-05-13. Successfully read inputs file ... Castro git describe: 22.05-16-gcd83f56f0 AMReX git describe: 22.05-16-g947306229 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.447834417 Restart time = 0.046346354 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053090051 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048180448 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056269764 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063350379 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066670125 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026160224 seconds Ending run at 08:24:04 UTC on 2022-05-13. Run time = 0.360972422 Run time without initialization = 0.31406367 Average number of zones advanced per microsecond: 4.173 Average number of zones advanced per microsecond per rank: 4.173 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.361 ... 0.361 ... 0.361 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0884 0.0884 0.0884 24.49% VisMF::Read() 3 0.03909 0.03909 0.03909 10.83% MLCellLinOp::applyBC() 1946 0.03463 0.03463 0.03463 9.59% MLPoisson::Fsmooth() 1440 0.02711 0.02711 0.02711 7.51% VisMF::Write(FabArray) 1 0.02473 0.02473 0.02473 6.85% StateData::FillBoundary(geom) 160 0.01133 0.01133 0.01133 3.14% MLCGSolver::bicgstab 36 0.01025 0.01025 0.01025 2.84% MultiFab::Dot() 484 0.009477 0.009477 0.009477 2.63% Castro::computeTemp() 30 0.00929 0.00929 0.00929 2.57% Castro::normalize_species() 30 0.008758 0.008758 0.008758 2.43% FabArray::setVal() 537 0.006704 0.006704 0.006704 1.86% FillBoundary_nowait() 1766 0.006157 0.006157 0.006157 1.71% MLCellLinOp::defineAuxData() 6 0.006141 0.006141 0.006141 1.70% MultiFab::LinComb() 690 0.006136 0.006136 0.006136 1.70% FabArray::ParallelCopy_nowait() 380 0.005888 0.005888 0.005888 1.63% StateDataPhysBCFunct::() 20 0.005313 0.005313 0.005313 1.47% MLPoisson::Fapply() 500 0.005067 0.005067 0.005067 1.40% Castro::enforce_min_density() 30 0.004687 0.004687 0.004687 1.30% Gravity::fill_multipole_BCs() 6 0.004659 0.004659 0.004659 1.29% Castro::estTimeStep() 10 0.004076 0.004076 0.004076 1.13% MLMG::addInterpCorrection() 180 0.003204 0.003204 0.003204 0.89% Amr::restart() 1 0.003063 0.003063 0.003063 0.85% amrex::average_down 180 0.002943 0.002943 0.002943 0.82% MultiFab::Xpay() 258 0.002886 0.002886 0.002886 0.80% Castro::do_advance_ctu() 5 0.002314 0.002314 0.002314 0.64% BndryData::define() 6 0.002103 0.002103 0.002103 0.58% Castro::reset_internal_energy(MultiFab) 30 0.00165 0.00165 0.00165 0.46% Amr::writePlotFile() 1 0.001515 0.001515 0.001515 0.42% Castro::construct_new_gravity_source() 5 0.001492 0.001492 0.001492 0.41% Castro::construct_old_gravity_source() 5 0.001066 0.001066 0.001066 0.30% Castro::initialize_advance() 5 0.0009229 0.0009229 0.0009229 0.26% MultiFab::Saxpy() 10 0.0009152 0.0009152 0.0009152 0.25% Castro::reset_internal_energy(Fab) 240 0.0009034 0.0009034 0.0009034 0.25% Gravity::get_old_grav_vector() 5 0.0008811 0.0008811 0.0008811 0.24% Castro::expand_state() 5 0.0008801 0.0008801 0.0008801 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008754 0.0008754 0.0008754 0.24% Gravity::get_new_grav_vector() 5 0.0008677 0.0008677 0.0008677 0.24% Castro::subcycle_advance_ctu() 5 0.0008455 0.0008455 0.0008455 0.23% MLCellLinOp::setLevelBC() 6 0.0008169 0.0008169 0.0008169 0.23% MLMG::ResNormInf() 42 0.0008085 0.0008085 0.0008085 0.22% MLMG::oneIter() 36 0.0007034 0.0007034 0.0007034 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007017 0.0007017 0.0007017 0.19% FabArray::mult() 22 0.0006436 0.0006436 0.0006436 0.18% FabArray::setDomainBndry() 20 0.0006359 0.0006359 0.0006359 0.18% MLCellLinOp::prepareForSolve() 6 0.0006328 0.0006328 0.0006328 0.18% MultiFab::contains_nan() 10 0.0005873 0.0005873 0.0005873 0.16% MLMG::prepareForSolve() 6 0.0005706 0.0005706 0.0005706 0.16% MLCellLinOp::smooth() 720 0.000528 0.000528 0.000528 0.15% MLCellLinOp::compGrad() 6 0.0004926 0.0004926 0.0004926 0.14% Castro::enforce_speed_limit() 30 0.0004075 0.0004075 0.0004075 0.11% Amr::InitAmr() 1 0.000407 0.000407 0.000407 0.11% FabArrayBase::CPC::define() 244 0.0003991 0.0003991 0.0003991 0.11% FabArrayBase::getCPC() 632 0.0003727 0.0003727 0.0003727 0.10% FabArray::FillBoundary() 1766 0.0003534 0.0003534 0.0003534 0.10% FabArrayBase::getFB() 1766 0.0002544 0.0002544 0.0002544 0.07% Gravity::solve_for_phi() 5 0.0002491 0.0002491 0.0002491 0.07% main() 1 0.000241 0.000241 0.000241 0.07% Gravity::update_max_rhs() 6 0.0002282 0.0002282 0.0002282 0.06% MLCellLinOp::apply() 500 0.0002104 0.0002104 0.0002104 0.06% CGSolver::sxay() 690 0.0001791 0.0001791 0.0001791 0.05% Castro::construct_new_source() 25 0.0001685 0.0001685 0.0001685 0.05% MLCellLinOp::defineBC() 6 0.0001528 0.0001528 0.0001528 0.04% Castro::create_source_corrector() 5 0.0001449 0.0001449 0.0001449 0.04% MultiFab::Copy() 6 0.0001447 0.0001447 0.0001447 0.04% FillPatchIterator::Initialize 20 0.0001444 0.0001444 0.0001444 0.04% MultiFab::max() 6 0.0001401 0.0001401 0.0001401 0.04% FabArray::ParallelCopy() 380 0.0001259 0.0001259 0.0001259 0.03% MLCGSolver::ParallelAllReduce 659 0.0001194 0.0001194 0.0001194 0.03% Amr::coarseTimeStep() 5 0.0001136 0.0001136 0.0001136 0.03% Castro::construct_new_gravity() 5 0.0001135 0.0001135 0.0001135 0.03% MLLinOp::defineGrids() 6 0.0001103 0.0001103 0.0001103 0.03% MLMG::MLRhsNormInf() 6 0.0001059 0.0001059 0.0001059 0.03% MLCellLinOp::correctionResidual() 216 0.0001033 0.0001033 0.0001033 0.03% Castro::advance() 5 0.0001001 0.0001001 0.0001001 0.03% Amr::timeStep() 5 9.617e-05 9.617e-05 9.617e-05 0.03% MLMG::mgVcycle() 36 8.742e-05 8.742e-05 8.742e-05 0.02% AmrLevel::restart() 1 7.731e-05 7.731e-05 7.731e-05 0.02% Castro::post_timestep() 5 7.502e-05 7.502e-05 7.502e-05 0.02% Castro::initialize_do_advance() 5 7.191e-05 7.191e-05 7.191e-05 0.02% Castro::finalize_advance() 5 6.892e-05 6.892e-05 6.892e-05 0.02% StateData::restartDoit() 4 6.806e-05 6.806e-05 6.806e-05 0.02% FabArrayBase::FB::FB() 26 5.63e-05 5.63e-05 5.63e-05 0.02% MLMG:computeResOfCorrection() 180 5.461e-05 5.461e-05 5.461e-05 0.02% MLMG::actualBottomSolve() 36 4.368e-05 4.368e-05 4.368e-05 0.01% Castro::post_restart() 1 4.126e-05 4.126e-05 4.126e-05 0.01% MLMG::mgVcycle_down::0 36 3.894e-05 3.894e-05 3.894e-05 0.01% Castro::clean_state() 30 3.741e-05 3.741e-05 3.741e-05 0.01% MLMG::mgVcycle_down::1 36 3.628e-05 3.628e-05 3.628e-05 0.01% MLMG::solve() 6 3.486e-05 3.486e-05 3.486e-05 0.01% MLMG::mgVcycle_down::2 36 3.484e-05 3.484e-05 3.484e-05 0.01% MLMG::mgVcycle_down::4 36 3.248e-05 3.248e-05 3.248e-05 0.01% MLMG::mgVcycle_down::3 36 3.153e-05 3.153e-05 3.153e-05 0.01% Castro::buildMetrics() 1 3.089e-05 3.089e-05 3.089e-05 0.01% Gravity::actual_multilevel_solve() 1 3.025e-05 3.025e-05 3.025e-05 0.01% Castro::construct_old_source() 25 2.976e-05 2.976e-05 2.976e-05 0.01% Castro::initMFs() 1 2.739e-05 2.739e-05 2.739e-05 0.01% MLMG::mgVcycle_up::4 36 2.699e-05 2.699e-05 2.699e-05 0.01% Castro::swap_state_time_levels() 5 2.643e-05 2.643e-05 2.643e-05 0.01% Amr::writeSmallPlotFile() 1 2.605e-05 2.605e-05 2.605e-05 0.01% MLCellLinOp::solutionResidual() 42 2.543e-05 2.543e-05 2.543e-05 0.01% MLMG::mgVcycle_up::0 36 2.429e-05 2.429e-05 2.429e-05 0.01% MLMG::mgVcycle_up::3 36 2.226e-05 2.226e-05 2.226e-05 0.01% MLMG::mgVcycle_up::2 36 2.198e-05 2.198e-05 2.198e-05 0.01% Castro::construct_old_gravity() 5 2.079e-05 2.079e-05 2.079e-05 0.01% MLMG::mgVcycle_up::1 36 2.057e-05 2.057e-05 2.057e-05 0.01% MLLinOp::define() 6 2.008e-05 2.008e-05 2.008e-05 0.01% Castro::finalize_do_advance() 5 1.773e-05 1.773e-05 1.773e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.772e-05 1.772e-05 1.772e-05 0.00% MLMG::mgVcycle_bottom 36 1.602e-05 1.602e-05 1.602e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.598e-05 1.598e-05 1.598e-05 0.00% FillPatchSingleLevel 20 1.459e-05 1.459e-05 1.459e-05 0.00% MLPoisson::define() 6 1.457e-05 1.457e-05 1.457e-05 0.00% MLMG::computeResidual() 36 1.432e-05 1.432e-05 1.432e-05 0.00% makeSFC 30 1.407e-05 1.407e-05 1.407e-05 0.00% Castro::do_new_sources() 5 9.977e-06 9.977e-06 9.977e-06 0.00% Amr::initSubcycle() 1 8.898e-06 8.898e-06 8.898e-06 0.00% DistributionMapping::Distribute() 31 8.084e-06 8.084e-06 8.084e-06 0.00% Castro::do_old_sources() 5 8.045e-06 8.045e-06 8.045e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.838e-06 6.838e-06 6.838e-06 0.00% Castro::check_for_nan() 10 6.043e-06 6.043e-06 6.043e-06 0.00% Castro::apply_source_to_state() 10 5.452e-06 5.452e-06 5.452e-06 0.00% MLPoisson::prepareForSolve() 6 4.564e-06 4.564e-06 4.564e-06 0.00% Gravity::swapTimeLevels() 5 4.283e-06 4.283e-06 4.283e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.976e-06 3.976e-06 3.976e-06 0.00% MLMG::computeMLResidual() 6 3.357e-06 3.357e-06 3.357e-06 0.00% MLMG::getGradSolution() 6 3.023e-06 3.023e-06 3.023e-06 0.00% Castro::computeNewDt() 5 2.934e-06 2.934e-06 2.934e-06 0.00% MLMG::buildFineMask() 6 2.933e-06 2.933e-06 2.933e-06 0.00% Gravity::set_mass_offset() 6 2.433e-06 2.433e-06 2.433e-06 0.00% MLMG::MLResNormInf() 6 2.31e-06 2.31e-06 2.31e-06 0.00% Castro::FluxRegCrseInit 5 2.012e-06 2.012e-06 2.012e-06 0.00% Castro::retry_advance_ctu() 5 1.936e-06 1.936e-06 1.936e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.063e-06 1.063e-06 1.063e-06 0.00% Castro::FluxRegFineAdd() 5 1.005e-06 1.005e-06 1.005e-06 0.00% Amr::init() 1 9.78e-07 9.78e-07 9.78e-07 0.00% AmrLevel::AmrLevel() 1 6.63e-07 6.63e-07 6.63e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.361 0.361 0.361 100.00% Amr::coarseTimeStep() 5 0.2877 0.2877 0.2877 79.69% Amr::timeStep() 5 0.286 0.286 0.286 79.24% Castro::advance() 5 0.2807 0.2807 0.2807 77.76% Castro::subcycle_advance_ctu() 5 0.2743 0.2743 0.2743 75.99% Castro::do_advance_ctu() 5 0.2735 0.2735 0.2735 75.76% Castro::construct_new_gravity() 5 0.1427 0.1427 0.1427 39.52% Gravity::solve_phi_with_mlmg() 6 0.1386 0.1386 0.1386 38.39% Gravity::solve_for_phi() 5 0.1352 0.1352 0.1352 37.45% Gravity::actual_solve_with_mlmg() 6 0.1338 0.1338 0.1338 37.07% MLMG::solve() 6 0.1217 0.1217 0.1217 33.71% MLMG::oneIter() 36 0.1147 0.1147 0.1147 31.79% MLMG::mgVcycle() 36 0.114 0.114 0.114 31.59% Castro::construct_ctu_hydro_source() 5 0.08839 0.08839 0.08839 24.49% MLCellLinOp::smooth() 720 0.0584 0.0584 0.0584 16.18% Amr::init() 1 0.04639 0.04639 0.04639 12.85% Amr::restart() 1 0.04639 0.04639 0.04639 12.85% MLCellLinOp::applyBC() 1946 0.04145 0.04145 0.04145 11.48% AmrLevel::restart() 1 0.0393 0.0393 0.0393 10.89% StateData::restartDoit() 4 0.03922 0.03922 0.03922 10.86% VisMF::Read() 3 0.03909 0.03909 0.03909 10.83% MLMG::mgVcycle_bottom 36 0.03503 0.03503 0.03503 9.70% MLMG::actualBottomSolve() 36 0.03502 0.03502 0.03502 9.70% MLCGSolver::bicgstab 36 0.03466 0.03466 0.03466 9.60% MLPoisson::Fsmooth() 1440 0.02711 0.02711 0.02711 7.51% Amr::writePlotFile() 1 0.02624 0.02624 0.02624 7.27% Castro::clean_state() 30 0.02573 0.02573 0.02573 7.13% VisMF::Write(FabArray) 1 0.02473 0.02473 0.02473 6.85% FillPatchIterator::Initialize 20 0.01943 0.01943 0.01943 5.38% FillPatchSingleLevel 20 0.01865 0.01865 0.01865 5.17% StateDataPhysBCFunct::() 20 0.01664 0.01664 0.01664 4.61% MLCellLinOp::apply() 500 0.01571 0.01571 0.01571 4.35% MLMG::mgVcycle_down::0 36 0.01524 0.01524 0.01524 4.22% MLMG::mgVcycle_up::0 36 0.01307 0.01307 0.01307 3.62% Castro::computeTemp() 30 0.01184 0.01184 0.01184 3.28% StateData::FillBoundary(geom) 160 0.01133 0.01133 0.01133 3.14% MLPoisson::define() 6 0.00983 0.00983 0.00983 2.72% Castro::initialize_do_advance() 5 0.009791 0.009791 0.009791 2.71% MultiFab::Dot() 484 0.009477 0.009477 0.009477 2.63% MLCellLinOp::correctionResidual() 216 0.009193 0.009193 0.009193 2.55% Castro::normalize_species() 30 0.008758 0.008758 0.008758 2.43% MLMG:computeResOfCorrection() 180 0.007922 0.007922 0.007922 2.19% MLMG::mgVcycle_down::1 36 0.007619 0.007619 0.007619 2.11% MLMG::mgVcycle_down::2 36 0.007396 0.007396 0.007396 2.05% Gravity::get_new_grav_vector() 5 0.007383 0.007383 0.007383 2.05% Castro::construct_old_gravity() 5 0.007155 0.007155 0.007155 1.98% Gravity::get_old_grav_vector() 5 0.007134 0.007134 0.007134 1.98% MLMG::mgVcycle_down::3 36 0.007014 0.007014 0.007014 1.94% MLCellLinOp::defineAuxData() 6 0.006873 0.006873 0.006873 1.90% FabArray::FillBoundary() 1766 0.006821 0.006821 0.006821 1.89% Castro::do_old_sources() 5 0.006797 0.006797 0.006797 1.88% FabArray::setVal() 537 0.006704 0.006704 0.006704 1.86% MLMG::mgVcycle_down::4 36 0.006702 0.006702 0.006702 1.86% FillBoundary_nowait() 1766 0.006467 0.006467 0.006467 1.79% FabArray::ParallelCopy() 380 0.006402 0.006402 0.006402 1.77% CGSolver::sxay() 690 0.006316 0.006316 0.006316 1.75% Castro::do_new_sources() 5 0.006298 0.006298 0.006298 1.74% FabArray::ParallelCopy_nowait() 380 0.006276 0.006276 0.006276 1.74% Castro::initialize_advance() 5 0.006207 0.006207 0.006207 1.72% MultiFab::LinComb() 690 0.006136 0.006136 0.006136 1.70% MLCGSolver::ParallelAllReduce 659 0.00568 0.00568 0.00568 1.57% MLMG::mgVcycle_up::2 36 0.005671 0.005671 0.005671 1.57% MLMG::mgVcycle_up::1 36 0.005584 0.005584 0.005584 1.55% Castro::expand_state() 5 0.005439 0.005439 0.005439 1.51% MLMG::addInterpCorrection() 180 0.005392 0.005392 0.005392 1.49% MLMG::mgVcycle_up::3 36 0.005362 0.005362 0.005362 1.49% MLMG::mgVcycle_up::4 36 0.005271 0.005271 0.005271 1.46% Castro::post_timestep() 5 0.005227 0.005227 0.005227 1.45% amrex::average_down 180 0.005163 0.005163 0.005163 1.43% MLPoisson::Fapply() 500 0.005067 0.005067 0.005067 1.40% Castro::enforce_min_density() 30 0.004687 0.004687 0.004687 1.30% Gravity::fill_multipole_BCs() 6 0.004659 0.004659 0.004659 1.29% Castro::estTimeStep() 10 0.004076 0.004076 0.004076 1.13% Castro::post_restart() 1 0.003847 0.003847 0.003847 1.07% Gravity::multilevel_solve_for_new_phi() 1 0.003707 0.003707 0.003707 1.03% Gravity::actual_multilevel_solve() 1 0.00369 0.00369 0.00369 1.02% MLCellLinOp::solutionResidual() 42 0.003229 0.003229 0.003229 0.89% MultiFab::Xpay() 258 0.002886 0.002886 0.002886 0.80% MLCellLinOp::defineBC() 6 0.002784 0.002784 0.002784 0.77% MLMG::prepareForSolve() 6 0.002761 0.002761 0.002761 0.76% MLMG::computeResidual() 36 0.002681 0.002681 0.002681 0.74% BndryData::define() 6 0.002631 0.002631 0.002631 0.73% Castro::reset_internal_energy(MultiFab) 30 0.002554 0.002554 0.002554 0.71% Castro::construct_new_source() 25 0.00166 0.00166 0.00166 0.46% Castro::computeNewDt() 5 0.00151 0.00151 0.00151 0.42% Castro::construct_new_gravity_source() 5 0.001492 0.001492 0.001492 0.41% Castro::construct_old_source() 25 0.001096 0.001096 0.001096 0.30% Castro::construct_old_gravity_source() 5 0.001066 0.001066 0.001066 0.30% Castro::apply_source_to_state() 10 0.0009207 0.0009207 0.0009207 0.26% MultiFab::Saxpy() 10 0.0009152 0.0009152 0.0009152 0.25% Castro::reset_internal_energy(Fab) 240 0.0009034 0.0009034 0.0009034 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008754 0.0008754 0.0008754 0.24% MLCellLinOp::setLevelBC() 6 0.0008169 0.0008169 0.0008169 0.23% MLMG::ResNormInf() 42 0.0008085 0.0008085 0.0008085 0.22% FabArrayBase::getCPC() 632 0.0007718 0.0007718 0.0007718 0.21% MLMG::getGradSolution() 6 0.0007607 0.0007607 0.0007607 0.21% MLCellLinOp::compGrad() 6 0.0007577 0.0007577 0.0007577 0.21% FabArray::mult() 22 0.0006436 0.0006436 0.0006436 0.18% MLPoisson::prepareForSolve() 6 0.0006374 0.0006374 0.0006374 0.18% FabArray::setDomainBndry() 20 0.0006359 0.0006359 0.0006359 0.18% MLCellLinOp::prepareForSolve() 6 0.0006328 0.0006328 0.0006328 0.18% Castro::check_for_nan() 10 0.0005934 0.0005934 0.0005934 0.16% MultiFab::contains_nan() 10 0.0005873 0.0005873 0.0005873 0.16% MLMG::computeMLResidual() 6 0.0005655 0.0005655 0.0005655 0.16% Gravity::update_max_rhs() 6 0.0004459 0.0004459 0.0004459 0.12% Amr::InitAmr() 1 0.0004159 0.0004159 0.0004159 0.12% Castro::enforce_speed_limit() 30 0.0004075 0.0004075 0.0004075 0.11% FabArrayBase::CPC::define() 244 0.0003991 0.0003991 0.0003991 0.11% FabArrayBase::getFB() 1766 0.0003107 0.0003107 0.0003107 0.09% Gravity::swapTimeLevels() 5 0.0002271 0.0002271 0.0002271 0.06% MLLinOp::define() 6 0.0001591 0.0001591 0.0001591 0.04% Castro::buildMetrics() 1 0.0001524 0.0001524 0.0001524 0.04% Castro::create_source_corrector() 5 0.0001449 0.0001449 0.0001449 0.04% MultiFab::Copy() 6 0.0001447 0.0001447 0.0001447 0.04% MultiFab::max() 6 0.0001401 0.0001401 0.0001401 0.04% MLLinOp::defineGrids() 6 0.0001391 0.0001391 0.0001391 0.04% MLMG::MLResNormInf() 6 0.0001343 0.0001343 0.0001343 0.04% MLMG::MLRhsNormInf() 6 0.0001059 0.0001059 0.0001059 0.03% Castro::finalize_advance() 5 7.194e-05 7.194e-05 7.194e-05 0.02% FabArrayBase::FB::FB() 26 5.63e-05 5.63e-05 5.63e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.773e-05 2.773e-05 2.773e-05 0.01% Castro::initMFs() 1 2.739e-05 2.739e-05 2.739e-05 0.01% Castro::swap_state_time_levels() 5 2.643e-05 2.643e-05 2.643e-05 0.01% Amr::writeSmallPlotFile() 1 2.605e-05 2.605e-05 2.605e-05 0.01% makeSFC 30 2.089e-05 2.089e-05 2.089e-05 0.01% Castro::finalize_do_advance() 5 1.773e-05 1.773e-05 1.773e-05 0.00% Amr::initSubcycle() 1 8.898e-06 8.898e-06 8.898e-06 0.00% DistributionMapping::Distribute() 31 8.084e-06 8.084e-06 8.084e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.238e-06 5.238e-06 5.238e-06 0.00% MLMG::buildFineMask() 6 2.933e-06 2.933e-06 2.933e-06 0.00% Gravity::set_mass_offset() 6 2.433e-06 2.433e-06 2.433e-06 0.00% Castro::FluxRegCrseInit 5 2.012e-06 2.012e-06 2.012e-06 0.00% Castro::retry_advance_ctu() 5 1.936e-06 1.936e-06 1.936e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.063e-06 1.063e-06 1.063e-06 0.00% Castro::FluxRegFineAdd() 5 1.005e-06 1.005e-06 1.005e-06 0.00% AmrLevel::AmrLevel() 1 6.63e-07 6.63e-07 6.63e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-16-g9473062293af) finalized