=== If no file names and line numbers are shown below, one can run addr2line -Cpfie my_exefile my_line_address to convert `my_line_address` (e.g., 0x4a6b) into file name and line number. Or one can use amrex/Tools/Backtrace/parse_bt.py. === Please note that the line number reported by addr2line may not be accurate. One can use readelf -wl my_exefile | grep my_line_address' to find out the offset for that line. 0: ./Castro3d.pgi.TPROF.CUDA.ex() [0x79e636] amrex::BLBackTrace::print_backtrace_info(_IO_FILE*) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_BLBackTrace.cpp:167 1: ./Castro3d.pgi.TPROF.CUDA.ex() [0x7a0da2] amrex::BLBackTrace::handler(int) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_BLBackTrace.cpp:71 2: ./Castro3d.pgi.TPROF.CUDA.ex() [0x4492d8] amrex::Gpu::synchronize() /raid/testing/castro-gpu/AMReX//Src/Base/AMReX_GpuDevice.H:148 3: ./Castro3d.pgi.TPROF.CUDA.ex() [0x76e665] amrex::MFIter::~MFIter() tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_MFIter.cpp:200 4: ./Castro3d.pgi.TPROF.CUDA.ex() [0x7341fa] void amrex::Add(amrex::FabArray&, amrex::FabArray const&, int, int, int, amrex::IntVect const&) /raid/testing/castro-gpu/AMReX//Src/Base/AMReX_FabArrayUtility.H:1337 5: ./Castro3d.pgi.TPROF.CUDA.ex() [0x7281bd] amrex::MultiFab::Add(amrex::MultiFab&, amrex::MultiFab const&, int, int, int, int) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_MultiFab.cpp:105 6: ./Castro3d.pgi.TPROF.CUDA.ex() [0x81d719] amrex::FillPatchIterator::~FillPatchIterator() inlined at tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_AmrLevel.cpp:2202 in amrex::AmrLevel::FillPatchAdd(amrex::AmrLevel&, amrex::MultiFab&, int, double, int, int, int, int) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_AmrLevel.cpp:1465 amrex::AmrLevel::FillPatchAdd(amrex::AmrLevel&, amrex::MultiFab&, int, double, int, int, int, int) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_AmrLevel.cpp:2202 7: ./Castro3d.pgi.TPROF.CUDA.ex() [0x474a12] Castro::do_advance_ctu(double, double, int, int) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/Castro_advance_ctu.cpp:106 8: ./Castro3d.pgi.TPROF.CUDA.ex() [0x47686e] Castro::subcycle_advance_ctu(double, double, int, int) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/Castro_advance_ctu.cpp:556 9: ./Castro3d.pgi.TPROF.CUDA.ex() [0x464b39] Castro::advance(double, double, int, int) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/Castro_advance.cpp:55 10: ./Castro3d.pgi.TPROF.CUDA.ex() [0x8113e5] amrex::Amr::timeStep(int, double, int, int, double) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_Amr.cpp:2112 11: ./Castro3d.pgi.TPROF.CUDA.ex() [0x8114ec] amrex::Amr::timeStep(int, double, int, int, double) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_Amr.cpp:2173 12: ./Castro3d.pgi.TPROF.CUDA.ex() [0x8058a0] amrex::Amr::coarseTimeStep(double) tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/AMReX_Amr.cpp:2439 13: ./Castro3d.pgi.TPROF.CUDA.ex() [0x40c4ca] main tmp_build_dir/s/3d.pgi.TPROF.CUDA.EXE/main.cpp:143 14: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fbb336041a3] __libc_start_main ??:0 15: ./Castro3d.pgi.TPROF.CUDA.ex() [0x4121ce] _start ??:0 ===== TinyProfilers ====== main() Amr::coarseTimeStep() Amr::timeStep() Amr::timeStep() Castro::advance() Castro::subcycle_advance_ctu() Castro::do_advance_ctu()