Backtrace should print line numbers

[wdconinc]

IMHO the custom backtrace is hardly ever helpful for my day to day work, in particular due to missing line numbers. E.g https://github.com/eic/EICrecon/actions/runs/3870649762/jobs/6597905508#step:5:658 (segfault on debug build when running inside valgrind) does not include line numbers, making it hard to figure out what's going on. When this is happening in a production job or in a CI system, we don't have easy access to the binaries to run addr2line, so having this in the logs would be helpful.

[nathanwbrei]

I sympathize with your complaints about the backtrace. This is not an easy feature to add because while backtrace() comes from execinfo.h, and the symbol names themselves come from libdl, the line numbers would come from libdwarf or libbfd, neither of which are dependencies of JANA2. I would prefer to avoid adding libdwarf for a variety of reasons which I can explain later.

Several alternatives spring to mind:

1. Generate core dumps. (This requires a small tweak in the segfault handler; the bigger question is whether you can enable core dumps on the production machines. I believe this is done by setting ulimit -c unlimited.

2. Fork/exec GDB directly from the segfault handler, potentially scripting it to retrieve any debugging information you need

3. Fork/exec an external program like addr2line which parses the DWARF data for us

What do you think of these alternatives?

References

• https://stackoverflow.com/questions/4636456/how-to-get-a-stack-trace-for-c-using-gcc-with-line-number-information
• https://stackoverflow.com/questions/3151779/best-way-to-invoke-gdb-from-inside-program-to-print-its-stacktrace
• http://www.alexonlinux.com/how-to-handle-sigsegv-but-also-generate-core-dump