Documentation on working with breakpad symbol files about symbolic HOT 10 CLOSED

Hi @bsergean, basically what you're describing is correct. The way we do it:

1. Run the minidump processor to get all modules involved in the crash
2. Optionally load CFI for these modules and run the processor again for better stack traces
3. Use symcaches to look up each frame and get the function / file / line info.

This process is mostly targeted at Sentry, where generating an initial (unsymbolicated) stack trace is separated from processing frames. Symbolic does not assume anything about your symbol handling, nor it helps with it.

To get an example for how to piece together the entire puzzle, you can have a look at the minidump_stackwalk example, which also scans for symbols in the file system, generates symcaches on the fly and then outputs the symbolicated stack trace. Note that it is not intended to be an exact reimplementation of the breakpad tool with the same name.

In python, there is no single test for the entire chain. However, have a look at test_linux_with_cfi for an example on how to load CFI and get the stack trace, and then at test_symcache for symbolicating a single frame. Note that you don't have to write the symcache to disk as you could use it directly.

Hope this helps. I will keep this issue open, as this part really deserves better documentation. As I said before, this library is currently targeted at the use cases we have in Sentry, and we are still making regular changes to its fundamentals.

from symbolic.

Excellent, thanks for the detailed answer.

I think a good way to progress for me would be to write a minidump_stackwalk.py which works the same way as the one from google breakpad. Possibly that one could be stuffed in the example folder ?

from symbolic.

@bsergean did you ever get around to writing the stackwalk tool? Looking into something similar (using breakpad in a Qt app that we want to dump into sentry)

from symbolic.

@asgoel Can I ask what's in your way of simply uploading minidumps to Sentry?

from symbolic.

@jan-auer nothing right now. We're planning on just uploading our symbols and sending minidumps to Sentry. But, we still want to make sure we have a good understanding of how the process work in case we ever decide to do it on our side (either before uploading to Sentry or if running Sentry/something else ourselves).

from symbolic.

@jan-auer is it expected to see a stark difference in the ability for the paid Sentry plan to symbolicate a minidump vs the ability for the breakpad stackwalker to do so? They are using the exact same symbols as far as I can tell, yet the stackwalker can resolve far many more than we're seeing when we send a dump to sentry. Let me know if this is better resolved through Sentry support.

from symbolic.

@asgoel Thanks for letting me now, we should definitely not get worse results than the breakpad stackwalker. That being said, minidump support is still in development and while our symbolic is already pretty feature-complete, we haven't integrated everything into sentry.io yet.

Symbolic is using parts of the Breakpad stackwalker internally (like almost all other tools out there), but we've made some changes to the way it works and are also preparing to switch to a different stack walker entirely. You _should_™️ get mostly the same results with a couple of key differences:

• At the moment, Sentry does not honor CFI (call frame information). This will make a huge difference on x86/i686 when compiling without frame pointers. Adding support for this is already in our pipeline, but I cannot give you a final timeline on this. See here for more information.
• For Darwin and Linux we can use native debug symbols in place of the ASCII .sym files if you upload them. In this case, we can also expand inlined function calls and sometimes get more accurate results. We use custom code to extract debugging entries and CFI, so results will definitely vary here.
• We're using a custom demangler instead of breakpad's own implemenation (which basically uses cxxabi if I remember correctly). So even if we extract the same stack trace, the demangled name might look slightly different (and hopefully more accurate).

I suppose the differences you're seeing is because of CFI. But to double-check, it would be great if you could send me some test files (symbols, minidumps and symbolicated stack traces) so I can verify.

from symbolic.

@jan-auer I can send you some test files. What's a good email to reach you at?

from symbolic.

Awesome! Please send them to [email protected]

from symbolic.

The reason was indeed missing call frame information (CFI). I was not aware that this is also such a big issue on arm. We are planning to address this in July or August, and you should start seeing improvements then. I will post an update to getsentry/sentry#8193 once we've got something ready.

Thank you for providing test data!

from symbolic.