android-tombstone-symbolication

Android Tombstone .NET Symbolication

Resolves native backtrace frames from .NET Android app crashes (MAUI, Xamarin, Mono) to function names, source files, and line numbers using ELF BuildIds and Microsoft's symbol server.

Inputs: Tombstone file or logcat crash output, llvm-symbolizer (from Android NDK or any LLVM 14+ toolchain), internet access for symbol downloads.

Do not use when: The crash is a managed .NET exception (visible in logcat with a managed stack trace), the crashing library is not a .NET component (e.g., libart.so), or the tombstone is from iOS.

---

Workflow

Step 1: Parse the Tombstone Backtrace

Each backtrace frame has this format:

#NN pc OFFSET  /path/to/library.so (optional_symbol+0xNN) (BuildId: HEXSTRING)

Extract: frame number, PC offset (hex, already library-relative), library name, and BuildId (32–40 hex chars).

Symbolicate all threads by default (background threads like GC/finalizer often have useful .NET frames). The crashing thread's backtrace is listed first; additional threads appear after --- --- --- markers.

Format notes:

• The script auto-detects #NN pc frame lines with or without a backtrace: header, and strips logcat timestamp/tag prefixes automatically.
• Logcat-captured tombstones often omit BuildIds. Recover via adb shell readelf -n, CI build artifacts, or the .NET runtime NuGet package.
• GitHub issue pastes may mangle #1 pc into issue links — replace org/repo#N pc with #N pc before saving to a file.
• If the script fails to parse a format, fall back to manual extraction of #NN pc OFFSET library.so (BuildId: HEX) tuples.

Step 2: Identify .NET Runtime Libraries

Filter frames to .NET runtime libraries:

Library	Runtime
libmonosgen-2.0.so	Mono (MAUI, Xamarin, interpreter)
libcoreclr.so	CoreCLR (JIT mode)
libSystem.*.so	.NET BCL native components (Native, Globalization.Native, IO.Compression.Native, Security.Cryptography.Native.OpenSsl, Net.Security.Native)

NativeAOT: No libcoreclr.so or libmonosgen-2.0.so — the runtime is statically linked into the app binary (e.g., libMyApp.so). The libSystem.*.so BCL libraries remain separate and can be symbolicated via the symbol server. For the app binary itself, you need the app's own debug symbols.

Skip libc.so, libart.so, and other Android system libraries unless the user specifically asks.

Step 3: Download Debug Symbols

For each unique .NET BuildId, download debug symbols:

https://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-<BUILDID>/_.debug

curl -sL "https://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-1eb39fc72918c7c6c0c610b79eb3d3d47b2f81be/_.debug" \
  -o libmonosgen-2.0.so.debug

Verify with file libmonosgen-2.0.so.debug — should show ELF 64-bit ... with debug_info, not stripped. If the download returns 404 or HTML, symbols are not published for that build. Do not add or subtract library base addresses — offsets in tombstones are already library-relative.

Step 4: Symbolicate Each Frame

llvm-symbolizer --obj=libmonosgen-2.0.so.debug -f -C 0x222098

Output:

ves_icall_System_Environment_FailFast
/__w/1/s/src/runtime/src/mono/mono/metadata/icall.c:6244

The /__w/1/s/ prefix is the CI workspace root — the meaningful path starts at src/runtime/, mapping to dotnet/dotnet VMR.

Step 5: Present the Symbolicated Backtrace

Combine original frame numbers with resolved function names and source locations:

#00  libc.so              abort+164
#01  libmonosgen-2.0.so   ves_icall_System_Environment_FailFast        (mono/metadata/icall.c:6244)
#02  libmonosgen-2.0.so   do_icall                                     (mono/mini/interp.c:2457)
#03  libmonosgen-2.0.so   mono_interp_exec_method                      (mono/mini/interp.c)

For unresolved frames (??), keep the original line with BuildId and PC offset.

Automation Script

scripts/Symbolicate-Tombstone.ps1 automates the full workflow:

pwsh scripts/Symbolicate-Tombstone.ps1 -TombstoneFile tombstone_01.txt -LlvmSymbolizer llvm-symbolizer

Flags: -CrashingThreadOnly (limit to crashing thread), -OutputFile path (write to file), -ParseOnly (report libraries/BuildIds/URLs without downloading), -SkipVersionLookup (skip runtime version identification).

---

Finding llvm-symbolizer

Check the Android NDK first: $ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/*/bin/llvm-symbolizer or $ANDROID_HOME/ndk/*/toolchains/llvm/prebuilt/*/bin/llvm-symbolizer. Also available via brew install llvm, apt install llvm, or xcrun --find llvm-symbolizer on macOS.

If unavailable, complete steps 1–3 and present the download commands and llvm-symbolizer commands for the user to run. Do not spend time installing LLVM.

---

Understanding the Output

CI source paths use these prefixes:

Path prefix	Maps to
/__w/1/s/src/runtime/	src/runtime/ in dotnet/dotnet VMR
/__w/1/s/src/mono/	src/mono/ in the VMR (older builds)
/__w/1/s/	VMR root

Runtime Version Identification

The script identifies the exact .NET runtime version by matching BuildIds against locally-installed runtime packs. It searches: SDK packs ($DOTNET_ROOT/packs/), NuGet cache (~/.nuget/packages/), and NuGet.org as an online fallback. When found, it extracts the version and source commit from the .nuspec <repository commit="..." /> element. Pass -SkipVersionLookup to disable. Requires llvm-readelf (auto-discovered from the NDK).

---

Validation

1. file <debug-file> shows ELF ... with debug_info, not stripped
2. At least one .NET frame resolves to a function name (not ??)
3. Resolved paths contain recognizable .NET runtime structure (e.g., mono/metadata/, mono/mini/)

Stop Signals

• No .NET frames found: Report parsed frames and stop.
• All frames resolved: Present symbolicated backtrace. Do not trace into source or attempt to build/debug the runtime.
• Symbols not available (404): One attempt per BuildId, then stop. Report unsymbolicated frames with BuildIds and offsets.
• llvm-symbolizer not available: Use -ParseOnly, present manual commands. Do not install LLVM.

Common Pitfalls

• Missing BuildIds: Logcat tombstones often omit BuildIds. Recover via: adb shell readelf -n /path/to/lib.so, CI build artifacts, or the runtime NuGet package (~/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.android-arm64/<version>/). Prefer pulling raw tombstone files (adb shell cat /data/tombstones/tombstone_XX) which always include BuildIds.
• Symbols not found (404): Pre-release/internal builds may not publish symbols. Check for local unstripped .so/.so.dbg in build artifacts or the NuGet runtime pack.
• NativeAOT: No runtime .so in the tombstone — runtime is in the app binary. libSystem.*.so BCL libraries still work with the symbol server; the app binary needs its own debug symbols.
• Wrong llvm-symbolizer version: Use LLVM 14+ for best DWARF compatibility.
• Multiple BuildIds: Each .NET library has its own BuildId — download symbols for each separately.