https://source.android.com/devices/tech/debug/#debuggerd
Debugging Native Android Platform Code
This page contains a summary of useful tools and related commands fordebugging, tracing, and profiling native Android platform code. The pageswithin this section contain detailed information on other debugging tools foruse during development of platform-level features.
For example, you may learn how to explore system services with Dumpsys and evaluate network and RAM use. Seethe subpages for tools and methods not described below.
debuggerd
When a dynamically-linked executable starts, several signal handlers areregistered that connect to debuggerd
(or debuggerd64)
in the event that signalis sent to the process. The debuggerd
process dumps registers and unwinds thestack.
It's possible for debuggerd
to attach only if nothing else isalready attached. This means that using tools like strace
orgdb
will prevent debuggerd
from working. Also, ifyou call prctl(PR_SET_DUMPABLE, 0)
you can preventdebuggerd
from attaching. This can be useful if you wish toexplicitly opt out of crash reporting.
Here is example output (with timestamps and extraneous information removed):
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** Build fingerprint: 'Android/aosp_flounder/flounder:5.1.51/AOSP/enh08201009:eng/test-keys' Revision: '0' ABI: 'arm' pid: 1656, tid: 1656, name: crasher >>> crasher <<< signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr -------- Abort message: 'some_file.c:123: some_function: assertion "false" failed' r0 00000000 r1 00000678 r2 00000006 r3 f70b6dc8 r4 f70b6dd0 r5 f70b6d80 r6 00000002 r7 0000010c r8 ffffffed r9 00000000 sl 00000000 fp ff96ae1c ip 00000006 sp ff96ad18 lr f700ced5 pc f700dc98 cpsr 400b0010 backtrace: #00 pc 00042c98 /system/lib/libc.so (tgkill+12) #01 pc 00041ed1 /system/lib/libc.so (pthread_kill+32) #02 pc 0001bb87 /system/lib/libc.so (raise+10) #03 pc 00018cad /system/lib/libc.so (__libc_android_abort+34) #04 pc 000168e8 /system/lib/libc.so (abort+4) #05 pc 0001a78f /system/lib/libc.so (__libc_fatal+16) #06 pc 00018d35 /system/lib/libc.so (__assert2+20) #07 pc 00000f21 /system/xbin/crasher #08 pc 00016795 /system/lib/libc.so (__libc_init+44) #09 pc 00000abc /system/xbin/crasher Tombstone written to: /data/tombstones/tombstone_06
This can be pasted into development/scripts/stack
to get a more detailed unwindwith line number information (assuming the unstripped binaries can be found).
Some libraries on the system are built with LOCAL_STRIP_MODULE :=keep_symbols
to provide usable backtraces directly from debuggerd
. This makesyour library or executable slightly larger, but not nearly as large as anunstripped version.
Note also the last line of debuggerd
output --- in addition to dumping asummary to the log, debuggerd
writes a full “tombstone” to disk. This containsa lot of extra information that can be helpful in debugging a crash, inparticular the stack traces for all the threads in the crashing process (notjust the thread that caught the signal) and a full memory map.
Crash dumps
If you don't have a specific crash that you're investigating right now,the platform source includes a tool for testing debuggerd
called crasher. Ifyou mm
in system/core/debuggerd/
you'll get both a crasher
and a crasher64
on your path (the latter allowing you to test64-bit crashes). Crasher can crash in a large number of interesting ways basedon the command line arguments you provide. Use crasher --help
to see the currently supported selection.
To introduce the difference pieces in a crash dump, let's work through the example above:
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
The line of asterisks with spaces is helpful if you're searching a logfor native crashes. The string "*** ***" rarely shows up in logs other thanat the beginning of a native crash.
Build fingerprint: 'Android/aosp_flounder/flounder:5.1.51/AOSP/enh08201009:eng/test-keys'
The fingerprint lets you identify exactly which build the crash occurredon. This is exactly the same as the ro.build.fingerprint
system property.
Revision: '0'
The revision refers to the hardware rather than the software. This isusually unused but can be useful to help you automatically ignore bugs knownto be caused by bad hardware. This is exactly the same as the ro.revision
system property.
ABI: 'arm'
The ABI is one of arm, arm64, mips, mips64, x86, or x86-64. This ismostly useful for the stack
script mentioned above, so that it knowswhat toolchain to use.
pid: 1656, tid: 1656, name: crasher >>> crasher <<<
This line identifies the specific thread in the process that crashed. Inthis case, it was the process' main thread, so the process ID and threadID match. The first name is the thread name, and the name surrounded by>>> and <<< is the process name. For an app, the process nameis typically the fully-qualified package name (such as com.facebook.katana),which is useful when filing bugs or trying to find the app in Google Play. Thepid and tid can also be useful in finding the relevant log lines precedingthe crash.
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
This line tells you which signal (SIGABRT) was received, and more abouthow it was received (SI_TKILL). The signals reported by debuggerd
are SIGABRT,SIGBUS, SIGFPE, SIGILL, SIGSEGV, and SIGTRAP. The signal-specific codes varybased on the specific signal.
Abort message: 'some_file.c:123: some_function: assertion "false" failed'
Not all crashes will have an abort message line, but aborts will. Thisis automatically gathered from the last line of fatal logcat output forthis pid/tid, and in the case of a deliberate abort is likely to give anexplanation of why the program killed itself.
r0 00000000 r1 00000678 r2 00000006 r3 f70b6dc8 r4 f70b6dd0 r5 f70b6d80 r6 00000002 r7 0000010c r8 ffffffed r9 00000000 sl 00000000 fp ff96ae1c ip 00000006 sp ff96ad18 lr f700ced5 pc f700dc98 cpsr 400b0010
The register dump shows the content of the CPU registers at the time thesignal was received. (This section varies wildly between ABIs.) How usefulthese are will depend on the exact crash.
backtrace: #00 pc 00042c98 /system/lib/libc.so (tgkill+12) #01 pc 00041ed1 /system/lib/libc.so (pthread_kill+32) #02 pc 0001bb87 /system/lib/libc.so (raise+10) #03 pc 00018cad /system/lib/libc.so (__libc_android_abort+34) #04 pc 000168e8 /system/lib/libc.so (abort+4) #05 pc 0001a78f /system/lib/libc.so (__libc_fatal+16) #06 pc 00018d35 /system/lib/libc.so (__assert2+20) #07 pc 00000f21 /system/xbin/crasher #08 pc 00016795 /system/lib/libc.so (__libc_init+44) #09 pc 00000abc /system/xbin/crasher
The backtrace shows you where in the code we were at the time ofcrash. The first column is the frame number (matching gdb's style wherethe deepest frame is 0). The PC values are relative to the location of theshared library rather than absolute addresses. The next column is the nameof the mapped region (which is usually a shared library or executable, butmight not be for, say, JIT-compiled code). Finally, if symbols are available,the symbol that the PC value corresponds to is shown, along with the offsetinto that symbol in bytes. You can use this in conjunction with objdump(1)
to find the corresponding assembler instruction.
Tombstones
Tombstone written to: /data/tombstones/tombstone_06
This tells you where debuggerd
wrote extra information.debuggerd
will keep up to 10 tombstones, cycling throughthe numbers 00 to 09 and overwriting existing tombstones as necessary.
The tombstone contains the same information as the crash dump, plus afew extras. For example, it includes backtraces for all threads (notjust the crashing thread), the floating point registers, raw stack dumps,and memory dumps around the addresses in registers. Most usefully it alsoincludes a full memory map (similar to /proc/pid/maps
). Here's anannotated example from a 32-bit ARM process crash:
memory map: (fault address prefixed with --->) --->ab15f000-ab162fff r-x 0 4000 /system/xbin/crasher (BuildId: b9527db01b5cf8f5402f899f64b9b121)
There are two things to note here. The first is that this line is prefixedwith "--->". The maps are most useful when your crash isn't just a nullpointer dereference. If the fault address is small, it's probably some variantof a null pointer dereference. Otherwise looking at the maps around the faultaddress can often give you a clue as to what happened. Some possible issuesthat can be recognized by looking at the maps include:
- Reads/writes past the end of a block of memory.
- Reads/writes before the beginning of a block of memory.
- Attempts to execute non-code.
- Running off the end of a stack.
- Attempts to write to code (as in the example above).
The second thing to note is that executables and shared libraries fileswill show the BuildId (if present) in Android M and later, so you can seeexactly which version of your code crashed. (Platform binaries include aBuildId by default since Android M. NDK r12 and later automatically pass-Wl,--build-id
to the linker too.)
ab163000-ab163fff r-- 3000 1000 /system/xbin/crasher ab164000-ab164fff rw- 0 1000 f6c80000-f6d7ffff rw- 0 100000 [anon:libc_malloc]
On Android the heap isn't necessarily a single region. Heap regions willbe labeled [anon:libc_malloc]
.
f6d82000-f6da1fff r-- 0 20000 /dev/__properties__/u:object_r:logd_prop:s0 f6da2000-f6dc1fff r-- 0 20000 /dev/__properties__/u:object_r:default_prop:s0 f6dc2000-f6de1fff r-- 0 20000 /dev/__properties__/u:object_r:logd_prop:s0 f6de2000-f6de5fff r-x 0 4000 /system/lib/libnetd_client.so (BuildId: 08020aa06ed48cf9f6971861abf06c9d) f6de6000-f6de6fff r-- 3000 1000 /system/lib/libnetd_client.so f6de7000-f6de7fff rw- 4000 1000 /system/lib/libnetd_client.so f6dec000-f6e74fff r-x 0 89000 /system/lib/libc++.so (BuildId: 8f1f2be4b37d7067d366543fafececa2) (load base 0x2000) f6e75000-f6e75fff --- 0 1000 f6e76000-f6e79fff r-- 89000 4000 /system/lib/libc++.so f6e7a000-f6e7afff rw- 8d000 1000 /system/lib/libc++.so f6e7b000-f6e7bfff rw- 0 1000 [anon:.bss] f6e7c000-f6efdfff r-x 0 82000 /system/lib/libc.so (BuildId: d189b369d1aafe11feb7014d411bb9c3) f6efe000-f6f01fff r-- 81000 4000 /system/lib/libc.so f6f02000-f6f03fff rw- 85000 2000 /system/lib/libc.so f6f04000-f6f04fff rw- 0 1000 [anon:.bss] f6f05000-f6f05fff r-- 0 1000 [anon:.bss] f6f06000-f6f0bfff rw- 0 6000 [anon:.bss] f6f0c000-f6f21fff r-x 0 16000 /system/lib/libcutils.so (BuildId: d6d68a419dadd645ca852cd339f89741) f6f22000-f6f22fff r-- 15000 1000 /system/lib/libcutils.so f6f23000-f6f23fff rw- 16000 1000 /system/lib/libcutils.so f6f24000-f6f31fff r-x 0 e000 /system/lib/liblog.so (BuildId: e4d30918d1b1028a1ba23d2ab72536fc) f6f32000-f6f32fff r-- d000 1000 /system/lib/liblog.so f6f33000-f6f33fff rw- e000 1000 /system/lib/liblog.so
Typically a shared library will have three adjacent entries. One will bereadable and executable (code), one will be read-only (read-onlydata), and one will be read-write (mutable data). The first columnshows the address ranges for the mapping, the second column the permissions(in the usual Unix ls(1)
style), the third column the offset into the file(in hex), the fourth column the size of the region (in hex), and the fifthcolumn the file (or other region name).
f6f34000-f6f53fff r-x 0 20000 /system/lib/libm.so (BuildId: 76ba45dcd9247e60227200976a02c69b) f6f54000-f6f54fff --- 0 1000 f6f55000-f6f55fff r-- 20000 1000 /system/lib/libm.so f6f56000-f6f56fff rw- 21000 1000 /system/lib/libm.so f6f58000-f6f58fff rw- 0 1000 f6f59000-f6f78fff r-- 0 20000 /dev/__properties__/u:object_r:default_prop:s0 f6f79000-f6f98fff r-- 0 20000 /dev/__properties__/properties_serial f6f99000-f6f99fff rw- 0 1000 [anon:linker_alloc_vector] f6f9a000-f6f9afff r-- 0 1000 [anon:atexit handlers] f6f9b000-f6fbafff r-- 0 20000 /dev/__properties__/properties_serial f6fbb000-f6fbbfff rw- 0 1000 [anon:linker_alloc_vector] f6fbc000-f6fbcfff rw- 0 1000 [anon:linker_alloc_small_objects] f6fbd000-f6fbdfff rw- 0 1000 [anon:linker_alloc_vector] f6fbe000-f6fbffff rw- 0 2000 [anon:linker_alloc] f6fc0000-f6fc0fff r-- 0 1000 [anon:linker_alloc] f6fc1000-f6fc1fff rw- 0 1000 [anon:linker_alloc_lob] f6fc2000-f6fc2fff r-- 0 1000 [anon:linker_alloc] f6fc3000-f6fc3fff rw- 0 1000 [anon:linker_alloc_vector] f6fc4000-f6fc4fff rw- 0 1000 [anon:linker_alloc_small_objects] f6fc5000-f6fc5fff rw- 0 1000 [anon:linker_alloc_vector] f6fc6000-f6fc6fff rw- 0 1000 [anon:linker_alloc_small_objects] f6fc7000-f6fc7fff rw- 0 1000 [anon:arc4random _rsx structure] f6fc8000-f6fc8fff rw- 0 1000 [anon:arc4random _rs structure] f6fc9000-f6fc9fff r-- 0 1000 [anon:atexit handlers] f6fca000-f6fcafff --- 0 1000 [anon:thread signal stack guard page]
Note that since Android 5.0 (Lollipop), the C library names most of its anonymous mappedregions so there are fewer mystery regions.
f6fcb000-f6fccfff rw- 0 2000 [stack:5081]
Regions named [stack:tid]
are the stacks for the given threads.
f6fcd000-f702afff r-x 0 5e000 /system/bin/linker (BuildId: 84f1316198deee0591c8ac7f158f28b7) f702b000-f702cfff r-- 5d000 2000 /system/bin/linker f702d000-f702dfff rw- 5f000 1000 /system/bin/linker f702e000-f702ffff rw- 0 2000 f7030000-f7030fff r-- 0 1000 f7031000-f7032fff rw- 0 2000 ffcd7000-ffcf7fff rw- 0 21000 ffff0000-ffff0fff r-x 0 1000 [vectors]
Whether you see [vector]
or [vdso]
depends on the architecture. ARM uses [vector], while all other architectures use [vdso].
Native Debugging with GDB
Debugging a running app
To connect to an already-running app or native daemon, use gdbclient
.
Current versions of gdbclient just require the process ID (PID). So to debug a process withPID 1234, simply run:
$ gdbclient 1234
The script will set up port forwarding, start the appropriategdbserver
on the device, start the appropriate gdb
onthe host, configure gdb
to find symbols, and connectgdb
to the remote gdbserver
.
Debugging a native process as it starts
If you want to debug a process as it starts, you’ll need to use gdbserver
or gdbserver64
manually, but that’s easy too:
$ adb shell gdbserver :5039 /system/bin/my_test_app Process my_test_app created; pid = 3460 Listening on port 5039
Identify the app’s PID from the gdbserver
output, and then inanother window:
$ gdbclient <app pid>
Then enter continue at the gdb
prompt.
Note that to debug a 64-bit process, you'll need to use gdbserver64
.The error messages from gdb
if you made the wrong choice are unhelpful(along the lines of Reply contains invalid hex digit 59
).
Debugging processes that crash
If you want debuggerd
to suspend crashed processes so you canattach gdb
, set the appropriate property:
$ adb shell setprop debug.db.uid 999999 # <= M $ adb shell setprop debug.debuggerd.wait_for_gdb true # > M
At the end of the usual crash output, debuggerd
will give youinstructions on how to connect gdb
using the typical command:
$ gdbclient <pid>
Debugging without symbols
If you don’t have symbols, sometimes gdb
will get confused about theinstruction set it is disassembling (ARM or Thumb). The instruction set that ischosen as the default when symbol information is missing can be switchedbetween ARM or Thumb like so:
$ set arm fallback-mode arm # or 'thumb'
Other tools
Valgrind
The following steps show you how to use Valgrind on Android. This tool suite contains anumber of tools including Memcheck for detecting memory-related errors in C andC++.
- To build Valgrind, run:
$ mmma -j6 external/valgrind
- Set up the temporary directory:
$ adb shell mkdir /data/local/tmp $ adb shell chmod 777 /data/local/tmp
- Run the system server with Valgrind:
$ adb shell setprop wrap.system_server "logwrapper valgrind" $ adb shell stop && adb shell start
- For debug symbols, push unstripped libraries to
/data/local/symbols
:$ adb shell mkdir /data/local/symbols $ adb push $OUT/symbols /data/local/symbols
- To use Valgrind during boot up, edit
out/target/product/XXXX/root/init.rc
andchange:
service example /system/bin/foo --arg1 --arg2
to:
service example /system/bin/logwrapper /system/bin/valgrind /system/bin/foo --arg1 --arg2
To see the effects, you need to create aboot.img
and reflash the device.
Systrace
See Systrace ondeveloper.android.com for deriving execution times of applications andother Android system processes.