[Android稳定性] NativeCrash分析(二)-Debug材料及工具

一.分析材料

1.1 tombstone文件

位置:data/tombstones/中,最多存10个,超过会从最旧的复写掉.

chengang@mi:~/Documents/gdb_file$ adb shell
phoenix:/ # cd data/tombstones/                                                                                                                
phoenix:/data/tombstones # ls -la
total 6155
drwxrwx--x  6 system     system    3488 2019-12-25 17:14 .
drwxrwx--x 55 system     system    4096 2019-12-25 15:27 ..
drwxrwx--x  2 system     system    3488 1970-02-27 06:56 dsps
drwxrwx--x  2 system     system    3488 1970-02-27 06:56 lpass
drwxrwx--x  2 system     system    3488 1970-02-27 06:56 modem
-rw-r-----  1 tombstoned system 1103345 2019-12-25 14:52 tombstone_00
-rw-r-----  1 tombstoned system 1272737 2019-12-25 15:28 tombstone_01
-rw-r-----  1 tombstoned system 1434693 2019-12-25 15:29 tombstone_02
-rw-r-----  1 tombstoned system 1245462 2019-12-25 17:10 tombstone_03
-rw-r-----  1 tombstoned system 1211508 2019-12-25 17:14 tombstone_04
drwxrwx--x  2 system     system    3488 1970-02-27 06:56 wcnss

tombstone信息:

日后详细的tombstone文件分析将写一篇,先在这立个flag

1.2 Symbol文件

GCC编译加-g参数编译器会在目标文件中加上调试信息符号的行号信息等,android项目上在out/target/product/xxxx/symbol下对应so的目录下:

使用strip命令去除debug等无用信息,瘦身作用.
多出来的section即是debug信息部分,包含符号所在行数.
看出symbol文件中有较多的debug section,掌握这些section的作用在分析问题时便可以获得更多的debug信息.

1.3 coredump

文件作用:
当进程意外终止时,系统可以将该进程的地址空间的内容及终止时的一些其他信息转储到coredump中,也是elf文件
可以使用GDB对进程异常时的现场进行调试查找问题.
抓取coredump的方法需要提前设置

coredump中提取oat文件的方法:https://www.cnblogs.com/YYPapa/p/6851259.html

二. 分析工具

2.1 addr2line

功能作用:用来分析单个pc地址对应的源码行数

工具位置:

chengang@mi:~$ which addr2line
/usr/bin/addr2line

使用方法:

chengang@mi:~$ addr2line -h
Usage: addr2line [option(s)] [addr(s)]
 Convert addresses into line number/file name pairs.
 If no addresses are specified on the command line, they will be read from stdin
 The options are:
  @<file>                Read options from <file>
  -a --addresses         Show addresses
  -b --target=<bfdname>  Set the binary file format
  -e --exe=<executable>  Set the input file name (default is a.out)
  -i --inlines           Unwind inlined functions
  -j --section=<name>    Read section-relative offsets instead of addresses
  -p --pretty-print      Make the output easier to read for humans
  -s --basenames         Strip directory names
  -f --functions         Show function names
  -C --demangle[=style]  Demangle function names
  -h --help              Display this information
  -v --version           Display the program's version
 
addr2line: supported targets: elf64-x86-64 elf32-i386 elf32-iamcu elf32-x86-64 a.out-i386-linux pei-i386 pei-x86-64 elf64-l1om elf64-k1om elf64-little elf64-big elf32-little elf32-big pe-x86-64 pe-bigobj-x86-64 pe-i386 plugin srec symbolsrec verilog tekhex binary ihex
Report bugs to <http://www.sourceware.org/bugzilla/>

使用举例:

注意如果有inline的函数,添加-i参数

2.2 ndk-stack

工具位置:在Sdk的ndk目录下,如果没有ndk目录,可以使用Sdk-manager下载Sdk-tool中的NDK support工具,即有ndk-stack脚本工具

chengang@mi:~/Android/Sdk$ find . -iname "ndk-stack"
./ndk/20.1.5948944/ndk-stack
./ndk/20.1.5948944/prebuilt/linux-x86_64/bin/ndk-stack

使用方法:

chengang@mi:~/Android/Sdk/ndk/20.1.5948944$ ./ndk-stack -h
usage: ndk-stack.py [-h] -sym SYMBOL_DIR [-i INPUT]
 
Symbolizes Android crashes.
 
optional arguments:
  -h, --help            show this help message and exit
  -sym SYMBOL_DIR, --sym SYMBOL_DIR
                        directory containing unstripped .so files
  -i INPUT, -dump INPUT, --dump INPUT
                        input filename
 
See <https://developer.android.com/ndk/guides/ndk-stack>.

使用举例:

当然也可以使用自己写的脚本,实现原理基本都是批量addr2line
https://blog.csdn.net/TaylorPotter/article/details/86522986

2.3 c++filt

功能作用:
对于被编译器转换过的函数名,可以通过c++filt工具查看原始函数

函数签名:所有的符号都以"_Z"开头,对于嵌套的名字(在命名空间或类里面的),后面紧跟"N",然后是各个名称空间和类的名字,每个名字前是名字字符串长度,再以E结尾,对于一个函数来说,他的参数列表紧跟在"E"后面,对于int即"i",void即"v"
工具位置:

chengang@mi:~/miui/miui_code/g7b_q_dev_11_20/prebuilts$ which c++filt
/usr/bin/c++filt

使用方法:

chengang@mi:~$ c++filt -h
Usage: c++filt [options] [mangled names]
Options are:
  [-_|--strip-underscore]     Ignore first leading underscore
  [-n|--no-strip-underscore]  Do not ignore a leading underscore (default)
  [-p|--no-params]            Do not display function arguments
  [-i|--no-verbose]           Do not show implementation details (if any)
  [-t|--types]                Also attempt to demangle type encodings
  [-s|--format {none,auto,gnu,lucid,arm,hp,edg,gnu-v3,java,gnat,dlang}]
  [@<file>]                   Read extra options from <file>
  [-h|--help]                 Display this information
  [-v|--version]              Show the version information
Demangled names are displayed to stdout.
If a name cannot be demangled it is just echoed to stdout.
If no names are provided on the command line, stdin is read.
Report bugs to <http://www.sourceware.org/bugzilla/>.

使用举例:
tombstone中的frame,方法名优化过:

还原后:

2.4 Objdump

功能作用:用来把相应的so变成汇编语言的asm文件
工具位置:

chengang@mi:~/miui/miui_code/g7b_q_dev_11_20/prebuilts$ find -iname Objdump
./gcc/linux-x86/aarch64/aarch64-linux-android-4.9/aarch64-linux-android/bin/objdump
./gcc/linux-x86/host/x86_64-w64-mingw32-4.8/x86_64-w64-mingw32/bin/objdump
./gcc/linux-x86/host/x86_64-linux-glibc2.17-4.8/x86_64-linux/bin/objdump
./gcc/linux-x86/x86/x86_64-linux-android-4.9/x86_64-linux-android/bin/objdump
./gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin/objdump
./gcc/linux-x86/arm/arm-eabi-4.8/arm-eabi/bin/objdump
./go/linux-x86/pkg/tool/linux_amd64/objdump
./go/linux-x86/src/cmd/objdump
./tools/gcc-sdk/objdump

使用方法:

chengang@mi:~/miui/miui_code/g7b_q_dev_11_20/prebuilts$ ./gcc/linux-x86/x86/x86_64-linux-android-4.9/x86_64-linux-android/bin/objdump --help
Usage: ./gcc/linux-x86/x86/x86_64-linux-android-4.9/x86_64-linux-android/bin/objdump <option(s)> <file(s)>
 Display information from object <file(s)>.
 At least one of the following switches must be given:
  -a, --archive-headers    Display archive header information
  -f, --file-headers       Display the contents of the overall file header
  -p, --private-headers    Display object format specific file header contents
  -P, --private=OPT,OPT... Display object format specific contents
  -h, --[section-]headers  Display the contents of the section headers
  -x, --all-headers        Display the contents of all headers
  -d, --disassemble        Display assembler contents of executable sections
  -D, --disassemble-all    Display assembler contents of all sections
  -S, --source             Intermix source code with disassembly
  -s, --full-contents      Display the full contents of all sections requested
  -g, --debugging          Display debug information in object file
  -e, --debugging-tags     Display debug information using ctags style
 ......

举例:
上面tombstone_04对应的cam chi override 库的objdump

2.5 IDA工具

https://blog.csdn.net/u012195899/article/details/52938353

2.6 Debuggerd

功能作用:查看目标进程的所有线程的当前调用栈
工具位置:

chengang@mi:~/Documents/gdb_file$ adb shell
phoenix:/ # which debuggerd
/system/bin/debuggerd
phoenix:/ #

使用方法:

phoenix:/ # debuggerd -h                                                                                                                       
usage: debuggerd [-bj] PID
 
-b, --backtrace    just a backtrace rather than a full tombstone
-j                 collect java traces
1|phoenix:/ #
 
 
chengang@mi:~/Documents/gdb_file$ adb shell ps -ef | grep camera
cameraserver  1042     1 0 14:51:52 ?     00:00:30 cameraserver
u0_a63       16781   661 0 15:56:27 ?     00:00:14 com.android.camera
cameraserver 24563     1 0 20:29:35 ?     00:00:01 android.hardware.camera.provider@2.4-service_64
chengang@mi:~/Documents/gdb_file$ adb shell debuggerd -b 24563 > camera.trace.txt

2.7 Oatdump

功能作用:解析oat文件
工具位置:
使用方法:
使用案例:

2.8 GDB

功能作用:暂停程序以调试程序
工具位置:

chengang@mi:~/miui/miui_code/g7b_q_dev_11_20/development/scripts$ ls -la | grep -i  "gdbclient*"
-rwxrwxr-x  1 chengang chengang    6119 11月 20 21:21 gdbclient
-rwxrwxr-x  1 chengang chengang   14033 11月 20 21:21 gdbclient.py

使用方法:

chengang@mi:~/miui/miui_code/g7b_q_dev_11_20$ gdbclient.py --help
usage: gdbclient.py [-h] [--adb ADB_PATH] [-a | -d | -e | -s SERIAL]
                    (-p PID | -n NAME | -r ...) [--port [PORT]]
                    [--user [USER]] [--setup-forwarding {gdb,vscode}]
                    [--env VAR=VALUE]
 
optional arguments:
  -h, --help            show this help message and exit
  --adb ADB_PATH        use specific adb command
  --port [PORT]         override the port used on the host [default: 5039]
  --user [USER]         user to run commands as on the device [default: root]
  --setup-forwarding {gdb,vscode}
                        Setup the gdbserver and port forwarding. Prints
                        commands or .vscode/launch.json configuration needed
                        to connect the debugging client to the server.
  --env VAR=VALUE       set environment variable when running a binary
 
device selection:
  -a                    directs commands to all interfaces
  -d                    directs commands to the only connected USB device
  -e                    directs commands to the only connected emulator
  -s SERIAL             directs commands to device/emulator with the given
                        serial
 
attach target:
  -p PID                attach to a process with specified PID
  -n NAME               attach to a process with specified name
  -r ...                run a binary on the device, with args

使用案例:GDB常用命令
常用的功能有:
1.设置断点(条件断点),打印调用栈
2.查看各线程等信息
线程调用栈,切换到调用栈的第n层,附近代码,汇编代码查看,寄存器查看,变量查看(值或类型),查看内存信息,查看变量偏移
3.单步调试
改变变量值
4.对coredump进行分析调试:coredump分析实操

三.总结

屠龙宝刀很厉害,但还是需要一定的内功才能发挥出宝刀的威力
需要一定的技术基础总结如下:

技术基础	说明	参考
操作系统	进程的调度,内存分配的知识,对理解程序运行很有帮助
ARM Architecture	针对android操作系统指令运行及问题分析帮助很大
链接,装载与库	计算机程序运行的基本原理，万变不离其宗	<程序员的自我修养>
ELF spec	elf里面提供很多debug信息，掌握可以方便使用arm-eabi-readelf来看这些debug信息
coredump机制	coredump是进程出问题的现场,是对debug NE最有效的材料
ptrace机制	ptrace是用一个进程debug另外一个进程的机制，这也是GDB的核心实现机制
Linux的signal机制	NE都会伴随signal的发出
业务逻辑	熟悉业务逻辑能更快的了解上下文以更快的分析场景和根本愿意
线程安全	线程安全是一类很大几率引起NE的因素