简述:
breakpad是一个比Linux core机制更强大的、用于记录程序崩溃时信息的工具。
breakpad是google开发的一个跨平台C/C++ dump捕获开源库和工具套件,可用来发布删除了编译器调试信息的应用程序。在程序崩溃时,将崩溃信息记录在一个小巧的微软minidump格式文件中,支持将其发送回服务器,并且可以从这些minidump和符号文件来生成C和C++堆栈跟踪。breakpad可以在程序崩溃时触发dump写入操作,也可以在没有触发dump时主动写dump文件。breakpad支持windows、linux、macos、android、ios等。目前已有Google Chrome, Firefox, Google Picasa, Camino, Google Earth等项目使用。
本文将主要介绍brekapad的源码编译和linux上breakpad的使用。
本地系统:Ubuntu 16.04
交叉编译器:arm-linux-gnueabihf
Qt版本:5.12.8
1、工作原理
1.1 工作示意图
工作示意图如上图所示。图片左上角是一个完整的应用程序,它包含了三部分:程序本身的代码、breakpad 客户端信息、调试信息。
1> Build System
在Build System中breakpad的symbol生成工具借助应用层序中的Debugging Information这一部分生成一个Google自己的符号文件,这个符号文件类似于windows上的pdb文件,最终呢在发布应用层序的时候使用strip将调式信息去除。
2> User's System
在User's System中运行的应用程序是通过strip去除了调式信息的,倘若应用程序发生Crash,Breakpad client就会写minidump文件到指定目录,并且可以将产生的minidump文件发送到远端服务器即Crash Colletcor。
3> Crash Collector
在Crash Collector就可以利用Build System中产生的symol文件和User's System中上报的minidump文件生成用户可读的stack trace。
1.2 breakpad 三大组件
breakpad有三个主要的组件:breakpad client, symbol dumper,processor module
1> breakpad client
breakpad client是breakpad的客户端静态库(即:libbreakpad_client.a),主要提供给应用程序使用。它的主要作用是在程序崩溃后,接管程序的异常处理(生成minidump文件以及上传minidump文件),具体来说,它主要做了两方面的事情。
a. 响应程序崩溃时接收到的signal,包括:SIGSEGV,SIGABRT,SIGFPE,SIGILL,SIGBUS。 (另外两个SIGSTOP,SIGKILL无法处理)
b. 获取程序崩溃那一刻的运行时信息,保存为一个minidump格式的文件。
2> symbol dumper
symbol dumper是调试信息文件生成程序(即:dump_syms)。主要是用来从可执行程序中提取与符号相关的信息,并保存为一种特定格式的文件(symbol file, 是google自己的格式)。
3> processor module
processor module是minidump 处理程序(即:minidump_stackwalk),它的作用就是根据coredump及symbol file,构建出可读的call stack。
2、源码编译
2.1 源码下载
1>下载一
下载breakpad源码 :git clone https://github.com/google/breakpad
下载breakpad依赖的三方库 LSS :git clone https://github.com/adelshokhy112/linux-syscall-support,将 LSS 中的linux_syscall_support.h文件放至breakpad/src/third_party/lss/ 目录下
2 > 下载二
2.2 编译
解压缩后可在源目录下编译
1> Windows平台编译使用google breakpad
2> Linux(Ubuntu)平台编译breakpad
./configure
make -j10
make install
编译输出完整信息请看附录一
3> Linux (arm)交叉编译
例如交叉编译器为 arm-linux-gnueabihf
./configure --prefix=/root/google_beakpad/beakpad-master/linux --host=arm-linux-gnueabihf
make -j10
make install
编译输出完整信息请看附录二
3、在Linux程序使用breakpad
breakpad分为 in-process
和 out-process,这里主要介绍
in-process。
3.1 加载库
1 > 将编译好后生成的include文件夹及lib文件夹拷贝到项目中
交叉编译的在--prefix生成路径下查找,如/root/google_beakpad/beakpad-master/linux
root@ubuntu:~/google_beakpad/beakpad-master/linux# ls include/
breakpad
root@ubuntu:~/google_beakpad/beakpad-master/linux# ls lib/
libbreakpad.a libbreakpad_client.a pkgconfig
root@ubuntu:~/google_beakpad/beakpad-master/linux# ls bin/
core2md MainboardTooling.sym minidump_dump sym_upload
dump_syms microdump_stackwalk minidump_stackwalk
MainboardTooling minidump-2-core minidump_upload
root@ubuntu:~/google_beakpad/beakpad-master/linux# ls share/
doc
root@ubuntu:~/google_beakpad/beakpad-master/linux#
本地直接编译的在/usr/local/下查找。
root@ubuntu:/usr/local# ls include/
breakpad gstreamer-1.0 valgrind
root@ubuntu:/usr/local# ls bin/
callgrind_annotate gst-discoverer-1.0 minidump_stackwalk
callgrind_control gst-inspect-1.0 minidump_upload
cg_annotate gst-launch-1.0 ms_print
cg_diff gst-play-1.0 sym_upload
cg_merge gst-stats-1.0 valgrind
core2md gst-typefind-1.0 valgrind-di-server
dump_syms microdump_stackwalk valgrind-listener
dump_syms_mac minidump-2-core vgdb
gst-device-monitor-1.0 minidump_dump
root@ubuntu:/usr/local#
2> 在.pro文件或.pri文件加载库和头文件
LIBS += $$PWD/breakpad/lib/arm/libbreakpad_client.a
INCLUDEPATH += $$PWD/breakpad/include/breakpad
3.2 函数调用
在main.cpp调用。
#include "client/linux/handler/exception_handler.h"
static bool dumpCallback(const google_breakpad::MinidumpDescriptor &descriptor,
void *context,
bool succeeded);
int main(int argc, char *argv[])
{
// 加载崩溃信息捕获
QString curDateTime = QDateTime::currentDateTime().toString("yyyyMMddhhmmss");
QDir dir(DMP_ROOT + curDateTime);
if(!dir.exists())
{
dir.mkpath(DMP_ROOT + curDateTime);
}
google_breakpad::MinidumpDescriptor descriptor(QString(DMP_ROOT + curDateTime).toStdString());
google_breakpad::ExceptionHandler eh(descriptor,
NULL,
dumpCallback,
NULL,
true,
-1);
//...
return a.exec();
}
static bool dumpCallback(const google_breakpad::MinidumpDescriptor &descriptor,
void *context,
bool succeeded)
{
Q_UNUSED(context)
printf("Dump path: %s\n", descriptor.path());
return succeeded;
}
3.3 生成symbol文件
通过上面的步骤还是只有dump文件,需要通过google提供的dump_syms二进制程序给应用程序生成symbol文件,之后通过symbol文件和dump文件就可以产生可读堆栈信息。
例如程序为project,则生成symbol文件命令为
root@ubuntu:~# dump_syms project > project.sym
为了将symbol符号文件与minidump_stackwalk工具一起使用,还需要将它们放在特定的目录结构中,才能够被正确读取。 符号文件的第一行包含生成此目录结构所需的信息。命令如下:
//生成符号文件
root@ubuntu:~# dump_syms project > project.sym
root@ubuntu:~# head -n1 project.sym
MODULE Linux arm A0D9BDF362F2A47AC66F9FFB3010F3260 project
root@ubuntu:~# mkdir -p ./symbols/project/A0D9BDF362F2A47AC66F9FFB3010F3260/
root@ubuntu:~# mv project.sym ./symbols/project/A0D9BDF362F2A47AC66F9FFB3010F3260
//根据minnidump和symbol文件生成stack trace
root@ubuntu:~# minidump_stackwalk xxx.dmp ./symbols/ > result.log
3.4 生成stack trace
根据minnidump和symbol文件生成stack trace。
root@ubuntu:~# minidump_stackwalk xxx.dmp ./symbols/ > result.log
最终生成的result.log就是可读堆栈信息文件。
3.3~3.4 使用脚本生成更简单
How To Add Breakpad To Your Linux Application
附录一 Linux(Ubuntu)编译完整输出信息
root@ubuntu:~/google_beakpad/beakpad-master# ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '0' is supported by ustar format... yes
checking whether GID '0' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether to enable maintainer-specific portions of Makefiles... no
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for ar... ar
checking the archiver (ar) interface... ar
checking dependency style of gcc... gcc3
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking how to run the C preprocessor... gcc -E
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ranlib... ranlib
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking a.out.h usability... yes
checking a.out.h presence... yes
checking for a.out.h... yes
checking sys/random.h usability... no
checking sys/random.h presence... no
checking for sys/random.h... no
checking for arc4random... no
checking for getrandom... no
checking whether g++ supports C++11 features by default... no
checking whether g++ supports C++11 features with -std=c++11... yes
checking whether C++ compiler accepts -Werror=unknown-warning-option... no
checking whether C++ compiler accepts -Wmissing-braces... yes
checking whether C++ compiler accepts -Wnon-virtual-dtor... yes
checking whether C++ compiler accepts -Woverloaded-virtual... yes
checking whether C++ compiler accepts -Wreorder... yes
checking whether C++ compiler accepts -Wsign-compare... yes
checking whether C++ compiler accepts -Wunused-local-typedefs... yes
checking whether C++ compiler accepts -Wunused-variable... yes
checking whether C++ compiler accepts -Wvla... yes
checking for O_CLOEXEC defined in fcntl.h... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating breakpad.pc
config.status: creating breakpad-client.pc
config.status: creating Makefile
config.status: creating src/config.h
config.status: executing depfiles commands
root@ubuntu:~/google_beakpad/beakpad-master# make -j10
depbase=`echo src/processor/stackwalk_common.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/stackwalk_common.o -MD -MP -MF $depbase.Tpo -c -o src/processor/stackwalk_common.o src/processor/stackwalk_common.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-crc32.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-crc32.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-crc32.o `test -f 'src/common/linux/crc32.cc' || echo './'`src/common/linux/crc32.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-dump_symbols.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-dump_symbols.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-dump_symbols.o `test -f 'src/common/linux/dump_symbols.cc' || echo './'`src/common/linux/dump_symbols.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-elf_symbols_to_module.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-elf_symbols_to_module.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-elf_symbols_to_module.o `test -f 'src/common/linux/elf_symbols_to_module.cc' || echo './'`src/common/linux/elf_symbols_to_module.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-elfutils.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-elfutils.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-elfutils.o `test -f 'src/common/linux/elfutils.cc' || echo './'`src/common/linux/elfutils.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-file_id.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-file_id.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-file_id.o `test -f 'src/common/linux/file_id.cc' || echo './'`src/common/linux/file_id.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-linux_libc_support.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-linux_libc_support.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-linux_libc_support.o `test -f 'src/common/linux/linux_libc_support.cc' || echo './'`src/common/linux/linux_libc_support.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-memory_mapped_file.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-memory_mapped_file.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-memory_mapped_file.o `test -f 'src/common/linux/memory_mapped_file.cc' || echo './'`src/common/linux/memory_mapped_file.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -g -O2 -std=c++11 -MT src/common/linux/src_tools_linux_dump_syms_dump_syms-safe_readlink.o -MD -MP -MF src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-safe_readlink.Tpo -c -o src/common/linux/src_tools_linux_dump_syms_dump_syms-safe_readlink.o `test -f 'src/common/linux/safe_readlink.cc' || echo './'`src/common/linux/safe_readlink.cc
depbase=`echo src/common/linux/http_upload.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/common/linux/http_upload.o -MD -MP -MF $depbase.Tpo -c -o src/common/linux/http_upload.o src/common/linux/http_upload.cc &&\
mv -f $depbase.Tpo $depbase.Po
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-linux_libc_support.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-linux_libc_support.Po
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-safe_readlink.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-safe_readlink.Po
depbase=`echo src/common/linux/symbol_upload.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/common/linux/symbol_upload.o -MD -MP -MF $depbase.Tpo -c -o src/common/linux/symbol_upload.o src/common/linux/symbol_upload.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cfi_to_module.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cfi_to_module.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cfi_to_module.o `test -f 'src/common/dwarf_cfi_to_module.cc' || echo './'`src/common/dwarf_cfi_to_module.cc
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-memory_mapped_file.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-memory_mapped_file.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cu_to_module.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cu_to_module.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cu_to_module.o `test -f 'src/common/dwarf_cu_to_module.cc' || echo './'`src/common/dwarf_cu_to_module.cc
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-crc32.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-crc32.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-dwarf_line_to_module.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf_line_to_module.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-dwarf_line_to_module.o `test -f 'src/common/dwarf_line_to_module.cc' || echo './'`src/common/dwarf_line_to_module.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-language.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-language.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-language.o `test -f 'src/common/language.cc' || echo './'`src/common/language.cc
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-elfutils.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-elfutils.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-md5.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-md5.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-md5.o `test -f 'src/common/md5.cc' || echo './'`src/common/md5.cc
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-elf_symbols_to_module.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-elf_symbols_to_module.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-module.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-module.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-module.o `test -f 'src/common/module.cc' || echo './'`src/common/module.cc
mv -f src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-language.Tpo src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-language.Po
mv -f src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-md5.Tpo src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-md5.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-path_helper.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-path_helper.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-path_helper.o `test -f 'src/common/path_helper.cc' || echo './'`src/common/path_helper.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-stabs_reader.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-stabs_reader.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-stabs_reader.o `test -f 'src/common/stabs_reader.cc' || echo './'`src/common/stabs_reader.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/src_tools_mac_dump_syms_dump_syms_mac-stabs_to_module.o -MD -MP -MF src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-stabs_to_module.Tpo -c -o src/common/src_tools_mac_dump_syms_dump_syms_mac-stabs_to_module.o `test -f 'src/common/stabs_to_module.cc' || echo './'`src/common/stabs_to_module.cc
mv -f src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-path_helper.Tpo src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-path_helper.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-bytereader.o -MD -MP -MF src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-bytereader.Tpo -c -o src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-bytereader.o `test -f 'src/common/dwarf/bytereader.cc' || echo './'`src/common/dwarf/bytereader.cc
mv -f src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf_line_to_module.Tpo src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf_line_to_module.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-dwarf2diehandler.o -MD -MP -MF src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf2diehandler.Tpo -c -o src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-dwarf2diehandler.o `test -f 'src/common/dwarf/dwarf2diehandler.cc' || echo './'`src/common/dwarf/dwarf2diehandler.cc
mv -f src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-stabs_reader.Tpo src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-stabs_reader.Po
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-dwarf2reader.o -MD -MP -MF src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf2reader.Tpo -c -o src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-dwarf2reader.o `test -f 'src/common/dwarf/dwarf2reader.cc' || echo './'`src/common/dwarf/dwarf2reader.cc
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/third_party/mac_headers -DHAVE_MACH_O_NLIST_H -g -O2 -std=c++11 -MT src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-elf_reader.o -MD -MP -MF src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-elf_reader.Tpo -c -o src/common/dwarf/src_tools_mac_dump_syms_dump_syms_mac-elf_reader.o `test -f 'src/common/dwarf/elf_reader.cc' || echo './'`src/common/dwarf/elf_reader.cc
mv -f src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-bytereader.Tpo src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-bytereader.Po
depbase=`echo src/processor/basic_code_modules.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/basic_code_modules.o -MD -MP -MF $depbase.Tpo -c -o src/processor/basic_code_modules.o src/processor/basic_code_modules.cc &&\
mv -f $depbase.Tpo $depbase.Po
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-file_id.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-file_id.Po
depbase=`echo src/processor/basic_source_line_resolver.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/basic_source_line_resolver.o -MD -MP -MF $depbase.Tpo -c -o src/processor/basic_source_line_resolver.o src/processor/basic_source_line_resolver.cc &&\
mv -f $depbase.Tpo $depbase.Po
mv -f src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cfi_to_module.Tpo src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf_cfi_to_module.Po
depbase=`echo src/processor/call_stack.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/call_stack.o -MD -MP -MF $depbase.Tpo -c -o src/processor/call_stack.o src/processor/call_stack.cc &&\
mv -f $depbase.Tpo $depbase.Po
mv -f src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf2diehandler.Tpo src/common/dwarf/.deps/src_tools_mac_dump_syms_dump_syms_mac-dwarf2diehandler.Po
depbase=`echo src/processor/cfi_frame_info.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/cfi_frame_info.o -MD -MP -MF $depbase.Tpo -c -o src/processor/cfi_frame_info.o src/processor/cfi_frame_info.cc &&\
mv -f $depbase.Tpo $depbase.Po
mv -f src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-module.Tpo src/common/.deps/src_tools_mac_dump_syms_dump_syms_mac-module.Po
depbase=`echo src/processor/disassembler_x86.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/disassembler_x86.o -MD -MP -MF $depbase.Tpo -c -o src/processor/disassembler_x86.o src/processor/disassembler_x86.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/processor/dump_context.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/dump_context.o -MD -MP -MF $depbase.Tpo -c -o src/processor/dump_context.o src/processor/dump_context.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/processor/dump_object.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/dump_object.o -MD -MP -MF $depbase.Tpo -c -o src/processor/dump_object.o src/processor/dump_object.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/processor/exploitability.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/exploitability.o -MD -MP -MF $depbase.Tpo -c -o src/processor/exploitability.o src/processor/exploitability.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/processor/exploitability_linux.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/exploitability_linux.o -MD -MP -MF $depbase.Tpo -c -o src/processor/exploitability_linux.o src/processor/exploitability_linux.cc &&\
mv -f $depbase.Tpo $depbase.Po
mv -f src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-dump_symbols.Tpo src/common/linux/.deps/src_tools_linux_dump_syms_dump_syms-dump_symbols.Po
depbase=`echo src/processor/exploitability_win.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/exploitability_win.o -MD -MP -MF $depbase.Tpo -c -o src/processor/exploitability_win.o src/processor/exploitability_win.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/processor/fast_source_line_resolver.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-compare -Wunused-local-typedefs -Wunused-variable -Wvla -Werror -fPIC -g -O2 -std=c++11 -MT src/processor/fast_source_line_resolver.o -MD -MP -MF $depbase.Tpo -c -o src/processor/fast_source_line_resolver.o src/processor/fast_source_line_resolver.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo src/processor/logging.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./src -I./src -Wmissing-braces -Wnon-virtual-dtor -Woverloaded-virtua