CentOS7 bcc 与 bpftrace环境搭建

CentOS7 bcc 与 bpftrace环境搭建

kernel

升级CentOS内核, 建议升级到 4.18+. 此次使用内核 5.12.17 作为测试版本.

***注: ***

• kernel 5.12 gcc 版本最低要求为 4.9;
• CentOS 7 altarch kernel 5.4.129-200.el7 测试通过 ;

BPF Features by Linux Kernel Version 列出了每个内核版本所支持的BPF特性

dwarves

git clone https://github.com/acmel/dwarves
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr ../dwarves
make 
make install

zstd

$ yum install zstd -y

建议使用github最新版本

$ git clone https://github.com/facebook/zstd
$ cd zstd 
$ make install

build kernel

启用以下配置项:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_FUNCTION_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_HAVE_KPROBES=y
CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y
CONFIG_DEBUG_FS=y

编译 & 安装:

$ make -j16
$ make modules_install -j16
$ sudo make install

llvm & clang tools

$ git clone -b llvmorg-10.0.1 --single-branch https://github.com/llvm/llvm-project.git 
$ mkdir llvm-10.0.1-build && cd llvm-10.0.1-build
$ cmake -G Ninja \
	-DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;libcxx;libcxxabi;libunwind;lldb;compiler-rt;lld;polly'  \
	-DCMAKE_INSTALL_PREFIX=/opt/llvm-10.0.1-el7.x86_64 \
	-DCMAKE_BUILD_TYPE=Release \
	-DLLVM_BINUTILS_INCDIR=/usr/include \
	-DLLVM_TARGETS_TO_BUILD="X86;BPF;WebAssembly" \
	-DLLVM_BUILD_LLVM_DYLIB=ON \
	-DLLVM_OPTIMIZED_TABLEGEN=ON \
	-DLLVM_PARALLEL_LINK_JOBS=1 \
    -DLLVM_LINK_LLVM_DYLIB:BOOL=ON \
	-DPYTHON_EXECUTABLE=/usr/bin/python3 '-DCMAKE_INSTALL_RPATH:BOOL=;' \
    -DCLANG_PLUGIN_SUPPORT:BOOL=ON \
    -DENABLE_LINKER_BUILD_ID:BOOL=ON \
    -DLLVM_ENABLE_EH=ON \
    -DLLVM_ENABLE_RTTI=ON
	../llvm-project/llvm
$ ninja
$ ninja install
$ export PATH=/opt/llvm-10.0.1-el7.x86_64/bin:$PATH

注意
DLLVM_LINK_LLVM_DYLIB 打开后， clang install 时会移除 RPATH 导致 clang 无法找到 libLLVM.so 有2种方法解决:

• 将安装目录${install-prefix}/lib 加入 ld.so.conf;
• 禁用DLLVM_LINK_LLVM_DYLIB ;

libbpf

$ git clone https://github.com/libbpf/libbpf
$ cd libbpf/src
$ make install

bcc

$ yum --enablerepo='centos-sclo-rh' install devtoolset-9 
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_USE_LIBBPF_PACKAGE:BOOL=TRUE -DPYTHON_CMD=python3 -DENABLE_LLVM_SHARED=1 -DCMAKE_INSTALL_PREFIX=/usr .
$ ninja 
$ ninja install

注意
建议使用 devtoolset-9 使用高版本 gcc;

配置-DCMAKE_USE_LIBBPF_PACKAGE:BOOL=TRUE 启用了bcc使用外部libbpf库；bcc 实际已通过submodule 内嵌了libbpf, 并同步了libbpf 版本, bcc release 有每个版本的详细说明;建议直接使用内嵌libbpf模式, 这样不会遇到libbpf版本不匹配的问题;

内嵌libbpf 编译:

$ git submodule init
$ git submodule update --remote
$ mkdir build && cd build
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DPYTHON_CMD=python3 -DENABLE_LLVM_SHARED=1 -DCMAKE_INSTALL_PREFIX=/usr ../
$ ninja 
$ ninja install

bpftrace

$ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_TESTING:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=OFF -DCMAKE_INSTALL_PREFIX=/usr .
$ make
$ make install

注意
bpftrace 默认使用bccAPI 作为加载bpf的接口, 只有当识别bcc失败时才会启用libbpf API.

test

注: bcc 安装后默认python默认使用的解释器路径依然是 /usr/bin/python, 需手动改为 /usr/bin/python3

$ find /usr/share/bcc/tools -maxdepth 1 -type f -exec sed -i "s/python$/python3/g" {} ';'

bcc hello_world.py

$ sudo /usr/share/bcc/examples/hello_world.py
b'        ksmtuned-1248    [015] d... 46906.257934: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-11919   [024] d... 46906.284752: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-1248    [015] d... 46906.417588: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-11921   [009] d... 46906.417973: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-11922   [026] d... 46906.424017: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-11921   [011] d... 46906.689568: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-11921   [011] d... 46906.689760: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-1248    [015] d... 46906.691458: bpf_trace_printk: Hello, World!'
b''
b'        ksmtuned-1248    [015] d... 46906.692855: bpf_trace_printk: Hello, World!'
b''
^CTraceback (most recent call last):
  File "/usr/share/bcc/examples/hello_world.py", line 12, in <module>
    BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 1339, in trace_print
    line = self.trace_readline(nonblocking=False)
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 1319, in trace_readline
    line = trace.readline(1024).rstrip()
KeyboardInterrupt

bcc tcpstates

$ sudo /usr/share/bcc/tools/tcpstates 
SKADDR           C-PID C-COMM     LADDR           LPORT RADDR           RPORT OLDSTATE    -> NEWSTATE    MS
ffff893426a62300 0     swapper/8  0.0.0.0         22    0.0.0.0         0     LISTEN      -> SYN_RECV    0.000
ffff893426a62300 0     swapper/8  192.168.22.170  22    192.168.22.42   44826 SYN_RECV    -> ESTABLISHED 0.029
ffff893426a62300 0     swapper/8  192.168.22.170  22    192.168.22.42   44826 ESTABLISHED -> CLOSE_WAIT  1998.229
ffff893426a62300 12185 sshd       192.168.22.170  22    192.168.22.42   44826 CLOSE_WAIT  -> LAST_ACK    27.110
ffff893426a62300 0     swapper/8  192.168.22.170  22    192.168.22.42   44826 LAST_ACK    -> CLOSE       0.052

bpftrace tcplift.bt

$ sudo bpftrace  /usr/share/bpftrace/tools/tcplife.bt 
Attaching 3 probes...
PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
11939 sshd       192.168.22.170  22    192.168.22.42   49230      2   244 4115
^C