【深度】nginx strace和gstack源码调试

一、前提

1 nginx编译安装
假设已经安装好了nginx服务器，切能正常启动访问，个人版本为 nginx version: nginx/1.14.0，使用strace/pstack进行调试（或者gdb，没调试过）。strace 和ltrace 可以查看一个应用程序在运行过程中所发起的系统调用

2 strace命令
strace - trace system calls and signals 跟踪系统调用和信号，最重要的参数 -p，
strace -p pid

Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). -p “pidof PROG” syntax is supported

详细参数，自行查看 man strace（yum install strace）

3 nginx配置
nginx采用master-worker模式，且只有一个worker进程，方便调试

worker_processes  1;
events {
    worker_connections  1024;
}

二、gstack 调试

ps aux | grep nginx | grep -v grep

www      13366  0.0  0.1  21784  2164 ?        S    Mar23   0:00 nginx: worker process
root     15221  0.0  0.1  21784  2132 ?        ts   Mar13   0:00 nginx: master process /usr/local/nginx/sbin/nginx

[@bx16-77-246 /usr/local/nginx]# gstack 13366
#0  0x00007fe98da830f3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x00000000004320b8 in ngx_epoll_process_events (cycle=0xd7e1b0, timer=18446744073709551615, flags=1) at src/event/modules/ngx_epoll_module.c:800
#2  0x0000000000429c48 in ngx_process_events_and_timers (cycle=cycle@entry=0xd7e1b0) at src/event/ngx_event.c:242
#3  0x0000000000430644 in ngx_worker_process_cycle (cycle=0xd7e1b0, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:750
#4  0x000000000042ed72 in ngx_spawn_process (cycle=cycle@entry=0xd7e1b0, proc=proc@entry=0x4305d3 <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x480ead "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:199
#5  0x000000000042f90b in ngx_start_worker_processes (cycle=cycle@entry=0xd7e1b0, n=1, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:359
#6  0x0000000000431358 in ngx_master_process_cycle (cycle=cycle@entry=0xd7e1b0) at src/os/unix/ngx_process_cycle.c:244
#7  0x000000000040c84b in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382

从main()函数到epoll_wait()函数的调用关系一目了然，说明：（调用栈从下自上）
1 入口是nginx.c文件里的main函数，解析命令行参数，初始化
2 根据ngx_process变量，进入分支ngx_master_process_cycle（另一个分支是ngx_single_process_cycle）
3 进入master后，就需要启动worker进程，调用ngx_start_worker_processes
4 调用ngx_spawn_process启动worker进行，并执行回调ngx_worker_process_cycle
5 进入worker主循环，其中最重要的逻辑是ngx_process_events_and_timers
6 调用ngx_worker_process_init，执行各个模块的init钩子，并设置action为ngx_epoll_process_events
7 epoll开始进入事件等待 __epoll_wait_nocancel

三、strace 调试

#参数1：9是epoll_create创建文件描述符，等待事件触发，1代表rdlist就绪队列有事件要处理
epoll_wait(9, [{EPOLLIN, {u32=14344048, u64=14344048}}], 512, -1) = 1

#接收client请求，3是socket的文件描述符
accept4(17, {sa_family=AF_INET, sin_port=htons(57695), sin_addr=inet_addr("10.2.82.135")}, [16], SOCK_NONBLOCK) = 3

#将3 （accept4 产生）加入到epoll的事件监听机制
epoll_ctl(9, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0

#等待就绪事件，1表示有事件要处理
epoll_wait(9, [{EPOLLIN, {u32=14344720, u64=14344720}}], 512, 60000) = 1

#从client接收数据，长度484字节
recvfrom(3, "GET /ae/test HTTP/1.1\r\nHost: tem"..., 16384, 0, NULL, NULL) = 484

#判断_not_exists_是否存在，-1表示不存在
stat("/usr/local/nginx/html/_not_exists_", 0x7fff9e42c1c0) = -1 ENOENT (No such file or directory)

#更新事件注册
epoll_ctl(9, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0

#初始化新的socket 4
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4

#对设备的I/O通道进行管理，0表示成功
ioctl(4, FIONBIO, [1])                  = 0

#将socket 4 加入epoll的事件监听机制
epoll_ctl(9, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344944, u64=14344944}}) = 0

# socket 4 建立连接
connect(4, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)

#rdlist就绪队列里有两个事件待处理
epoll_wait(9, [{EPOLLOUT, {u32=14344720, u64=14344720}}, {EPOLLOUT, {u32=14344944, u64=14344944}}], 512, 60000) = 2

#获取套接字的选项信息
# 第一个参数：socket套接字
# 第二个参数：套接字层次
# 第三个参数：要获取的信息项
getsockopt(4, SOL_SOCKET, SO_ERROR, [0], [4]) = 0

#响应头509字节内容写入socket4缓冲区
writev(4, [{"GET /ae/test HTTP/1.0\r\nX-Forward"..., 509}], 1) = 509

#rdlist就绪队列里有一个事件待处理，即socket 3
epoll_wait(9, [{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=14344944, u64=14344944}}], 512, 60000) = 1

#从socket 4 读取响应头及内容
recvfrom(4, "HTTP/1.0 200 OK\r\nContent-Length:"..., 4096, 0, NULL, NULL) = 4096
readv(4, [{"m' => 'input',\n  "..., 4096}], 1) = 4096
readv(4, [{"  'file_size' => 0,\n "..., 4096}], 1) = 4096
readv(4, [{"put',\r\n      'rule' "..., 4096}], 1) = 4096
readv(4, [{"#39;2' => array(\r\n       "..., 4096}], 1) = 4096
readv(4, [{"   'file_size' => arr"..., 4096}], 1) = 4096
readv(4, [{"\r\n        'file_size' =&"..., 4096}], 1) = 4096
readv(4, [{"            'predownload&#39"..., 4096}], 1) = 4096
readv(4, [{"y (\n      'label' => "..., 4096}], 1) = 4096

#向socket 3 写入数据
writev(3, [{"HTTP/1.1 200 OK\r\nServer: nginx/1"..., 165}, {"<!DOCTYPE html>\n<html>\n  <head>\n"..., 3949}, {"m&#39; =&gt; &#39;input&#39;,\n  "..., 4096}], 3) = 8210
writev(3, [{"  &#39;file_size&#39; =&gt; 0,\n "..., 4096}, {"put&#39;,\r\n      &#39;rule&#39; "..., 4096}], 2) = 8192
writev(3, [{"#39;2&#39; =&gt; array(\r\n       "..., 4096}, {"   &#39;file_size&#39; =&gt; arr"..., 4096}], 2) = 8192
writev(3, [{"\r\n        &#39;file_size&#39; =&"..., 4096}, {"            &#39;predownload&#39"..., 4096}], 2) = 8192
writev(3, [{"y (\n      &#39;label&#39; =&gt; "..., 4096}], 1) = 4096
readv(4, [{"  array(\n                    &#3"..., 4096}, {"9;file_ext&#39; =&gt; &#39;mp4,g"..., 4096}, {"\r\n    <div class=\"col-md-1\">9</d"..., 4096}, {"=&gt; array(\r\n    &#39;gotourl&#"..., 4096}, {"tle\" id=\"exampleModalScrollableT"..., 4096}, {"", 4096}, {"", 4096}, {"", 4096}, {"", 4096}], 9) = 17810
readv(4, [{"", 2670}, {"", 4096}, {"", 4096}, {"", 4096}, {"", 4096}], 5) = 0
writev(3, [{"  array(\n                    &#3"..., 4096}, {"9;file_ext&#39; =&gt; &#39;mp4,g"..., 4096}, {"\r\n    <div class=\"col-md-1\">9</d"..., 4096}, {"=&gt; array(\r\n    &#39;gotourl&#"..., 4096}, {"tle\" id=\"exampleModalScrollableT"..., 1426}], 5) = 17810

#关闭socket 4
close(4)                                = 0
write(28, "10.2.82.135 - - [24/Mar/2020:12:"..., 220) = 220
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0

#查看epoll的rdlist就绪队列是否有事件，0表示无
epoll_wait(9, [], 512, 65000)           = 0

#关闭socket 3
close(3)                                = 0

#继续等待事件触发
epoll_wait(9,

整个过程，你应该清晰的知道epoll对象上的rdlist就绪队列和rbtree的变化情况

延伸阅读
nginx 源码注释项目

参考文档
1 epoll_create
2 strace工具
3 nginx 源码注释