nginx和haproxy代理时对后端的长连接分析

由于代理服务器上开启了长连接后，可以减少每次与后端服务器的三次握手，提高后端服务器的效率。所以nginx在1.1.4版本中终于增加了upstream对keepalive的支持，而haproxy原本对后端是支持长连接的（但是这个时候由于不能把client的IP加到header转发给后端服务器造成会丢失客户端的IP）。今天简单测试了一下。基本的结构是

A:

Client –> nginx(proxy) –>nginx(server)

B:

Client –> haproxy(proxy) –>nginx(server)

对比的结构

Client –>nginx(server)

先测试一下client直接和server端能否保持住长连接，我是直接在server端抓包观察，确定OK后再进行nginx和haproxy的测试。

nginx proxy的配置如下：

server {

location / {

root html;

proxy_set_header Connection "";

proxy_http_version 1.1;

proxy_intercept_errors on;

proxy_set_header Host $http_host;

proxy_set_header ORIG_CLIENT_IP $remote_addr;

index index.html,index.htm;

proxy_pass http://httpd;

}

}

upstream httpd {

server 172.189.85.156:1080;

keepalive 4;

least_conn;

}

在server端tcpflow抓包后可以看到

0172.189.085.161.50970-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50970: HTTP/1.1 304 Not Modified

0172.189.085.161.50971-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50971: HTTP/1.1 404 Not Found

0172.189.085.161.50972-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50972: HTTP/1.1 304 Not Modified

0172.189.085.161.50973-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50973: HTTP/1.1 404 Not Found

0172.189.085.161.50974-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50974: HTTP/1.1 304 Not Modified

0172.189.085.161.50975-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50975: HTTP/1.1 404 Not Found

0172.189.085.161.50976-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50976: HTTP/1.1 304 Not Modified

0172.189.085.161.50977-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50977: HTTP/1.1 404 Not Found

0172.189.085.161.50978-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50978: HTTP/1.1 304 Not Modified

0172.189.085.161.50979-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50979: HTTP/1.1 404 Not Found

0172.189.085.161.50980-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50980: HTTP/1.1 304 Not Modified

0172.189.085.161.50981-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50981: HTTP/1.1 404 Not Found

0172.189.085.161.50982-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50982: HTTP/1.1 304 Not Modified

0172.189.085.161.50983-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50983: HTTP/1.1 404 Not Found

0172.189.085.161.50984-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50984: HTTP/1.1 304 Not Modified

0172.189.085.161.50985-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50985: HTTP/1.1 404 Not Found

0172.189.085.161.50986-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50986: HTTP/1.1 304 Not Modified

0172.189.085.161.50987-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50987: HTTP/1.1 404 Not Found

0172.189.085.161.50988-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50988: HTTP/1.1 304 Not Modified

0172.189.085.161.50989-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50989: HTTP/1.1 404 Not Found

0172.189.085.161.50990-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50990: HTTP/1.1 304 Not Modified

0172.189.085.161.50991-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50991: HTTP/1.1 404 Not Found

从上面可以看到代理服务器每次连接后端服务器使用的端口都是不同的，这个时候我就郁闷了，难道nginx对后端开启不了长连接？

测试了一两个小时我终于想到是不是加了最小连接数的那个算法导致的。果然把那行去掉后就OK了。

[最新的状况是 把 keepalive 4;设置放在least_conn之后就可以了 ]

server {

location / {

root html;

proxy_set_header Connection "";

proxy_http_version 1.1;

proxy_intercept_errors on;

proxy_set_header Host $http_host;

proxy_set_header ORIG_CLIENT_IP $remote_addr;

index index.html,index.htm;

proxy_pass http://httpd;

}

}

upstream httpd {

server 172.189.85.156:1080;

keepalive 4;

}

0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified

0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found

0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified

0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found

0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified

0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found

0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified

0172.189.085.161.50998-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 404 Not Found

0172.189.085.161.50998-0172.189.085.156.01080: GET /index.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.50998: HTTP/1.1 304 Not Modified

然后我又测试了一下haproxy。主要是之前对haproxy的理解有点误区。一方面doc里面说了

A last improvement in the communications is the pipelining mode. It still uses

keep-alive, but the client does not wait for the first response to send the

second request. This is useful for fetching large number of p_w_picpaths composing a

page :

[CON] [REQ1] [REQ2] … [RESP1] [RESP2] [CLO] …

This can obviously have a tremendous benefit on performance because the network

latency is eliminated between subsequent requests. Many HTTP agents do not

correctly support pipelining since there is no way to associate a response with

the corresponding request in HTTP. For this reason, it is mandatory for the

server to reply in the exact same order as the requests were received.

By default HAProxy operates in a tunnel-like mode with regards to persistent

connections: for each connection it processes the first request and forwards

everything else (including additional requests) to selected server. Once

established, the connection is persisted both on the client and server

sides. Use “option http-server-close” to preserve client persistent connections

while handling every incoming request individually, dispatching them one after

another to servers, in HTTP close mode. Use “option httpclose” to switch both

sides to HTTP close mode. “option forceclose” and “option

http-pretend-keepalive” help working around servers misbehaving in HTTP close

mode.

另外一方面就是大家都知道haproxy对后端是没有keepalive的，目前连测试版也不支持。。。这个就让我觉得很是疑惑。

今天其实主要是为了测试一下haproxy，配置文件如下

frontend httpf

bind *:1689

default_backend httpd

backend httpd

mode http

option forwardfor header ORIG_CLIENT_IP

balance roundrobin

server ser2 172.189.85.156:1080

frontend httpf2

bind *:1690

default_backend httpd2

backend httpd2

mode http

option httpclose

balance roundrobin

option forwardfor header ORIG_CLIENT_IP

server ser2 172.189.85.156:1080

不过抓包后直接看到了当访问1689端口的时候对后端服务器的连接是一直保持的，但是显示不出客户端的IP（没有转发ORIG_CLIENT_IP）,

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified

0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found

0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified

0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found

0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified

0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found

0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified

0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found

0172.189.085.161.51026-0172.189.085.156.01080: GET /index2.html HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 304 Not Modified

0172.189.085.161.51026-0172.189.085.156.01080: GET /favicon.ico HTTP/1.1

0172.189.085.156.01080-0172.189.085.161.51026: HTTP/1.1 404 Not Found

访问1690端口，即是配置了httpclose的时候发现能显示客户端的IP，但是每个连接都在请求完成后就关闭了，抓包的记录就不贴了。

看来有问题还是需要自己抓包来实际分析才知道究竟是怎么个情况。

后记：

今天统计了一下做代理的nginx似乎每个链接对后端请求处理100个后就断掉了，仔细查了一下wiki发现这个实际上是后端nginx的默认设置的问题。

可以吧后端的nginx的配置改成keepalive_requests 1000,这样代理服务器向后端的连接能每个处理1000个再中断。

转载于:https://blog.51cto.com/hxl2009/1325150