1.异常分析:
haproxy集群作为负载代理,底层挂着多个服务程序,20w个用户同时并发请求,在100%请求协议为http情况下服务良好,无异常;当请求为websocket 75% http30%情况下,出现大量503服务不可以异常。
首先查看底层服务器,底层服务器运行平稳,压力未满,对应的503请求未发送到底层服务器,查看haproxy日志,发现大量如下日志:
>> - 503 625990 use 0.000304s
haproxy完全未将请求发送给底层服务器,haproxy接到这个请求后立马就返回503。神奇的现象,首先可以判断为haproxy问题;
到底是什么问题导致了haproxy在底层服务器健康并且压力低的情况下任性的返回503呢,开启纠结之旅,排查服务器配置:
[root@A33 ~]# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515033
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 627680
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 60240
virtual memory
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515033
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 627680
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 60240
virtual memory