Nginx TCP监控和自动限流

接着昨天的步子。今天写完监控和限流。主要通过解析tcp_access.log来实时监控各个端口的网络流量情况。对于流量超标的端口暂时封闭，隔天恢复。

1. Nginx TCP日志字段解析

log_time worker_process_pid client_ip host_ip accept_time upstream_ip bytes_read bytes_write

2011/08/02 06:19:07 [5972] 127.0.0.1 0.0.0.0:1982 2011/08/02 06:18:19 172.19.0.129:80 80 236305

log_time: The current time when writing this log. 

          The log action is called when the proxy session is closed.

worker_process_pid: the pid of worker process

client_ip: the client ip

host_ip: the server ip and port

accept_time: the time when the server accepts client's connection

upstream_ip: the upstream server's ip

bytes_read: the bytes read from client

bytes_write: the bytes written to client

2. Python日志解析及监控脚本

# -*- coding: utf-8 -*-
"""
"""
import os
import time
import json

INPUT=0.0
OUTPUT=0.0
TATAL=0.0
PORT_MAP = {}
FORBID_MAP = {}
LIMIT = 8192.0

f=open("/usr/local/nginx/logs/tcp_access.log","r")
l = open("/data/nginx/forbid.list", "a+")

for fine in l :
	FORBID_MAP[fine.replace("\n","")]=fine.replace("\n","")

for line in f :
	arr = line.split(" ")
	PORT_MAP
	port = arr[4].split(":")[1]

	if PORT_MAP.has_key(port):  
		PORT_MAP[port]["INPUT"] += float(arr[8])
		PORT_MAP[port]["OUTPUT"] += float(arr[9])
	else : 
		PORT_MAP[port] = {
			"INPUT" : float(arr[8]),
			"OUTPUT" : float(arr[9])
		}

	INPUT += long(arr[8])
	OUTPUT += long(arr[9])
	TATAL += long(arr[8]) + long(arr[9]) 

for (k,v) in PORT_MAP.items():
	v["LOGTIME"] = time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time()))
	v["TATAL"] = round((v["INPUT"]+v["OUTPUT"])/1024/1024,2)
	v["INPUT"] = round(v["INPUT"]/1024/1024,2)
	v["OUTPUT"] = round(v["OUTPUT"]/1024/1024,2)
	if v["TATAL"] > LIMIT and (not FORBID_MAP.has_key(k)):
		os.system("firewall-cmd --permanent --remove-port="+k+"/tcp ")
		os.system("firewall-cmd --reload ")
		l.write(k+"\n")

PORT_MAP["SUMMARY"] = {
	"LOGTIME" : time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),
	"INPUT" : round(INPUT/1024/1024,2),
	"OUTPUT" : round(OUTPUT/1024/1024,2),
	"TATAL" : round(TATAL/1024/1024,2)
}

JSONSTR = json.dumps(PORT_MAP);
w = open("/data/nginx/result.json", "a+")
w.write(JSONSTR+",\n")

脚本会将日志解析为端口的流量信息实时汇总。格式为：

{
    "1234": {
        "INPUT": 0.05,
        "TATAL": 0.39,
        "LOGTIME": "2016-03-22 02:52:07",
        "OUTPUT": 0.34
    },
    "2234": {
        "INPUT": 31.66,
        "TATAL": 753.82,
        "LOGTIME": "2016-03-22 02:52:07",
        "OUTPUT": 722.16
    },
    "SUMMARY": {
        "INPUT": 31.71,
        "TATAL": 754.21,
        "LOGTIME": "2016-03-22 02:52:07",
        "OUTPUT": 722.5
    }
}

配合一个5秒循环脚本，定时监控流量。如端口流量超标，通过防火墙关闭该端口。

#!/bin/bash

while true
do
	python /usr/local/shell/analyst.py
	sleep 5
done

3. Nginx 日志切分和端口重开

#!/bin/bash

# 1. move host.access.log to host.access_20120821.log
logs_path="/usr/local/nginx/logs/"
mv ${logs_path}access.log ${logs_path}access_$(date -d "yesterday" +"%Y%m%d").log
mv ${logs_path}tcp_access.log ${logs_path}tcp_access_$(date -d "yesterday" +"%Y%m%d").log
kill -USR1 `cat ${logs_path}nginx.pid`

# 2. delete all host.access_2012xxxx.log which overtime 10 days
#find ${logs_path} -name 'access_*.log' -type f -mtime +10 -exec rm {} \;

# clear forbid.list ,reopen ports
> /data/nginx/forbid.list
firewall-cmd --permanent --add-port=1234/tcp
firewall-cmd --permanent --add-port=2234/tcp
firewall-cmd --permanent --add-port=3234/tcp
firewall-cmd --permanent --add-port=4234/tcp
firewall-cmd --reload  

配置定时任务每天凌晨执行

1 0 * * * sh /usr/local/shell/nginxcutlogs.sh