Nginx分析日志三剑客脚本

Nginx分析日志三剑客脚本

1.数据模板

101.226.61.184 - - [22/Nov/2015:11:02:00 +0800] "GET /mobile/sea-modules/gallery/zepto/1.1.3/zepto.js HTTP/1.1" 200 24662 "http://m.oldboyedu.com.cn/mobile/theme/oldboy/home/index.html" "Mozilla/5.0 (Linux; U; Android 5.1.1; zh-cn; HUAWEI CRR-UL00 Build/HUAWEICRR-UL00) AppleWebKit/533.1 (KHTML, like Gecko)Version/4.0 MQQBrowser/5.4 TBS/025478 Mobile Safari/533.1 MicroMessenger/6.3.7.51_rbb7fa12.660 NetType/3gnet Language/zh_CN"
101.226.61.184 - - [22/Nov/2015:11:02:00 +0800] "GET /mobile/theme/oldboy/common/js/baiduAnalytics.js HTTP/1.1" 200 526 "http://m.oldboyedu.com.cn/mobile/theme/oldboy/home/index.html" "Mozilla/5.0 (Linux; U; Android 5.1.1; zh-cn; HUAWEI CRR-UL00 Build/HUAWEICRR-UL00) AppleWebKit/533.1 (KHTML, like Gecko)Version/4.0 MQQBrowser/5.4 TBS/025478 Mobile Safari/533.1 MicroMessenger/6.3.7.51_rbb7fa12.660 NetType/3gnet Language/zh_CN"
27.154.190.158 - - [22/Nov/2015:11:02:00 +0800] "GET /mobile/theme/oldboy/product/images/arrow.png HTTP/1.1" 200 3084 "http://m.oldboyedu.com.cn/mobile/theme/oldboy/product/productCategoryIndex.html" "Mozilla/5.0 (iPad; CPU OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1"
101.226.61.184 - - [22/Nov/2015:11:02:00 +0800] "GET /mobile/static/common/src/publicMethod.js HTTP/1.1" 200 7391 "http://m.oldboyedu.com.cn/mobile/theme/oldboy/home/index.html" "Mozilla/5.0 (Linux; U; Android 5.1.1; zh-cn; HUAWEI CRR-UL00 Build/HUAWEICRR-UL00) AppleWebKit/533.1 (KHTML, like Gecko)Version/4.0 MQQBrowser/5.4 TBS/025478 Mobile Safari/533.1 MicroMessenger/6.3.7.51_rbb7fa12.660 NetType/3gnet Language/zh_CN"
​

2.要求

2.1统计访问 IP 次数

cat access.log | awk '{print$1}' |sort|uniq -c|sort -rn

2.2 统计访问访问大于 100 次的 IP

cat access.log | awk '{print$1}' |sort|uniq -c|sort -rn|awk -F " " '$1>100 {print $0}'

2.3 统计访问 IP 次数并排序取前 10：

cat access.log | awk '{print$1}' |sort|uniq -c|sort -rn|awk -F " " '$1>100 {print $0}'|head -10

2.4 统计时间段访问最多的 IP

awk '$4>="[9/Apr/2018:00:00:01" && $4<="[9/Apr/2018:23:59:59" {a[$1]++}END{for(i in a)print a[i],i|"sort -k1 -nr|head -n10"}' access.log

2.5 统计上一分钟访问量

date=$(date -d '-1 minute' +%d/%b/%Y:%H:%M);awk -vdate=$date '$0~date{c++}END{print c}' access.log

2.6 统计每个 IP 访问状态码数量

awk '{a[$1" "$9]++}END{for(i in a)print i,a[i]}' access.log

2.7 统计访问 IP 是 404 状态次数

awk '{if($9~/404/)a[$1" "$9]++}END{for(i in a)print i,a[i]}' access.log