Linux 基础实战（二）

nginx log 解析实战 (一)

找出所有404和500错误日志，统计错误日志行数

217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 404 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 404 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.51 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 500 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 0.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 404 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 404 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /topics HTTP/1.1" 500 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 2.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /topics HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 11.53 A"
217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 1.53 A"
217.138.222.100 - - [11/Feb/2022:13:22:11 +0000] "GET /topics HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 0.51 A"
217.138.222.100 - - [11/Feb/2022:13:22:11 +0000] "GET /topics HTTP/1.1" 200 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36 2.11 A"

1. grep 前缀+关键字, egrep 代表或逻辑，一旦前缀不匹配会过滤bad case
2. awk 结合grep 切割指定的列
3. 单独用awk

cat nginx.log | egrep 'HTTP/1.1" 404|HTTP/1.1" 500' | wc -l
cat nginx.log | awk '{print $9}' | egrep '404|500' | wc -l
awk '$9~/400||500/ {print $9}' nginx.log | wc -l
awk '$9~/400||500/{t+=1}END{print t}' nginx.log

可以用less, /反例（200） 来测试上述脚本是否有bad case

cat nginx.log | awk '{print $9}' | egrep '404|500' | less
awk '$9~/400||500/ {print $9}' nginx.log | less

awk pattern

• awk 理论上可以代替grep
• awk ‘pattern{action}’
• awk ‘BEGIN{}END{}’ 开始和结束
• awk ‘/Running/’ 正则匹配
• awk ‘/aa/,/bb/’ 区间选择
• awk ‘$2~/xxx/’ 字段匹配
• awk ‘NR==2’ 取第二行
• awk ‘NR>1’ 去除第一行

nginx log 解析实战 (二)

找出url 访问量最高 前3个

less nginx.log |awk '{print $7}'| sort |uniq -c | sort -r |head -3

  7 /favicon.ico
  4 /favi/p4
  1 /favi/p3

awk 获取每列列号

取文件里第一行所有列， awk ‘NR==1{for(i=1;i<NF;i++) {print “line"i”:" $i}}’ nginx.log

awk 计算指定路径/topics 的平均响应时间（倒数第二列）

less nginx.log | awk '$7=="/topics"{total+=$(NF-1);i++}END{print total/i}'

$(NF) 最后一列，-1为倒数第二列
不建议用grep “/topics” 会过滤到一些脏数据