日志内容如下:
1.119.144.106 - - [03/Jan/2019:12:47:50 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.001" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:50 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.001" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:51 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.001" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:51 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.000" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:51 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.001" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:51 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.000" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:52 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.001" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:53 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.001" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
1.119.144.106 - - [03/Jan/2019:12:47:53 +0800] image.aa.com "GET /image/public/index.php/index/uploads/handle HTTP/1.1 0.000" 500 5 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" -
123.125.71.82 - - [03/Jan/2019:12:47:57 +0800] image.aa.com "GET /Uploads/150827/55dee28ce9dbf.jpg HTTP/1.1 0.211" 200 509831 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" -
220.181.108.169 - - [03/Jan/2019:12:48:03 +0800] image.aa.com "GET /Uploads/160622/576a046300906.jpg HTTP/1.1 1.567" 200 49965 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" -
115.239.212.137 - - [03/Jan/2019:12:48:15 +0800] image.aa.com "GET /image/public/uploads/170109/587398dc20d79.JPG HTTP/1.1 0.362" 200 985944 "-" "Mozilla/5.0 (Windows NT 5.1; rv:8.0.1) Gecko/20100101 Firefox/8.0.1" -
123.125.71.114 - - [03/Jan/2019:12:48:31 +0800] image.aa.com "GET /Uploads/170505/590c2d515b50a.jpg HTTP/1.1 1.406" 200 25317 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" -
220.243.135.80 - - [03/Jan/2019:12:48:38 +0800] image.aa.com "GET /image/public/uploads/171129/5a1ec9255a89e.jpg HTTP/1.1 0.108" 200 39174 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" -
220.243.135.143 - - [03/Jan/2019:12:48:38 +0800] image.aa.com "GET /Uploads/170804/5984129b7b4fb.jpg HTTP/1.1 0.172" 200 33351 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" -
220.243.135.140 - - [03/Jan/2019:12:48:49 +0800] image.aa.com "GET /Uploads/161222/585b9f24c4a99.JPG HTTP/1.1 0.223" 200 381442 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" -
112.80.254.51 - - [03/Jan/2019:12:48:52 +0800] image.aa.com "GET /Uploads/170806/5985f7ac2fb94.jpg HTTP/1.1 0.179" 200 72244 "-" "Mozilla/5.0 (Windows NT 5.1; rv:8.0.1) Gecko/20100101 Firefox/8.0.1" -
123.125.71.109 - - [03/Jan/2019:12:48:57 +0800] image.aa.com "GET /Uploads/170516/591ac3201f954.jpg HTTP/1.1 3.026" 200 76574 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" -
220.243.136.128 - - [03/Jan/2019:12:49:24 +0800] image.aa.com "GET /image/public/uploads/170330/58dc7a95a5a70.jpg HTTP/1.1 0.065" 200 59287 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" -
220.243.135.128 - - [03/Jan/2019:12:49:24 +0800] image.aa.com "GET /image/public/uploads/170321/58d0c185e5668.jpg HTTP/1.1 0.073" 200 102566 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" -
第一列是IP地址,现在我想根据IP统计它的数量,就可以写一个shell命令如下:
cat logs/xxx.log | awk '{print $(1)}' | sort | uniq -c | sort -k 1 -n -r|head -10
解释一下上面的命令,
cat logs/xxx.log就是输出要统计的日志。
awk后面跟一个指令,awk '{print $(1)}'就是打印出日志内容的第几列。$1就是第一列
扩展:$(NF)是总列数,那么如果根据倒数第二列统计,就是$(NF-1)。
sort就是对内容进行排序,默认是自然顺序排序。
uniq指令用于排重,而是只适用于相邻两行相同的情况。所以一般结合sort使用。即先sort排序再排重。
uniq -u是只显示唯一的记录行。uniq -c是显示有重复记录的情况。
sort -k 1 -n -r这个指令,参看下面sort指令参数的详细说明
sort选项与参数:
-f :忽略大小写的差异,例如 A 与 a 视为编码相同;
-b :忽略最前面的空格符部分;
-M :以月份的名字来排序,例如 JAN, DEC 等等的排序方法;
-n :使用『纯数字』进行排序(默认是以文字型态来排序的);
-r :反向排序;
-u :就是 uniq ,相同的数据中,仅出现一行代表;
-t :分隔符,默认是用 [tab] 键来分隔;
-k :以哪个区间 (field) 来进行排序的意思
所以 sort -k 1 -n -r 指令的意思就是对第一列按照纯数字逆序排序。
这个纯数字是哪里来的呢?是uniq -c来的,原来剩下一列就是IP了,当执行uniq -c指令时,它会统计重复记录的次数并把这次数显示在第一列。所以现在有两列了,第一列是重复次数,第二列是IP。所以这里是按照重复次数排序。
head -10这个不用说了吧,显示前10行。