使用logstash传输处理日志输出到ES，再通过Kibana做分析

目前发现网上找到了大部分kibana操作都是旧版本，已不适用于新版本的kibana操作和使用
（且发现部分数据处理过程中跟现有部分日志内容脱节）

本文重点是传输日志+处理数据+kibana操作

（对于ES+kibana搭建以及logstash原理不多做介绍）

重点内容
传输日志=logstash使用以及基本的操作规范，sincedb_path意义和常见问题。
数据处理=logstash导入输出的过程中有2个要点必须明确
1、filter grok正则，把数据导入过程中分不同的类，用于kibana数据分类处理。
2、timestamp必须是日志中的时间（要做一步额外处理），否则默认会以ES导入数据时间为基准，从而影响数据维度
3、kibana基本功能，比如状态码统计，URL次数统计，IP统计等。

1. 传输日志

这边贴一个http样例日志
224.165.101.202 - - [25/Sep/2020:09:40:00 +0000] “GET /category/electronics?from=20 HTTP/1.1” 200 132 “/category/electronics” “Mozilla/5.0 (Windows NT 6.0; rv:10.0.1) Gecko/20100101 Firefox/10.0.1”
28.201.42.165 - - [25/Sep/2020:09:40:05 +0000] “GET /item/electronics/3452 HTTP/1.1” 200 117 “/category/games” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; YTB730; GTB7.2; EasyBits GO v1.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)”
120.45.68.92 - - [25/Sep/2020:09:40:10 +0000] “GET /category/music HTTP/1.1” 200 118 “/category/books?from=20” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11”
40.216.111.67 - - [25/Sep/2020:09:40:15 +0000] “GET /category/books HTTP/1.1” 200 91 “/category/finance” “Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)”
212.102.68.187 - - [25/Sep/2020:09:40:20 +0000] “GET /category/sports HTTP/1.1” 200 86 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11”
120.21.65.131 - - [25/Sep/2020:09:40:25 +0000] “GET /category/electronics HTTP/1.1” 200 133 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11”
216.81.198.88 - - [25/Sep/2020:09:40:30 +0000] “GET /category/giftcards HTTP/1.1” 200 67 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11”
112.57.120.179 - - [25/Sep/2020:09:40:35 +0000] “GET /category/electronics HTTP/1.1” 200 126 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.1) Gecko/20100101 Firefox/10.0.1”
64.201.227.148 - - [25/Sep/2020:09:40:40 +0000] “GET /category/cameras HTTP/1.1” 200 64 “-” “Mozilla/5.0 (Windows NT 6.0; rv:10.0.1) Gecko/20100101 Firefox/10.0.1”

logstash代码：

input{