Logstash 是开源的服务器端数据处理管道,能够同时从多个来源采集数据,转换数据,然后将数据发送到您最喜欢的“存储库”中。
上传logstash并解压缩
tar -zxvf logstash-7.3.1.tar.gz
编辑配置文件
vi test.conf
input {
#标准输入
stdin {
#通用选项,用于输入数据的编解码器
codec => plain{ charset => "UTF-8" }
}
#从文件读取数据
file {
type => "json"
path => "/data/logstash/logdata/test.log"
#选择logstash开始读取文件的位置,begining或者end
start_position => "beginning"
codec => plain{ charset => "UTF-8" }
}
}
filter{
#解析json
json {
source => "message"
}
#将timestamp转为时间类型,并重新定义一个createdate字段来存储
date {
match => [ "timestamp", "UNIX_MS" ]
target => "createdate"
}
}
output {
#将事件存储到es中
elasticsearch { hosts => ["192.168.98.128:9200"] }
#标准输出,将事件输出到屏幕上
stdout { codec => rubydebug }
}
测试数据文件如下
vi test.log
{"name": "zhangsan", "age": 21, "addr": "北京", "country": "中国", "timestamp": "1532933162361", "salary": 10000}
{"name": "tom", "age":20,"addr":"纽约", "country": "美国", "timestamp": "1532933163361", "salary": 12000}
{"name": "wangwu", "age":19,"addr":"上海", "country": "中国", "timestamp": "1532933163361", "salary": 11000}
{"name": "dabai", "age":19,"addr":"上海", "country": "中国", "timestamp": "1532933162361", "salary": 7000}
{"name": "xiaoming", "age":19,"addr":"北京", "country": "中国", "timestamp": "1532933165361", "salary": 15000}
{"name": "xiaohong", "age": 21, "addr": "北京", "country": "中国", "timestamp": "1532933162361", "salary": 9000}
{"name": "jack", "age":20,"addr":"纽约", "country": "美国", "timestamp": "1532933162361", "salary": 14000}
测试配置文件是否正确
./bin/logstash -t -f config/test.conf
启动logstash 并读取log文件,存入es
./bin/logstash -f config/test.conf
控制台输入hello 世界回车,自动存入es
遇到的问题:
重启logstash时,要想重新读取日志文件。
需要手工删除logstash-7.3.1/data/plugins/inputs目录下的file目录