建立了个测试java项目spring-boot-log,启动项目
java -jar spring-boot-log.jar
此项目会将产生的项目日志存到/home/alen/application/logs/spring-boot-log-info.log
启动
bin/logstash -f hello.conf
codec的使用( Coder/decoder 两个单词首字母缩写)
Codec: 解码编码 数据格式
好处 更方便logstash与支持自定义数据格式的运维产品进行使用
logstash更细化的处理流程
input->decode->filter->encode->output
hello.conf
input {
# 从文件读取日志信息 输送到控制台
file {
path => "/home/alen/application/logs/spring-boot-log-info.log"
#codec => "json" ## 以JSON格式读取日志
type => "elasticsearch"
start_position => "beginning"
}
}
# filter {
#
# }
output {
# 标准输出
# stdout {}
# 输出进行格式化,采用Ruby库来解析日志
stdout { codec => rubydebug }
}
访问项目
logstash打印结果
{
"host" => "localhost.localdomain",
"@timestamp" => 2018-09-22T13:23:23.946Z,
"@version" => "1",
"message" => "2018-09-22 21:23:22.539 [http-nio-8070-exec-5] INFO com.alen.log.controller.HelloController - 打印日志",
"path" => "/home/alen/application/logs/spring-boot-log-info.log",
"type" => "elasticsearch"
}
filter使用
例子
切割插件mutate,随意输入一串以|分割的字符,比如 "123|000|ttter|sdfds*=123|dfwe
配置二 test_filter.conf
input {
# 从文件读取日志信息 输送到控制台
file {
path => "/home/alen/application/logs/spring-boot-log-info.log"
#codec => "json" ## 以JSON格式读取日志
type => "elasticsearch"
start_position => "beginning"
}
}
filter {
mutate {
split => ["message", "|"]
}
}
output {
# 标准输出
# stdout {}
# 输出进行格式化,采用Ruby库来解析日志
stdout { codec => rubydebug }
}
启动
bin/logstash -f test_filter.conf
打印结果message会是根据“|”分割为一个数组
{
"path" => "/home/alen/application/logs/spring-boot-log-info.log",
"message" => [
[0] "2018-09-22 21:45:08.853 [http-nio-8070-exec-2] INFO com.alen.log.controller.HelloController - 打印日志"
],
"host" => "localhost.localdomain",
"@version" => "1",
"@timestamp" => 2018-09-22T13:45:17.204Z,
"type" => "elasticsearch"
}
从日志文件中读取日志,输出到elasticsearch集群中
启动
bin/logstash -f test_es.conf
配置文件test_es.conf
input {
file {
path => "/home/alen/application/logs/spring-boot-log-info.log"
start_position => "beginning" #从文件开始处读写
}
# stdin {} #可以从标准输入读数据
}
filter {
#Only matched data are send to output.
}
output {
elasticsearch {
#The operation on ES 文档https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
action => "index"
hosts => ["127.0.0.1:9200"] #ElasticSearch host, can be array.
index => "es-message-%{+YYYY.MM.dd}" #The index to write data to.
}
#同时打印到控制台
stdout{codec => rubydebug}
}
访问项目spring-boot-log产生日志
通过elasticsearch查看
访问:http://localhost:9200/es-message-2018.09.23/_search?q=*&pretty
返回
{
"took" : 18,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "es-message-2018.09.23",
"_type" : "doc",
"_id" : "Gaf7A2YB_AQwvE2K6Z6A",
"_score" : 1.0,
"_source" : {
"message" : "2018-09-23 09:12:10.314 [http-nio-8070-exec-6] INFO com.alen.log.controller.HelloController - 打印日志",
"@version" : "1",
"path" : "/home/alen/application/logs/spring-boot-log-info.log",
"@timestamp" : "2018-09-23T01:12:11.250Z",
"host" : "localhost.localdomain"
}
}
]
}
}
logstash官方文档地址