logstash采集输送日志

建立了个测试java项目spring-boot-log,启动项目

 java -jar spring-boot-log.jar 

此项目会将产生的项目日志存到/home/alen/application/logs/spring-boot-log-info.log

启动

bin/logstash -f hello.conf 

    codec的使用( Coder/decoder 两个单词首字母缩写）
        Codec: 解码编码 数据格式  
        好处 更方便logstash与支持自定义数据格式的运维产品进行使用
    logstash更细化的处理流程
        input->decode->filter->encode->output

hello.conf

input {
 # 从文件读取日志信息 输送到控制台
file {
      path => "/home/alen/application/logs/spring-boot-log-info.log"
	#codec => "json" ## 以JSON格式读取日志
	type => "elasticsearch"
	start_position => "beginning"
	}
}

# filter {
#
# }

output {
	# 标准输出 
	# stdout {}
	# 输出进行格式化，采用Ruby库来解析日志   
	 stdout { codec => rubydebug }
}

访问项目

http://localhost:8070/hello

logstash打印结果

{
          "host" => "localhost.localdomain",
    "@timestamp" => 2018-09-22T13:23:23.946Z,
      "@version" => "1",
       "message" => "2018-09-22 21:23:22.539 [http-nio-8070-exec-5] INFO  com.alen.log.controller.HelloController - 打印日志",
          "path" => "/home/alen/application/logs/spring-boot-log-info.log",
          "type" => "elasticsearch"
}

filter使用
        例子
            切割插件mutate，随意输入一串以|分割的字符，比如 "123|000|ttter|sdfds*=123|dfwe

        配置二 test_filter.conf

input {
 # 从文件读取日志信息 输送到控制台
file {
      path => "/home/alen/application/logs/spring-boot-log-info.log"
	#codec => "json" ## 以JSON格式读取日志
	type => "elasticsearch"
	start_position => "beginning"
	}
}

filter {  
    mutate {  
        split => ["message", "|"]  
    }  
} 

output {
	# 标准输出 
	# stdout {}
	# 输出进行格式化，采用Ruby库来解析日志   
	 stdout { codec => rubydebug }
}

启动

bin/logstash -f test_filter.conf

打印结果message会是根据“|”分割为一个数组

{
          "path" => "/home/alen/application/logs/spring-boot-log-info.log",
       "message" => [
        [0] "2018-09-22 21:45:08.853 [http-nio-8070-exec-2] INFO  com.alen.log.controller.HelloController - 打印日志"
    ],
          "host" => "localhost.localdomain",
      "@version" => "1",
    "@timestamp" => 2018-09-22T13:45:17.204Z,
          "type" => "elasticsearch"
}

从日志文件中读取日志，输出到elasticsearch集群中

启动

bin/logstash -f test_es.conf 

配置文件test_es.conf

input {
        file {
                path => "/home/alen/application/logs/spring-boot-log-info.log"
                start_position => "beginning" #从文件开始处读写
        }
#       stdin {}  #可以从标准输入读数据
}
filter {
  #Only matched data are send to output.
}
output {
  elasticsearch {
    #The operation on ES 文档https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
    action => "index"          
    hosts  => ["127.0.0.1:9200"]       #ElasticSearch host, can be array.
    index => "es-message-%{+YYYY.MM.dd}"        #The index to write data to.
  }
   #同时打印到控制台
  stdout{codec => rubydebug} 
}

访问项目spring-boot-log产生日志

通过elasticsearch查看

访问：http://localhost:9200/es-message-2018.09.23/_search?q=*&pretty

返回

{
  "took" : 18,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "es-message-2018.09.23",
        "_type" : "doc",
        "_id" : "Gaf7A2YB_AQwvE2K6Z6A",
        "_score" : 1.0,
        "_source" : {
          "message" : "2018-09-23 09:12:10.314 [http-nio-8070-exec-6] INFO  com.alen.log.controller.HelloController - 打印日志",
          "@version" : "1",
          "path" : "/home/alen/application/logs/spring-boot-log-info.log",
          "@timestamp" : "2018-09-23T01:12:11.250Z",
          "host" : "localhost.localdomain"
        }
      }
    ]
  }
}

logstash官方文档地址

https://www.elastic.co/guide/en/logstash/current/index.html