logstash

https://segmentfault.com/a/1190000016591126

https://elasticsearch.cn/question/8038

 

https://www.qedev.com/linux/317121.html。

 

docker构建ELK。https://hub.docker.com/r/sebp/elk

docker-compose.yml

elk:
  image: sebp/elk
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"

如需使用filebeat发送日志。需要进入启动elk镜像后的容器，将 /etc/pki/tls/certs/logstash-beats.crt 拷贝到宿主机,然后配置filebeat时需要指定该文件。

该镜像产生3类日志：

/var/log/kibana/kibana5.log
/var/log/elasticsearch/elasticsearch.log
/var/log/logstash/logstasg-plain.log

 

遇到的坑。

Filebeat

fibeat6.3之后Filebeat prospectors 重命名到 inputs。

filebeat.yml

output:
  logstash:
    hosts:
      - 127.0.0.1:5044
  timeout: 15
  ssl:
    certificate_authorities:
      - /etc/filebeat/logstash-beats.crt

filebeat.inputs:
- type: log
  paths:
    - /var/log/system.log
    - /var/log/wifi.log
- type: log
  paths:
    - "/var/log/apache2/*"
  fields:
    apache: true
  fields_under_root: true
  enabled: true
processors:
  - drop_fields:
      fields: ["agent","ecs","input","agent"]

https://blog.csdn.net/yk20091201/article/details/90768632

filebeat除了发送日志，还有一些内置字段,需要移除的话 添加drop_fields

 

logstash.conf

input { 
  beats {
        port => 5044
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
        ssl_key => "/etc/pki/tls/private/logstash-beats.key"
    } 

}

filter {

    grok {
        match => ["message","\[%{HTTPDATE:request_time}\]"]
    }
    grok {
        match => {"log" => ["\/(?<app.name>[\w|-]+)\."]}
    }
    grok {
        match => {"message" => "%{LOGLEVEL:log.level}"}
    }
    date {
        locale => "zh"
        match => ["request_time","yyyy-MM-dd HH:mm:ss Z"]
        target => "@timestamp"
    }
    


}

output {
  elasticsearch { 
    hosts => ["localhost:9200"]
    index => "applogs-%{+YYYY.MM.dd}"
 }
  stdout { codec => rubydebug }
}