配置logstash-6.6.1.tar.gz

1.首先从官网下载

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.1.tar.gz

2.解压文件

tar -zxf logstash-6.6.1.tar.gz

3.配置logstash-6.6.1

input {
  kafka {
    bootstrap_servers => "172.18.2.139:9092,172.18.2.140:9092,172.18.2.141:9092"
    topics => ["nginxaccess","nginxerror"]
    group_id => "nginx"
    codec => "json"
    consumer_threads => 1
    decorate_events => true

  }
}

filter {
  json {
    source => "message"
    remove_field => [ "message" ]
  }
}

output {
  elasticsearch {
    hosts => ["172.18.2.144:9200"]
    index => "logs-%{+YYYY.MM.dd}"
    workers => 1
    user => elastic
    password => tWbWZc7NE3wYqS6DvSu4
  }
}

 

收集nginx的日志

 

4.启动命令

nohup /usr/local/logstash-6.6.1/bin/logstash -f /usr/local/logstash-6.6.1/config/logstash.conf --config.reload.automatic &

[2019-03-05T11:42:10,939][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 2.1.0
[2019-03-05T11:42:10,939][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : eec43959745f444f
[2019-03-05T11:42:11,208][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: zHRy03jZTBmhhYTuI883Xw
[2019-03-05T11:42:11,210][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=nginx] Discovered group coordinator hb-db-test2:9092 (id: 2147483646 rack: null)
[2019-03-05T11:42:11,212][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-0, groupId=nginx] Revoking previously assigned partitions []
[2019-03-05T11:42:11,213][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=nginx] (Re-)joining group
[2019-03-05T11:42:11,229][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-05T11:42:13,920][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=nginx] Successfully joined group with generation 10
[2019-03-05T11:42:13,922][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-0, groupId=nginx] Setting newly assigned partitions [nginxerror-0, nginxacce

 

 

启动成功。