logstash从kafka采集数据

1、修改logstash.conf

1.1添加kafka输入

bootstrap_servers：kafka broker地址

topics：kafka topic名称

    input{
      kafka {
        bootstrap_servers => ["xxx.xxx.xxx.xxx:9092"]
        auto_offset_reset => "latest"
        consumer_threads => 3
        decorate_events => true
        topics => ["xieyun1"]
        codec => "json"
      }
    }

1.2添加输出到es

参数说明：

hosts：es主机地址+端口号

index：输出到es的index索引名称

    output {
      if [index] == "xieyun1"{
        elasticsearch {
          hosts => ["elasticsearch-logging.kube-system:9200"]
          index => "kafkalog-%{+YYYY.MM.dd}"
        }
      }
    }

参考文章：

使用log-pilot进行日志收集 - 乔克爱运维 - 博客园

GitHub - AliyunContainerService/log-pilot: Collect logs for docker containers