logstash-将消息从file文件抽取到kafka

从多个文件抽取日志信息到kafka

input {
  file {
    type => "production-log"
    path => ["/var/log/gitlab/gitlab-rails/production_json.log"]
  }
  file {
    type => "shell-log"
    path => ["/var/log/gitlab/gitlab-shell/gitlab-shell.log"]
  }
}

output {
  if [type] == "production-log" {
    kafka {
      bootstrap_servers  => "ip1:9092,ip2:9092,ip3:9092"
      topic_id => "gitlab-platform-production-log"
      codec => plain {
        format => "%{message}"
      }
    }
  }
  if [type] == "shell-log" {
    kafka {
      bootstrap_servers  => "ip1:9092,ip2:9092,ip3:9092"
      topic_id => "gitlab-platform-shell-log"
      codec => plain {
        format => "%{message}"
      }
    }
  }
}

logstash kafka的输入输出插件默认codec为json格式。输出时，logstash会自动添加timestamp、hostname等json格式信息。如果输出信息只想要原本的message信息，则需要修改指定codec，如下：

codec => plain {
format => "%{message}"
}