ELK系列-如何从S3上收集一天的日志

---

一、配置Logstash

input {
   s3 {
     aws_credentials_file => "/app/aws-credentials.yaml"
     region => "us-west-1"
     bucket => "aws-test-qa"
     prefix => "QA_APILOG_2021-04/ip-10-0-1-10/2021-04-14"
     sincedb_path => "/app/logstash/.qa-kong-logs-s3-sincedb"
     type => "qa-kong-logs-s3"
   }
}
filter {
    if [type] == "qa-kong-logs-s3" {
        grok {
            match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:msg_content}" }
        }
 
        json {
            source => "msg_content"
            remove_field => ["message","msg_content"]
        }
 
        date {
            match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
            target => "@timestamp"
            remove_field => [ "timestamp" ]
        }
         
        if "_grokparsefailure" in [tags] {
            drop { }
        }
    }
}
output {
    if [type] == "qa-kong-logs-s3" {
        elasticsearch {
            hosts => ["localhost:9200"]
            user => "qa"
            password => "xxxxx"
            index => "qa-kong-logs-s3-%{+YYYY.MM.dd}"
        }
        stdout { codec => rubydebug }
    }
}

二、测试配置

 sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/qa-kong-logs-s3.conf

几秒钟后，如果没有语法错误，它应该会显示Configuration OK。否则，尝试读取错误输出，查看Logstash配置有什么问题

三、重启Logstash

  sudo systemctl restart logstash
  sudo systemctl enable logstash

四、检查Elasticsearch中的索引

  curl http://localhost:9200/_cat/indices/qa-kong-logs-s3*