Elastic测试笔记：logstash读取redis，指定时间字段

环境

前端filebeat 读取nginx日志（json格式），输出到中间redis，后端logstash从redis读取并解析。

## filebeat.yml
- input_type: log
  paths:
    - /var/log/nginx/prod_access.json
  document_type: "access_log"    ## 定义了type字段
  fields:
    log_source: "prod_nginx"
output.redis:
  hosts: "redis.dev"
  port: 37000
  key: "%{[fields.log_source]}"

## logstash配置文件
input {
    redis {
        host => "192.168.100.34"
        port => 6379
        password => "helloworld"
        key => "prod_nginx"
        data_type => "list"
    }
}
filter {
    mutate {
            gsub => ["message", "\\x", "\\\x"]    ##由于nginx日志存在中文字符，故对原始的\\x字符再增加\进行转意
            remove_field => ["beat"]
        }
    json {
        source => "message"
        remove_field => ["message"]
    }
    data {
        locale => "en"
        match => ["localtime", "dd/MMM/YYYY:HH:mm:ss Z"]
    }
    geoip {
        source => "clientip"
    }
}
output {
    if [type] == "access_log" {
        elasticsearch {
            hosts => "elk.dev:9200"
            index => "shopweb-%{+YYYYMMdd}"
        }
        file {
            path => "/data/aliyun/applogs/nginx_prod/prod_access.log"
            codec => line { format => "%{message}"}
        }
    }
}

预定义template

在向elasticsearch开始索引之前，创建模板用于定义其中的地理位置，时间字段

PUT /_template/prod_nginx
{
    "template": "prod_nginx*",
    "order": 0,
    "settings": {
      "index.number_of_replicas": "1",
      "index.number_of_shards": "3"
    },
    "mappings": {
      "access_log": {
        "properties": {
          "localtime": {
            "type": "date",
            "format": "dd/MMM/yyyy:HH:mm:ss Z"
          },
          "clientip": {
            "type": "ip"
          },
          "geoip": {
            "properties": {
                "location": {
                    "type": "geo_point"
                }
            }
          }
        }
      }
    }
}