Elasticsearch将一份原始日志，取出部分字段，另存一份到新索引

一、实现思路

1、logstash处完整多输出一份日志到指定索引

2、ingest node 针对指定索引做字段删除

二、具体配置

1、logstash output 追加以下内容

  if [@metadata][kafka][topic] =~ "loginsuccess|efunpf-paysuccess" {
    elasticsearch {
        hosts => ["10.20.91.21:9200","10.20.91.22:9200","10.20.91.23:9200"]
        index => "part_%{[@metadata][kafka][topic]}-%{+YYYY.MM}"
        user => "elastic"
        password => "xxxxxx"
    }
  }

2、ingest pipeline 配置

官方提示： 删除现有字段。如果一个字段不存在，将引发异常。

Remove processor | Elasticsearch Guide [8.9] | Elastic

所以加上这些参数：ignore_failure、ignore_missing

PUT _ingest/pipeline/loginsuccess-pipeline
{
  "description": "efunpf-part",
  "processors": [
    {
      "remove": {
        "ignore_failure": true,
        "ignore_missing":true ,
        "field": [
          "thirdId",
          "packageName",
          "apps",
          "idCard",
          "referrer",
          "@version",
          "signature",
          "timeZone",
          "unionId",
          "gpid",
          "partner",
          "adjAdId","region","platForm","androidid","mac","userArea","imei","gameVersion","host"
        ]
      }
    }
  ]
}

3、ES模板定义

PUT _template/part-loginsuccess
{
    "order" : 0,
    "index_patterns" : [
      "part_efunpf-loginsuccess*"
    ],
    "settings" : {
      "index.default_pipeline": "loginsuccess-pipeline"
    },
    "mappings" : {
    },
    "aliases" : { }
  }

三、验证数据

原始数据

新索引数据