Elasticsearch：如何处理 ingest pipeline 中的异常

于 2020-11-17 21:01:35 发布
阅读量3.6k 收藏 6
点赞数 1
分类专栏： Elastic Elasticsearch 文章标签： elasticsearch 大数据
本文为博主原创文章，未经博主允许不得转载。
本文链接：https://blog.csdn.net/UbuntuTouch/article/details/109749068
版权
Elastic 同时被 2 个专栏收录
1496 篇文章 903 订阅
订阅专栏
Elasticsearch
1023 篇文章 596 订阅
订阅专栏

在我之前的文章 “如何在 Elasticsearch 中使用 pipeline API 来对事件进行处理” 中，我详细地介绍了如何创建并使用一个 ingest pipeline。简单地说 pipeline 是一系列处理器的定义，这些处理器将按照声明的顺序执行。 pipeline 包含两个主要字段：描述和处理器列表：

在这里，特别需要指出的是 pipeline 是运行于 ingest node 之上的。所有的 ingest pipeline 被保存于 cluster state 中。

Pipeline 是如何工作的

下面是一个定义 pipeline 的例子：

PUT _ingest/pipeline/apache-log
{
  "description": "This is an example for apache logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{COMMONAPACHELOG}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"]
      }
    },
    {
      "remove": {
        "field": "message"
      }
    }
  ]
}

上面的 processors 将被依次执行。我们可以使用如下的例子来进行调用：

PUT logs/_doc/1?pipeline=apache-log
{
  "message": "83.149.9.216 - - [17/May/2015:10:05:03 +0000] \"GET / HTTP/1.1\" 200 24"
}

上面的命令的输出为：

{
  "took" : 20,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logs",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "request" : "/",
          "auth" : "-",
          "ident" : "-",
          "verb" : "GET",
          "@timestamp" : "2015-05-17T10:05:03.000Z",
          "response" : "200",
          "bytes" : "24",
          "clientip" : "83.149.9.216",
          "httpversion" : "1.1",
          "timestamp" : "17/May/2015:10:05:03 +0000"
        }
      }
    ]
  }
}

我们可以通过如下的命令来查询已经被 apache-log pipeline 处理过的文档：

GET logs/_doc/1

上面的命令将返回：

{
  "_index" : "logs",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 2,
  "_seq_no" : 1,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "request" : "/",
    "auth" : "-",
    "ident" : "-",
    "verb" : "GET",
    "@timestamp" : "2015-05-17T10:05:03.000Z",
    "response" : "200",
    "bytes" : "24",
    "clientip" : "83.149.9.216",
    "httpversion" : "1.1",
    "timestamp" : "17/May/2015:10:05:03 +0000"
  }
}

从上面我们可以看出来：经过 apache-log 这个 pipeline 的一组 processors，我们成功地把 log 进行结构化并丰富我们的数据。我们通过 grok processor 把数据进行结构化；通过 date processor 把 @timestamp 设置为和 timestamp 一样的值；通过 remove 把 message 字段去掉。

在设计 pipeline 时，我们很少情况下直接让它作用于我们的文档。在更多的情况下，我们希望通过一些测试文档来检验我们的 pipeline 的正确性。否则一个不正确的 pipeline 会把我们的数据搞坏。我们可以通过 _simulate 来进行检测。针对我们的情况：

POST _ingest/pipeline/apache-log/_simulate
{
  "docs": [
    {
      "_source": {
        "message": "83.149.9.216 - - [17/May/2015:10:05:03 +0000] \"GET / HTTP/1.1\" 200 24"
      }
    }
  ]
}

在上面 docs 可以定义各种可能的文档类型来进行测试。它是一个数组。我们可以同时定义多个文档来进行测试。上面的命令的返回结果是：

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "request" : "/",
          "auth" : "-",
          "ident" : "-",
          "verb" : "GET",
          "@timestamp" : "2015-05-17T10:05:03.000Z",
          "response" : "200",
          "bytes" : "24",
          "clientip" : "83.149.9.216",
          "httpversion" : "1.1",
          "timestamp" : "17/May/2015:10:05:03 +0000"
        },
        "_ingest" : {
          "timestamp" : "2020-11-17T11:09:35.351117Z"
        }
      }
    }
  ]
}

我们可以看到模拟出来的结果。

上面的个文档经过一组 pipeline processor 的处理，我们不能看出来每个 processor 的处理结果。这个时候，我们可以添加上  verbose 来进行查看每个 processor 的处理情况：

POST _ingest/pipeline/apache-log/_simulate?verbose
{
  "docs": [
    {
      "_source": {
        "message": "83.149.9.216 - - [17/May/2015:10:05:03 +0000] \"GET / HTTP/1.1\" 200 24"
      }
    }
  ]
}

上面的返回的结果是：

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "apache-log",
              "timestamp" : "2020-11-17T11:11:43.039149Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "success",
          "doc" : {
            "_index" : "_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 24""",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "apache-log",
              "timestamp" : "2020-11-17T11:11:43.039149Z"
            }
          }
        },
        {
          "processor_type" : "remove",
          "status" : "success",
          "doc" : {
            "_index" : "_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "apache-log",
              "timestamp" : "2020-11-17T11:11:43.039149Z"
            }
          }
        }
      ]
    }
  ]
}

上面详细地记录了每个 processor 所处理的结果。它可以非常方便地让我对每个 processor 的运行进行分解，并对我们的错误进行排查。

如何处理 pipeline 错误

当我们使用 pipeline 处理一个文档的时候，有时并不是所有的文档都很规范，那么这个时候就会出现文档不能被正确解析或者处理的情况：

当它不能正常解析的时候，它会返回客户端一个错误的信息，表明它不能被正确地处理。这是一种默认的动作。另外一种处理方式是，我们可以通过 on_failure 来处理我们的错误：

当错误发生后，我们可以创建另外一组 processor 来处理我们的错。在通常的情况下，我们可以使用 set processor 来对文档做一些相关的处理。比如它可以帮我们把错误的文档信息记录下来，并保存于另外一个索引之中。之后我们可以检查这个索引，并根据错误信息来解决这个问题。在这种情况下：

在这种情况下，下面的 processor 将不被执行。在这种情况下，客户端将不再收到失败的消息。我们可以在 on_failure 中把错误的信息存放于另外一个索引之中。我们可以定义一组的 processor 来处理这个错误，比如：

在上面，我们可以通过 remove 以及 set 两个 processor 对失败的文档进行处理。我们甚至可以针对这个 failure 的 processor 组再进行额外的 on_failure 处理。在上面，我们可以通过 set process 来做一些处理，比如如果当前的 date 是错误的，我们可以设置一个默认的日期，或者使用当前的日期，让后重新让这个文档进入队列进行处理：

这个完全依赖于你自己的业务需求来进行处理。

下面，我将以一个例子来进行演示：

GET _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

在上面，我们把文档写入到 my_index 之中，尽管我们只是模拟：

{
  "docs" : [
    {
      "doc" : {
        "_index" : "my_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "request" : "/",
          "auth" : "-",
          "ident" : "-",
          "verb" : "GET",
          "@timestamp" : "2015-05-17T10:05:03.000Z",
          "response" : "200",
          "bytes" : "24",
          "clientip" : "83.149.9.216",
          "httpversion" : "1.1",
          "timestamp" : "17/May/2015:10:05:03 +0000"
        },
        "_ingest" : {
          "timestamp" : "2020-11-17T11:55:43.679709Z"
        }
      }
    }
  ]
}

在上面我们可以看出来经过 grok 的处理，bytes 是一个字符串。我们可以通过 convert processor 来把这个字段转变为整数：

GET _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

那么现在的结果是：

{
  "docs" : [
    {
      "doc" : {
        "_index" : "my_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "request" : "/",
          "auth" : "-",
          "ident" : "-",
          "verb" : "GET",
          "@timestamp" : "2015-05-17T10:05:03.000Z",
          "response" : "200",
          "bytes" : 24,
          "clientip" : "83.149.9.216",
          "httpversion" : "1.1",
          "timestamp" : "17/May/2015:10:05:03 +0000"
        },
        "_ingest" : {
          "timestamp" : "2020-11-17T12:01:38.662559Z"
        }
      }
    }
  ]
}

从上面，我们可以看出来 bytes 现在变为整型值了。当然我们也可以如法炮制，把上面的接口调用添加  verbose 参数来查看每个 processor 的执行情况。为了调试的方便，我们甚至可以对每个 processor 添加一个 tag，这样当我们使用 verbose 时可以很轻松地知道是那个 processor：

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

在上面，我们针对 response 的 convert processor 添加了一个叫做 convert_response 的 tag。这样当我们搜寻 convert_response 更加容易，否则有两个 convert processor，我们不容易区分，尽管执行是按照次序先后执行的。

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:07:36.432606Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 24""",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:07:36.432606Z"
            }
          }
        },
        {
          "processor_type" : "remove",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:07:36.432606Z"
            }
          }
        },
        {
          "processor_type" : "convert",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : "200",
              "bytes" : 24,
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:07:36.432606Z"
            }
          }
        },
        {
          "processor_type" : "convert",
          "status" : "success",
          "tag" : "convert_reponse",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : 200,
              "bytes" : 24,
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:07:36.432606Z"
            }
          }
        }
      ]
    }
  ]
}

我们从上面的输出中可以看到 convert_response 的 tag。

下面，我们来模拟一个错误的文档，从而使得 processor 不能被正确地解析。我们把文档中 2015 中的 “5” 去掉:

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/201:10:05:03 +0000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

显然这样会造成一个不能被正确解析的文档。返回的错误如下：

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "error",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : """Provided Grok expressions do not match field value: [83.149.9.216 - - [17/May/201:10:05:03 +0000] "GET / HTTP/1.1" 200 24]"""
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : """Provided Grok expressions do not match field value: [83.149.9.216 - - [17/May/201:10:05:03 +0000] "GET / HTTP/1.1" 200 24]"""
          }
        }
      ]
    }
  ]
}

当我们发送这样的错误时，我们很容易发现这个问题。上面显示 grok pattern 不匹配。我们可以对文档进行如下的修改：

 "message": """83.149.9.216 - - [17/May/2015:10:05:03 +200] "GET / HTTP/1.1" 200 24"""

在上面，我们把时间中的 +0000 修改为 +000，也就是少了一个 0。我们接下来运行：

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

上面的命令返回的结果为：

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:20:43.098763Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "error",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "unable to parse date [17/May/2015:10:05:03 +000]"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "unable to parse date [17/May/2015:10:05:03 +000]",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "failed to parse date field [17/May/2015:10:05:03 +000] with format [dd/MMM/yyyy:HH:mm:ss Z]",
              "caused_by" : {
                "type" : "date_time_parse_exception",
                "reason" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21"
              }
            }
          }
        }
      ]
    }
  ]
}

这次显然和之前的是不一样的。 grok pattern 能正确地解析我们的文档，但是我们的 date  processor 解析时间出现了问题。

处理这种问题，我们有两种方法：

  • pipeline 级别来处理
  • processor 级别来处理

Pipeline 级别来处理

我们在 pipeline 的后面添加一个 on_failure:

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

在上面，我添加了如下的代码：

    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed"
        }
      }
    ]

 在这里，我们指定了另外一个索引叫做 failed。执行上面的 pipeline:

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:25:50.517958Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "error",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "unable to parse date [17/May/2015:10:05:03 +000]"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "unable to parse date [17/May/2015:10:05:03 +000]",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "failed to parse date field [17/May/2015:10:05:03 +000] with format [dd/MMM/yyyy:HH:mm:ss Z]",
              "caused_by" : {
                "type" : "date_time_parse_exception",
                "reason" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21"
              }
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "failed",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:25:50.517958Z",
              "on_failure_processor_type" : "date"
            }
          }
        }
      ]
    }
  ]
}

显然第一步是成功的，第二步有错误，紧接着它执行了 on_failure，并在里面执行了 set processor 把索引修改为 failed。你之后可以直接在 failed 索引中进行查看。

            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:25:50.517958Z",
              "on_failure_processor_type" : "date"
            }

在上面它指出来在 ingest 是的一个错误信息，我们可以接着记录这个错误的信息：

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed"
        }
      },
      {
        "set": {
          "tag": "mark_failure",
          "field": "failure",
          "value": {
            "message": "{{_ingest.on_failure_message}}"     
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

在上面我们设置 failure 字段，并记录一个 object。运行上面的 pipeline:

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:39:09.206999Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "error",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "unable to parse date [17/May/2015:10:05:03 +000]"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "unable to parse date [17/May/2015:10:05:03 +000]",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "failed to parse date field [17/May/2015:10:05:03 +000] with format [dd/MMM/yyyy:HH:mm:ss Z]",
              "caused_by" : {
                "type" : "date_time_parse_exception",
                "reason" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21"
              }
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "failed",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:39:09.206999Z",
              "on_failure_processor_type" : "date"
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "tag" : "mark_failure",
          "doc" : {
            "_index" : "failed",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "failure" : {
                "message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21"
              },
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:39:09.206999Z",
              "on_failure_processor_type" : "date"
            }
          }
        }
      ]
    }
  ]
}

显然在上面的 _source 中新增加了一个叫做 failure 的字段。它含有相应的错误信息。由于上面的 failure 是一个 object, 事实上我们可以为它添加多个字段，比如：

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed"
        }
      },
      {
        "set": {
          "tag": "mark_failure",
          "field": "failure",
          "value": {
            "message": "{{_ingest.on_failure_message}}",
            "processor": "{{_ingest.on_failure_processor_type}}"
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

我们添加了字段 processor，这样我们可以更容易知道是哪个 processor 出了问题：

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:42:27.811805Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "error",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "unable to parse date [17/May/2015:10:05:03 +000]"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "unable to parse date [17/May/2015:10:05:03 +000]",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "failed to parse date field [17/May/2015:10:05:03 +000] with format [dd/MMM/yyyy:HH:mm:ss Z]",
              "caused_by" : {
                "type" : "date_time_parse_exception",
                "reason" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21"
              }
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "failed",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:42:27.811805Z",
              "on_failure_processor_type" : "date"
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "tag" : "mark_failure",
          "doc" : {
            "_index" : "failed",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "failure" : {
                "message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
                "processor" : "date"
              },
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:42:27.811805Z",
              "on_failure_processor_type" : "date"
            }
          }
        }
      ]
    }
  ]
}

上面的这种处理是在 pipeline 级的处理。

Processor 级处理

我们直接可以针对每个 processor 进行错误的捕获及处理。比如针对 date process:

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ],
          "on_failure": [
            {
              "set": {
                "tag": "set_default_date",
                "field": "@timestamp",
                "value": "{{_ingest.timestamp}}"
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer"
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed"
        }
      },
      {
        "set": {
          "tag": "mark_failure",
          "field": "failure",
          "value": {
            "message": "{{_ingest.on_failure_message}}",
            "processor": "{{_ingest.on_failure_processor_type}}"
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24"""
      },
      "_index": "my_index"
    }
  ]
}

在上面，我们为 date processor 添加了如下的 on_failure 代码：

      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ],
          "on_failure": [
            {
              "set": {
                "tag": "set_default_date",
                "field": "@timestamp",
                "value": "{{_ingest.timestamp}}"
              }
            }
          ]
        }
      }

当错误发生时，我们直接使用 _ingest.timestamp 作为 @timestamp 的值。运行上面的 pipeline:

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:49:49.720153Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "error",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "unable to parse date [17/May/2015:10:05:03 +000]"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "unable to parse date [17/May/2015:10:05:03 +000]",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "failed to parse date field [17/May/2015:10:05:03 +000] with format [dd/MMM/yyyy:HH:mm:ss Z]",
              "caused_by" : {
                "type" : "date_time_parse_exception",
                "reason" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21"
              }
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "tag" : "set_default_date",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +000] "GET / HTTP/1.1" 200 24""",
              "@timestamp" : "2020-11-17T12:49:49.720153Z",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "Text '17/May/2015:10:05:03 +000' could not be parsed at index 21",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:49:49.720153Z",
              "on_failure_processor_type" : "date"
            }
          }
        },
        {
          "processor_type" : "remove",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2020-11-17T12:49:49.720153Z",
              "response" : "200",
              "bytes" : "24",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:49:49.720153Z"
            }
          }
        },
        {
          "processor_type" : "convert",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2020-11-17T12:49:49.720153Z",
              "response" : "200",
              "bytes" : 24,
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:49:49.720153Z"
            }
          }
        },
        {
          "processor_type" : "convert",
          "status" : "success",
          "tag" : "convert_reponse",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2020-11-17T12:49:49.720153Z",
              "response" : 200,
              "bytes" : 24,
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:49:49.720153Z"
            }
          }
        }
      ]
    }
  ]
}

显然在这次运行中，当错误发生时 set_default_date，被调用，并且 @timestamp" : "2020-11-17T12:49:49.720153Z。显然是 ingest pipeline 被执行的时间。这个和之前的文档中的时间相差很远。这个完全依赖于你自己的业务设计，看你具体想使用什么值。

接下来，我们假如我们已经修正了我们的时间，重新变为 +0000。我们把 bytes 的数值修改为一个不可以转换为数值的字符，比如 "-".

    "message": """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 -"""

重新运行 pipeline，我们将会发现错误信息：

          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "field [bytes] not present as part of path [bytes]"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "field [bytes] not present as part of path [bytes]"
          }

如法炮制，我们可以为这个 processor 定制一个 on_failure:

GET _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{COMMONAPACHELOG}"
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/yyyy:HH:mm:ss Z"
          ],
          "on_failure": [
            {
              "set": {
                "tag": "set_default_date",
                "field": "@timestamp",
                "value": "{{_ingest.timestamp}}"
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "integer",
          "on_failure":[
            {
              "set": {
                "field": "bytes",
                "value": -1
              }
            }
          ]
        }
      },
      {
        "convert": {
          "tag": "convert_reponse",
          "field": "response",
          "type": "integer"
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed"
        }
      },
      {
        "set": {
          "tag": "mark_failure",
          "field": "failure",
          "value": {
            "message": "{{_ingest.on_failure_message}}",
            "processor": "{{_ingest.on_failure_processor_type}}"
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 -"""
      },
      "_index": "my_index"
    }
  ]
}

我们添加了如想的代码：

      {
        "convert": {
          "field": "bytes",
          "type": "integer",
          "on_failure":[
            {
              "set": {
                "field": "bytes",
                "value": -1
              }
            }
          ]
        }
      }

也就是说，当错误发生后，我们直接把 bytes 设置为 -1：

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "grok",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "response" : "200",
              "clientip" : "83.149.9.216",
              "verb" : "GET",
              "httpversion" : "1.1",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 -""",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:59:19.385189Z"
            }
          }
        },
        {
          "processor_type" : "date",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "message" : """83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET / HTTP/1.1" 200 -""",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : "200",
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:59:19.385189Z"
            }
          }
        },
        {
          "processor_type" : "remove",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "auth" : "-",
              "ident" : "-",
              "response" : "200",
              "clientip" : "83.149.9.216",
              "verb" : "GET",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:59:19.385189Z"
            }
          }
        },
        {
          "processor_type" : "convert",
          "status" : "error",
          "error" : {
            "root_cause" : [
              {
                "type" : "illegal_argument_exception",
                "reason" : "field [bytes] not present as part of path [bytes]"
              }
            ],
            "type" : "illegal_argument_exception",
            "reason" : "field [bytes] not present as part of path [bytes]"
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : "200",
              "bytes" : -1,
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "on_failure_message" : "field [bytes] not present as part of path [bytes]",
              "on_failure_processor_tag" : null,
              "timestamp" : "2020-11-17T12:59:19.385189Z",
              "on_failure_processor_type" : "convert"
            }
          }
        },
        {
          "processor_type" : "convert",
          "status" : "success",
          "tag" : "convert_reponse",
          "doc" : {
            "_index" : "my_index",
            "_type" : "_doc",
            "_id" : "_id",
            "_source" : {
              "request" : "/",
              "auth" : "-",
              "ident" : "-",
              "verb" : "GET",
              "@timestamp" : "2015-05-17T10:05:03.000Z",
              "response" : 200,
              "bytes" : -1,
              "clientip" : "83.149.9.216",
              "httpversion" : "1.1",
              "timestamp" : "17/May/2015:10:05:03 +0000"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-11-17T12:59:19.385189Z"
            }
          }
        }
      ]
    }
  ]
}

我们可以从上面的输出结果中看出来 byte 已经被设置为 -1。

好了今天的分享就到这里。希望大家知道如何来处理 pipeline 的错误，并做相应的处理。

Elastic 中国社区官方博客
关注
专栏目录
博客
Elasticsearch 开放推理 API 新增阿里云 AI 搜索支持
09-19 1413
我们很高兴地宣布，Elasticsearch 开放推理 API 新增了阿里云 AI 搜索的集成功能。此项工作使 Elastic 用户能够直接连接到阿里云 AI 平台。使用 Elasticsearch 向量数据库构建 RAG 应用程序的开发人员可以使用 semantic_text 存储和使用托管在阿里云 AI 搜索平台上的模型生成的密集和稀疏嵌入。此外，Elastic 用户现在可以集成访问重新排名模型，以增强语义重新排名和LLM 系列。
博客
Elastic 线下 Meetup 将于 2024 年 9 月 21 号在成都举办
08-29 1638
2024 Elastic Meetup 成都站活动，由 Elastic、腾讯、新智锦绣联合举办，现诚邀广大技术爱好者及开发者参加。
博客
Elastic：如何成为一名 Elastic 认证工程师，Elastic 认证分析师及 Elastic 认证可观测性工程师
10-28 2万+
Elasticsearch 无疑是是目前世界上最为流行的大数据搜索引擎。根据 DB - Engines 的统计，Elasticsearch 雄踞排行榜第一名，并且市场还在不断地扩大：能够成为一名 Elastic 认证工程师也是很多开发者的梦想。这个代表了 Elastic 的最高认证，在业界也得到了很高的认知度。得到认证的工程师，必须除了具有丰富的 Elastic Stack 知识，而且必须有丰富的操作及有效的解决问题的能力。拥有这个认证证书，也代表了个人及公司的荣誉。针对个人的好处是，你可以..
博客
Elastic：开发者上手指南
02-25 15万+
你们好，我是Elastic的刘晓国。如果大家想开始学习Elastic的话，那么这里将是你理想的学习园地。在我的博客几乎涵盖了你想学习的许多方面。在这里，我来讲述一下作为一个菜鸟该如何阅读我的这些博客文章。我们可以按照如下的步骤来学习：1)Elasticsearch简介：对Elasticsearch做了一个简单的介绍2)Elasticsearch中的一些重要概念:cluster, n..........................................................
博客
Elastic：培训视频 - ​在生产环境中配置 Fleet Server 和 Elastic Agent 之间的安全
01-06 1万+
在这篇文章中，我将会把我写的有些内容录制成视频，供大家参考。希望对大家有所帮助。优酷的视频频道地址在这里。Elastic 简介及Elastic Stack 安装：优酷，腾讯 Elastic Stack docker 部署：优酷，腾讯 Elasticsearch中的一些重要概念（Cluster/Shards/Replica/Document/Type/Index）：优酷，腾讯 开始使用El...............
博客
Elasticsearch 简介
08-08 17万+
Elasticsearch是一个非常强大的搜索引擎。它目前被广泛地使用于各个IT公司。Elasticsearch是由Elastic公司创建并开源维护的。它的开源代码位于https://github.com/elastic/elasticsearch。同时，Elastic公司也拥有Logstash及Kibana开源项目。这个三个开源项目组合在一起，就形成了 ELK软件栈。他们三个共同形成了一个强大的...
博客
Observability：构建下一代托管接入服务
09-21 803
随着无服务器（serverless）的引入，向 Elastic Cloud 发送可观察性数据变得越来越容易。你可以在中创建一个可观察性无服务器项目，并将可观察性信号直接发送到 Elasticsearch 或通过我们一直称之为托管接收服务（类似于 Elastic APM 服务器的完全托管解决方案）发送。你不再需要担心 APM 服务器的大小和配置。相反，你可以将数据直接发送到托管接收服务，让我们的平台为你完成艰难的扩展工作。
博客
LangChain 和 Elasticsearch 加速构建 AI 检索代理
09-20 1176
​Elastic 和 LangChain 很高兴地宣布发布新的 LangGraph 检索代理模板，旨在简化需要代理使用 Elasticsearch 进行代理检索的生成式人工智能 (GenAI) 代理应用程序的开发。此模板预先配置为使用 Elasticsearch，允许开发人员使用 LangChain 和 Elasticsearch 快速构建代理。要立即开始，请访问 github 上的项目：https://github.com/langchain-ai/retrieval-agent-template
博客
Elasticsearch：检索增强生成背后的重要思想
09-20 1817
星期天晚上 10 点，我九年级的女儿哭着冲进我的房间。她说她对代数一无所知，注定要失败。我进入超级妈妈模式，却发现我一点高中数学知识都不记得了。于是，我做了任何一位超级妈妈在 2024 年都会做的事情，前往 ChatGPT 寻求帮助。这些生成式 AI 聊天机器人太棒了。我很快就得到了关于如何解决她所有问题的详细解释。“但是妈妈，”她抱怨道，“我们不是这样做的！”如果我能让 ChatGPT 的答案与班级的特定做事方式保持一致就好了……
博客
Elasticsearch：一次生产集群 ES Watcher 失效的深度排查与分析 - 全过程剖析与解决方案
09-19 1321
Watcher是 Elasticsearch 提供的一项监控和告警服务，允许用户定义、管理警报规则，并持续跟踪网络和数据的变化。基于 Elasticsearch 的分布式架构，Watcher 可以对基础设施、索引数据、集群健康等多种指标进行实时监控和自动化告警。当预设条件被触发时，Watcher 会及时发送通知。这项功能特别适用于日志数据的监控、应用性能追踪，以及其他需要自动化响应的场景，帮助运维人员快速响应潜在问题，提升系统稳定性和可靠性。1）定时监控数据。
博客
Elastic 的 OpenTelemetry PHP 发行版简介
09-18 1296
宣布 OpenTelemetry PHP 的 Elastic 发行版的第一个 alpha 版本。在本篇博文中了解使用 OpenTelemetry 来检测 PHP 应用程序是多么简单。我们很高兴推出 OpenTelemetry PHP 的 Elastic Distribution 的第一个 alpha 版本。在这篇文章中，你将了解如何轻松安装和设置 PHP 应用程序的监控。
博客
Elasticsearch：Open Crawler 现已进入测试阶段
09-18 1911
今天，我们发布了 Open Crawler 0.2 版，该版本也已升级为测试版！Open Crawler 最初于作为技术预览版发布（0.1 版）。从那时起，我们一直在迭代产品并添加了几个新功能。要访问这些更改，你可以使用最新的 Docker 工件，或直接从 GitHub 存储库下载源代码。按照我们文档中的设置说明开始使用。
博客
Observability：日志管理的最佳实践 - 利用日志更快地解决问题
09-17 1108
在当今快速发展的软件开发环境中，高效的日志管理对于维护系统可靠性和性能至关重要。随着基础架构和应用程序组件的不断扩展和复杂化，运营和开发团队的职责也不断增加且越来越复杂。这篇博文概述了有效日志管理的最佳实践，以应对数据量不断增长、基础架构复杂以及快速解决问题的需求等挑战。
博客
Kibana：如何使用魔法公式创建具有影响力的可视化效果？（第 1 部分）
09-16 872
在上一篇博文《作为非设计师设计直观的 Kibana 仪表板》中，我们强调了创建直观仪表板的重要性。它展示了简单的更改（分组主题、更改类型图表等）如何对理解数据产生影响。在提供Kibana 数据分析或课程等课程时，我们强调了这篇博文以及这些更改如何帮助将重要信息带到表面。我喜欢一种互补的方法来实现这一目标：使用两种颜色将最高数据值与常见数据值区分开来。为了说明这个想法，我们将使用示例航班数据数据集。现在，让我们比较两个可视化，按航班总数排名前 10 个目的地国家。哪个可视化的影响更大？
博客
Observability：OpenTelemetry Java Agent Elastic 分布正式发布
09-15 1346
Elastic 宣布 OpenTelemetry (OTel) Java Agent 的 Elastic 发行版正式发布，这是一个与 OTel 完全兼容的代理，具有一组丰富实用的附加功能。随着 Elastic 继续致力于 OpenTelemetry (OTel)，我们很高兴地宣布) 正式上市。
博客
了解 BSI IT Grundschutz：使用 GenAI 搜索你的（私人）PDF 宝藏的秘诀
09-14 1114
有没有想过如何让多年来收集的文档可以通过语义而不是关键字进行搜索？处理充满有价值信息的 PDF 可能具有挑战性，尤其是在分块和创建跨多种语言的可搜索数据时。这篇博文将指导你将文档集合转换为 AI 驱动的语义搜索系统。我们将探索 Elastic 如何为 RAG 提供端到端解决方案：提取 PDF、将其处理成块、向量化内容，并提供一个交互式游乐场来查询和与数据交互。探索如何让你的信息不仅更易于访问，而且真正具有对话性。你将能够测试如何轻松创建相关信息摘要，了解每个给出的答案的相关性分数，并最大限度地了解创建答
博客
用于安全研究的 Elastic Container Project
09-13 1308
​Elastic Stack 是一个模块化数据分析生态系统。虽然这允许工程灵活性，但建立开发实例进行测试可能很麻烦。建立 Elastic Stack 的最简单方法是使用 Elastic Cloud - 这是完全一站式的。但是，在某些情况下，Elastic Cloud 可能不适用于你的测试环境。为了帮助解决这个问题，本博客将为你提供必要的信息，以便快速轻松地建立本地、完全容器化、TLS 安全的 Elastic Stack，并启用 Fleet 和检测引擎。你将能够创建 Fleet 策略，在本地主机或 VM 上
博客
使用 Elastic 和 LM Studio 的 Herding Llama 3.1
09-13 1981
最新的 LM Studio 0.3 更新使 Elastic 的安全 AI Assistant 能够更轻松、更快速地与 LM Studio 托管模型一起运行。在这篇博客中，Elastic 和 LM Studio 团队将向你展示如何在几分钟内开始使用。如果你在同一网络上或在你的机器上本地工作，则不再需要设置代理。
博客
Elastic 线上 Meetup “Elastic 认证及 Watcher 使用分享” 将于 9 月 18 号举办
09-13 687
Elastic 线上 Meetup “Elastic 认证及 Watcher 使用分享” 将于 9 月 18 号举办
博客
Elasticsearch 开放 inference API 为 Hugging Face 添加了原生分块支持
09-13 1355
​借助 Elasticsearch 开放推理 API，你可以使用 Hugging Face 的推理端点（Inference Endpoints）在 Elasticsearch 之外执行推理。这样你就可以使用 Hugging Face 的可扩展基础架构，包括在 GPU 和 AI 加速器上执行推理的能力。使用 Hugging Face 生成的嵌入的能力是作为 Elasticsearch 8.11 中的第一个开放推理 API 集成引入的，从那时起，我们一直在努力工作，并使用更强大的功能对其进行了更新，让您可以事
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值