Elasticsearch 慢日志管理手册

Elasticsearch 慢日志管理

Elasticsearch慢日志包括Search慢日志和Index慢日志，但只用于分片级别，即只应用于数据节点。

1. 慢日志配置

开启慢日志需要定义具体动作（query、fetch 或 index）、期望的事件记录等级（INFO、WARN、DEBUG 等）、以及时间阈值。用户可以根据业务场景，开启和调整相关配置。

1. Search慢日志

Elasticsearch中的Search分两个阶段进行：

• query：收集相关结果的文档ID，收集完成后仅返回与搜索匹配的文档的ID，不会返回字段或其他信息。
• fetch：使用来自query阶段的文档ID查询真实文档，完成整体的Search请求。

分片级的慢日志允许将慢速Search（包括query阶段和fetch阶段）记录到专用日志文件中，因此可全面了解query阶段和fetch阶段所花费的时间，并能够检查整个Search本身。

当业务出现查询耗时久的情况时，需查看Searching慢日志进行排查。
查询耗时越久，集群资源消耗越大。当日志存在大量的慢日志，需要排查集群资源及负载情况，获取瓶颈项进行及时处理，以保证集群的稳定性。

1. 配置慢日志监控阈值

# 示例1：在线配置所有索引的Search慢日志监控阈值
# curl -X PUT "<ES_HOST>:<ES_PORT>/_all/_settings?pretty" -H 'Content-Type: application/json' -d'
{
   
    "index.search.slowlog.threshold.query.warn": "200ms",
    "index.search.slowlog.threshold.query.info": "100ms",
    "index.search.slowlog.threshold.query.debug": "500ms",
    "index.search.slowlog.threshold.query.trace": "1s",
    "index.search.slowlog.threshold.fetch.warn": "200ms",
    "index.search.slowlog.threshold.fetch.info": "100ms",
    "index.search.slowlog.threshold.fetch.debug": "500ms",
    "index.search.slowlog.threshold.fetch.trace": "1s",
    "index.search.slowlog.level": "info"
}'

# 示例2：在线配置索引shakespeare的Search慢日志阈值

1)查看索引shakespeare的配置：
# curl -X GET "<ES_HOST>:<ES_PORT>/shakespeare/_settings?pretty"
{
   
  "shakespeare" : {
   
    "settings" : {
   
      "index" : {
   
        "creation_date" : "1649916911093",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "B-kIRD0mQ7m4kJsrhqNdKQ",
        "version" : {
   
          "created" : "6010199"
        },
        "provided_name" : "shakespeare"
      }
    }
  }
}

2)为索引shakespeare配置慢日志监控阈值
# curl -X PUT "<ES_HOST>:<ES_PORT>/shakespeare/_settings?pretty" -H 'Content-Type: application/json' -d'
{
   
    "index.search.slowlog.threshold.query.warn": "200ms",
    "index.search.slowlog.threshold.query.info": "100ms",
    "index.search.slowlog.threshold.query.debug": "500ms",
    "index.search.slowlog.threshold.query.trace": "1s",
    "index.search.slowlog.threshold.fetch.warn": "200ms",
    "index.search.slowlog.threshold.fetch.info": "100ms",
    "index.search.slowlog.threshold.fetch.debug": "500ms",
    "index.search.slowlog.threshold.fetch.trace": "1s",
    "index.search.slowlog.level": "info"
}'
{
   
  "acknowledged" : true
}


3)确认索引shakespeare的配置已生效
# curl -X GET "<ES_HOST>:<ES_PORT>/shakespeare/_settings?pretty"
{
   
  "shakespeare" : {
   
    "settings" : {
   
      "index" : {
   
        "search" : {
   
          "slowlog" : {
   
            "level" : "info",
            "threshold" : {
   
              "fetch" : {
   
                "warn" : "200ms",
                "trace" : "1s",
                "debug" : "500ms",
                "info" : "100ms"
              },
              "query" : {
   
                "warn" : "200ms",
                "trace" : "1s",
                "debug" : "500ms",
                "info" : "100ms"
              }
            }
          }
        },
        "number_of_shards" : "5",
        "provided_name" : "shakespeare",
        "creation_date" : "1649916911093",
        "number_of_replicas" : "1",
        "uuid" : "B-kIRD0mQ7m4kJsrhqNdKQ",
        "version" : {
   
          "created" : "6010199"
        }
      }
    }
  }
}


4)ES日志提示如下：
[2022-04-14T14:43:37,218][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.query.warn] from [-1] to [200ms]
[2022-04-14T14:43:37,219][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.query.info] from [-1] to [100ms]
[2022-04-14T14:43:37,219][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.query.debug] from [-1] to [500ms]
[2022-04-14T14:43:37,219][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.query.trace] from [-1] to [1s]
[2022-04-14T14:43:37,219][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.fetch.warn] from [-1] to [200ms]
[2022-04-14T14:43:37,220][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.fetch.info] from [-1] to [100ms]
[2022-04-14T14:43:37,220][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.fetch.debug] from [-1] to [500ms]
[2022-04-14T14:43:37,220][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.threshold.fetch.trace] from [-1] to [1s]
[2022-04-14T14:43:37,220][INFO ][o.e.c.s.IndexScopedSettings] [<ES_HOST>-10701] updating [index.search.slowlog.level] from [TRACE] to [info]

# 示例3：编辑配置文件以全局配置Search慢日志监控阈值（分别执行的query和fetch阶段设置阈值）（需要重启生效）

# vim elasticsearch.yml
新增如下配置内容：

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.search.slowlog.level: info  # 从5.6起，官网配置示例中新增此配置

注：设置慢日志时允许控制将记录日志的日志级别( warn、info、debug、trace) ，多个级别的好处是能够快速发现违反特定阈值的慢日志。正常情况下，可以通过设置适当的阈值来关闭"more verbose"的日志记录器，从而配置Search慢日志和Index慢日志级别。如仅需index.indexing.slowlog.level = INFO，那么可以配置index.indexing.slowlog.threshold.index.debug和index.indexing.slowlog.threshold.index.trace为-1，即关闭这两个级别的监控。

2. 配置慢日志文件

# 示例1：低版本（2.4版本及以下版本）默认是在logging.yml文件中配置搜索慢日志文件
index_search_slow_log_file:
  type: dailyRollingFile
  file: ${
   path.logs}/${
   cluster.name}_index_search_slowlog.log
  datePattern: "'.'yyyy-MM-dd"
  layout:
    type: pattern
    conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

# 示例4：5.0及以上版本默认是在log4j2.properties文件中配