elasticsearch报错index read-only

背景

线上服务器的Elasticsearch服务大量报错,查询数据没问题,但是新增或者修改数据时,返回如下错误:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
  },
  "status": 403
}

原因

由于磁盘空间不足,导致Elasticsearch触发磁盘保护,强制将所有索引设置成了只读状态,相关参数见官方文档:Disk-based Shard Allocation

主要相关参数如下:

  • cluster.routing.allocation.disk.threshold_enabled:是否启动根据磁盘空间自动分配,默认为true
  • cluster.routing.allocation.disk.watermark.low:控制磁盘使用率的低水位,默认是85%,当一个节点的磁盘空间使用率超过85%,就不会给该节点分配新的shard
  • cluster.routing.allocation.disk.watermark.high:控制磁盘使用率的高水位,默认是90%,当一个节点的磁盘空间使用率超过90%,就会将该节点的部分shard转移到其他节点上
  • cluster.routing.allocation.disk.watermark.flood_stage:洪水水位线,默认为95%,当一个节点的磁盘空间使用率超过95%,就会把所有索引设为只读。这是最后一个保护措施,索引的只读状态必须通过人工手动解除
  • cluster.info.update.interval:检查磁盘使用率的频率,默认30s

本地模拟磁盘空间不足,当磁盘空间占用超过85%时,Elasticsearch输出如下日志:

[2020-05-16T02:22:38,049][INFO ][o.e.c.r.a.DiskThresholdMonitor] [DYUbwoP] low disk watermark [85%] exceeded on [DYUbwoPHSAWB08omk4MmEA][DYUbwoP][/opt/elasticsearch-6.8.0/data/nodes/0] 
free: 929.3mb[14.6%], replicas will not be assigned to this node

当磁盘空间占用超过90%时,Elasticsearch输出如下日志:

[2020-05-16T02:26:38,124][WARN ][o.e.c.r.a.DiskThresholdMonitor] [DYUbwoP] high disk watermark [90%] exceeded on [DYUbwoPHSAWB08omk4MmEA][DYUbwoP][/opt/elasticsearch-6.8.0/data/nodes/0] 
free: 599.9mb[9.4%], shards will be relocated away from this node

当磁盘空间占用超过95%时,Elasticsearch输出如下日志:

[2020-05-16T02:27:38,155][WARN ][o.e.c.r.a.DiskThresholdMonitor] [DYUbwoP] flood stage disk watermark [95%] exceeded on [DYUbwoPHSAWB08omk4MmEA][DYUbwoP][/opt/elasticsearch-6.8.0/data/nodes/0] 
free: 316.1mb[4.9%], all indices on this node will be marked read-only
[2020-05-16T02:29:11,336][WARN ][o.e.x.m.e.l.LocalExporter] [DYUbwoP] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_161]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_161]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_161]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_161]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_161]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_161]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_161]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_161]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_161]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) ~[?:?]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:665) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:470) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:465) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:945) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:785) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:171) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:99) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:121) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:440) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:553) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:256) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:607) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:411) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.0.jar:6.8.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
	at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:208) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.blockExceptions(TransportReplicationAction.java:254) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:99) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:779) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 19 more

解决

紧急给Elasticsearch的硬盘扩容,扩容完毕后执行以下语句关闭索引的只读状态:

PUT _all/_settings
{
  "index.blocks.read_only_allow_delete": null
}

如果来不及给Elasticsearch硬盘扩容,可以先关闭磁盘分配保护,让最后仅有的5%的磁盘空间缓冲一点时间,然后再给硬盘扩容。

  1. 关闭磁盘分配保护
PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.threshold_enabled": false
  }
}
  1. 关闭索引的只读状态
PUT _all/_settings
{
  "index.blocks.read_only_allow_delete": null
}
  1. 磁盘扩容完毕后,启用磁盘分配保护
PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.threshold_enabled": true
  }
}
©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页