机房意外掉电导致Elasticsearch的部分index无数据的修复过程

最新推荐文章于 2023-07-28 09:13:34 发布

宰祖宣

最新推荐文章于 2023-07-28 09:13:34 发布

阅读量967

点赞数

分类专栏： BigData Linux 运维文章标签： elasticsearch 大数据

本文链接：https://blog.csdn.net/zaizuxuan/article/details/129089544

版权

Linux 同时被 3 个专栏收录

18 篇文章 0 订阅

订阅专栏

BigData

7 篇文章 1 订阅

订阅专栏

运维

4 篇文章 0 订阅

订阅专栏

环境 :华为大数据集群FusionInsight V100R002C800SPC200、Elasticsearch 6.1.3、Kibana
问题产生原因：因机房意外掉电导致集群部分机器两次掉电导致Elasticsearch重启，Elasticsearch重启后看似正常但某些index无数据。经排查判断为Elasticsearch的部分index变成只读导致的。

解决步骤：

检查集群所有index是否有只读的情况：Kibana执行GET /_settings得到的结果中有index显示结果为read_only_allow_delete:"true"，表明此index为只读

	  "index_name": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "2000"
          }
        },
        "refresh_interval": "180s",
        "translog": {
          "flush_threshold_size": "1GB",
          "sync_interval": "120s",
          "durability": "async"
        },
        "blocks": {
          "read_only_allow_delete": "true",
          "write": "false"
        },
        "provided_name": "index_name",
        "max_result_window": "100000",
        "creation_date": "1591003803952",
        "unassigned": {
          "node_left": {
            "delayed_timeout": "30m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "4h3Hzw6fQ3ajh8WW1iDTOQ",
        "version": {
          "created": "6010399"
        },
        "routing": {
          "allocation": {
            "total_shards_per_node": "4"
          }
        },
        "number_of_shards": "48",
        "merge": {
          "scheduler": {
            "max_thread_count": "1"
          },
          "policy": {
            "segments_per_tier": "20",
            "max_merge_at_once": "20"
          }
        },
        "max_slices_per_scroll": "5000"
      }
    }
  },

修改只读：Kibana执行以下命令，结果返回true

PUT index_name/_settings
                {
                  "index.blocks.read_only_allow_delete":null
                }

验证：Kibana执行命令:GET jz_resource_3009/_settings查看结果里面不含以下内容则正常

"blocks": {
          "read_only_allow_delete": "true",
          "write": "false"
        },

结果为以下

{
  "index_name": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "2000"
          }
        },
        "refresh_interval": "180s",
        "translog": {
          "flush_threshold_size": "1GB",
          "sync_interval": "120s",
          "durability": "async"
        },
        "blocks": {
          "write": "false"
        },
        "provided_name": "index_name",
        "max_result_window": "100000",
        "creation_date": "1591003803952",
        "unassigned": {
          "node_left": {
            "delayed_timeout": "30m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "4h3Hzw6fQ3ajh8WW1iDTOQ",
        "version": {
          "created": "6010399"
        },
        "routing": {
          "allocation": {
            "total_shards_per_node": "4"
          }
        },
        "number_of_shards": "48",
        "merge": {
          "scheduler": {
            "max_thread_count": "1"
          },
          "policy": {
            "segments_per_tier": "20",
            "max_merge_at_once": "20"
          }
        },
        "max_slices_per_scroll": "5000"
      }
    }
  }
}