ES数据备份到HDFS

1. 插件安装

使用hdfs进行快照备份,依赖于插件:Hadoop HDFS Repository Plugin

• 插件下载地址

  https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-hdfs/repository-hdfs-6.5.3.zip

• 离线http安装

  ./elasticsearch/bin/elasticsearch-plugin install http://10.0.6.122/elk-plugins/repository-hdfs-6.5.3.zip
  

• 安装完成后,重启集群所有节点

2. 创建仓库

• keytab文件

  1. 在所有节点的/home/elk/elasticsearch/config/下创建目录repository-hdfs,有则不用创建
  2. 将keytab文件放在repository-hdfs目录下,并改名为krb5.keytab.(文件名固定,就叫krb5.keytab)

• 创建hdfs仓库

  # 创建仓库
  PUT _snapshot/my_hdfs_repository
  {
    "type": "hdfs",
    "settings": {
      "uri": "hdfs://cdhdev01.nosql.tech:8020/",
      "path": "elasticsearch/repositories/my_hdfs_repository",
      "security.principal": "faith"
    }
  }
  # 查看仓库
  GET /_snapshot/my_hdfs_repository
  # 3. 查看仓库验证情况
  POST /_snapshot/my_hdfs_repository/_verify
  
  

3. 创建快照

#  创建快照
# PUT /_snapshot/my_backup/<snapshot-{now/d}>
# PUT /_snapshot/my_backup/%3Csnapshot-%7Bnow%2Fd%7D%3E
PUT /_snapshot/my_hdfs_repository/%3Cpdm_doc-%7Bnow%2Fd%7D%3E?wait_for_completion=true
{
  "indices": "pdm_doc",
  "ignore_unavailable": true,
  "include_global_state": false
}
# 查看快照
GET /_snapshot/my_hdfs_repository/pdm_doc-*
# 查看当前正在执行的快照
GET /_snapshot/my_hdfs_repository/_current

4. 快照恢复

# 全部恢复
POST /_snapshot/my_hdfs_repository/my_index-2019.01.29/_restore
# 指定恢复规则
POST /_snapshot/my_hdfs_repository/my_index-2019.01.29/_restore
{
  "indices": "my_index",
  "ignore_unavailable": true,
  "include_global_state": true,
  "rename_pattern": "my_index",
  "rename_replacement": "restored_my_index"
}