Elasticsearch-5.4.0安装repository-hdfs,快速在不同集群之间恢复数据

官方文档

https://www.elastic.co/guide/en/elasticsearch/plugins/5.4/repository-hdfs.html

下载环境包

wget https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-hdfs/repository-hdfs-5.4.0.zip

解压后得到elasticsearch文件夹,将该文件夹移动到es安装目录下plugins文件夹下并改名为repository-hdfs

如果集群设置了权限,需要进行安全修改

vim plugins/repository-hdfs/plugin-security.policy

添加下列信息

  permission java.lang.RuntimePermission "accessDeclaredMembers";
  permission java.lang.RuntimePermission "getClassLoader";
  permission java.lang.RuntimePermission "shutdownHooks";
  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
  permission javax.security.auth.AuthPermission "doAs";
  permission javax.security.auth.AuthPermission "getSubject";
  permission javax.security.auth.AuthPermission "modifyPrivateCredentials";
  permission java.security.AllPermission;
  permission java.util.PropertyPermission "*", "read,write";
  permission javax.security.auth.PrivateCredentialPermission "org.apache.hadoop.security.Credentials * \"*\"", "read";

vim conf/jvm.options
添加下列信息

-Djava.security.policy=file:////usr/local/elasticsearch-5.4.0/plugins/repository-hdfs/plugin-security.policy

每个节点都进行上述操作
重启集群

Snapshot官方api,了解更多详情

https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-snapshots.html#modules-snapshots

有两个es集群cluster_1和cluster_2,将索引index_1从cluster_1快速恢复到另一个集群cluster_2

分别在两个集群创建仓库

PUT http://cluster_1_esnode1:9200/_snapshot/my_backup

{
  "type": "hdfs",
  "settings": {
    "uri": "hdfs://namenode:8020/",
    "path": "/elasticsearch/respositories/my_hdfs_repository/cluster_1",
    "conf.dfs.client.read.shortcircuit": "true",
    "conf.dfs.domain.socket.path": "/var/lib/hadoop-hdfs/dn_socket"
  }
}

PUT http://cluster_2_esnode1:9200/_snapshot/my_backup

{
  "type": "hdfs",
  "settings": {
    "uri": "hdfs://namenode:8020/",
    "path": "/elasticsearch/respositories/my_hdfs_repository/cluster_2",
    "conf.dfs.client.read.shortcircuit": "true",
    "conf.dfs.domain.socket.path": "/var/lib/hadoop-hdfs/dn_socket"
  }
}

在cluster_1创建快照

PUT http://cluster_1_esnode1:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true

{
  "indices": "index_1"
}

复制快照

hdfs -dfs -cp /elasticsearch/respositories/my_hdfs_repository/cluster_1* /elasticsearch/respositories/my_hdfs_repository/cluster_2

在cluster_2恢复快照

POST http://cluster_2_esnode1:9200/_snapshot/my_backup/snapshot_1/_restore