Elasticsearch数据安全_CCR跨集群备份(Cross Cluster Replication)

需求前提：

1. 两个ES集群之间的网络是互通的，可以互相操作的
2. 两个集群都具有CCR的许可证(白银级别，需要付费的服务)
3. 如果打开了安全验证(Elasticsearch security is enabled)

         3.1- 对于从集群的索引，用户必须有write, monitor, and manage_follow_index的权利

         3.2- 对于主集群的索引，用户必须有read and monitor的权利

         3.3- 对于整个集群而言，用户必须有manage_ccr的权利

 

操作步骤：

1. 在从集群执行命令创建远程主集群链接
2. 在从集群创建index同步主集群的数据(可以设置同步的参数等，参数解释见https://blog.csdn.net/zhang5324496/article/details/107162699)
3. 解除从集群index同步主集群数据

    假设主集群(leader cluster)的host和port如下:

     CCR-cluster-leader-lab-vm-01:9300,
     CCR-cluster-leader-lab-vm-02:9300,
     CCR-cluster-leader-lab-vm-03:9300,
     CCR-cluster-leader-lab-vm-04:9300,
     CCR-cluster-leader-lab-vm-05:9300,
     CCR-cluster-leader-lab-vm-06:9300

    从集群(follower cluster)的host和port如下：

      CCR-cluster-follower-lab-vm-01:9300,
      CCR-cluster-follower-lab-vm-02:9300,
      CCR-cluster-follower-lab-vm-03:9300,
      CCR-cluster-follower-lab-vm-04:9300,
      CCR-cluster-follower-lab-vm-05:9300,
      CCR-cluster-follower-lab-vm-06:9300

1. 创建远程集群(leader cluster)

在从集群(follower cluster)上执行命令，以kibana为例

样板如下:

<leader_name>: 设置主集群的名字

<leader_cluster>： 主集群的地址(host+port)

 
PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "<leader_name>": {
          "seeds": [
            <leader_cluster>
          ]
        }
      }
    }
  }
}

示例：

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "leadertest": {
          "seeds": [
            "CCR-cluster-leader-lab-vm-01:9300",
            "CCR-cluster-leader-lab-vm-02:9300",
            "CCR-cluster-leader-lab-vm-03:9300",
            "CCR-cluster-leader-lab-vm-04:9300",
            "CCR-cluster-leader-lab-vm-05:9300",
            "CCR-cluster-leader-lab-vm-06:9300"
          ]
        }
      }
    }
  }
}

   2.检测两个集群是否连通

在从集群执行

本文例子:

GET /_remote/info
 
Expected output:
{
  "leadertest" : {
    "seeds" : [
      "CCR-cluster-leader-lab-vm-01:9300",
      "CCR-cluster-leader-lab-vm-02:9300",
      "CCR-cluster-leader-lab-vm-03:9300",
      "CCR-cluster-leader-lab-vm-04:9300",
      "CCR-cluster-leader-lab-vm-05:9300",
      "CCR-cluster-leader-lab-vm-06:9300"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

   3.创建从集群索引 同步主集群(包括手动常见和自动创建)

     3.1-手动创建从index同步主集群索引的数据

     在从集群执行命令 (参数解释详见https://blog.csdn.net/zhang5324496/article/details/107162699)

PUT /<follower_index>/_ccr/follow
{
  "remote_cluster" : "<leader_cluster>",
  "leader_index" : "<leader_index>",
  "max_read_request_operation_count" : 5120,
  "max_read_request_size" : "32mb",
  "max_outstanding_read_requests" : 12,
  "max_write_request_operation_count" : 5120,
  "max_write_request_size" : "9223372036854775807b",
  "max_outstanding_write_requests" : 9,
  "max_write_buffer_count" : 2147483647,
  "max_write_buffer_size" : "512mb",
  "max_retry_delay" : "500ms",
  "read_poll_timeout" : "1m"
}

本文实例:

remote_cluster: 步骤1 取的主集群别称(名字) ， 本例名字为 leadertest

leader_inde： 需要同步的目标索引(在主集群中)， 本例索引名为 cross-test

PUT /cross-test/_ccr/follow
{
  "remote_cluster": "leadertest",
  "leader_index": "cross-test",
  "max_read_request_operation_count" : 5120,
  "max_read_request_size" : "32mb",
  "max_outstanding_read_requests" : 12,
  "max_write_request_operation_count" : 5120,
  "max_write_request_size" : "9223372036854775807b",
  "max_outstanding_write_requests" : 9,
  "max_write_buffer_count" : 2147483647,
  "max_write_buffer_size" : "512mb",
  "max_retry_delay" : "500ms",
  "read_poll_timeout" : "1m"
}
 
Expected output:
{
  "follow_index_created" : true,
  "follow_index_shards_acked" : false,
  "index_following_started" : false
}

手动创建从集群的索引去同步主集群索引数据完毕

   3.2- 自动创建从集群索引去同步主集群索引

PUT /_ccr/auto_follow/<auto_follow_pattern_name>
{
  "remote_cluster" : "<remote_cluster>",
  "leader_index_patterns" :
  [
    "<leader_index_pattern>"
  ],
  "follow_index_pattern" : "<follow_index_pattern>"，
  "settings": {
    "index.number_of_replicas": 0
  },
  "max_read_request_operation_count" : 1024,
  "max_outstanding_read_requests" : 16,
  "max_read_request_size" : "1024k",
  "max_write_request_operation_count" : 32768,
  "max_write_request_size" : "16k",
  "max_outstanding_write_requests" : 8,
  "max_write_buffer_count" : 512,
  "max_write_buffer_size" : "512k",
  "max_retry_delay" : "10s",
  "read_poll_timeout" : "30s"
}

<auto_follow_pattern_name>: 必填项，自动创建从集群索引集合的名称，本例值为 cross_auto_test

<remote_cluster>: 必填项，步骤1定义的主集群名称， 本例为 leadertest

<leader_index_patterns>: 可选项，同步目标的索引模型， 本例为 cross*

<follow_index_pattern>: 可选项，同步生成的从索引名称模型，如果值是{{leader_index}}，则从索引名称和主索引名称一致，本例为{{ leader_index }}

PUT /_ccr/auto_follow/cross_auto_test
{
  "remote_cluster": "leadertest",
  "leader_index_patterns": [
    "cross*"
  ],
  "follow_index_pattern": "{{leader_index}}"
}

获取信息，验证是否创建成功

GET /_ccr/auto_follow/
 
Expected output:
{
  "patterns" : [
    {
      "name" : "cross_auto_test",
      "pattern" : {
        "remote_cluster" : "leadertest",
        "leader_index_patterns" : [
          "cross*"
        ],
        "follow_index_pattern" : "{{leader_index}}"
      }
    }
  ]
}

4. 将从集群的从索引恢复成正常的索引(不会去同步主索引数据)

    4.1- 恢复手动创建的从索引

POST /<follower_index>/_ccr/unfollow

    4.2- 恢复自动创建的从索引

DELETE /_ccr/auto_follow/<auto_follow_pattern_name>