Elasticsearch使用快照进行跨集群索引迁移

最新推荐文章于 2024-10-09 18:27:18 发布

云平_Stephen

最新推荐文章于 2024-10-09 18:27:18 发布

阅读量459

点赞数 4

文章标签： elasticsearch jenkins 大数据

本文链接：https://blog.csdn.net/qq_43303221/article/details/141964030

版权

1、创建NFS共享存储目录服务端

安装 NFS服务
yum install -y nfs-utils
创建目录并授权

mkdir -p /data/es-backup
chmod 666 /data/es-backup
chown nobody /data/es-backup

导出文件系统

vim /etc/exports
/data08/esnfs_data *(rw,sync,insecure,no_subtree_check,no_root_squash)

/data/nfs (rw,sync,insecure,no_subtree_check,no_root_squash)说明：
1、/data/nfs： nfs server 目录
2、：表示所有的服务器都可以挂载该nfs，当然也可以指定ip地址/ip地址段
3、rw：读写权限
4、sync：同步写入,即数据写入服务器后再返回响应，保证数据的可靠性和一致性
5、insecure：表示不进行端口校验，允许客户端使用非保留端口进行连接
6、no_subtree_check: 禁止子树检查
7、no_root_squash: 表示禁用 root 用户映射机制，即来自客户端的 root 用户被映射为匿名用户而没有特权

启动服务

systemctl start nfs-server
systemctl enable nfs-server

查看验证

#exportfs -rv
exporting *:/data/es-backup

nfs服务端配置

yum install -y nfs-utils

mkdir /data08/esnfs_data
chmod 666  /data08/esnfs_data
chown nobody  /data08/esnfs_data

2、ES节点挂载共享目录并修改参数重启

安装软件包
yum install -y nfs-utils
测试挂载

mkdir /esdata_backup

[root@secpaas108 ~]# showmount -e 10.45.151.130
Export list for 10.45.151.130:
/data08/esnfs_data *

mount

mount -t nfs 10.45.151.130:/data08/esnfs_data /esdata_backup
chown elastic:elastic /esdata_backup

修改集群的配置，每个节点都需要修改：

vim ./elasticsearch/elasticsearch.yml
path.repo: ["/esdata_backup"]

设置开机自动挂载

vim /etc/fstab
10.45.151.130:/data/es-backup /esdata_backup nfs defaults 0 0

修改所属组
chown elasticsearch:elasticsearch /mnt/es-backup

3、老集群创建快照

1. 创建一个快照库

curl -v -XPUT 'http://10.45.151.108:9200/_snapshot/test-repo' -H "Content-Type: application/json" -d '
> {
>     "type": "fs",
>     "settings": {
>         "location": "/esdata_backup"
>     }
> }'

其中test-repo 是快照库的名字，/esdata_backup是我们上面挂载的目录

结果输出为:

* About to connect() to 10.45.151.108 port 9200 (#0)
*   Trying 10.45.151.108...
* Connected to 10.45.151.108 (10.45.151.108) port 9200 (#0)
> PUT /_snapshot/test-repo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.45.151.108:9200
> Accept: */*
> Content-Type: application/json
> Content-Length: 83
>
* upload completely sent off: 83 out of 83 bytes
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 21
<
* Connection #0 to host 10.45.151.108 left intact
{"acknowledged":true}

1. 查看快照库
  查看刚刚创建的快照库

curl -X GET "http://10.45.151.108:9200/_snapshot?pretty"
{
  "test-repo" : {
    "type" : "fs",
    "settings" : {
      "location" : "/esdata_backup"
    }
  }
}

1. 在快照库中创建一个索引的快照

计划将此索引迁移：

[elastic@secpaas108 ~]$  curl "10.45.151.108:9200/_cat/indices?v"|grep aman_gwtest0815
1green  open   gwtest0815                    UMdP7fHARlSoXvaujTKgKg  48   1   91560000            0     26.6gb         13.3gb

创建快照命令：

curl -v -XPUT 'http://10.45.151.108:9200/_snapshot/test-repo/snapshot-gwtest0815?wait_for_completion=true' -H "Content-Type: application/json" -d '
{
"indices": "gwtest0815"
}'

其中：
test-repo: 上面创建的快照库名字
snapshot-gwtest0815：索引快照的名字
wait_for_completion=true：是否等待命令成功再返回
indices：索引名称

注意：index的状态必须是非red

输出如下：

* About to connect() to 10.45.151.108 port 9200 (#0)
*   Trying 10.45.151.108...
* Connected to 10.45.151.108 (10.45.151.108) port 9200 (#0)
> PUT /_snapshot/test-repo/snapshot-gwtest0815?wait_for_completion=true HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.45.151.108:9200
> Accept: */*
> Content-Type: application/json
> Content-Length: 33
>
* upload completely sent off: 33 out of 33 bytes
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 440
<
* Connection #0 to host 10.45.151.108 left intact
{"snapshot":{"snapshot":"snapshot-gwtest0815","uuid":"Q8PKCQeMQlqloy9dcj_ccA","version_id":6082399,"version":"6.8.23","indices":["gwtest0815"],"include_global_state":true,"stat      e":"SUCCESS","start_time":"2024-08-15T08:27:25.842Z","start_time_in_millis":1723710445842,"end_time":"2024-08-15T08:28:51.155Z","end_time_in_millis":1723710531155,"duration_in_millis":85

3、在新集群恢复快照

1）创建一个与老集群相同名字的快照库：

curl -v -XPUT 'http://10.45.185.80:9200/_snapshot/test-repo' -H "Content-Type: application/json" -d '
{
    "type": "fs",
    "settings": {
        "location": "/esdata_backup"
    }
}'

注意：每次老集群创建快照以后，如果冷备集群已经有对应的快照库，就无法识别热备新创建的快照，需要先把快照库删除，重新创建一下，才能识别
附删除快照库命令：删除一个快照库：curl -XDELETE 'http://10.45.151.108:9200/_snapshot/test-repo'

命令输出如下

* About to connect() to 10.45.185.80 port 9200 (#0)
*   Trying 10.45.185.80...
* Connected to 10.45.185.80 (10.45.185.80) port 9200 (#0)
> PUT /_snapshot/test-repo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.45.185.80:9200
> Accept: */*
> Content-Type: application/json
> Content-Length: 83
>
* upload completely sent off: 83 out of 83 bytes
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< Warning: 299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."
< content-type: application/json; charset=UTF-8
< content-length: 21
<
* Connection #0 to host 10.45.185.80 left intact
{"acknowledged":true}

1. 查看已创建的快照

curl -X GET "http://10.45.185.80:9200/_snapshot/test-repo/snapshot-gwtest0815?pretty"

可以查看到老集群创建的快照的基本信息

{
  "snapshots" : [
    {
      "snapshot" : "snapshot-gwtest0815",
      "uuid" : "Q8PKCQeMQlqloy9dcj_ccA",
      "repository" : "test-repo",
      "version_id" : 6082399,
      "version" : "6.8.23",
      "indices" : [
        "aman_gwtest0815"
      ],
      "data_streams" : [ ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2024-08-15T08:27:25.842Z",
      "start_time_in_millis" : 1723710445842,
      "end_time" : "2024-08-15T08:28:51.155Z",
      "end_time_in_millis" : 1723710531155,
      "duration_in_millis" : 85313,
      "failures" : [ ],
      "shards" : {
        "total" : 48,
        "failed" : 0,
        "successful" : 48
      },
      "feature_states" : [ ]
    }
  ],
  "total" : 1,
  "remaining" : 0
}

1. 利用快照恢复索引数据至老集群

curl -XPOST "http://10.45.185.80:9200/_snapshot/test-repo/snapshot-gwtest0815/_restore"
{"accepted":true}

1. 等待数据恢复完成

查看ES 索引状态：

curl -X GET 'http://10.45.185.80:9200/_cat/indices?v=true&pretty'
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   gwtest0815   gkXrNy08QPijMvGwRAeZTQ  48   1

此时索引已经创建，但是shard尚未恢复完成，可以用health命令查看进度

curl -X GET 'http://10.45.185.80:9200/_cat/health?v=true&pretty'
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1723712508 09:01:48  elasticsearch yellow         12         9    198 196    0   19       43             0                  -                 76.2%

curl -X GET 'http://10.45.185.80:9200/_cat/health?v=true&pretty'
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1723712656 09:04:16  elasticsearch green          12         9    260 210    0    0        0             0                  -                100.0%

恢复完成以后，ES 索引状态正常

[elastic@gwtest80 ~]$  curl -X GET 'http://10.45.185.80:9200/_cat/indices?v=true&pretty'
green  open   gwtest0815   gkXrNy08QPijMvGwRAeZTQ  48   1   91560000            0     26.6gb         13.3gb

至此我们就完成了ES索引跨集群迁移