基于 Elasticsearch 的 snapshot 迁移ES集群数据

以下迁移⽅案基于 Elasticsearch 的 snapshot。

源 ES 集群拓扑	⽬标 ES 集群拓扑
192.168.244.10:9200	192.168.244.10:9500
192.168.244.20:9200	192.168.244.20:9500
192.168.244.30:9200	192.168.244.30:9500

1. 创建共享⽂件系统

以 OCI 上为例

2. 将共享磁盘挂载到各个 ES 节点上

sudo yum install nfs-utils
sudo mkdir -p /mnt/FileSystem-20221108-1352-23
sudo mount 192.168.244.85:/FileSystem-20221108-1352-23 /mnt/FileSystem-20221108-1352-23

以上是 OCI 上的操作步骤，AWS 上应该类似。

3. 创建 ES 的备份仓库

# cd /mnt/FileSystem-20221108-1352-23/
# mkdir -p es_backup_20221108
# chown -R es.es /mnt/FileSystem-20221108-1352-23/es_backup_20221108/

4. 修改源 ES 集群的配置

4.1 修改源 ES 集群的配置⽂件

# cd /usr/local/es/9200/es-test/elasticsearch-7.4.2/config/
# vim elasticsearch.yml
path.repo: ["/mnt/FileSystem-20221108-1352-23/es_backup_20221108"]

4.2 重启 ES 节点

# ps -ef | grep java
es 12128 1 1 09:51 ? 00:03:33 /usr/local/es/9200/estest/
elasticsearch-7.4.2/jdk/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -
XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -
Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -
XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -
Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -
Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -
Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -
Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-15652940960214243721 -
XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -
XX:ErrorFile=logs/hs_err_pid%p.log -
Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize
=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=pooled -
XX:MaxDirectMemorySize=2147483648 -Des.path.home=/usr/local/es/9200/estest/
elasticsearch-7.4.2 -Des.path.conf=/usr/local/es/9200/es-test/elasticsearch-
7.4.2/config -Des.distribution.flavor=default -Des.distribution.type=tar -
Des.bundled_jdk=true -cp /usr/local/es/9200/es-test/elasticsearch-7.4.2/lib/*
org.elasticsearch.bootstrap.Elasticsearch -d
# kill 12128
# su - es
# /usr/local/es/9200/es-test/elasticsearch-7.4.2/bin/elasticsearch -d

4.3 等待集群状态变为 green

# curl localhost:9200/_cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign
pending_tasks max_task_wait_time active_shards_percent
1667917616 14:26:56 es-test green 3 3 12 6 0 0 0
0 - 100.0%

4.4 修改其它 ES 节点的配置⽂件并滚动重启

5. 创建 repository

curl -X PUT "192.168.244.10:9200/_snapshot/my_repository?pretty" -H 'Content-Type:
application/json' -d'
{
"type": "fs",
"settings": {
"location": "/mnt/FileSystem-20221108-1352-23/es_backup_20221108"
}
}'

{
"acknowledged" : true
}

查看源集群的索引情况

# curl 192.168.244.10:9200/_cat/indices
green open test_data JIf6Y6p4TYaX_x-Vo1vtAg 1 1 200000 0 35.6mb 17.8mb
green open test_data1 J6J8nKJaSnWJj2lOPolVkQ 1 1 100000 0 17.9mb 8.9mb
green open test_data2 sa7cFLJiSAOzsrjuSPA58Q 1 1 276000 0 49.4mb 24.7mb
green open test_data_2 yvpkeVLYSweZQn6bxkapYw 1 1 2059000 0 355.7mb 177.8mb
green open test_data_3 sh3s93FMRxyuAjA5WfIRjw 1 1 111000 0 19.8mb 9.9mb
green open test_data_4 kfc3bRRDTCmu_o1pRs3KwQ 1 1 327000 0 58.3mb 29.1mb

6. 创建快照

curl -X PUT "192.168.244.10:9200/_snapshot/my_repository/my_snapshot?
wait_for_completion=true&pretty"
{
"snapshot" : {
"snapshot" : "my_snapshot",
state 显示 SUCCESS 代表快照成功。
查看 my_repository 下 snapshot 的状态
"uuid" : "hU5aYZwUR5uvYptrExNmHg",
"version_id" : 7040299,
"version" : "7.4.2",
"indices" : [
"test_data1",
"test_data_3",
"test_data_4",
"test_data",
"test_data2",
"test_data_2"
],
"include_global_state" : true,
"state" : "SUCCESS",
"start_time" : "2022-11-08T14:47:19.136Z",
"start_time_in_millis" : 1667918839136,
"end_time" : "2022-11-08T14:47:24.938Z",
"end_time_in_millis" : 1667918844938,
"duration_in_millis" : 5802,
"failures" : [ ],
"shards" : {
"total" : 6,
"failed" : 0,
"successful" : 6
}
}
}

state 显示 SUCCESS 代表快照成功。

查看 my_repository 下 snapshot 的状态

# curl -X GET "192.168.244.10:9200/_snapshot/my_repository/*?verbose=false&pretty"
{
"snapshots" : [
{
"snapshot" : "my_snapshot",
"uuid" : "hU5aYZwUR5uvYptrExNmHg",
"indices" : [
"test_data",
"test_data1",
"test_data2",
"test_data_2",
"test_data_3",
"test_data_4"
],
"state" : "SUCCESS"
}
]
}

7. 拷⻉数据

把源 ES 集群创建好的 snapshot 拷⻉⾄⽬标 ES 集群创建好的仓库中

# cp -r es_backup_20221108 es_backup_new_20221108/
# chown -R es.es es_backup_20221108 es_backup_new_20221108/

这⾥只是演示，实际需要通过 scp 之类的⼯具进⾏拷⻉

8. 调整新集群的配置

主要是添加 path.repo

# cd /usr/local/es/9500/es-test-new/elasticsearch-7.4.2/config/
# vim elasticsearch.yml
path.repo: ["/mnt/FileSystem-20221108-1352-23/es_backup_new_20221108"]
# kill 2510
# su - es
$ /usr/local/es/9500/es-test-new/elasticsearch-7.4.2/bin/elasticsearch -d
$ curl 192.168.244.10:9500/_cat/health
1667919766 15:02:46 es-test-new green 3 3 6 3 0 0 0 0 - 100.0%

滚动修改其它节点的配置，并重启。
调整⽅式同源集群⼀样。

9. 新集群创建 repository

curl -X PUT "192.168.244.10:9500/_snapshot/my_repository?pretty" -H 'Content-Type:
application/json' -d'
{
"type": "fs",
"settings": {
"location": "/mnt/FileSystem-20221108-1352-23/es_backup_new_20221108"
}
}'

查看快照情况

$ curl -X GET "192.168.244.10:9500/_snapshot/my_repository/*?verbose=false&pretty"
{
"snapshots" : [
{
"snapshot" : "my_snapshot",
"uuid" : "hU5aYZwUR5uvYptrExNmHg",
"indices" : [
"test_data",
"test_data1",
"test_data2",
"test_data_2",
"test_data_3",
"test_data_4"
],
"state" : "SUCCESS"
}
]
}

10. 执⾏恢复操作

$ curl -X POST "192.168.244.10:9500/_snapshot/my_repository/my_snapshot/_restore?
wait_for_completion=true&pretty"

{
"snapshot" : {
"snapshot" : "my_snapshot",
"indices" : [
"test_data1",
"test_data",
"test_data2",
"test_data_2",
"test_data_3",
"test_data_4"
],
"shards" : {
"total" : 6,
"failed" : 0,
"successful" : 6
}
}
}

11. 查看索引恢复情况

$ curl 192.168.244.10:9500/_cat/indices
green open test_data JeNZfkSrRwCFVUFQB2mctA 1 1 200000 0 35.6mb 17.8mb
green open test_data1 KE4zqB1kRm2Xu8KLG98ZnA 1 1 100000 0 17.9mb 8.9mb
green open test_data2 3RDshwA-S-S-3n4CYIQ-4g 1 1 276000 0 49.4mb 24.7mb
green open test_data_2 jOGUZLjtSnSi4Y7P9Rcubw 1 1 2059000 0 355.7mb 177.8mb
green open test_data_3 GyNVSUOJTPSZWWEXGJCuBQ 1 1 111000 0 19.8mb 9.9mb
green open test_data_4 8UzXBMiZQtmEeCWSOcvqEg 1 1 327000 0 58.3mb 29.1mb

注意事项

迁移完毕后，如果共享磁盘后续不会继续挂载，切记要删除备份仓库（my_repository）和配置⽂件中的
path.repo 配置。否则，实例起来的时候，如果发现 path.repo 对应的⽬录不存在，会直接报错，不会正常启动。

$ curl -X DELETE "192.168.244.10:9200/_snapshot/my_repository?pretty"
{
"acknowledged" : true
}