Elasticsearch Snapshot操作

qq_21569821

已于 2024-08-13 11:01:42 修改

阅读量176

点赞数 3

分类专栏： Elasticsearch 文章标签： elasticsearch jenkins 大数据

于 2024-08-06 10:32:29 首次发布

本文链接：https://blog.csdn.net/qq_21569821/article/details/140946496

版权

Elasticsearch 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

ES Snapshot

Snapshot是一个正在运行的ES cluster的一个备份。可以对cluster中的单个index分别创建快照，也可以对整个集群中的所有index创建快照，然后存储在一个共享的的文件系统(shared filesystem)的存储库(repository)中，ES提供了很多的plugins支持远程存储库(remote repositories),比如S3,Azure, Google Cloud Storage以及其他更多。

Snapshot是增量创建的(Snapshots are taken incrementally)，因此可以频繁切高效的对cluster创建快照，因为在创建快照时，它能够避免拷贝任何已经存在repository的数据(先前创建snapshot时相同的数据)。

Snapshot可以通过restore API恢复到一个运行的ES cluster中。在restore时，可以修改restored index name，也可以修改index的settings，ES在snapshot创建与恢复(restore)方面提供了很大的灵活性(flexibility)。

不能通过直接拷贝集群(cluster)中所有节点的数据目录来进行备份，通过该方式进行数据备份在恢复时，可能提示数据毁坏或者文件丢失(reporting corruption and/or missing files)，或者提示成功，但是会丢失数据(silently lost)。备份集群的唯一可靠方式是使用snapshot与恢复功能(using the snapshot and restore funcitonality)。

1.Snapshot查看

1.1 list plugins(can find S3 plugin)

可以看到安装了repository-s3, snapshot创建时需要repository，这个插件可以将snapshot存储到S3上。

//GET _cat/plugins?format=json
GET _cat/plugins

master-new-a-10-246-24-90   analysis-nori 6.8.2
master-new-a-10-246-24-90   discovery-ec2 6.8.2
master-new-a-10-246-24-90   repository-s3 6.8.2
master-new-c-10-210-146-153 analysis-nori 6.8.2
master-new-c-10-210-146-153 discovery-ec2 6.8.2
master-new-c-10-210-146-153 repository-s3 6.8.2
client-new-b-10-246-45-203  analysis-nori 6.8.2
client-new-b-10-246-45-203  discovery-ec2 6.8.2
client-new-b-10-246-45-203  repository-s3 6.8.2
client-new-a-10-246-29-150  analysis-nori 6.8.2
client-new-a-10-246-29-150  discovery-ec2 6.8.2
client-new-a-10-246-29-150  repository-s3 6.8.2
data-a-10-246-29-182        analysis-nori 6.8.2
data-a-10-246-29-182        discovery-ec2 6.8.2
data-a-10-246-29-182        repository-s3 6.8.2
master-new-b-10-246-62-124  analysis-nori 6.8.2
master-new-b-10-246-62-124  discovery-ec2 6.8.2
master-new-b-10-246-62-124  repository-s3 6.8.2
client-new-c-10-210-130-216 analysis-nori 6.8.2
client-new-c-10-210-130-216 discovery-ec2 6.8.2
client-new-c-10-210-130-216 repository-s3 6.8.2
data-c-10-210-141-58        analysis-nori 6.8.2
data-c-10-210-141-58        discovery-ec2 6.8.2
data-c-10-210-141-58        repository-s3 6.8.2
data-b-10-246-34-214        analysis-nori 6.8.2
data-b-10-246-34-214        discovery-ec2 6.8.2
data-b-10-246-34-214        repository-s3 6.8.2

1.2 list repositories

GET _cat/repositories?format=json

[
  {
    "id" : "test_elasticsearch_snapshot",
    "type" : "s3"
  }
]

1.3 Get Repository Detail

GET _snapshot/test_elasticsearch_snapshot

{
  "maui_elasticsearch_snapshot" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "corp-es",
      "base_path" : "elasticsearch_backup/test_elasticsearch/test_elasticsearch_snapshot"
    }
  }
}

也可以查看多个repository，通过逗号(comma)分隔，也支持*通配符

GET /_snapshot/repo*,*backup*

1.4 List snapshots for a repository

GET _cat/snapshots/test_elasticsearch_snapshot?format=json

[
  {
    "id" : "test_elasticsearch-20240804-1911",
    "status" : "SUCCESS",
    "start_epoch" : "1722798693",
    "start_time" : "19:11:33",
    "end_epoch" : "1722798707",
    "end_time" : "19:11:47",
    "duration" : "14.7s",
    "indices" : "5",
    "successful_shards" : "30",
    "failed_shards" : "0",
    "total_shards" : "30"
  },
  {
    "id" : "test_elasticsearch-20240805-1911",
    "status" : "SUCCESS",
    "start_epoch" : "1722885098",
    "start_time" : "19:11:38",
    "end_epoch" : "1722885113",
    "end_time" : "19:11:53",
    "duration" : "14.5s",
    "indices" : "5",
    "successful_shards" : "30",
    "failed_shards" : "0",
    "total_shards" : "30"
  }
]

另一种方式

GET /_snapshot/test_elasticsearch_snapshot/_all

[
  {
    "id" : "test_elasticsearch-20240804-1911",
    "status" : "SUCCESS",
    "start_epoch" : "1722798693",
    "start_time" : "19:11:33",
    "end_epoch" : "1722798707",
    "end_time" : "19:11:47",
    "duration" : "14.7s",
    "indices" : "5",
    "successful_shards" : "30",
    "failed_shards" : "0",
    "total_shards" : "30"
  },
  {
    "id" : "test_elasticsearch-20240805-1911",
    "status" : "SUCCESS",
    "start_epoch" : "1722885098",
    "start_time" : "19:11:38",
    "end_epoch" : "1722885113",
    "end_time" : "19:11:53",
    "duration" : "14.5s",
    "indices" : "5",
    "successful_shards" : "30",
    "failed_shards" : "0",
    "total_shards" : "30"
  }
]

又一种方式：

GET /_cat/snapshots/maui_elasticsearch_snapshot?v&s=id

id                                status start_epoch start_time end_epoch  end_time duration indices successful_shards failed_shards total_shards
test_elasticsearch-20240711-1911 SUCCESS 1720725096  19:11:36   1720725105 19:11:45     9.2s       4                25             0           25
test_elasticsearch-20240712-1911 SUCCESS 1720811496  19:11:36   1720811512 19:11:52    16.2s       4                25             0           25
test_elasticsearch-20240713-1911 SUCCESS 1720897894  19:11:34   1720897910 19:11:50    16.4s       4                25             0           25

1.5 Get Snapshot Detail

// _snapshot/{repository}/{snapshotName}
GET _snapshot/test_elasticsearch_snapshot/test_elasticsearch-20240711-1911

{
  "snapshots" : [
    {
      "snapshot" : "test_elasticsearch-20240711-1911",
      "uuid" : "67P8sP1cSW2SMwZm_ZCTdQ",
      "version_id" : 6080299,
      "version" : "6.8.2",
      "indices" : [
        "test-management",
        "ticket-master-task-2021-10-29"
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2024-07-11T19:11:36.091Z",
      "start_time_in_millis" : 1720725096091,
      "end_time" : "2024-07-11T19:11:45.384Z",
      "end_time_in_millis" : 1720725105384,
      "duration_in_millis" : 9293,
      "failures" : [ ],
      "shards" : {
        "total" : 25,
        "failed" : 0,
        "successful" : 25
      }
    }
  ]
}

2. Repository

在创建snapshot或者使用snapshot restore API恢复时，必须先注册snapshot repository.
推荐的做法是为每一个大版本(major version)创建一个新的snapshot repository.
如果为多个ES cluster注册相同的snapshot repository，那么只能设置一个cluster有write权限，所有其他的cluster应该设置为readonly mode.
Snapshot的格式(format)在major version之间可能发生变更，如果多个集群安装了不同版本的ES，使用相同的snapshot repository时，某个版本的集群写入的snapshot对其他版本的集群可能不可见(not visible)，那么这个snapshot repository中的数据可能会被破坏(corrupted)。

2.1 注册(Register)

Shared file system
type为"fs"时，location指定的路径或者它路径中的任意一个父路径必须在所有的master 与data nodes中settings通过path.repo注册。

PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "/mount/backups/my_fs_backup_location",
        "compress": true
    }
}
// 也可以用相对路径，相对settings文件中的path.repo配置的路径
PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "my_fs_backup_location",
        "compress": true
    }
}

支持的参数

Param	Desc
location	Location of the snapshots. Mandatory.
compress	Turns on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. Defaults to true.
chunk_size	Big files can be broken down into chunks during snapshotting if needed. The chunk size can be specified in bytes or by using size value notation, i.e. 1g, 10m, 5k. Defaults to null (unlimited chunk size).
max_restore_bytes_per_sec	Throttles per node restore rate. Defaults to 40mb per second.
max_snapshot_bytes_per_sec	Throttles per node snapshot rate. Defaults to 40mb per second.
readonly	Makes repository read-only. Defaults to false