vmbackup和vmrestore是何方神圣？

最新推荐文章于 2024-07-07 02:04:31 发布

运维开发故事

最新推荐文章于 2024-07-07 02:04:31 发布

阅读量1.3k

点赞数

分类专栏： kubernetes 文章标签：监控程序

本文链接：https://blog.csdn.net/wanger5354/article/details/122710413

版权

kubernetes 专栏收录该内容

122 篇文章

订阅专栏

微信公众号：运维开发故事，作者：华仔

背景

在之前的文章中，给大家介绍了vmstorage、vminsert、vmselect、vmui、vmagent，分别代表了数据的存储、添加、查询、可视化以及数据的采集。想想是不是还差点什么？没错！还缺少数据的备份和还原。那么今天，它们来了！接下来，请让我为大家一一介绍。

vmbackup

vmbackup是一个数据备份的工具，可以从vmstorage即时的快照中去备份vm数据，同时支持将备份数据存储到存储系统，如：

GCS. Example: gs:///<path/to/backup>
S3. Example: s3:///<path/to/backup>
Any S3-compatible storage such as MinIO, Ceph or Swift. See these docs for details
Local filesystem. Example: fs://</absolute/path/to/backup>

vmbackup支持增量备份和全备。当备份数据存储路径已经有了之前的备份数据，增量备份将自动进行；而全备则可以通过-origin指定相同的远端存储去加速备份。因为在这种情况下，可以在远端存储进行新备份与已经存在的备份之间进行共有数据的复制，从而节省了时间和其他花销。备份进程可以在任何时间被打断，但当你使用vmbackup使用同样的参数进行再次备份时，备份将从打断处自动恢复。同时，通过vmbackup备份的数据能够被vmrestore用来恢复数据。

vmrestore

vmrestore被用于从通过vmbackup备份的数据中进行恢复数据，跟vmbackup备份进程一样，恢复数据进程也可以在任何时候被打断；当再次使用相同的命令和参数去恢复数据时，将从断点处自动恢复。原vmstorage数据存储的的数据将被恢复的数据所替换。

实操

数据备份

1、创建快照

上面提到过，vmbackup需要从即时的快照去备份数据，因此想要备份，我们需要先创建快照。vmstorage给我们提供了API去创建快照。使用以下的方式：

# request
curl http://<victoriametrics-addr>:8428/snapshot/create # 需要根据实际部署修改地址和端口

# response
{"status":"ok","snapshot":"<snapshot-name>"}

下面来实操一下：

[root@test ~]# curl http://172.16.47.122:8482/snapshot/create
{"status":"ok","snapshot":"20220119051931-16CA0D126A870F8B"}

我们再看看vmstorage的日志：

2022-01-19T05:19:31.767Z info VictoriaMetrics/lib/storage/storage.go:304 creating Storage snapshot for "/vm-data"...
2022-01-19T05:19:31.771Z info VictoriaMetrics/lib/storage/table.go:143 creating table snapshot of "/vm-data/data"...
2022-01-19T05:19:31.773Z info VictoriaMetrics/lib/storage/partition.go:1549 creating partition snapshot of "/vm-data/data/small/2022_01" and "/vm-data/data/big/2022_01"...
2022-01-19T05:19:31.920Z info VictoriaMetrics/lib/storage/partition.go:1570 created partition snapshot of "/vm-data/data/small/2022_01" and "/vm-data/data/big/2022_01" at "/vm-data/data/small/snapshots/20220119051931-16CA0D126A870F8B/2022_01" and "/vm-data/data/big/snapshots/20220119051931-16CA0D126A870F8B/2022_01" in 0.146 seconds
2022-01-19T05:19:31.920Z info VictoriaMetrics/lib/storage/table.go:171 created table snapshot for "/vm-data/data" at ("/vm-data/data/small/snapshots/20220119051931-16CA0D126A870F8B", "/vm-data/data/big/snapshots/20220119051931-16CA0D126A870F8B") in 0.149 seconds
2022-01-19T05:19:31.921Z info VictoriaMetrics/lib/mergeset/table.go:1161 creating Table snapshot of "/vm-data/indexdb/16CA0D126A874608"...
2022-01-19T05:19:31.963Z info VictoriaMetrics/lib/mergeset/table.go:1230 created Table snapshot of "/vm-data/indexdb/16CA0D126A874608" at "/vm-data/indexdb/snapshots/20220119051931-16CA0D126A870F8B/16CA0D126A874608" in 0.041 seconds
2022-01-19T05:19:31.963Z info VictoriaMetrics/lib/mergeset/table.go:1161 creating Table snapshot of "/vm-data/indexdb/16CA0D126A874607"...
2022-01-19T05:19:31.966Z info VictoriaMetrics/lib/mergeset/table.go:1230 created Table snapshot of "/vm-data/indexdb/16CA0D126A874607" at "/vm-data/indexdb/snapshots/20220119051931-16CA0D126A870F8B/16CA0D126A874607" in 0.003 seconds
2022-01-19T05:19:31.971Z info VictoriaMetrics/lib/storage/storage.go:361 created Storage snapshot for "/vm-data" at "/vm-data/snapshots/20220119051931-16CA0D126A870F8B" in 0.204 seconds

最后去vmstorage的存储路径下看看文件：

# 我这里是直接去我的nfs下vmstorage对应的存储path下查看的。
# 可以很清楚看到我们之前的快照
[root@test ~]# cd /nfs_data/pvc-b06968ca-f31e-4d93-b16d-cf66a10c64d9/
[root@test pvc-b06968ca-f31e-4d93-b16d-cf66a10c64d9]# ls -al
total 24
drwxr-xr-x 6 root root 4096 Jan 14 13:49 .
drwxr-xrwx 6 root root 4096 Jan 14 13:49 ..
drwxr-xr-x 4 root root 4096 Jan 14 13:49 data
-rw-r--r-- 1 root root    0 Jan 14 13:49 flock.lock
drwxr-xr-x 5 root root 4096 Jan 14 13:49 indexdb
drwxr-xr-x 2 root root 4096 Jan 14 13:49 metadata
drwxr-xr-x 3 root root 4096 Jan 19 13:19 snapshots
[root@test pvc-b06968ca-f31e-4d93-b16d-cf66a10c64d9]# cd snapshots/
[root@test snapshots]# ls -al
total 12
drwxr-xr-x 3 root root 4096 Jan 19 13:19 .
drwxr-xr-x 6 root root 4096 Jan 14 13:49 ..
drwxr-xr-x 4 root root 4096 Jan 19 13:19 20220119051931-16CA0D126A870F8B
[root@test snapshots]# cd 20220119051931-16CA0D126A870F8B/
[root@test 20220119051931-16CA0D126A870F8B]# ls -al
total 16
drwxr-xr-x 4 root root 4096 Jan 19 13:19 .
drwxr-xr-x 3 root root 4096 Jan 19 13:19 ..
drwxr-xr-x 2 root root 4096 Jan 19 13:19 data
lrwxrwxrwx 1 root root   55 Jan 19 13:19 indexdb -> ../../indexdb/snapshots/20220119051931-16CA0D126A870F8B
drwxr-xr-x 2 root root 4096 Jan 19 13:19 metadata

其实也可以通过vmstorage提供的API去查看我们刚刚创建的快照

# 到这里我们的快照已经准备就绪了
[root@test ~]# curl http://172.16.47.122:8482/snapshot/list
{"status":"ok","snapshots":[
"20220119051931-16CA0D126A870F8B"

2、数据备份

有了上面的快照，下面我们来进行备份,我这边没有s3、gcs，直接用local filesystem，将其备份到本地/opt目录下

# 通过刚刚创建的快照进行备份
./vmbackup-prod -storageDataPath=/vm-data -snapshotName=20220119051931-16CA0D126A870F8B -dst=fs:///opt

# 输出如下
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:12 build version: vmbackup-20210715-111047-tags-v1.63.0-0-g61cc13c16
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:13 command line flags
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:28 flag "concurrency"="10" (is_set=false)
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:28 flag "configFilePath"="" (is_set=false)
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:28 flag "configProfile"="" (is_set=false)
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:28 flag "credsFilePath"="" (is_set=false)
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:28 flag "customS3Endpoint"="" (is_set=false)
2022-01-19T05:45:29.709Z info VictoriaMetrics/lib/logger/flag.go:28 flag "dst"="fs:///opt" (is_set=true)
...省略...
00_22197_20220119051721.456_20220119051724.766_16CA152B5133B4AD/timestamps.bin", file_size: 0, offset: 0, size: 0} from src fslocal "/vm-data/snapshots/20220119051931-16CA0D126A870F8B" to dst fsremote "/opt"
2022-01-19T05:45:30.374Z info VictoriaMetrics/lib/backup/actions/backup.go:164 uploaded 125373885 out of 125373885 bytes from src fslocal "/vm-data/snapshots/20220119051931-16CA0D126A870F8B" to dst fsremote "/opt" in 627.488079ms
2022-01-19T05:45:30.374Z info VictoriaMetrics/lib/backup/actions/backup.go:171 backup from src fslocal "/vm-data/snapshots/20220119051931-16CA0D126A870F8B" to dst fsremote "/opt" with origin fsnil is complete; backed up 125373885 bytes in 0.664 seconds; deleted 0 bytes; server-side copied 0 bytes; uploaded 125373885 bytes

# 再去/opt目录下确保数据备份正确
/vm-data # cd /opt/
/opt # ls -al
total 24
drwxr-xr-x    1 root     root          4096 Jan 19 05:45 .
drwxr-xr-x    1 root     root          4096 Jan 14 05:49 ..
-rw-------    1 root     root             2 Jan 19 05:45 backup_complete.ignore
drwx------    4 root     root          4096 Jan 19 05:45 data
drwx------    4 root     root          4096 Jan 19 05:45 indexdb
drwx------    3 root     root          4096 Jan 19 05:45 metadata

3、创建第二个快照

[root@test /]# curl http://172.16.47.122:8482/snapshot/create
{"status":"ok","snapshot":"20220119055254-16CA0D126A870F8C"}

4、增量备份

# 备份
./vmbackup-prod -storageDataPath=/vm-data -snapshotName=20220119055254-16CA0D126A870F8C  -dst=fs:///opt
# 日志输出
...省略...
2022-01-19T05:55:18.548Z info VictoriaMetrics/lib/backup/actions/backup.go:171 backup from src fslocal "/vm-data/snapshots/20220119055254-16CA0D126A870F8C" to dst fsremote "/opt" with origin fsnil is complete; backed up 127597222 bytes in 0.214 seconds; deleted 2714902 bytes; server-side copied 0 bytes; uploaded 4938239 bytes

从日志上不难看出，增量备份时，delete file 的值不为0，这里删除的是第一次快照中存在，但在第二次快照中不会存在的数据；另外
花费的时间也明显小于咱们第一次全备。

5、再做一个快照，测试server-side copy

# 创建快照
[root@test ~]# curl http://172.16.47.122:8482/snapshot/create
{"status":"ok","snapshot":"20220119073821-16CA0D126A870F8D"}

6、全备，备份时添加-origin参数

# 备份，添加-origin指定之前备份数据所在目录
./vmbackup-prod -storageDataPath=/vm-data -snapshotName=20220119073821-16CA0D126A870F8D  -dst=fs:///ss -origin=fs:///opt

# 从命令输出来看，确定有进行server-side copy
...省略...
2022-01-19T07:44:56.657Z info VictoriaMetrics/lib/backup/actions/backup.go:171 backup from src fslocal "/vm-data/snapshots/20220119073821-16CA0D126A870F8D" to dst fsremote "/ss" with origin fsremote "/opt" is complete; backed up 131747249 bytes in 0.190 seconds; deleted 0 bytes; server-side copied 121316947 bytes; uploaded 10430302 bytes

数据恢复

上面我们创建了多个快照，也进行了多次备份。这里我选择从fs:///ss目录下的备份进行恢复到/tt目录下

# 进行恢复
./vmrestore-prod  -src=fs:///ss -storageDataPath=/tt
# vm停止，再进行数据恢复；恢复后，vm再次重启时-storageDataPath参数需要指定为/tt目录，即恢复的数据所在的目录
# 控制台输出
...省略...
2022-01-19T07:58:32.117Z info VictoriaMetrics/lib/logger/flag.go:28 flag "src"="fs:///ss" (is_set=true)
2022-01-19T07:58:32.117Z info VictoriaMetrics/lib/logger/flag.go:28 flag "storageDataPath"="/tt" (is_set=true)
...省略...
2022-01-19T07:58:32.751Z info VictoriaMetrics/lib/backup/actions/restore.go:185 downloaded 131747249 out of 131747249 bytes from fsremote "/ss" to fslocal "/tt" in 624.892637ms
2022-01-19T07:58:32.751Z info VictoriaMetrics/lib/backup/actions/restore.go:192 restored 131747249 bytes from backup in 0.635 seconds; deleted 0 bytes; downloaded 131747249 bytes

# 查看/tt目录下的恢复的数据
/vm-data # ls -al /tt/
total 20
drwxr-xr-x    5 root     root          4096 Jan 19 07:58 .
drwxr-xr-x    1 root     root          4096 Jan 19 07:44 ..
drwx------    4 root     root          4096 Jan 19 07:58 data
-rw-r--r--    1 root     root             0 Jan 19 07:58 flock.lock
drwx------    4 root     root          4096 Jan 19 07:58 indexdb
drwx------    2 root     root          4096 Jan 19 07:58 metadata

注：在进行恢复数据期间，vm应该停止。