prometheus 异常退出 报错：opening storage failed

prometheus 异常退出日志如下：

 解决方法：

1. 进入到prometheus数据目录内，移走报错文件夹，再次启动后日志如下：

ts=2022-06-30T03:52:56.187Z caller=manager.go:956 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-06-30T03:52:56.187Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2022-06-30T03:52:56.187Z caller=main.go:1068 level=info msg="Notifier manager stopped"
ts=2022-06-30T03:52:56.187Z caller=main.go:1077 level=error err="opening storage failed: reloadBlocks: 15 errors: corrupted block 01G2XWKD8X0VPKWNJGN5YE64E7: read symbols: invalid checksum; corrupted block 01G3E6EMA9NMC5P3GEASJD7A4H: invalid magic number 0; corrupted block 01G1TFMH0WD2YXH0GJ9D86QKFY: invalid magic number 0; corrupted block 01G22VDVS27547SZ1BRMG8R4JY: invalid magic number 0; corrupted block 01G3VG5NRWAVJ8WGN0PZ2T5TAH: read symbols: invalid checksum; corrupted block 01G548SQ8V8YFVCR5XP3XS6GGW: read symbols: invalid checksum; corrupted block 01G313KA0WPYMK53APAPGGQ3RD: read symbols: invalid checksum; corrupted block 01G361H0RV5HA7ZMJA37GJJ605: invalid magic number 0; corrupted block 01G36P4714CSMRBZJMV5XV2W63: read symbols: invalid checksum; corrupted block 01G3X07RGVREACT0YTJP3HMVH2: read symbols: invalid checksum; corrupted block 01G578XWRW3E0VSY0X35FMMDPD: read symbols: invalid checksum; corrupted block 01G32CSNGWD0SAFSDF2HSXT6J5: read symbols: invalid checksum; corrupted block 01G37AQC8WVV2AWQNZWNF05WYA: read symbols: invalid checksum; corrupted block 01G386698WS46Q9566K0X9VX25: invalid magic number 0; corrupted block 01G4A9YT0WJDKKWEM33B017D0T: read symbols: invalid checksum"

 2. 将日志中的所有错误块移走

$ mv 01G313KA0WPYMK53APAPGGQ3RD 01G361H0RV5HA7ZMJA37GJJ605 01G36P4714CSMRBZJMV5XV2W63 01G3X07RGVREACT0YTJP3HMVH2 01G578XWRW3E0VSY0X35FMMDPD 01G32CSNGWD0SAFSDF2HSXT6J5 01G37AQC8WVV2AWQNZWNF05WYA 01G386698WS46Q9566K0X9VX25 01G4A9YT0WJDKKWEM33B017D0T bak/

3. 再次启动成功。

ts=2022-06-30T03:56:12.892Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G313KA0WPYMK53APAPGGQ3RD err="open /data/01G313KA0WPYMK53APAPGGQ3RD/meta.json: no such file or directory"
ts=2022-06-30T03:56:13.355Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G32CSNGWD0SAFSDF2HSXT6J5 err="open /data/01G32CSNGWD0SAFSDF2HSXT6J5/meta.json: no such file or directory"
ts=2022-06-30T03:56:15.150Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G361H0RV5HA7ZMJA37GJJ605 err="open /data/01G361H0RV5HA7ZMJA37GJJ605/meta.json: no such file or directory"
ts=2022-06-30T03:56:15.381Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G36P4714CSMRBZJMV5XV2W63 err="open /data/01G36P4714CSMRBZJMV5XV2W63/meta.json: no such file or directory"
ts=2022-06-30T03:56:15.641Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G37AQC8WVV2AWQNZWNF05WYA err="open /data/01G37AQC8WVV2AWQNZWNF05WYA/meta.json: no such file or directory"
ts=2022-06-30T03:56:16.006Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G386698WS46Q9566K0X9VX25 err="open /data/01G386698WS46Q9566K0X9VX25/meta.json: no such file or directory"
ts=2022-06-30T03:56:26.474Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G3X07RGVREACT0YTJP3HMVH2 err="open /data/01G3X07RGVREACT0YTJP3HMVH2/meta.json: no such file or directory"
ts=2022-06-30T03:56:33.312Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G4A9YT0WJDKKWEM33B017D0T err="open /data/01G4A9YT0WJDKKWEM33B017D0T/meta.json: no such file or directory"
ts=2022-06-30T03:56:46.171Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G578XWRW3E0VSY0X35FMMDPD err="open /data/01G578XWRW3E0VSY0X35FMMDPD/meta.json: no such file or directory"
ts=2022-06-30T03:57:08.413Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-06-30T03:57:35.681Z caller=head.go:520 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: corruption in head chunk file /data/chunks_head/001701: checksum mismatch expected:00000000, actual:cfead048"
ts=2022-06-30T03:57:35.684Z caller=head.go:689 level=info component=tsdb msg="Deleting mmapped chunk files"
ts=2022-06-30T03:57:35.764Z caller=head.go:699 level=info component=tsdb msg="Deletion of mmap chunk files successful, reattempting m-mapping the on-disk chunks"
ts=2022-06-30T03:57:35.869Z caller=head.go:536 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=27.456523956s
ts=2022-06-30T03:57:35.869Z caller=head.go:542 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-06-30T03:58:34.193Z caller=head.go:578 level=info component=tsdb msg="WAL checkpoint loaded"
ts=2022-06-30T03:58:48.103Z caller=db.go:750 level=warn component=tsdb msg="Encountered WAL read error, attempting repair" err="read records: corruption in segment /data/wal/00003413 at 77221215: snappy: corrupt input"
ts=2022-06-30T03:58:48.103Z caller=wal.go:364 level=warn component=tsdb msg="Starting corruption repair" segment=3413 offset=77221215
ts=2022-06-30T03:58:48.455Z caller=wal.go:372 level=warn component=tsdb msg="Deleting all segments newer than corrupted segment" segment=3413
ts=2022-06-30T03:59:54.766Z caller=wal.go:394 level=warn component=tsdb msg="Rewrite corrupted segment" segment=3413
ts=2022-06-30T04:00:44.492Z caller=main.go:958 level=info fs_type=65735546
ts=2022-06-30T04:00:44.492Z caller=main.go:961 level=info msg="TSDB started"