[hbase案例]从冷备份的hdfs数据中恢复到原来的hbase表

最新推荐文章于 2024-07-01 07:23:54 发布

宝罗Paul

最新推荐文章于 2024-07-01 07:23:54 发布

阅读量1.5k

点赞数 1

分类专栏：大数据文章标签： hbase 大数据运维

本文链接：https://blog.csdn.net/qq_31598113/article/details/79565957

版权

大数据专栏收录该内容

55 篇文章 6 订阅

订阅专栏

【hadoop版本：2.7.1 ；hbase版本：1.1.2】

之前我已经在关闭整个hbase集群时，通过hadoop distcp对本集群的hbase指定表数据做了个全量冷备份，那么这次要从已有的全量备份文件中恢复hbase数据，需要三步：

Step 1. 使用distcp命令将所备份的文件拷贝回原来所在HBase目录就可以

Step 2. 确保权限正确，例如有关的HBase目录和文件的所有者是hbase用户

Step 3. 开启hbase集群后，修复元数据、分配表数据到有关的region server

# 之前冷备份的数据存放在本地hdfs集群的/backup/bk_20180122目录下

[hbase@goodMaster-1 ~]$ hdfs dfs -ls /backup/bk_20180122

Found 1 items

drwxr-xr-x - hdfs hdfs 0 2018-01-22 21:14 /backup/bk_20180122/data

[hbase@goodMaster-1 ~]$ hdfs dfs -ls /backup/bk_20180122/data/hbase_data/all_info

Found 11 items

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/chenshan1_test

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/export_import_table_test

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/urls_30

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/urls_7

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/test_1

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/test_2

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/monthly_statistics

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/high_class_users

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/week_jobs_data

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/ios_users

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk_20180122/data/hbase_data/all_info/users_info

[hbase@goodMaster-1 ~]$ hdfs dfs -ls /backup/bk_20180122/data/hbase_data/all_info/users_info | wc -l

[hbase@goodMaster-1 ~]$ hdfs dfs -du -h /backup/bk_20180122/data/hbase_data/all_info/ | grep --color users_info

9.8 G /backup/bk_20180122/data/hbase_data/all_info/users_info

Step 1.

[hbase@goodMaster-1 ~]$ hadoop distcp hdfs://hdfs-ha/backup/bk_20180122/data/hbase_data/all_info/users_info hdfs://hdfs-ha/apps/hbase/data/hbase_data/all_info > ~/distcprecover.log 2>&1 &

[1] 6928

Step 2.

# 检查权限，必须保证该表对应hdfs目录下的文件owner都是hbase用户。若不是则hadoop fs -chown -R hbase:hdfs /apps/hbase

[hbase@goodMaster-1 ~]$ hdfs dfs -ls /apps/hbase | awk '{print $3,$4}' | grep hbase | head -n 2

hbase hdfs

[hbase@goodMaster-1 ~]$ hdfs dfs -ls /apps/hbase | awk '{print $3}' | grep hdfs # 说明owner都是hbase用户而不是hdfs用户

[hbase@goodMaster-1 ~]$ less ~/fix_log

Step 3.

# 确认权限没问题就在Active Hmaster上修复元数据，还要把表数据重新分配到相关区域(会自动按照hbase的相关机制进行分配，具体如何分配的请上网搜)

[hbase@goodMaster-1 ~]$ hbase hbck -fixMeta -fixAssignments "all_info:users_info" > ~/fix_log 2>&1 &

[1] 11747

[hbase@kgoodMaster-1 ~]$ jobs

[1]+ Running hbase hbck -fixMeta -fixAssignments "all_info:users_info" > ~/fix_log 2>&1 &

[hbase@goodMaster-1 ~]$ jobs

[1]+ Done hbase hbck -fixMeta -fixAssignments "all_info:users_info" > ~/fix_log 2>&1

[hbase@goodMaster-1 ~]$ less ~/fix_log # 当出现Deployed on [your_region_server] 相关字样时，说明该表已成功部署到region server

# 去hbase shell验证一下

hbase(main):003:0> scan "all_info:users_info" , LIMIT => 3

ROW COLUMN+CELL

0 column=cf:confidence, timestamp=1502878819524, value=1.0

0 column=cf:label, timestamp=1502878819524, value=\xE4\xB8\xAD\xE5\x9B\xBD,\xE6\xB1\x9F\xE8\x8B\x8F,\xE6\xB7\xAE\xE5\xAE\x89,,\xE7\x94\xB5\xE4\xBF\xA1

00000000000000000000000000000000 column=cf:confidence, timestamp=10000, value=0.5

00000000000000000000000000000000 column=cf:label, timestamp=10000, value=\xE4\xB8\xAD\xE5\x9B\xBD,\xE5\xB9\xBF\xE4\xB8\x9C,\xE4\xBD\x9B\xE5\xB1\xB1,,\xE7\x94\xB5\xE4\xBF\xA1

00000051992B3C6B48DF65CFCD00F570 column=cf:confidence, timestamp=1502895476079, value=1.0

00000051992B3C6B48DF65CFCD00F570 column=cf:label, timestamp=1502895476079, value=\xE4\xB8\xAD\xE5\x9B\xBD,\xE5\xB9\xBF\xE4\xB8\x9C,\xE7\x8F\xA0\xE6\xB5\xB7,,\xE7\x94\xB5\xE4\xBF\xA1

3 row(s) in 0.1810 seconds

hbase(main):009:0> count "all_info:users_info", INTERVAL => 5000000

Current count: 5000000, row: 12CB6A9E9CCD42674CAEBF68CB59B644

Current count: 10000000, row: 2597BB29A9B94ED48D86287575917903

Current count: 15000000, row: 38615F1F568A52017E65EDA1639CFF55

Current count: 20000000, row: 4B2C059D1DA69C5BA780E431F7A88F22

Current count: 25000000, row: 5DF4FAD9697292B6DED164C14D6D19A3

Current count: 30000000, row: 70BBDC41E4C5757FB8D26618F22698F2

Current count: 35000000, row: 8385DC4053D07A1910B815B2D445F6E1

Current count: 40000000, row: 964C61B9B42B41FEDCE8F5073638B483

Current count: 45000000, row: A9126076D2B29BDF5BE335A4B884FA69

Current count: 50000000, row: BBDBF49A21E278D46DB774490C3EABCE