HBase备份之ExportSnapshot或CopyTable

原创 2014年08月13日 16:41:52

文章《HBase备份之导入导出》介绍了使用HBase的自带工具Export和Import来实现在主集群和从集群之间拷贝表的目的。本篇介绍一种相比导入导出而言,更快速的一种备份办法。即ExportSnapshot。

1、ExportSnapshot

和Export类似,ExportSnapshot也是使用MapReduce方式来进行表的拷贝。不过和Export不同,ExportSnapshot导出的是表的快照。我们可以使用ExportSnapshot将表的快照数据先导出到从集群,然后再从集群中使用restore_snapshot命令恢复快照,即可实现表在主从集群之间的复制工作。具体的操作步骤如下:

1)在主集群中为表建立快照

$ cd $HBASE_HOME/  
$ bin/hbase shell  
2014-08-13 15:59:12,495 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.98.2-hadoop2, r1591526, Wed Apr 30 20:17:33 PDT 2014

hbase(main):001:0> snapshot 'test_table', 'test_table_snapshot'
0 row(s) in 0.3370 seconds

2)使用ExportSnapshot命令导出快照数据

$ cd $HBASE_HOME/  
$ bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot test_table_snapshot -copy-to hdfs://follow_cluster_namenode:8082/hbase
其中,test_table_snapshot为刚建的快照名,hdfs://follow_cluster_namenode:8082/hbase为从集群的hbase的hdfs根目录的全路径。

ExportSnapshot命令也可以限定mapper个数,如下:

$ bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot test_table_snapshot -copy-to hdfs://follow_cluster_namenode:8082/hbase -mapers n
还可以限定拷贝的流量,如下:

$ bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot test_table_snapshot -copy-to hdfs://follow_cluster_namenode:8082/hbase -mapers n -bandwidth 200
上面的例子将拷贝的流量限定为200M。

执行ExportSnapshot命令之后的输出很长,部分如下:

2014-08-13 16:08:26,318 INFO  [main] mapreduce.Job: Running job: job_1407910396081_0027
2014-08-13 16:08:33,494 INFO  [main] mapreduce.Job: Job job_1407910396081_0027 running in uber mode : false
2014-08-13 16:08:33,495 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2014-08-13 16:08:41,567 INFO  [main] mapreduce.Job:  map 100% reduce 0%
2014-08-13 16:08:42,581 INFO  [main] mapreduce.Job: Job job_1407910396081_0027 completed successfully
2014-08-13 16:08:42,677 INFO  [main] mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=116030
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=1386
		HDFS: Number of bytes written=988
		HDFS: Number of read operations=7
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=3
	Job Counters 
		Launched map tasks=1
		Rack-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=13518
		Total time spent by all reduces in occupied slots (ms)=0
	Map-Reduce Framework
		Map input records=1
		Map output records=0
		Input split bytes=174
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=23
		CPU time spent (ms)=1860
		Physical memory (bytes) snapshot=323575808
		Virtual memory (bytes) snapshot=1867042816
		Total committed heap usage (bytes)=1029177344
	org.apache.hadoop.hbase.snapshot.ExportSnapshot$Counter
		BYTES_COPIED=988
		BYTES_EXPECTED=988
		FILES_COPIED=1
	File Input Format Counters 
		Bytes Read=224
	File Output Format Counters 
		Bytes Written=0
2014-08-13 16:08:42,685 INFO  [main] snapshot.ExportSnapshot: Finalize the Snapshot Export
2014-08-13 16:08:42,697 INFO  [main] snapshot.ExportSnapshot: Verify snapshot validity
2014-08-13 16:08:42,698 INFO  [main] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 16:08:42,713 INFO  [main] snapshot.ExportSnapshot: Export Completed: test_table_snapshot
3)到从集群中恢复快照
$ cd $HBASE_HOME/  
$ bin/hbase shell  
2014-08-13 16:16:13,817 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.98.2-hadoop2, r1591526, Wed Apr 30 20:17:33 PDT 2014

hbase(main):001:0> restore_snapshot 'test_table_snapshot'
0 row(s) in 16.4940 seconds
4)查看表是否恢复成功

hbase(main):002:0> list
TABLE                                                                                                                               test_table                                                                                                                                                      
1 row(s) in 1.0460 seconds

=> ["test_table"]
另外,还可以通过scan或count命令进行检验。

快照恢复操作一般会很快,相比较Export和Import需要导出和导入两次MapReduce任务才能完成表的复制来讲,使用ExportSnapshot会快很多。

2、CopyTable

首先,看一下CopyTable命令的使用方法

$ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable
Usage: CopyTable [general options] [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] <tablename>

Options:
 rs.class     hbase.regionserver.class of the peer cluster
              specify if different from current cluster
 rs.impl      hbase.regionserver.impl of the peer cluster
 startrow     the start row
 stoprow      the stop row
 starttime    beginning of the time range (unixtime in millis)
              without endtime means from starttime to forever
 endtime      end of the time range.  Ignored if no starttime specified.
 versions     number of cell versions to copy
 new.name     new table's name
 peer.adr     Address of the peer cluster given in the format
              hbase.zookeeer.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
 families     comma-separated list of families to copy
              To copy from cf1 to cf2, give sourceCfName:destCfName. 
              To keep the same name, just give "cfName"
 all.cells    also copy delete markers and deleted cells

Args:
 tablename    Name of the table to copy

Examples:
 To copy 'TestTable' to a cluster that uses replication for a 1 hour window:
 $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=1265875194289 --endtime=1265878794289 --peer.adr=server1,server2,server3:2181:/hbase --families=myOldCf:myNewCf,cf2,cf3 TestTable 
For performance consider the following general options:
-Dhbase.client.scanner.caching=100
-Dmapred.map.tasks.speculative.execution=false
可以看到,它支持设定需要复制的表的时间范围,cell的版本,也可以指定列簇,设定从集群的地址等。

对于上面的test_table表,我们可以使用如下命令进行拷贝:

$ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=slave1,slave2,slave3:2181:/hbase  test_table

注意:在使用上述语句之前,需要在从集群建立一个模式和主集群表test_table相同的表。

使用上述语句的部分执行结果如下:

2014-08-13 16:18:21,812 INFO  [main] mapreduce.Job: Running job: job_1407910396081_0062
2014-08-13 16:18:29,955 INFO  [main] mapreduce.Job: Job job_1407910396081_0062 running in uber mode : false
2014-08-13 16:18:29,957 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2014-08-13 16:18:36,005 INFO  [main] mapreduce.Job:  map 100% reduce 0%
2014-08-13 16:18:37,029 INFO  [main] mapreduce.Job: Job job_1407910396081_0062 completed successfully
2014-08-13 16:18:37,137 INFO  [main] mapreduce.Job: Counters: 37
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=117527
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=88
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=1
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=0
	Job Counters 
		Launched map tasks=1
		Rack-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=9740
		Total time spent by all reduces in occupied slots (ms)=0
	Map-Reduce Framework
		Map input records=1
		Map output records=1
		Input split bytes=88
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=254
		CPU time spent (ms)=1810
		Physical memory (bytes) snapshot=345137152
		Virtual memory (bytes) snapshot=1841782784
		Total committed heap usage (bytes)=1029177344
	HBase Counters
		BYTES_IN_REMOTE_RESULTS=34
		BYTES_IN_RESULTS=34
		MILLIS_BETWEEN_NEXTS=254
		NOT_SERVING_REGION_EXCEPTION=0
		NUM_SCANNER_RESTARTS=0
		REGIONS_SCANNED=1
		REMOTE_RPC_CALLS=3
		REMOTE_RPC_RETRIES=0
		RPC_CALLS=3
		RPC_RETRIES=0
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=0
然后,就可以对比主集群中的表和从集群中对应的表数据是否一致。

转载请注明出处:http://blog.csdn.net/iAm333


版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/iAm333/article/details/38538073

HBase Snapshot 相关操作原理

1. 前言 HBase 从0.95开始引入了 Snapshot,Snapshot 相关操作主要是对 table 做备份与还原等,这在容灾方面是必不可少的,而由于在实际工作中遇到了部分问题需要阅读...
  • t894690230
  • t894690230
  • 2016-08-04 18:20:46
  • 1759

HBase快照(Snapshot)技术

什么是快照快照就是一份元信息的合集,允许管理员恢复到表的先前状态。快照不是表的复制而是一个文件名称列表,因而不会复制数据。 完全快照恢复是指恢复到之前的“表结构”以及当时的数据,快照之后发生的数据不...
  • scgaliguodong123_
  • scgaliguodong123_
  • 2015-07-09 17:56:27
  • 4512

hbase 快照和集群备份的NB方法

HBase Snapshots允许你对一个表进行快照(即可用副本),它不会对Region Servers产生很大的影响,它进行复制和 恢复操作的时候不包括数据拷贝。导出快照到另外的集群也不会对Regi...
  • cuilanbo
  • cuilanbo
  • 2015-01-29 17:05:17
  • 3839

HBase Snapshot解析

Snapshot即是快照,代表某个时刻数据库中表的状态,就像是照片一样,记录了该时刻所有的数据。 HBase可以对某个时刻的表建立snapshot,过后可以恢复到该snapshot的状态,也可以用s...
  • huang_quanlong
  • huang_quanlong
  • 2016-03-13 15:05:25
  • 1388

hbase两种CopyTable 和export备份恢复用例

创建表 hbase(main):001:0> create 't1',{NAME => 'f1', VERSIONS => 2},{NAME => 'f2', VERSIONS => 2} h...
  • haoxiaoyan
  • haoxiaoyan
  • 2016-11-11 18:04:07
  • 1468

大数据技术-HBase:使用CopyTable在线备份HBase表数据

CopyTable是hbase提供的一个很有用的备份工具。主要可以用于集群内部表备份,远程集群备份,表数据增量备份,部分结构数据部分等。其依赖于hadoop mapreduce,使用标准的hbase ...
  • huanggang028
  • huanggang028
  • 2015-05-22 15:18:32
  • 4217

【HBase运维系列】集群间的数据拷贝-copytable

场景:数据从老环境拷贝到新环境,两个集群建是可以通讯的 集群间数据备份包括两种,一种是全停止拷贝,一种是动态备份。 动态备份还包括replication、copytable以及export imp...
  • WeiJonathan
  • WeiJonathan
  • 2013-12-24 16:33:18
  • 6932

HBase中的拷贝表功能

在使用HBase时,建立表的记录数目很多,想对表进行split操作,但又想以一个大的region来进行比较。所以想到拷贝表。找找命令,HBase有拷贝表的命令:CopyTable ...
  • anhuidelinger
  • anhuidelinger
  • 2013-12-15 20:06:47
  • 7632

Hbase集群间数据迁移方法总结

呵呵,今天花了一天的时间查资料做测试,略微的总结了一下hbase数据迁移的方法。 一、需要在hbase集群停掉的情况下迁移 步骤:(1)执行hadoop distcp -f file...
  • jingling_zy
  • jingling_zy
  • 2012-05-10 17:59:35
  • 19289

Hbase表备份——Snapshot

-- 内存数据刷盘 flush'paycore.pay_test' --禁用表 disable'paycore.pay_test' -- 备份快照 snapshot'paycor...
  • yingcuicy
  • yingcuicy
  • 2017-12-12 10:26:06
  • 98
收藏助手
不良信息举报
您举报文章:HBase备份之ExportSnapshot或CopyTable
举报原因:
原因补充:

(最多只允许输入30个字)