Hbase表export和import

最新推荐文章于 2024-07-25 09:48:00 发布

crj2951

最新推荐文章于 2024-07-25 09:48:00 发布

阅读量471

点赞数

文章标签：大数据 java ui

1、查看要做测试的表test

hbase(main):002:0> scan 'test'
ROW COLUMN+CELL
row-01 column=cf1:id, timestamp=1442020353563, value=1
row-01 column=cf1:name, timestamp=1442020382276, value=aaa
row-02 column=cf1:id, timestamp=1442020360143, value=2
row-02 column=cf1:name, timestamp=1442020388494, value=bbb
row-03 column=cf1:id, timestamp=1442020364496, value=3
row-03 column=cf1:name, timestamp=1442020393616, value=ccc
row-04 column=cf1:id, timestamp=1442020369002, value=4
row-04 column=cf1:name, timestamp=1442020398557, value=ddd
row-05 column=cf1:id, timestamp=1442020373493, value=5
row-05 column=cf1:name, timestamp=1442020404131, value=eee
5 row(s) in 0.7520 seconds

2、使用Hadoop的export功能导出test表

首先查看JAR表的有哪些帮助选项

grid@master1:~$ hadoop jar /usr/local/hbase/hbase-0.94.27.jar -h
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/collect/Multimap
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Multimap
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 6 more

报错java.lang.NoClassDefFoundError: com/google/common/collect/Multimap，找不到Hbase类

grid@master1:~$ cp /usr/local/hbase/lib/guava-11.0.2.jar /usr/local/hadoop/lib

再次执行，还是报错

grid@master1:~$ hadoop jar /usr/local/hbase/hbase-0.94.27.jar -h
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
at java.lang.Class.getMethod0(Class.java:2856)
at java.lang.Class.getMethod(Class.java:1668)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.<init>(ProgramDriver.java:56)
at org.apache.hadoop.util.ProgramDriver.addClass(ProgramDriver.java:99)
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 12 more

还是有报错，但是报错的信息与之前不一样了，说明复制的JAR包是正确的，这次是说找不到Zookeeper

grid@master1:~$ cp /usr/local/zookeeper/zookeeper-3.4.5.jar /usr/local/hadoop/lib

复制Zookeeper的JAR包到Hadoop的lib目录中，再次执行

grid@master1:~$ hadoop jar /usr/local/hbase/hbase-0.94.27.jar -h
Unknown program '-h' chosen.
Valid program names are:
CellCounter: Count cells in HBase table
completebulkload: Complete a bulk data load.
copytable: Export a table from local cluster to peer cluster
export: Write table data to HDFS.
import: Import data written by Export.
importtsv: Import data in TSV format.
rowcounter: Count rows in HBase table
verifyrep: Compare the data from tables in two different clusters. WARNING: It doesn't work for incrementColumnValues'd cells since the timestamp is changed after being appended to the log.

3、正式执行Hbase export表的操作

grid@master1:~$ hadoop jar /usr/local/hbase/hbase-0.94.27.jar export -D mapred.output.compress=true -D mapred.output.compression.condec=org.apache.hadoop.io.compress.BZip2Codec -D mapred.output.compression.type=BLOCK test /backup/test 2147483647
15/09/13 01:35:49 INFO mapreduce.Export: versions=2147483647, starttime=0, endtime=9223372036854775807, keepDeletedCells=false
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.NoClassDefFoundError: com/google/protobuf/Message
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addHBaseDependencyJars(TableMapReduceUtil.java:657)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(TableMapReduceUtil.java:694)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:151)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:228)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:94)
at org.apache.hadoop.hbase.mapreduce.Export.createSubmittableJob(Export.java:95)
at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
... 10 more
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.Message
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 23 more

还是报错，真是麻烦啊，这次是protobuf包

grid@master1:~$ cp /usr/local/hbase/lib/protobuf-java-2.4.0a.jar /usr/local/hadoop/lib

复制缺少JAR包protobuf，再次执行export

grid@master1:~$ hadoop jar /usr/local/hbase/hbase-0.94.27.jar export -D mapred.output.compress=true -D mapred.output.compression.condec=org.apache.hadoop.io.compress.BZip2Codec -D mapred.output.compression.type=BLOCK test /backup/test 2147483647
.....................................................
15/09/13 01:48:57 INFO mapred.JobClient: Running job: job_201509122338_0002
15/09/13 01:48:58 INFO mapred.JobClient: map 0% reduce 0%
15/09/13 01:49:18 INFO mapred.JobClient: map 100% reduce 0%
15/09/13 01:49:23 INFO mapred.JobClient: Job complete: job_201509122338_0002
15/09/13 01:49:23 INFO mapred.JobClient: Counters: 29
.....................................................

查看MR的UI，可以看到有一个job，是执行了export_test操作

4、为import操作做准备，truncate表test，此操作，是先disable表，在drop表，最后create表

hbase(main):004:0> truncate 'test'
Truncating 'test' table (it may take a while):
- Disabling table...
- Dropping table...
- Creating table...
0 row(s) in 3.5160 seconds
hbase(main):005:0> scan 'test'
ROW COLUMN+CELL
0 row(s) in 0.0270 seconds

5、执行import操作，导入test表

grid@master1:~/sh$ hadoop jar /usr/local/hbase/hbase-0.94.27.jar import test /backup/test
.....................................................................
15/09/13 02:01:14 INFO mapreduce.TableOutputFormat: Created table instance for test
15/09/13 02:01:14 INFO input.FileInputFormat: Total input paths to process : 1
15/09/13 02:01:15 INFO mapred.JobClient: Running job: job_201509122338_0003
15/09/13 02:01:16 INFO mapred.JobClient: map 0% reduce 0%
15/09/13 02:01:36 INFO mapred.JobClient: map 100% reduce 0%
15/09/13 02:01:41 INFO mapred.JobClient: Job complete: job_201509122338_0003
..................................................................