最近研究了一下hive与hbase的整合,使用的都是各自的最新release, hive0.13, hbase0.96.2,整合的过程其实挺简单的,大致需要注意的地方如下:
1. hive的配置文件hive-site.xml需要添加的内容:
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/grid/hive/lib/hive-hbase-handler-0.13.0.jar,file:///home/grid/hive/lib/hbase-client-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-common-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-common-0.96.2-hadoop2-tests.jar,file:///home/grid/hive/lib/hbase-protocol-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-server-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/htrace-core-2.04.jar,file:///home/grid/hive/lib/zookeeper-3.4.6.jar,file:///home/grid/hive/lib/protobuf-java-2.5.0.jar,file:///home/grid/hive/lib/guava-11.0.2.jar</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>server1,server2</value>
</property>
2.安装hbase的过程不再累述,可以参考http://blog.csdn.net/codestinity/article/details/6947464
最后在双方查询都能查到数据,但是在hive中向hbase插入数据的时候出现了问题,错误信息如下:
java.io.FileNotFoundException: File does not exist: hdfs://*.*.*.*:9000/home/grid/hbase/lib/hbase-hadoop-compat-0.96.2-hadoop2.jar
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:99)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:740)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://*.*.*.*:9000/home/grid/hbase/lib/hbase-hadoop-compat-0.96.2-hadoop2.jar)'
Execution failed with exit status: 1
Obtaining error information
Task failed!
Task ID:
Stage-0
Logs:
/tmp/root/hive.log
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
这是需要将使用的jar包上传到hdfs文件系统中,所以,需要哪个就上传哪个吧,用put命令。
最后成功实现hive与hbase的整合。