前提:已经搭建好hadoop+hive,hadoop+hbase+zookeeper环境,现在来集成hive+hbase,相关集成介绍请查看前几篇文章介绍。
将对hbase-0.94.3和hive-0.9.0进行集成,使用的hadoop-1.0.3
1.首先将hbase-0.94.3.jar和zookeeper-3.4.3.jar复制到hive/lib目录下
注意:如果hive/lib下已经存在这两个文件的其他版本(例如zookeeper-3.3.3.jar),建议删除后使用hbase下的相关版本
命令:
cd /home/yf/hbase-0.94.3
cp hbase-0.94.3.jar /home/yf/hive-0.9.0/lib/
cd /home/yf/zookeeper-3.4.3
cp zookeeper-3.4.3.jar /home/yf/hive-0.9.0/lib/
2.在hive/conf下hive-site.xml文件中添加如下的内容:
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/yf/hive-0.9.0/lib/hive-hbase-handler-0.9.0.jar,file:///home/yf/hive-0.9.0/lib/hbase-0.94.3.jar,file:///home/yf/hive-0.9.0/lib/zookeeper-3.4.3.jar</value>
</property>
这个需要根据自己的具体文件路径指定
3.拷贝hbase-0.94.3.jar到所有hadoop节点(包括master)的hadoop/lib下
cd /home/yf/hbase-0.94.3
cp hbase-0.94.3.jar /home/yf/hadoop-1.0.3/lib
4.拷贝hbase/conf下的hbase-site.xml文件到所有hadoop节点(包括master)的hadoop/conf下
cp /home/yf/hbase-0.94.3/conf/hbase-site.xml /home/yf/hadoop-1.0.3/conf
5.单结点启动hive,伪分布式属于单结点:
bin/hive-hiveconf hbase.master=localhost:60000
集群启动:
bin/hive-hiveconf hbase.zookeeper.quorum=slave
6.创建hbase识别的数据,这个操作在hive下进行:
create table hive_hbase(key string,age int,sex int) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with SERDEPROPERTIES("hbase.columns.mapping" =":key,cf1:age,cf1:sex")tblproperties("hbase.table.name"= "hbase_hive");
7.在hbase下面查看数据:
list
8.创建临时表person:
createtable person(name string, age int, sex int) row format delimitedfields terminated by '\t' stored as textfile;
9.往临时表中添加数据:
loaddata local inpath '/home/hadoop/Desktop/data.txt' overwrite intotable person;
10.往hive_hbase表中添加数据:
insertoverwrite table hive_hbase select * from person;
11.查看表中数据:
select* from person limit 5;
12.hbase中查看数据:
scan'hbase_hive'
hive> CREATE TABLE hbase_table_1(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "xyz");
FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:122)
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:73)
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:147)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:398)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:538)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3305)