Hive 数据入库到HBase

最新推荐文章于 2024-01-04 19:03:27 发布

q79969786

最新推荐文章于 2024-01-04 19:03:27 发布

阅读量3.7k

点赞数

分类专栏： hadoop相关文章标签： hfile Hive Hadoop

本文链接：https://blog.csdn.net/q79969786/article/details/39736649

版权

本文详细介绍了如何将Hive中的数据批量导入到HBase中，包括创建HQL脚本、建立HBase表、处理数据导入过程中可能遇到的问题，如分区文件错误、RowKey设置不当、文件未排序等，并提供了相关的解决方案和参考资料。

摘要由CSDN通过智能技术生成

测试数据：http://www.nber.org/patents/apat63_99.zip

测试环境：hadoop-2.3 + hive-0.13.1 + hbase-0.98.4

测试效率：6列6亿的Hive表数据半小时

创建hfile.hql

drop table hbase_splits;
CREATE EXTERNAL TABLE IF NOT EXISTS hbase_splits(partition STRING)
ROW FORMAT
  SERDE 'org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe'
STORED AS
  INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveNullValueSequenceFileOutputFormat'
LOCATION '/tmp/hbase_splits_out';

-- create a location to store the resulting HFiles
drop table hbase_hfiles;
CREATE TABLE if not exists hbase_hfiles(rowkey STRING, pageviews STRING, bytes STRING)
STORED AS
  INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.hbase.HiveHFileOutputFormat'
TBLPROPERTIES('hfile.family.path' = '/tmp/hbase_hfiles/w');

ADD JAR /root/hive-0.13.1/lib/hive-contrib-0.13.1.jar; 
SET mapred.reduce.tasks=1;

CREATE TEMPORARY FUNCTION row_seq AS 'org.apache.hadoop.hive.contrib.udf.UDFRowSequence';

-- input file contains ~4mm records. Sample it so as to produce 5 input splits.
INSERT OVERWRITE TABLE hbase_splits
SELECT PATENT FROM
  (SELECT PATENT, row_seq() AS seq FROM apat tablesample(bucket 1 out of 1000 on PATENT) s
order by PATENT
limit 10000000) x
WHERE (seq % 300) = 0
ORDER BY PATENT
LIMIT 4;

-- after this is finished, combined the splits file:
dfs -rmr  /tmp/hbase_splits;
dfs -cp /tmp/hbase_splits_out/* /tmp/hbase_splits;

ADD JA