Hive文件格式

最新推荐文章于 2024-07-01 21:31:18 发布

松门一枝花

最新推荐文章于 2024-07-01 21:31:18 发布

阅读量4.7k

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/zengmingen/article/details/52627086

版权

大数据专栏收录该内容

132 篇文章 3 订阅

订阅专栏

Hive有四种文件格式：TextFile，SequenceFile，RCFile，ORC

TextFile

默认的格式，文本格式。

SequenceFile

简介

见：http://blog.csdn.net/zengmingen/article/details/52242768

操作

 
 hive (zmgdb)>create table t2(str string) stored assequencefile; 

OK

  Time taken: 0.299 seconds 

 
 hive (zmgdb)> desc formatted t2; 

OK

 
 .............................. 
  

 
 # Storage Information 

 
 SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe 

 
 InputFormat: org.apache.hadoop.mapred.SequenceFileInputFormat 

 
 OutputFormat: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat 

 
 Compressed: No 

 
 sequenceFile的表导入数据不能用load， 

  [root@hello110 data]# vi test_data 

3

we

ew

e

re

  er51 

2

  hive (zmgdb)> 
  load data local inpath '/data/test_data' into table t1; 

  Loading data to table zmgdb.t1 

OK

  Time taken: 1.498 seconds 

  hive (zmgdb)> 
 load data local inpath '/data/test_data' into table t2; 

  FAILED: SemanticException Unable to load data to destination table. Error: The file that you are trying to load 
 does not match the file format of the destination table. 

 
 要用 INSERT OVERWRITE TABLE test2 SELECT * FROM test1;开启mapreduce保存 

  hive (zmgdb)> 
 insert overwrite table t2 select * from t1; 

  WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. 

  Query ID = hadoop_20160914215205_992081a3-1783-4052-8da8-53e6097a2775 

  Total jobs = 3 

  Launching Job 1 out of 3 

  Number of reduce tasks is set to 0 since there's no reduce operator 

  Starting Job = job_1473855624724_0001, Tracking URL = http://hello110:8088/proxy/application_1473855624724_0001/ 

  Kill Command = /home/hadoop/app/hadoop-2.7.2/bin/hadoop job -kill job_1473855624724_0001 

  Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0 

  2016-09-14 21:52:22,073 Stage-1 map = 0%, reduce = 0% 

  2016-09-14 21:52:43,733 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.9 sec 

  MapReduce Total cumulative CPU time: 2 seconds 900 msec 

  Ended Job = job_1473855624724_0001 

  Stage-4 is selected by condition resolver. 

  Stage-3 is filtered out by condition resolver. 

  Stage-5 is filtered out by condition resolver. 

  Moving data to directory hdfs://hello110:9000/user/hive/warehouse/zmgdb.db/t2/.hive-staging_hive_2016-09-14_21-52-05_274_2207100662758769951-1/-ext-10000 

  Loading data to table zmgdb.t2 

  MapReduce Jobs Launched: 

  Stage-Stage-1: Map: 1 Cumulative CPU: 2.9 sec HDFS Read: 3844 HDFS Write: 1534 SUCCESS 

  Total MapReduce CPU Time Spent: 2 seconds 900 msec 

OK

  t1.str 

  Time taken: 43.709 seconds 

  hive (zmgdb)> select * from t2; 

OK

  t2.str 

1

2

2

43

4

dds

ads

  fdsdsf 

fds

ad

 
 查看hdfs里sequencefile的原文件 

 
 sequencefile的底层保存的是二进制格式，0101010101的。 

RCFile

一种行列存储相结合的存储方式。首先，其将数据按行分块，保证同一个record在一个块上，避免读一个记录需要读取多个block。其次，块数据列式存储，有利于数据压缩和快速的列存取。

hive (zmgdb)> create table rc_t1(id string) stored as rcfile;
OK
Time taken: 0.334 seconds

hive (zmgdb)> desc formatted rc_t1;
OK
col_name data_type comment
# col_name data_type comment

id string

# Detailed Table Information
Database: zmgdb
Owner: hadoop
CreateTime: Fri Sep 23 19:21:15 CST 2016
LastAccessTime: UNKNOWN
Retention: 0
Location: hdfs://hello110:9000/user/hive/warehouse/zmgdb.db/rc_t1
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
numFiles 0
numRows 0
rawDataSize 0
totalSize 0
transient_lastDdlTime 1474629675

# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe
InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
Time taken: 0.135 seconds, Fetched: 30 row(s)
hive (zmgdb)> insert overwrite table rc_t1 select * from t2;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = hadoop_20160923192210_96320492-f8bf-483a-83c4-b9874fd05ef4
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1474629517907_0001, Tracking URL = http://hello110:8088/proxy/application_1474629517907_0001/
Kill Command = /home/hadoop/app/hadoop-2.7.2/bin/hadoop job -kill job_1474629517907_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2016-09-23 19:22:22,091 Stage-1 map = 0%, reduce = 0%
2016-09-23 19:22:28,446 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.83 sec
MapReduce Total cumulative CPU time: 1 seconds 830 msec
Ended Job = job_1474629517907_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hello110:9000/user/hive/warehouse/zmgdb.db/rc_t1/.hive-staging_hive_2016-09-23_19-22-10_649_8279187505632970863-1/-ext-10000
Loading data to table zmgdb.rc_t1
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 1.83 sec HDFS Read: 4755 HDFS Write: 876 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 830 msec
OK
t2.id
Time taken: 19.126 seconds

hive (zmgdb)> select * from rc_t1;
OK
rc_t1.id
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

ORC

是RCfile的优化。自带了压缩和索引

存储总结

textfile 存储空间消耗比较大，并且压缩的text 无法分割和合并查询的效率最低,可以直接存储，加载数据的速度最高

sequencefile 存储空间消耗大,压缩的文件可以分割和合并查询效率高，需要通过text文件转化来加载

rcfile 存储空间最小，查询的效率最高，需要通过text文件转化来加载，加载的速度最低