一、hive sink概述
hive sink与hdfs sink 想对比,hive sink可以近实时的把数据采集到hive表中,hdfs sink要构建hive外部表去关联hdfs路径,并且实时性没辣么高。
二、注意事项
1、Hive表必须设置bucket并且 stored as orc
2、flume配置的hive列名必须都是小写,即fieldnames的配置都必须是小写
3、要手动构建分区,即autoCreatePartitions = false
三、Configure hive sink
```
a1.sinks.k2.type = hive
a1.sinks.k2.channel = c2
#hive元存储的url
a1.sinks.k2.hive.metastore = thrift://192.168.3.150:9083
#hive表库名
a1.sinks.k2.hive.database = test
#hive表表名
a1.sinks.k2.hive.table = ods_table
#hive表分区,逗号分隔,%Y代表2018,&y代表18
a1.sinks.k2.hive.partition = %Y-%m-%d
#此处自动创建分区必须关闭,否则会报错。使用手动构建分区
a1.sinks.k2.autoCreatePartitions = false
#使用本地时间(而不是事件头的时间戳)
a1.sinks.k2.useLocalTimeStamp = false
#a1.sinks.k2.round = true
#a1.sinks.k2.roundValue = 1
#a1.sinks.k2.roundUnit = minute
a1.sinks.k2.serializer = DELIMITED
#切记切记,一定要记得转义
a1.sinks.k2.serializer.delimiter = "\\001"
#a1.sinks.k2.serializer.serdeSeparator = "\\001"
#在Flume配置的Hive 列名必须都为小写字母。Hive表必须设置bucket并且 stored as orc。
a1.sinks.k2.serializer.fieldnames = dstype,id,type,lastuploadtime
```
四、hive
create table test.ods_table(
dsType string ,
id string ,
type string ,
lastUploadTime string
)
partitioned by (dt string)
clustered by (id) into 2 buckets
stored as orc
TBLPROPERTIES ('transactional'='true');
alter table test.ods_table add if not exists partition ( dt='2018-05-18');