Flume使用Hive作为Sink总结

项目中打算使用Flume把数据直接传到Hive表而不是HDFS上。使用Hive作为Sink，Flume版本为1.7.0。

开始时候遇到Failed connecting to EndPoint错误，Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed

原来使用Hive做为Sink，Hive表必须cluster by bucket。

修改表之后重启Flume，又遇到

org.apache.flume.EventDeliveryException: java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.AcidOutputFormat 错误。

看了下源码实现AcidOutputFormat的类只有OrcOutputFormat，那么答案显而易见，Hive表需要stored as orc。

重启之后又遇到错误，

Failed connecting to EndPoint

Caused by: org.apache.hive.hcatalog.streaming.InvalidColumn: Column 'URL' not found in table for input field 81

原来在Flume配置的Hive 列名必须都为小写字母。

再重启之后成功运行。

总结如下，Hive表必须设置bucket并且 stored as orc。

Flume Hive Sink：

tier1.sinks.sink_sitevisit.channel = channel_sitevisit
tier1.sinks.sink_sitevisit.type = hive
tier1.sinks.sink_sitevisit.hive.metastore = thrift://localhost:9083
tier1.sinks.sink_sitevisit.hive.database = sitevisit
tier1.sinks.sink_sitevisit.hive.table = visitlog
tier1.sinks.sink_sitevisit.hive.partition = %Y-%m-%d
tier1.sinks.sink_sitevisit.hive.txnsPerBatchAsk = 10
tier1.sinks.sink_sitevisit.hive.batchSize = 1500
tier1.sinks.sink_sitevisit.serializer = DELIMITED
tier1.sinks.sink_sitevisit.serializer.fieldnames = visitor_id,user_id,session_id,url,keyword
tier1.sinks.sink_sitevisit.roundUnit = hour
tier1.sinks.sink_sitevisit.roundValue = 1

Hive table：

create table weblogs ( id int , msg string )
    partitioned by (continent string, country string, time string)
    clustered by (id) into 2 buckets
    stored as orc;