Flume采集数据集成到Hive分桶表问题总结

Exception in thread “main” java.lang.NoClassDefFoundError: org/antlr/runtime/RecognitionException
解决：拷贝hive中的jar到flume的lib下
antlr-3.4.jar
antlr-runtime-3.4.jar

Caused by: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
解决：
ag1.sinks.sink1.useLocalTimeStamp=true

Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
解决：
拷贝hive中的lib的libfb303-0.9.3.jar到flume/lib

Cannot stream to table that has not been bucketed : {metaStoreUri=‘thrift://master:9083’, database=‘default’, table=‘t_pages’, partitionVals=[] }
解决：
flume必须连接hive的分桶表

org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.AcidOutputFormat
解决：
创建hive表加入：stored as orc;

还有就是报hive表字段和flume配置中字段找不到，因为hive表只能识别到小写，所以修改flume配置为：
awen.sinks.k1.serializer.fieldnames = datestring,username,operatione,userip,url,title,beforeurl,equipment,airip都

需要导入的jar包总结
libfb303-0.9.3.jar
antlr-3.4.jar
antlr-runtime-3.4.jar

hadoop-auth-2.5.0-cdh5.3.3.jar
hadoop-common-2.5.0-cdh5.3.3.jar
hadoop-hdfs-2.5.0-cdh5.3.3.jar
hadoop-hdfs-nfs-2.6.0-cdh5.12.2.jar
hadoop-mapred-0.22.0.jar 没找到

hive-cli-1.1.0-cdh5.12.2.jar
hive-common-1.1.0-cdh5.12.2.jar
hive-exec-1.1.0-cdh5.12.2.jar
hive-hcatalog-core-1.1.0-cdh5.12.2.jar
hive-hcatalog-pig-adapter-1.1.0-cdh5.12.2.jar
hive-hcatalog-server-extensions-1.1.0-cdh5.12.2.jar
hive-hcatalog-streaming-1.1.0-cdh5.12.2.jar
hive-metastore-1.1.0-cdh5.12.2.jar
htrace-core-3.2.0-incubating.jar

commons-configuration-1.9.jar
commons-io-2.4.jar
common-1.1.0.jar 没找到