flume读取kafka写入hive异常解决-Unable to deliver event. Exception follows.

用flume访问kafka，写入hive数据库时，启动命令：bin/flume-ng agent -n agent -c conf -f conf/flume.properties -Dflume.root.logger=DEBUG,console

配置文件是安装flume官方文档配置的，很全，地址：http://flume.apache.org/FlumeUserGuide.html#kafka-sink

启动flume之后，报错：

ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri=’thrift://192.168.100.223:9000’, database=’hivedb’, table=’t_traffic_his’, partitionVals=[2017-07-12] }
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:267)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri=’thrift://192.168.100.223:9000’, database=’hivedb’, table=’t_traffic_his’, partitionVals=[2017-07-12] }
at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:99)
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
… 3 more
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri=’thrift://192.168.100.223:9000’, database=’hivedb’, table=’t_traffic_his’, partitionVals=[2017-07-12] }
at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:383)
at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:86)
… 6 more
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:456)
at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:376)
… 7 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:434)
… 8 more
Caused by: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:436)
at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:430)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:550)
at 

之前已经把hive和hadoop的相关jar包拷贝到flume_HOME/lib下

(根据报错信息--java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter，把文件中所有jar包全部拷贝过去了)

数据也写不到hive中。

最后，启动命令增加--classpath参数：  
 

--classpath  "/home/huser/hive-2.1.1/hcatalog/share/hcatalog/*"

再次启动，kafka写入数据，hive可以查询到信息，但是之前有部分数据没有读到，flume配置的是agent.sources.skafka.kafka.consumer.auto.offset.reset = earliest

可能是offset在flume报错时也记录了吧，有了解的请留言告诉我，多谢！