用flume访问kafka,写入hive数据库时,启动命令:bin/flume-ng agent -n agent -c conf -f conf/flume.properties -Dflume.root.logger=DEBUG,console
配置文件是安装flume官方文档配置的,很全,地址:http://flume.apache.org/FlumeUserGuide.html#kafka-sink
启动flume之后,报错:
ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri=’thrift://192.168.100.223:9000’, database=’hivedb’, table=’t_traffic_his’, partitionVals=[2017-07-12] }
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:267)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri=’thrift://192.168.100.223:9000’, database=’hivedb’, table=’t_traffic_his’, partitionVals=[2017-07-12] }
at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:99)
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
… 3 more
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri=’thrift://192.168.100.223:9000’, database=’hivedb’, table=’t_traffic_his’, partitionVals=[2017-07-12] }
at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:383)
at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:86)
… 6 more
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:456)
at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:376)
… 7 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:434)
… 8 more
Caused by: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:436)
at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:430)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:550)
at
之前已经把hive和hadoop的相关jar包拷贝到flume_HOME/lib下
(根据报错信息--java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter,把文件中所有jar包全部拷贝过去了)
数据也写不到hive中。
最后,启动命令增加--classpath参数:
--classpath "/home/huser/hive-2.1.1/hcatalog/share/hcatalog/*"
再次启动,kafka写入数据,hive可以查询到信息,但是之前有部分数据没有读到,flume配置的是agent.sources.skafka.kafka.consumer.auto.offset.reset = earliest
可能是offset在flume报错时也记录了吧,有了解的请留言告诉我,多谢!