flume收集日志到hive遇到问题总结

本文总结了使用Flume收集日志到Hive时可能遇到的问题及解决方案,包括检查.bash_profile、验证jar包、检查hive-site.xml的配置,并提供了Hive表的创建示例,强调表需要分桶并开启事务。
摘要由CSDN通过智能技术生成

异常如下:

19/01/18 02:12:51 WARN hive.HiveSink: k1 : Failed connecting to EndPoint {metaStoreUri='thrift://wangfutai:9083', database='hive', table='flume2', partitionVals=[] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://wangfutai:9083', database='hive', table='flume2', partitionVals=[] }
	at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
	at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
	at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
	at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.sink.hive.HiveWriter$TxnBatchException: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://wangfutai:9083', database='hive', table='flume2', partitionVals=[] }
	at org.apache.flume.sink.hive.HiveWriter.nextTxnBatch(HiveWriter.java:400)
	at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:90)
	... 6 more
Caused by: org.apache.hive.hcatalog.streaming.TransactionBatchUnAvailable: Unable to acquire transaction batch on end point: {metaStoreUri='thrift://wangfutai:9083', database='hive', table='flume2', partitionVals=[] }
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:514)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:464)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatchImpl(HiveEndPoint.java:351)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatch(HiveEndPoint.java:331)
	at org.apache.flume.sink.hive.HiveWriter$9.call(HiveWriter.java:395)
	at org.apache.flume.sink.hive.HiveWriter$9.call(HiveWriter.java:392)
	at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPool
  • 0
    点赞
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值