使用SparkStreaming整合Flume时使用pull模式报错及解决

最新推荐文章于 2022-08-03 11:57:33 发布

Horizon_Zy

最新推荐文章于 2022-08-03 11:57:33 发布

阅读量824

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/Horizon_Zy/article/details/79338113

版权

大数据专栏收录该内容

6 篇文章 0 订阅

订阅专栏

错误信息如下：

2018-02-19 20:27:19,451 (Spark Sink Processor Thread - 2) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:59)] Spark was unable to successfully process the events. Transaction is being rolled back.
2018-02-19 20:27:19,451 (New I/O  worker #1) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:59)] Received an error batch - no events were received from channel!
2018-02-19 20:27:19,452 (Spark Sink Processor Thread - 3) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:80)] Error while processing transaction.
java.lang.IllegalStateException: begin() called when transaction is OPEN!
	at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
	at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:114)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:113)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.populateEvents(TransactionProcessor.scala:113)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:243)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:43)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

自己明明按照spark官网跟着做却报这个错误，花了三个小时终于找到错误了。flume的lib文件夹下本来自带一个scala-libary的jar包。后来自己拷贝一个2.11的jar包进去，没有删除原来的jar包，导致jar包冲突，所有引发这个问题。

解决：将flume_home下的lib中的老的版本的scala-libary的jar包删除，重新运行，问题解决。