spark-streaming+kafka 出现org.apache.spark.SparkException: java.lang.IllegalArgumentException

最新推荐文章于 2024-04-22 09:19:11 发布
最新推荐文章于 2024-04-22 09:19:11 发布
阅读量1.7k 收藏
点赞数
分类专栏： spark-streaming 文章标签： spark-streaming kafka java.lang.IllegalArgumentExcep
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u010379814/article/details/80115737
版权


启动spark-streaming 出现以下错误的原因：

             可能是项目使用的jar包中的kafka版本跟服务器端的kafka版本不一致，解决方法：检查spark集群中kafka的jar版本号，调整项目中引用的kafka版本

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 24.0 failed 1 times, most recent failure: Lost task 0.0 in stage 24.0 (TID 24, localhost, executor driver): java.lang.IllegalArgumentException
	at java.nio.Buffer.limit(Buffer.java:267)
	at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
	at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
	at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.Range.foreach(Range.scala:141)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
	at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
	at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
	at scala.collection.immutable.Range.foreach(Range.scala:141)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
	at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
	at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
	at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:192)
	at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:208)
	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:292)
	at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
	at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1668)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1627)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1616)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1862)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1875)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1888)
	at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1328)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.take(RDD.scala:1302)
	at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1430)
	at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1430)
	at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1430)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1429)
	at com.test.spark.main.xxx$$anonfun$createStreamingContext$1.apply(xxx.scala:82)
	at com.test.spark.main.xxx$$anonfun$createStreamingContext$1.apply(xxx.scala:81)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
	at scala.util.Try$.apply(Try.scala:161)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:223)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException
	at java.nio.Buffer.limit(Buffer.java:267)
	at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
	at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
	at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.Range.foreach(Range.scala:141)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
	at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
	at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
	at scala.collection.immutable.Range.foreach(Range.scala:141)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
	at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
	at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
	at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:192)
	at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:208)
	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:292)
	at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
	at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
	... 3 more

星月永随
关注
专栏目录
第114课:SparkStreaming+Kafka+Spark SQL+TopN+Mysql+KafkaOffsetMonitor电商广告点击综合案例实战(详细内幕版本)
段智华的博客
06-05 1万+
第114课:Spark Streaming电商广告点击综合案例实战(测试版本) /* 王家林老师授课http://weibo.com/ilovepains  每天晚上20：00YY频道现场授课频道68917580*/ 综合案例的实战我们分步进行： 1.Mysql 中建立表 ：
Spark-Kudu报错：kudu.client.NonRecoverableException: The table was deleted: Table deleted at xxx
九师兄
07-04 1433
1.美图 2.背景 一个spark 写入到kudu 的程序，开始运行正常，后来报错了，错误如下 Exchange hashpartitioning(magr_pos_id#549, 12) +- *(17) Project [data_dt#524, acct_id#525, corp_scale#65, magr_pos_id#549] +- SortMergeJoin [CUST_ID...
spark 开启ESS报 java.lang.IllegalArgumentException: Unknown message type: 10
weixin_48231806的博客
06-02 1055
spark external shuffle service 高版本不兼容低版本的shuffle服务
Caused by: java.lang.IllegalArgumentException: Cannot grow BufferHolder by size 1752 because the siz
WGS.
03-06 3474
报错如下： Caused by: java.lang.IllegalArgumentException: Cannot grow BufferHolder by size 9384 because the size after growing exceeds size limitation 2147483632 不能按大小9384增加BufferHolder，因为增长后的大小超过了大小限制2147483632 参考链接： https://docs.microsoft.com/zh-cn/azure/da
spark报错java.lang.IllegalArgumentException: Can‘t get JDBC type for array＜string＞
qq_45087487的博客
08-05 564
2、原因：hive表中元素为数组类型，mysql不支持，需要将类型转换一下就可（比如转换成字符串等等）1、使用spark任务将hive中的表数据存入到mysql中报如下错误。3、方案：进行备份之前将hive中为数组类型的数据结构进行转换一下。
pyspark 使用pandas_udf 报错：java.lang.IllegalArgumentException
anglemanyi的博客
06-13 647
pyspark使用pandas_udf时的一个坑 2.运行报错： 3. 解决办法 报错原因：pyarrow 版本太高。解决办法：卸载pyarrow，安装低版本的pyarrow，这里安装0.12.0 程序就正常了。
使用idea编辑sparkSql的程序运行出现java.lang.IllegalArgumentException: java.net.URISyntaxException
w3045872817的博客
10-25 4216
这个是具体错误：Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:D:/idea/work/sparkTest/spark-warehouse at org.apache.hadoop.f
spark+kafka报错java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative
a1776868的博客
05-28 2672
在执行kafka+spark程序时报错以下内容 ERROR JobScheduler: Error generating jobs for time 1588074000000 ms java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative at scala.Predef$.require(Predef.scala:224) at org.apache.spark.streaming.sch
六（1）、spark遇到的问题
Angular_need的博客
04-12 4271
1、意外退出spark-shell，而不是quit，然后再输入spark-shell命令的时候，报错： 19/04/11 13:42:32 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. jps查看，SparkSubmit在，查看4040端口也是被SparkSubmit占用...
spark：sparkstreaming 0.10版本 案例1：从 kafka 采集数据，并存储到 Hbase Demo示例 / 案例2：实时更新mysql
不花的花和尚的博客
06-19 806
pom： <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/
SparkStreaming消费kafka中的数据保存到HBase中简单demo
09-30 1949
package cn.zhangyu import cn.zhangyu.HbaseStreaming.createTable import cn.zhangyu.utils.JsonUitls import org.apache.hadoop.hbase.client.{ConnectionFactory, Put, Table} import org.apache.hadoop.hbase.u...
SparkException: org.apache.spark.streaming.dstream.MappedDStream has not been initialized
mtj66的博客，交流WX:SpringBreeze1104
12-16 2738
在使用故障恢复的时候采用此方法进行业务逻辑进行恢复的时候，所有的业务逻辑应该放在 functionToCreateContext 函数内部才能实现checkpoint目录数据的恢复。    eFormat import java.util.Date import org.apache.commons.logging.LogFactory import org.apache
解决spark任务报错:java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative
a1142939032的博客
03-08 3604
如题，这个任务的报错是有由于消费方的消费条数和注册中心的不一致导致的。解决方法仅供参考哈： 1、先到kafka集群查看消费偏移量 ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list [集群地址:对应端口] --topic [topic名] （Pid(分区id)、Offset(当前已消费的条数)、LogSize(总条数)、Lag(未消费的条数)） 2、修改注册中心zookeeper 的偏移量 a、先进入zookeeper ...
Kafka报错ERROR Exiting Kafka due to fatal exception during startup
最新发布
qq_48817343的博客
04-22 2916
找到你的logs文件目录下然后找到meta.properties并删除。
Spark报错:Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo
Sun's Blog
06-07 738
问题 在yarn集群上提交一个spark-submit的任务，最后抛出一个异常 解决 原因： 因为在之前的项目中，在hadoop中的core-site.xml 和mapred-site.xml文件配置了lzo格式的压缩，这就导致上传到hdfs 的文件自动被压缩为lzo了。所以当使用提交spark-submit任务时，需要访问HDFS上的文件，而spark自身没有lzo的jar包所以无法找到。 方法一： 软链接到spark下的jars目录，注意自己的hadoop-lzo-0.4.20.jar的目录！ ln
kafka消息无法正常消费，且报异常：java.lang.IllegalArgumentException: Path must not end with / character
热门推荐
ado1986的专栏
12-02 1万+
背景 昨天同事在重启消费者进程后，导致后台消息无法正常消费，看日志，有报数据库的异常：org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection；另外偶尔还有kafka异常：java.lang.IllegalArgumentException: Path must not end
ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)[已解决]
m0_65077254的博客
02-13 4306
ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)[已解决]
Hadoop Spark HA issue -- java.lang.IllegalArgumentException: java.net.UnknownHostException
vivasugar的博客
10-22 392
Usually, we copy core-site.xml and hdfs-site.xml from ~/hadoop/etc/ folder into ~/spark/conf/. Then sync the files into sub-nodes and restart spark server. It didn’t work. It would return the error me...
Spark Streaming Kafka CreateDirectDStreaming 遇见的问题
吟啸徐行
12-01 3746
问题1： spark-submit 提交任务报错如下： 分析：起初我的spark 集群是部署在yarn上，所以在spark-env和spark-default下配置了hadoop相关参数。最后我想使用spark standalone模式跑程序，就把spark-env和spark-default下的hadoop相关参数 注释掉了。之后提交程序提示： Exception in thread "
spark-streaming-kafka-0-8_2.11-2.1.0.jar下载
07-27
要下载 spark-streaming-kafka-0-8_2.11-2.1.0.jar，首先需要打开一个可用的Web浏览器，并前往一个可靠的源（如Apache官方网站或Maven中央存储库）。然后，按照以下步骤进行下载： 1. 在浏览器的搜索栏中键入...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值