spark+kafka报错java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative

最新推荐文章于 2021-05-12 17:26:21 发布

big data programmer

最新推荐文章于 2021-05-12 17:26:21 发布

阅读量2.6k

点赞数 1

分类专栏：大数据文章标签： kafka spark

本文链接：https://blog.csdn.net/a1776868/article/details/106398135

版权

当使用Spark处理Kafka数据时遇到`java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative`错误。该问题可能并非由于删除Kafka topic引起，常规解决方案如重建topic和消费者组无效。通过对错误信息的深入理解，发现是topic中没有记录导致。通过向topic写入数据后，程序成功启动。

摘要由CSDN通过智能技术生成

在执行kafka+spark程序时报错以下内容
在这里插入图片描述 ERROR JobScheduler: Error generating jobs for time 1588074000000 ms java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative at scala.Predef$.require(Predef.scala:224) at org.apache.spark.streaming.scheduler.StreamInputInfo.<init>(InputInfoTracker.scala:38) at org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.compute(DirectKafkaInputDStream.scala:233) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341) at org.apache.spark.streaming.dst