spark-02_java.lang.NoClassDefFoundError_org.apache.spark.streaming.kafka.Kafk

最新推荐文章于 2022-05-19 20:32:02 发布

秋雪夜雨寒

最新推荐文章于 2022-05-19 20:32:02 发布

阅读量335

点赞数

文章标签： spark

本文链接：https://blog.csdn.net/longyinyushi/article/details/103370021

版权

在spark中本地调试时遇到如果问题

19/11/20 17:08:49 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@73e132e0{/metrics/json,null,AVAILABLE}
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils$
	at org.vincent.streaming.kafkaStreamKafka$.main(kafkaStreamKafka.scala:44)
	at org.vincent.streaming.kafkaStreamKafka.main(kafkaStreamKafka.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaUtils$
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 11 more

spark本地运行脚本内容如下

/data/server/spark/bin/spark-submit \
    --master local[2] \
    --num-executors 2\
    --executor-memory 1g \
    --executor-cores 2 \
    --driver-memory 1g \
    --jars /data/server/spark/jars/spark-streaming_2.11-2.0.2.jar \
    --class org.vincent.streaming.kafkaStreamKafka ./spark_test-ver-1.0.jar

在网上查阅资料说是需要在运行脚本中手动添加一下依赖，最终解决方法如下，即添加--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0

/data/server/spark/bin/spark-submit \
    --master local[2] \
    --num-executors 2\
    --executor-memory 1g \
    --executor-cores 2 \
    --driver-memory 1g \
    --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0 \
    --repositories http://maven.aliyun.com/nexus/content/groups/public/ \
    --jars /data/server/spark/jars/spark-streaming_2.11-2.0.2.jar \
    --class org.vincent.streaming.kafkaStreamKafka ./spark_test-ver-1.0.jar