报错1:
java.lang.UnsatisfiedLinkError:org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
解决方法:
- 点击 Edit Configurations => Configuration
- 如图,在VM options:后面输入:-Djava.library.path=$HADOOP_HOME/lib/native
报错2:
ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
解决方法:
- 创建spark上下文时设置一个属性,set(“spark.serializer”, “org.apache.spark.serializer.KryoSerializer”)
// 创建Spark上下文
SparkConf conf = new SparkConf()
.setMaster("local[2]")
.setAppName("NewsRealtimeStatSpark")
.set("spark.serializer","org.apache.spark.serializer.KryoSerializer");
报错3:
java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext
解决方法:
- 问题就出在依赖的范围:provided,这表示编译和测试时有效,并且该jar包在运行时由服务器提供,因此只需要将这一行去掉即可,如下图,去掉: