spark-kafka-es项目开发问题汇总

1、spark连接kafka报错：

 Exception in thread "main" java.lang.ExceptionInInitializerError
     at org.apache.spark.streaming.dstream.InputDStream.<init>(InputDStream.scala:78)
     at org.apache.spark.streaming.dstream.ReceiverInputDStream.<init>(ReceiverInputDStream.scala:42)
     at org.apache.spark.streaming.kafka.KafkaInputDStream.<init>(KafkaInputDStream.scala:56)
     at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:91)
     at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:66)
     at com.webank.KafkaTest$.main(KafkaTest.scala:26)
     at com.webank.KafkaTest.main(KafkaTest.scala)
    Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.1
     at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
     at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
     at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)
     at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
     at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
     ... 7 more
    18/12/20 17:07:56 INFO SparkContext: Invoking stop() from shutdown hook

这是由于Jackson这个工具库的版本不一致导致的。解决方法：首先在Kafka的依赖项中排除对于Jackon的依赖，从而阻止Maven自动导入高版本的库，手动添加较低版本Jackon库的依赖项。

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.11</artifactId>
  <version>1.0.0</version>
  <exclusions>
    <exclusion>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>*</artifactId>
    </exclusion>
  </exclusions>
</dependency>

<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-core</artifactId>
  <version>2.6.6</version>
</dependency>

2.创建多个sparkContext报错：

org.apache.spark.SparkException: Task not serializable

spark分布式算子中(map, filter, flatMap 等) 使用了外部的变量，但是这个变量不能序列化。深入SparkContext的源代码得知，sparkContext继承了Logging接口，并实现了ExecutorAllocationClient接口，这两个接口都没有实现Serializable接口，所以就会出问题 ‘Task not serializable’；可以使用@transient标注即可；

3.spark启动报错，测试类运行没问题，代码里面运行就报错：

java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
 at java.lang.Class.getDeclaredMethods0(Native Method)
 at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
 at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
 at java.lang.Class.getMethod0(Class.java:3018)
 at java.lang.Class.getMethod(Class.java:1784)
 at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
 at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 ... 7 more
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" 

解决发现pom.xml spark-streaming_2.11改provided为compile：

 <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.11</artifactId>
      <version>2.1.0</version>
      <scope>provided</scope>
  </dependency>

改为：

   <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-streaming_2.11</artifactId>
          <version>2.1.0</version>
          <scope>compile</scope>
      </dependency>

因为对于scope=compile的情况（默认scope),也就是说这个项目在编译，测试，运行阶段都需要这个spark-streaming_2.11对应的jar包在classpath中。而对于scope=provided的情况，则可以认为这个provided是目标容器已经provide这个artifact。换种说法就是你在把这个项目打成war包时，scope=provided的jar包，不会出现在WEB-INFO/lib目录下，而scope=compile的jar包，会放到WEB-INFO/lib目录这样可以避免想servlet-api.jar这样的jar包冲突，但是如果遇到idea下默认不提供servlet-api.jar时，使用scope=provided就会缺少对应的jar包，也就是我遇到的问题找不到类。