本地Spark跑不会出现什么问题,但是切换到服务器上各种问题。
异常
java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.f$3 of type org.apache.spark.api.java.function.FlatMapFunction in instance of org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1
修改pom.xml文件在project中添加下方xml,然后右击项目执行-> Run as -> maven claen && maven install
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
添加jars路径
String jarPath = "打包成功之后的项目jar文件"
JavaSparkContext jsc = new JavaSparkContext(new SparkConf()
.setMaster("spark://服务器地址:端口")
.setJars(new String[] {jarPath})
.setAppName("Classification"));
然后在执行
END: