今天再用Maven来编写Spark程序的时候,在使用了SQLContext的API之后,每次编译就是不通过。看错误信息提示了bad symbolic reference. A signature in package.class refers to type compileTimeOnly。这个错误。在网上找了一大堆,都没有找到这个问题的真正解决方法。最后,无意中看了一个博客,最后解决了,所以在这里记录一下。在自己的pom文件里面加上这个。
<properties>
<jackson.version>2.4.2</jackson.version>
<scala.version>2.10.6</scala.version>
<slf4j.version>1.7.7</slf4j.version>
<scala.maven.version>2.11.1</scala.maven.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<id>compile-scala</id>
<phase>compile</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile-scala</id>
<phase>test-compile</phase>
<goals>
<goal>add-source</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>com.typesafe.scala-logging</groupId>
<artifactId>scala-logging_2.11</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.2</version>
</dependency>
</dependencies>
之前只是加了Spark Core和Spark SQL的依赖。没有加scala方面的依赖。最后加入了scala相关的依赖后,就可以运行了。估计是打包的时候,Spark里面的依赖包用到了scala的资源。虽然你的项目中有scala配置。但是打包的时候,是用不到的。所以要在pom里面加上scala的相关依赖包。