idea sbt Spark环境搭建

最近在参考厦门大学数据库实验室的博客http://dblab.xmu.edu.cn/blog/1492-2/进行spark sbt idea开发环境搭建，博客中使用的spark2.1.0版本，而个人使用了目前最新的2.2.0版本，所以build.sbt中的配置信息没有沿用以下内容

name := "SBTTest"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"

而是使用了以下信息

name := "SBTTest"

version := "1.0"

scalaVersion := "2.12.3"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"

然后在依赖库安装的过程中遇到了点小问题

Error:Error while importing SBT project:<br/>...<br/><pre>[warn] ==== local-preloaded: tried
[warn]   file:////root/.sbt/preloaded/org/apache/spark/spark-sql_2.12/2.2.0/spark-sql_2.12-2.2.0.pom
[info] Resolving org.scala-lang#scala-compiler;2.12.3 ...
[info] Resolving org.scala-lang#scala-reflect;2.12.3 ...
[info] Resolving org.scala-lang.modules#scala-xml_2.12;1.0.6 ...
[info] Resolving jline#jline;2.14.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-sql_2.12;2.2.0: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Unresolved dependencies path:
[warn]      org.apache.spark:spark-sql_2.12:2.2.0 (/root/spark/sbtHello/build.sbt#L7-8)
[warn]        +- default:sbthello_2.12:1.0
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql_2.12;2.2.0: not found
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql_2.12;2.2.0: not found
[error] Total time: 39 s, completed 2017-8-10 12:31:09</pre><br/>See complete log in <a href="file:/root/.IntelliJIdea2017.2/system/log/sbt.last.log">file:/root/.IntelliJIdea2017.2/system/log/sbt.last.log</a>

针对此问题，后来在官网上http://spark.apache.org/docs/latest/quick-start.html#self-contained-applications发现使用的是如下配置信息

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0"

然后根据官网的重新配置发现既然可以加载依赖库。以上配置中最明显的有两处信息不一样，一个是scalaVersion版本不一样，另一个是依赖库中的spark-core和spark-sql。

经测试发现，无法加载主要原因是scalaVersion版本引起，使用最新2.12的版本无法加载，按照官网使用2.11的可以，而依赖库中使用spark-core或使用spark-sql不影响加载。