spark-1-windows中用scala编写spark应用程序

1 IDEA中配置SBT

参考sbt-2-IDEA中配置创建scala项目sbt打包

2 创建scala的sbt项目

2.1 创建指定版本参数的项目

注意下面的配置。
自动生成如下目录。

2.2 添加build.sbt依赖

不需要在windows系统中安装spark。
直接使用IntellIj IDEA下载相关依赖，开发即可。
需要再build.sbt中添加依赖。
(1)添加spark的依赖

name := "zidingyi"

version := "0.1"

scalaVersion := "2.11.12"

val sparkVersion = "2.4.5"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "1.0.0"

变更build.sbt文件后，进行更新。

2.3 编辑代码

进入src/main/scala，右键，新建Scala Class，mytest，object。

(2)代码

import org.apache.spark.{SparkConf, SparkContext}
object mytest {
  def main(args: Array[String]): Unit = {
    val config = new SparkConf().setAppName("wc").setMaster("local[*]")
    val sc = new SparkContext(config)
    sc.setLogLevel("WARN")
    var arr = Array("Runoob", "Baidu", "Google")
    val data = sc.parallelize(arr)
    data.foreach(println)
  }
}

然后直接运行，即可。

3 异常及解决

(3)异常不影响使用
20/07/10 17:12:43 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.