1 IDEA中配置SBT
2 创建scala的sbt项目
2.1 创建指定版本参数的项目
注意下面的配置。
自动生成如下目录。
2.2 添加build.sbt依赖
不需要在windows系统中安装spark。
直接使用IntellIj IDEA下载相关依赖,开发即可。
需要再build.sbt中添加依赖。
(1)添加spark的依赖
name := "zidingyi"
version := "0.1"
scalaVersion := "2.11.12"
val sparkVersion = "2.4.5"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "1.0.0"
变更build.sbt文件后,进行更新。
2.3 编辑代码
进入src/main/scala,右键,新建Scala Class,mytest,object。
(2)代码
import org.apache.spark.{SparkConf, SparkContext}
object mytest {
def main(args: Array[String]): Unit = {
val config = new SparkConf().setAppName("wc").setMaster("local[*]")
val sc = new SparkContext(config)
sc.setLogLevel("WARN")
var arr = Array("Runoob", "Baidu", "Google")
val data = sc.parallelize(arr)
data.foreach(println)
}
}
然后直接运行,即可。
3 异常及解决
(3)异常不影响使用
20/07/10 17:12:43 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.