个人安装配置sbt流程及遇到的问题

个人安装使用sbt流程及遇到的问题

安装流程

1、官网下载sbt ， https://www.scala-sbt.org/download.html

2、测试sbt是否可用

//进入sbt目录
cd /opt/apps/sbt
./sbt sbtVison

显示无法找到,查看sbt目录发现sbt文件在bin目录下，改为

./bin/sbt sbtVersion

3、回到主目录，创建sparkapp文件夹作为应用程序根目录

cd ~
mkdir ./sparkapp
mkdir -p ./sparkapp/src/main/scala

进入sparkapp目录下查看创建情况

cd sparkapp
find .

.
./src
./src/main
./src/main/scala

4、进入/src/main/scala创建SipmleApp.scala

/*  SimpleApp.scala  */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp{
        def main(args: Array[String]){
                val logFile = "hdfs://hadoop0:9000/data/word/in/word"
                val conf = new SparkConf().setAppName("Simple Application")
                val sc = new SparkContext(conf)
                val logData = sc.textFile(logFile,2).cache()
                val numAs = logData.filter(_.contains("a")).count()
                val numBs = logData.filter(_.contains("b")).count()
                println("Lines with a: %s,Lines with b: %s".format(numAs,numBs))

        }
}

5、使用sbt对Scala程序进行打包

​ 回到sparkapp，创建simple.sbt

name := "Simple Project"
version := "1.0"
scalaVersion := "2.12.15"
libraryDependencies += "org.apache.spark" %% "spark-core" %"3.3.0"

输入命令/opt/apps/sbt/bin/sbt package,编译成功就会出现如下信息

[info] Loading project definition from /root/sparkapp/project
[info] Loading settings for project sparkapp from simple.sbt ...
[info] Set current project to Simple Project (in build file:/root/sparkapp/)
[success] Total time: 1 s, completed Oct 18, 2022 12:10:02 AM

遇到的问题

1、./bin/sbt sbtVersion命令后会开始下载

出现Getting org.Scala-sbt 1.3.8 (this may take some times)...后很长时间不动

在sbt目录下创建repositories文件，并添加

[repositories]
local
huaweicloud-maven: https://repo.huaweicloud.com/repository/maven/
maven-central: https://repo1.maven.org/maven2/
sbt-plugin-repo: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]

2、下载scala-2.12.10报错 error: not found: value scalaversion scalaversion := "2.12.15"

打开simple.sbt，去掉scalaVersion := "2.12.15"一行

3、编译时报错

在spark官网 https://spark.apache.org/docs/latest/streaming-programming-guide.html#a-quick-example 找到

将simple.sbt中的libraryDependencies += "org.apache.spark" %% "spark-core" %"3.3.0"改为
libraryDependencies += "org.apache.spark" % "spark-streaming_2.12" % "3.3.0" % "provided"