大数据开发之如何用Scala进行spark开发

1、开发第一个Spark程序
1)创建一个Spark Context
2)加载数据
3)把每一行分割成单词
4)转换成pairs并且计数

2、wordCount程序

import org.apache.spark.{SparkConf, SparkContext}


/**
  * @author
  * @date 2020-05-11 20:19
  * @version 1.0
  */
def main(args: Array[String]) {


  val conf = new SparkConf().setAppName("wordcount")
  val sc= new SparkContext(conf)


  val input=sc.textFile("/Users/navyliu/Downloads/spark/helloSpark.txt")


  val lines = input.flatMap(line=>line.split(" "))
  val count = lines.map(word=>(word,1)).reduceByKey{case (x,y)=>x+y}
  val output = count.saveAsTextFile("/Users/navyliu/Downloads/spark/helloSparkRes")


}

3、打包
配置jar包大数据培训
build

 

然后build

 

4、启动集群
1)启动master

./sbin/start-master.sh

 

master启动之后,可以访问下面地址:
http://localhost:8080/

 

2)启动worker

./bin/spark-class

启动work要用到这个地址:

spark://deMacBook-Pro.local:7077

启动work的命令为:

./bin/spark-class org.apache.spark.deploy.worker.Worker spark://navydeMacBook-Pro.local:7077

 

查看进程

 

也可以在页面中看到worker进程

 

3)提交作业

./bin/spark-submit

提交命令

./bin/spark-submit --master spark://deMacBook-Pro.local:7077 --class CountWord untitled.jar

把生成的jar包拷到当前目录

 

提交作业的命令为:

./spark-2.4.5-bin-hadoop2.7/bin/spark-submit --master spark://deMacBook-Pro.local:7077 --class CountWord untitled.jar

结果为:

 

可以看到生成了文件目录helloSparkRes

 

 

jobs任务截图:

 

5、遇到的问题

如果报这种错误:

dyld: lazy symbol binding failed: Symbol not found: ____chkstk_darwin

Referenced from: /private/var/folders/91/6g1y3wp163jbr7tgdrqgkjg80000gn/T/liblz4-java-8731604412047028366.dylib

Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ____chkstk_darwin

Referenced from: /private/var/folders/91/6g1y3wp163jbr7tgdrqgkjg80000gn/T/liblz4-java-8731604412047028366.dylib

Expected in: /usr/lib/libSystem.B.dylib

原因是scala版本与spark版本冲突，重新下载spark，版本是2.4.5(spark-2.4.5-bin-hadoop2.7)，对应的scala是scala-2.12.11.

重新运行即可.

作者：归来少年666