spark wordcount

访问本地数据

在/usr/local/spark/mycode/创建一个文件
vi /usr/local/spark/mycode/word1.txt 输入一些单词，作为需要统计的数据

进入spark-shell

val textFile = sc.textFile("file:///usr/local/spark/mycode/word.txt")

textFile.first(); //显示文本的第一行

textFile.saveAsTextFile("file:///usr/local/spark/mycode/wordcopy") //把文件数据写到wordcopy目录下， 目录下会有part-00000为数据内容

访问hdfs上数据

把word.txt上传到hdfs
创建文件夹

hadoop fs -mkdir /input
上传
hadoop fs -put word.txt /input

进入spark-shell

val textFile = sc.textFile(“hdfs://localhost:9000/input/word.txt”)
上面路径也可以简写成:
val textFile = sc.textFile(“/input/word.txt”)

单词统计

两种方式可以进行单词统计操作

在spar-shell下方式

val wordCount = textFile.flatMap(line => line.split(” “)).map(word => (word,1)).reduceByKey((a,b) => a+b)

wordCount.collect()

输出内容

Array[(String, Int)] = Array((python,1), (hello,3), (java,1), (spark,1))

通过submit方式

创建单词文件
mkdir -p /usr/local/spark/mycode
vi /usr/local/spark/mycode/word.txt

hello python
hello java
hello spark

创建scala工程的多级目录, 这里的wordcount就相当于程序的主目录
mkdir -p /usr/local/spark/mycode/wordcount/src/main/scala

在scala目录下编写test.scala
vi ~/src/main/scala/test.scala
内容如下：

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object WordCount {
        def main(args: Array[String]){
                val inputFile = "file:///usr/local/spark/mycode/word.txt"
                val conf = new SparkConf().setAppName("WordCount").setMaster("local[2]")
                val sc = new SparkContext(conf)
                val textFile = sc.textFile(inputFile)
                val wordCount = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a+b)
                wordCount.foreach(println)
        }
}

在工程主目录wordcount下创建simple.sbt
内容为：

name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"

在工程主目录wordcount下执行命令,进行打包
/usr/local/sbt/sbt package

执行结束后会有如下提示

[info] Done packaging.
[success] Total time: 101 s, completed Nov 2, 2017 9:21:56 AM

进入工程目录下查看生成的jar文件(simple-project_2.11-1.0.jar)
cd /usr/local/spark/mycode/wordcount/target/scala-2.11

接着通过spark-submit 执行任务
/usr/local/spark/bin/spark-submit –class “WordCount” /usr/local/spark/mycode/wordcount/target/scala-2.11/simple-project_2.11-1.0.jar

在输出的信息中可以看到如下结果:

(spark,1)
(python,1)
(hello,3)
(java,1)