ubuntu下Sprak(IDE)wordcount例子

一、进入IDE界面

cd ~/Downloads/idea/bin

idea.sh

 

二、建立scala项目

Step 1：导入Spark-hadoop对应的包，次选择“File”–> “Project Structure” –> “Libraries”，选择“+”，将spark-hadoop 对应的包导入:

 

点击“OK”确认：

点击“OK”:

IDEA工作完成后会发现Spark的jar包导入到了我们的工程中：

第二步、写scala代码实现wordcount

import SparkContext

import SparkContext._
object Wordcount {
def main(args: Array[String]) {
val sc = new SparkContext("local", "Simple App", "home\\yuanqin\\Downloads\\spark-0.9.0-incubating-bin-hadoop1",null)
//List("lib/spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar")
val logFile = sc.textFile("hdfs://Master:9000/user/root/input/file.txt") // Should be some file on your system
val count=logFile.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
println(count)
count.saveAsTextFile("home\\yuanqin\\output\\2014")

}
}

第三步、右击run as....

 

 

转载于:https://www.cnblogs.com/yuanqin/p/4060562.html