scala打印服务器消息,Spark中使用Scala实现WordCount业务

Spark中使用Scala实现WordCount业务

创建一个Project

sbt选择1.0.4

Scala选择2.11.8

配置路径

Project Sources

Dependencies

新建object

MyScalaWordCount.scala

本地模式

object MyScalaWordCount {

def main(args: Array[String]): Unit = {

val conf = new SparkConf().setAppName("MyScalaWordCount").setMaster("local");

//创建一个SparkContext对象

val sc = new SparkContext(conf)

//执行WordCount

val result = sc.textFile("hdfs://bigdata02:9000/wordcount.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)

//打印在屏幕上

result.foreach(println)

//释放资源

sc.stop()

}

}

导出Jar包在服务器上运行

MyScalaWordCount.scala

生成jar包

object MyScalaWordCount {

def main(args: Array[String]): Unit = {

val conf = new SparkConf().setAppName("MyScalaWordCount");

//创建一个SparkContext对象

val sc = new SparkContext(conf)

//执行WordCount

val result = sc.textFile(args(0)).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)

//打印在屏幕上

result.foreach(println)

//释放资源

sc.stop()

}

}

打包操作

Project Sources

Artifacts

Build Artifacts

导出成功

上传Jar包到服务器并执行

cd /home/bigdata/apps/spark-2.1.0-bin-hadoop2.7

./bin/spark-submit --master spark://bigdata02:7077 --class nx.MyScalaWordCount /home/bigdata/data/SparkScalaWork.jar hdfs://bigdata02:9000/wordcount.txt hdfs://bigdata02:9000/output/spark/wc0928

hdfs dfs -cat /output/spark/wc0928/part-00000