1、代码编写
if (args.length != 3 ){
println("usage is org.test.WordCount <master> <input> <output>")
return
}
val sc = new SparkContext(args(0), "WordCount",
System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_TEST_JAR")))
val textFile = sc.textFile(args(1))
val result = textFile.flatMap(line => line.split("\\s+"))
.map(word => (word, 1)).reduceByKey(_ + _)
result.saveAsTextFile(args(2))
2、导出jar包,这里我命名为WordCount.jar
3、运行
bin/spark-submit --master spark://master:7077 /home/zkpk/Desktop/WordCount.jar hdfs://master:9000/dedup_in hdfs://master:9000/dedup_out
先记录下,第三部还没执行,抽空试试