基于eclipse的Spark IDE可在 http://scala-ide.org/ 下载。
以WordCount为例:
-
- package com.lxw.test
-
- import org.apache.spark.{SparkConf, SparkContext}
- import SparkContext._
-
- object WordCount {
- def main (args: Array[String]) {
- if(args.length < 2) {
- println("Usage: WordCount ")
- System.exit(1)
- }
- val hdfsIn = args(0);
- val hdfsOut = args(1);
- val sc = new SparkContext(new SparkConf().setAppName("WordCount"))
- val srcData = sc.textFile(hdfsIn)
- val result = srcData.flatMap(_.split("\\s+")).map((_,1)).reduceByKey(_+_)
- result.saveAsTextFile(hdfsOut)
- }
- }
在eclipse中将程序打成普通的Java jar包即可。
在Spark的一台Client机器上使用spark-submit来提交运行jar包:
-
- $SPARK_HOME/bin/spark-submit \
- --name "lxw1234-wordcount" \
- --master spark://192.168.1.130:7077 \
- --executor-memory 1G \
- --class com.lxw.test.WordCount \
- /home/lxw1234/lxw-spark.jar /logs/site/2015-05-14/ /tmp/lxwoutput
本文转自:http://lxw1234.com/archives/2015/05/215.htm