最简单一个demo
//读取hdfs,然后写入hdfs
package com.spark.FirstApp
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
object WordCountTest {
def main(args: Array[String]) {
val inputFile = "hdfs://192.168.1.105:9000/word.txt"
val conf = new SparkConf().setAppName("WordCount111").setMaster("spark://centos701:7077")
val sc = new SparkContext(conf)
val textFile = sc.textFile(inputFile,1)
textFile.saveAsTextFile("hdfs://192.168.1.105:9000/test/hello.txt")
}
}
IDEAJ中Build->BuidProject->Build Artifacts->Build,然后找到jar包。
在客户机中提交
./spark-submit –class com.spark.FirstApp.WordCountTest /Users/liuchao/Desktop/20171211.jar
然后查看hdfs发现文件成功写入