Interlij 13编译Spark程序生成jar包

最新推荐文章于 2022-11-18 23:16:04 发布

ckl_soft

最新推荐文章于 2022-11-18 23:16:04 发布

阅读量1.9k

点赞数 1

分类专栏：技巧 Spark

本文链接：https://blog.csdn.net/cklsoft/article/details/38360377

版权

Spark 同时被 2 个专栏收录

11 篇文章 0 订阅

订阅专栏

技巧

7 篇文章 0 订阅

订阅专栏

1、创建项目sp

create new project->scala->NOT SBT->next->设置项目名称'sp'

2、导入相关jar包

File->Project Structure->Libraries->点绿色'+'->java->找到spark-assembly-1.0.0-hadoop2.2.0.jar->OK

按照类似的方法导入scala-compiler.jar, scala-library.jar, scala-reflect.jar //这些位于scala的安装目录下的lib目录

3、创建scala源文件HdfsWC.scala

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
class HdfsWC {
  def main(args: Array[String]) {
    val sc = new SparkContext(args(0)/*"yarn-standalone"*/,"myWordCount",System.getenv("SPARK_HOME"),null)
    //List("lib/spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar")
    val logFile = sc.textFile(args(1))//"hdfs://master:9101/user/root/spam.data") // Should be some file on your system
    //  val file = sc.textFile("D:\\test.txt")
    val counts = logFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
    //   println(counts)
    counts.saveAsTextFile(args(2)/*"hdfs://master:9101/user/root/out"*/)
  }
}