spark程序提交到集群上_win10下将spark的程序提交给远程集群中运行

一，开发环境：

操作系统：win19 64位

IDE：IntelliJ IDEA

JDK：1.8

scala:scala-2.10.6

集群：linux上cdh集群，其中spark为1.5.2,hadoop:2.6.0(其实我也想用spark最新版和hadoop的最新版，但1.6以前有spark-assembly-1.x.x-hadoop2.x.x.jar)

二，实现步骤：

1，设置maven的pom.xml

4.0.0

spark

test

1.0-SNAPSHOT

2008

2.10.6

scala-tools.org

Scala-Tools Maven2 Repository

http://scala-tools.org/repo-releases

scala-tools.org

Scala-Tools Maven2 Repository

http://scala-tools.org/repo-releases

junit

junit

4.12

org.specs

specs

1.2.5

test

commons-logging

commons-logging

1.1.1

jar

org.apache.commons

commons-lang3

3.1

log4j

log4j

1.2.9

org.apache.spark

spark-core_2.10

1.5.2

org.apache.spark

spark-sql_2.10

1.5.2

org.apache.hadoop

hadoop-client

2.6.0

org.apache.spark

spark-mllib_2.10

1.5.2

org.apache.spark

spark-hive_2.10

1.5.2

org.apache.spark

spark-streaming_2.10

1.5.2

org.scala-lang

scala-library

2.10.6

src/main/scala

src/test/scala

org.scala-tools

maven-scala-plugin

compile

testCompile

${scala.version}

-target:jvm-1.5

org.apache.maven.plugins

maven-eclipse-plugin

true

ch.epfl.lamp.sdt.core.scalabuilder

ch.epfl.lamp.sdt.core.scalanature

org.eclipse.jdt.launching.JRE_CONTAINER

ch.epfl.lamp.sdt.launching.SCALA_CONTAINER

org.scala-tools

maven-scala-plugin

${scala.version}

2，编写简单程序：

object test {

def main(args: Array[String]): Unit={

val conf= new SparkConf().setMaster("spark://xxxxx:7077").setAppName("test")

val sc= newSparkContext(conf)

sc.addJar("E:\\sparkTest\\out\\artifacts\\sparkTest_jar\\sparkTest.jar")

val count= sc.parallelize(1 to 4).filter { _ =>val x=math.random

val y=math.random

x*x + y*y < 1}.count()

println(s"Pi is roughly ${4.0 * count / 4}")

sc.stop()

}

}

3,打jar包，即：file->projectStruct->Artifacts->Build->Build Artifacts，点击run运行即可(刚刚试试了下，发现不要jar也能运行，只是控制台还没结果输出？)

4，pom.xml的spar