一,开发环境:
操作系统:win19 64位
IDE:IntelliJ IDEA
JDK:1.8
scala:scala-2.10.6
集群:linux上cdh集群,其中spark为1.5.2,hadoop:2.6.0(其实我也想用spark最新版和hadoop的最新版,但1.6以前有spark-assembly-1.x.x-hadoop2.x.x.jar)
二,实现步骤:
1,设置maven的pom.xml
4.0.0
spark
test
1.0-SNAPSHOT
2008
2.10.6
scala-tools.org
Scala-Tools Maven2 Repository
http://scala-tools.org/repo-releases
scala-tools.org
Scala-Tools Maven2 Repository
http://scala-tools.org/repo-releases
junit
junit
4.12
org.specs
specs
1.2.5
test
commons-logging
commons-logging
1.1.1
jar
org.apache.commons
commons-lang3
3.1
log4j
log4j
1.2.9
org.apache.spark
spark-core_2.10
1.5.2
org.apache.spark
spark-sql_2.10
1.5.2
org.apache.hadoop
hadoop-client
2.6.0
org.apache.spark
spark-mllib_2.10
1.5.2
org.apache.spark
spark-hive_2.10
1.5.2
org.apache.spark
spark-streaming_2.10
1.5.2
org.scala-lang
scala-library
2.10.6
src/main/scala
src/test/scala
org.scala-tools
maven-scala-plugin
compile
testCompile
${scala.version}
-target:jvm-1.5
org.apache.maven.plugins
maven-eclipse-plugin
true
ch.epfl.lamp.sdt.core.scalabuilder
ch.epfl.lamp.sdt.core.scalanature
org.eclipse.jdt.launching.JRE_CONTAINER
ch.epfl.lamp.sdt.launching.SCALA_CONTAINER
org.scala-tools
maven-scala-plugin
${scala.version}
2,编写简单程序:
object test {
def main(args: Array[String]): Unit={
val conf= new SparkConf().setMaster("spark://xxxxx:7077").setAppName("test")
val sc= newSparkContext(conf)
sc.addJar("E:\\sparkTest\\out\\artifacts\\sparkTest_jar\\sparkTest.jar")
val count= sc.parallelize(1 to 4).filter { _ =>val x=math.random
val y=math.random
x*x + y*y < 1}.count()
println(s"Pi is roughly ${4.0 * count / 4}")
sc.stop()
}
}
3,打jar包,即:file->projectStruct->Artifacts->Build->Build Artifacts,点击run运行即可(刚刚试试了下,发现不要jar也能运行,只是控制台还没结果输出?)
4,pom.xml的spar