Spark0.91编译记录maven

最新推荐文章于 2014-11-20 20:48:48 发布

OopsOutOfMemory

最新推荐文章于 2014-11-20 20:48:48 发布

阅读量1.8k

点赞数

分类专栏： spark 文章标签： spark 分布式 scala maven rdd

本文链接：https://blog.csdn.net/oopsoom/article/details/23968361

版权

spark 专栏收录该内容

41 篇文章 7 订阅

订阅专栏

最近新的版本发布了，听说修复了很多bug，专业人士说直接可以弃用0.81版本，直接0.91。

那就试试把。

第一步，编译，选用maven编译

1.ubuntu12.04

2.spark0.91

3.scala2.10.3

victor@victor-ubuntu:~/software/spark$ export MAVEN_OPTS="-Xmx1g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

victor@victor-ubuntu:~/software/spark$ mvn -Pyarn -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 -DskipTests clean package

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM .......................... SUCCESS [5.697s]
[INFO] Spark Project Core ................................ SUCCESS [5:11.903s]
[INFO] Spark Project Bagel ............................... SUCCESS [40.452s]
[INFO] Spark Project GraphX .............................. SUCCESS [2:11.054s]
[INFO] Spark Project ML Library .......................... SUCCESS [1:47.131s]
[INFO] Spark Project Streaming ........................... SUCCESS [2:37.548s]
[INFO] Spark Project Tools ............................... SUCCESS [30.485s]
[INFO] Spark Project REPL ................................ SUCCESS [1:40.824s]
[INFO] Spark Project YARN Parent POM ..................... SUCCESS [27.180s]
[INFO] Spark Project YARN Stable API ..................... SUCCESS [55.995s]
[INFO] Spark Project Assembly ............................ SUCCESS [1:04.813s]
[INFO] Spark Project External Twitter .................... SUCCESS [28.858s]
[INFO] Spark Project External Kafka ...................... SUCCESS [30.422s]
[INFO] Spark Project External Flume ...................... SUCCESS [28.940s]
[INFO] Spark Project External ZeroMQ ..................... SUCCESS [33.690s]
[INFO] Spark Project External MQTT ....................... SUCCESS [21.725s]
[INFO] Spark Project Examples ............................ SUCCESS [2:32.963s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22:10.599s
[INFO] Finished at: Thu Apr 17 22:46:20 CST 2014
[INFO] Final Memory: 80M/518M
[INFO] ------------------------------------------------------------------------

第二步,导入intellij idea

1. Import project选择文件路径

2.Import project from external model ->> choose maven

3. FInished