Spark安装试用

Spark作为分布式流处理的新选择，学习一下。

1. 下载Spark

http://spark.apache.org/downloads.html

2. 解压

    解压到某个目录。比如spark

iMac:spark-1.3.0 bruce$
iMac:spark-1.3.0 bruce$ ls
CHANGES.txt		dev			project
CONTRIBUTING.md		docker			python
LICENSE			docs			repl
NOTICE			ec2			sbin
README.md		examples		sbt
assembly		external		scalastyle-config.xml
bagel			extras			sql
bin			graphx			streaming
build			make-distribution.sh	tools
conf			mllib			tox.ini
core			network			yarn
data			pom.xml

3. 编译spark

http://spark.apache.org/docs/latest/building-spark.html

iMac:spark-1.3.0 bruce$ ./build/mvn -DskipTests clean package

exec: curl --progress-bar -L http://archive.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
#####################################                                     51.4%
######################################################################## 100.0%
exec: curl --progress-bar -L http://downloads.typesafe.com/zinc/0.3.5.3/zinc-0.3.5.3.tgz
###                                                                        4.2%
#####################                                                     29.6%
######################################                                    53.2%
#######################################################                   77.1%
######################################################################    98.3%
######################################################################## 100.0%
exec: curl --progress-bar -L http://downloads.typesafe.com/scala/2.10.4/scala-2.10.4.tgz
###                                                                        4.2%
#######                                                                   10.6%
###########                                                               15.5%
################                                                          23.5%
###################                                                       27.4%
#######################                                                   32.6%
##########################                                                36.4%
##########################                                                37.0%
############################                                              39.2%
###############################                                           43.5%
##################################                                        47.9%
####################################                                      50.9%
#######################################                                   55.1%
#########################################                                 58.0%
###########################################                               61.0%
###############################################                           66.1%
###################################################                       71.2%
#######################################################                   77.3%
#############################################################             84.9%
################################################################          89.9%
###################################################################       94.4%
######################################################################    97.5%
######################################################################## 100.0%
[INFO] Scanning for projects...
Downloading: https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom
Downloaded: https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom (15 KB at 2.1 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM
[INFO] Spark Project Networking
[INFO] Spark Project Shuffle Streaming Service
[INFO] Spark Project Core
[INFO] Spark Project Bagel
[INFO] Spark Project GraphX
[INFO] Spark Project Streaming
[INFO] Spark Project Catalyst
[INFO] Spark Project SQL
[INFO] Spark Project ML Library
[INFO] Spark Project Tools
[INFO] Spark Project Hive
[INFO] Spark Project REPL
[INFO] Spark Project Assembly
[INFO] Spark Project External Twitter
[INFO] Spark Project External Flume Sink
[INFO] Spark Project External Flume
[INFO] Spark Project External MQTT
[INFO] Spark Project External ZeroMQ
[INFO] Spark Project External Kafka
[INFO] Spark Project Examples
[INFO] Spark Project External Kafka Assembly
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Parent POM 1.3.0
[INFO] ------------------------------------------------------------------------

通过maven，下载相关的maven自身，spark，scala等等组件。

最后看到：

Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 0 ms
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 14.317 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 13.390 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  3.719 s]
[INFO] Spark Project Core ................................. SUCCESS [ 34.729 s]
[INFO] Spark Project Bagel ................................ SUCCESS [  2.022 s]
[INFO] Spark Project GraphX ............................... SUCCESS [  4.443 s]
[INFO] Spark Project Streaming ............................ SUCCESS [  8.416 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [  8.455 s]
[INFO] Spark Project SQL .................................. SUCCESS [  9.504 s]
[INFO] Spark Project ML Library ........................... SUCCESS [ 11.437 s]
[INFO] Spark Project Tools ................................ SUCCESS [  1.242 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 30.343 s]
[INFO] Spark Project REPL ................................. SUCCESS [  1.976 s]
[INFO] Spark Project Assembly ............................. SUCCESS [01:08 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [  4.485 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [  3.560 s]
[INFO] Spark Project External Flume ....................... SUCCESS [  2.919 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [  5.343 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [  3.758 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [  3.571 s]
[INFO] Spark Project Examples ............................. SUCCESS [01:17 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 14.539 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:29 min
[INFO] Finished at: 2015-04-07T15:40:57+08:00
[INFO] Final Memory: 50M/167M
[INFO] ------------------------------------------------------------------------

大功告成！Spark编译完成。