（1）JDK 1.8 安装
在根目录下创建sparkLearning目前，后续所有相关软件都放置在该目录下，代码如下：
[root@slave01 /]# mkdir /sparkLearning
[root@slave01 /]# ls
bin etc lib media proc selinux sys var
boot hadoopLearning lib64 mnt root sparkLearning tmp
dev home lost+found opt sbin srv usr
将共享目录中的jdk安装包复制到/sparkLearning目录
[root@slave01 share]# cp /mnt/hgfs/share/jdk-8u40-linux-x64.gz /sparkLearning/
[root@slave01 share]# cd /sparkLearning/
//解压
[root@slave01 sparkLearning]# tar -zxvf jdk-8u40-linux-x64.gz
设置环境变量：
[root@slave01 sparkLearning]# vim /etc/profile在文件最后添加：
export JAVA_HOME=/sparkLearning/jdk1.8.0_40
export PATH=${JAVA_HOME}/bin:$PATH测试配置是否成功：
[root@slave01 sparkLearning]# source /etc/profile
[root@slave01 sparkLearning]# $SCALA_HOME
bash: /sparkLearning/scala-2.10.4: is a directory
[root@slave01 sparkLearning]# scala -version
Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL（2）Scala 2.10.4 安装
//复制文件到sparkLearning目录下
[root@slave01 sparkLearning]# cp /mnt/hgfs/share/scala-2.10.4.tgz .
//解压
[root@slave01 sparkLearning]# tar -zxvf scala-2.10.4.tgz > /dev/null
[root@slave01 sparkLearning]# vim /etc/profile将/etc/profile文件末尾内容修改如下：
export JAVA_HOME=/sparkLearning/jdk1.8.0_40
export SCALA_HOME=/sparkLearning/scala-2.10.4
export PATH=${JAVA_HOME}/bin:${SCALA_HOME}/bin:$PATH测试Scala是否安装成功
//使修改后的配置生效
[root@slave01 sparkLearning]# source /etc/profile
//环境变量是否已经设置
[root@slave01 sparkLearning]# $JAVA_HOME
bash: /sparkLearning/jdk1.8.0_40: is a directory
//测试java是否安装配置成功
[root@slave01 sparkLearning]# java -version
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)（3）Zookeeper-3.4.5 集群搭建
[root@slave01 sparkLearning]# cp /mnt/hgfs/share/zookeeper-3.4.5.tar.gz .
[root@slave01 sparkLearning]# tar -zxvf zookeeper-3.4.5.tar.gz > /dev/null
[root@slave01 sparkLearning]# cp zookeeper-3.4.5/conf/zoo_sample.cfg zoo.cfg
[root@slave01 sparkLearning]# vim zoo.cfg
修改dataDir为：
dataDir=/sparkLearning/zookeeper-3.4.5/zookeeper_data
在文件末尾添加如下内容：
server.1=slave01.example.com:2888:3888
server.2=slave02.example.com:2888:3888
server.3=slave03.example.com:2888:3888创建ZooKeeper集群数据保存目录
[root@slave01 sparkLearning]# cd zookeeper-3.4.5/
[root@slave01 zookeeper-3.4.5]# mkdir zookeeper_data
[root@slave01 zookeeper-3.4.5]# cd zookeeper_data/
[root@slave01 zookeeper_data]# touch myid
[root@slave01 zookeeper_data]# echo 1 > myid 如此便完成配置，下面对集群进行测试：
//在slave03.example.com主机上
[root@slave03 ~]# cd /sparkLearning/zookeeper-3.4.5/bin
//启动slave03.example.com上的ZooKeeper
[root@slave03 bin]# ./zkServer.sh start
JMX enabled by default
Using config: /sparkLearning/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@slave03 bin]# ./zkServer.sh status
JMX enabled by default
Using config: /sparkLearning/zookeeper-3.4.5/bin/../conf/zoo.cfg
Mode: leader
//在slave02.example.com主机上
[root@slave02 bin]# ./zkServer.sh start
JMX enabled by default
Using config: /sparkLearning/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
//查看zookeeper集群状态，如果Mode显示为follower或leader则表明配置成功
[root@slave02 bin]# ./zkServer.sh status
JMX enabled by default
Using config: /sparkLearning/zookeeper-3.4.5/bin/../conf/zoo.cfg
Mode: follower
//在slave01.example.com主机上
[root@slave01 bin]# ./zkServer.sh start
JMX enabled by default
Using config: /sparkLearning/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@slave01 bin]# ./zkServer.sh status
JMX enabled by default
Using config: /sparkLearning/zookeeper-3.4.5/bin/../conf/zoo.cfg
Mode: follower如果出现Error contacting service. It is probably not running.错误，一次检测：
1）zookeeper是集群的概念，务必保证2台或2台以上的机器正常启动，才有主从的概念（leader或者是follower）。
另外jps这个命令是"可以列出本机所有Java进程的pid",因为zookeeper是java程序，所以通过jps命令来查看。
正常启动的话会显示：
[root@server02 bin]# jps
3076 Jps
2979 QuorumPeerMain
2）如果确定已经至少是2台机器成功启动，仍然出现这种结果，则按照以下的方式可以解决。
a)关闭防火墙：service iptables stop
#查看防火墙状态
service iptables status
#关闭防火墙
service iptables stop
（3）将Hadoop 2.4.1添加到环境变量
使用命令：vim hadoop-env.sh 将环境变量信息修改如下，在export JAVA_HOME修改为：
export JAVA_HOME=/sparkLearning/jdk1.8.0_40
（4）修改core-site.xml文件
利用vim core-site.xml命令，文件内容如下：
<configuration>
<!-- 指定hdfs的nameservice为ns1 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<!-- 指定hadoop临时目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/sparkLearning/hadoop-2.4.1/tmp</value>
</property>
<!-- 指定zookeeper地址 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>slave01.example.com:2181,slave02.example.com:2181,slave03.example.com:2181</value>
</property>
</configuration>（5）修改hdfs-site.xml文件
vim hdfs-site.xml内容如下：
<configuration>
<!--指定hdfs的nameservice为ns1，需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!-- ns1下面有两个NameNode，分别是nn1，nn2 -->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>slave01.example.com:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>slave01.example.com:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>slave02.example.com:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>slave02.example.com:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://slave01.example.com:8485;slave02.example.com:8485;slave03.example.com:8485/ns1</value>
</property>
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/sparkLearning/hadoop-2.4.1/journal</value>
</property>
<!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔离机制超时时间 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>（4）修改mapred-site.xml文件
[root@slave01 hadoop]# cp mapred-site.xml.template mapred-site.xmlvim mapred-site.xml修改文件内容如下:
<configuration>
<!-- 指定mr框架为yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration> （6）修改yarn-site.xml文件
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- 开启RM高可靠 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>SparkCluster</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>slave01.example.com</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>slave02.example.com</value>
</property>
<!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>
</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>（7）修改slaves文件
slave01.example.com
slave02.example.com
slave03.example.com（8）配置文件拷贝到其它服务器
//slave01.example.com上的配置文件拷贝到slave02.example.com
[root@slave01 hadoop]# scp -r /etc/profile slave02.example.com:/etc/profile
profile 100% 2027 2.0KB/s 00:00
[root@slave01 hadoop]# scp -r /sparkLearning/hadoop-2.4.1 slave02.example.com:/sparkLearning/
//slave01.example.com上的配置文件拷贝到slave03.example.com
[root@slave01 hadoop]# scp -r /etc/profile slave03.example.com:/etc/profile
profile 100% 2027 2.0KB/s 00:00
[root@slave01 hadoop]# scp -r /sparkLearning/hadoop-2.4.1 slave03.example.com:/sparkLearning/
（9）启动journalnode
//使用下列命令启动journalnode
[root@slave01 hadoop]# hadoop-daemons.sh start journalnode
slave02.example.com: starting journalnode, logging to /sparkLearning/hadoop-2.4.1/logs/hadoop-root-journalnode-slave02.example.com.out
slave03.example.com: starting journalnode, logging to /sparkLearning/hadoop-2.4.1/logs/hadoop-root-journalnode-slave03.example.com.out
slave01.example.com: starting journalnode, logging to /sparkLearning/hadoop-2.4.1/logs/hadoop-root-journalnode-slave01.example.com.out
//JournalNode进程存在，启动成功
[root@slave01 hadoop]# jps
11261 JournalNode
11295 Jps
[root@slave01 hadoop]# ssh slave02.example.com
Last login: Fri Sep 18 05:33:05 2015 from slave01.example.com
[root@slave02 ~]# jps
6598 JournalNode
6795 Jps
[root@slave02 ~]# ssh slave03.example.com
Last login: Fri Sep 18 05:33:26 2015 from slave02.example.com
[root@slave03 ~]# jps
5876 JournalNode
6047 Jps
[root@slave03 ~]#
（10）格式化HDFS（集群机器都需要执行）
登录slave02.example.com服务器，执行下列命令
[root@slave02 ~]# hdfs namenode -format
//下面是执行结果
15/09/18 06:05:26 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = slave02.example.com/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.4.1
STARTUP_MSG: classpath = /sparkLearning/hadoop-2.4.1/etc/hadoop:/sparkLearning/hadoop-........省略无关信息...............
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common -r 1604318; compiled by 'jenkins' on 2014-06-21T05:43Z
STARTUP_MSG: java = 1.8.0_40
.....................................................省略.....
/sparkLearning/hadoop-2.4.1/tmp/dfs/name has been successfully formatted.
15/09/18 06:05:30 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/09/18 06:05:30 INFO util.ExitUtil: Exiting with status 0
15/09/18 06:05:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at slave02.example.com/127.0.0.1
************************************************************/
（11）格式化HDFS信息复制到slave03.example.com服务器
[root@slave02 ~]# scp -r /sparkLearning/hadoop-2.4.1/tmp/ slave01.example.com:/sparkLearning/hadoop-2.4.1/
fsimage_0000000000000000000.md5 100% 62 0.1KB/s 00:00
seen_txid 100% 2 0.0KB/s 00:00
fsimage_0000000000000000000 100% 350 0.3KB/s 00:00
VERSION 100% 200 0.2KB/s 00:00 （12）格式化ZK(在slave02.example.com上执行即可)
[root@slave02 hadoop]# hdfs zkfc -formatZK
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /sparkLearning/hadoop-2.4.1/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
......省略无关信息...............
//执行成功
15/09/18 06:14:22 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
15/09/18 06:14:22 INFO zookeeper.ZooKeeper: Session: 0x34fe096c3ca0000 closed
15/09/18 06:14:22 INFO zookeeper.ClientCnxn: EventThread shut down
（13）启动HDFS(在slave02.example.com上执行)
[root@slave02 hadoop]# start-dfs.sh
[root@slave02 hadoop]# jps
7714 QuorumPeerMain
6598 JournalNode
8295 DataNode
8202 NameNode
8716 Jps
8574 DFSZKFailoverController
[root@slave02 hadoop]# ssh slave01.example.com
Last login: Thu Aug 27 06:24:16 2015 from slave01.example.com
[root@slave01 ~]# jps
13744 DataNode
13681 NameNode
11862 QuorumPeerMain
14007 Jps
13943 DFSZKFailoverController
13851 JournalNode
[root@slave03 ~]# jps
5876 JournalNode
7652 Jps
7068 DataNode
6764 QuorumPeerMain
（14）启动YARN(在slave01.example.com上执行)
//slave01.example.com
[root@slave01 ~]# start-yarn.sh
...输出省略.....
[root@slave01 ~]# jps
14528 Jps
13744 DataNode
13681 NameNode
14228 NodeManager
11862 QuorumPeerMain
13943 DFSZKFailoverController
14138 ResourceManager
13851 JournalNode
//slave02.example.com
[root@slave02 ~]# jps
11216 Jps
10656 JournalNode
7714 QuorumPeerMain
11010 NodeManager
10427 DataNode
10844 DFSZKFailoverController
10334 NameNode
//slave03.example.com
[root@slave03 ~]# jps
8610 JournalNode
8791 NodeManager
8503 DataNode
9001 Jps
6764 QuorumPeerMain
（15）查看hadoop运行管理界面
打开浏览器，输入http://slave01.example.com:8088/，可以得到hadoop集群管理界面: 
输入http://slave01.example.com:50070 可以得到HDFS管理界面 
至此Hadoop集群配置成功
3. Spark 1.5.0 集群部署
（1）将Spark添加到环境变量
[root@slave01 hadoop]# cp /mnt/hgfs/share/spark-1.5.0-bin-hadoop2.4.tgz /sparkLearning/
[root@slave01 sparkLearning]# tar -zxvf spark-1.5.0-bin-hadoop2.4.tgz > /dev/null
[root@slave01 sparkLearning]# vim /etc/profile将/etc/profile内容修改如下：
export JAVA_HOME=/sparkLearning/jdk1.8.0_40
export SCALA_HOME=/sparkLearning/scala-2.10.4
export HADOOP_HOME=/sparkLearning/hadoop-2.4.1
export SPARK_HOME=/sparkLearning/spark-1.5.0-bin-hadoop2.4
export PATH=${JAVA_HOME}/bin:${SCALA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${SPARK_HOME}/bin:${SPARK_HOME}/sbin:$PATH
（2）将Spark添加到环境变量
[root@slave01 sparkLearning]# cd spark-1.5.0-bin-hadoop2.4/conf
[root@slave01 conf]# ls
docker.properties.template metrics.properties.template spark-env.sh.template
fairscheduler.xml.template slaves.template
log4j.properties.template spark-defaults.conf.template
//复制模板文件
[root@slave01 conf]# cp spark-env.sh.template spark-env.sh
[root@slave01 conf]# vim spark-env.sh在spark-env.sh文件中添加如下内容：
export JAVA_HOME=/sparkLearning/jdk1.8.0_40
export SCALA_HOME=/sparkLearning/scala-2.10.4
export HADOOP_CONF_DIR=/sparkLearning/hadoop-2.4.1/etc/hadoop[root@slave01 conf]# cp slaves.template slaves
[root@slave01 conf]# vim slaves
slaves文件内容如下：
# A Spark Worker will be started on each of the machines listed below.
slave01.example.com
slave02.example.com
slave03.example.com（3）将配置信息复制到其它服务器
[root@slave01 sparkLearning]# scp /etc/profile slave02.example.com:/etc/profile
profile 100% 2123 2.1KB/s 00:00
[root@slave01 sparkLearning]# scp /etc/profile slave03.example.com:/etc/profile
profile 100% 2123 2.1KB/s 00:00
[root@slave01 sparkLearning]# vim /etc/profile
[root@slave01 sparkLearning]# scp -r spark-1.5.0-bin-hadoop2.4 slave02.example.com:/sparkLearning/
...执行过程省略.....
[root@slave01 sparkLearning]# scp -r spark-1.5.0-bin-hadoop2.4 slave03.example.com:/sparkLearning/
...执行过程省略.....
（4）启动Spark集群
因为本人机器上装了Ambari Server，占用了8080端口，而Spark Master默认端是8080，因此将sbin/start-master.sh中的SPARK_MASTER_WEBUI_PORT修改为8888
if [ "$SPARK_MASTER_WEBUI_PORT" = "" ]; then
SPARK_MASTER_WEBUI_PORT=8888
fi[root@slave01 sbin]# ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /sparkLearning/spark-1.5.0-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-slave01.example.com.out
slave03.example.com: starting org.apache.spark.deploy.worker.Worker, logging to /sparkLearning/spark-1.5.0-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave03.example.com.out
slave02.example.com: starting org.apache.spark.deploy.worker.Worker, logging to /sparkLearning/spark-1.5.0-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave02.example.com.out
slave01.example.com: starting org.apache.spark.deploy.worker.Worker, logging to /sparkLearning/spark-1.5.0-bin-hadoop2.4/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave01.example.com.out
[root@slave01 sbin]# jps
13744 DataNode
13681 NameNode
14228 NodeManager
16949 Master
11862 QuorumPeerMain
13943 DFSZKFailoverController
14138 ResourceManager
13851 JournalNode
17179 Jps
17087 Worker
浏览器中输入slave01.example.com:8888 
但是在启动过程中出现了错误，查看日志文件
[root@slave02 logs]# more spark-root-org.apache.spark.deploy.worker.Worker-1-slave02.example.com.out
日志内容中包括下列错误：
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://
sparkMaster@slave01.example.com:7077/), Path(/user/Master)]
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.
scala:65)
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.
scala:63)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExe
cutor.scala:55)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatched
Execute(Future.scala:74)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:1
20)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(F
uture.scala:73)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala
:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scal
a:248)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:266)
at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:533)
at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:569)
.....省略.....................没找到具体原因，在ubuntu 10.04服务器上进行相同的配置，集群搭建却成功了（心中一万头…..），运行界面如下: 
（5）测试Spark集群
采用下列命上传spark-1.5.0-bin-hadoop2.4目录下的README.md文件到相应的根目录。
hadoop dfs -put README.md 如下图： 
进入/spark-1.5.0-bin-hadoop2.4/bin目录，启动./spark-shell，如下图所示： 
执行REDME.md文件的wordcount操作：
scala> val textCount = sc.textFile(“README.md”).filter(line => line.contains(“Spark”)).count()
如下图： 
执行结果如下图： 
至此，Spark 1.5集群搭建成功。
原文地址：https://blog.csdn.net/lovehuangjiaju/article/details/48494737
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
