主要内容
- Ubuntu 10.04 系统设置
- ZooKeeper集群搭建
- Hadoop-2.4.1集群搭建
- Spark 1.4.0集群搭建
假设已经安装好Ubuntu操作系统
Ubuntu 10.04设置
1.主机规划
主机名 | IP地址 | 进程号 |
---|---|---|
SparkMaster | 192.168.1.103 | ResourceManager DataNode、NodeManager、JournalNode、QuorumPeerMain |
SparkSlave01 | 192.168.1.101 | ResourceManager DataNode、NodeManager、JournalNode、QuorumPeerMain NameNode、DFSZKFailoverController(zkfc) |
SparkSlave02 | 192.168.1.102 | DataNode、NodeManager、JournalNode、QuorumPeerMain NameNode、DFSZKFailoverController(zkfc) |
**说明:
1.在hadoop2.0中通常由两个NameNode组成,一个处于active状态,另一个处于standby状态。Active NameNode对外提供服务,而Standby NameNode则不对外提供服务,仅同步active namenode的状态,以便能够在它失败时快速进行切换。
hadoop2.0官方提供了两种HDFS HA的解决方案,一种是NFS,另一种是QJM。这里我们使用简单的QJM。在该方案中,主备NameNode之间通过一组JournalNode同步元数据信息,一条数据只要成功写入多数JournalNode即认为写入成功。通常配置奇数个JournalNode
这里还配置了一个zookeeper集群,用于ZKFC(DFSZKFailoverController)故障转移,当Active NameNode挂掉了,会自动切换Standby NameNode为standby状态
2.hadoop-2.2.0中依然存在一个问题,就是ResourceManager只有一个,存在单点故障,hadoop-2.4.1解决了这个问题,有两个ResourceManager,一个是Active,一个是Standby,状态由zookeeper进行协调**
2. 修改主机名称设置
利用vi /etc/hostname修改主机名称
3. 修改主机IP地址
利用vi /etc/network/interfaces修改主要IP
主机 | /etc/network/interfaces文件内容 |
---|---|
SparkMaster | auto loiface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.103 netmask 255.255.255.0 gateway 192.168.1.1 |
SparkSlave01 | auto loiface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.101 netmask 255.255.255.0 gateway 192.168.1.1 |
SparkSlave02 | auto loiface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.102 netmask 255.255.255.0 gateway 192.168.1.1 |
4. 修改域名解析服务器
由于需要联网安装OpenSSH等实现名密码登录,因此这边需要配置对应的域名解析服务器
主机 | /etc/resolv.conf文件内容 |
---|---|
SparkMaster | domain localdomain search localdomain nameserver 8.8.8.8 |
SparkSlave01 | domain localdomain search localdomain nameserver 8.8.8.8 |
SparkSlave02 | domain localdomain search localdomain nameserver 8.8.8.8 |
5.修改主机名与IP地址映射
主机 | /etc/resolv.conf文件内容 |
---|---|
SparkMaster | 127.0.0.1 SparkMaster localhost.localdomain localhost 192.168.1.101 SparkSlave01 192.168.1.102 SparkSlave02 192.168.1.103 SparkMaster ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts |
SparkSlave01 | 127.0.0.1 SparkSlave01 localhost.localdomain localhost 192.168.1.101 SparkSlave01 192.168.1.102 SparkSlave02 192.168.1.103 SparkMaster ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts |
SparkSlave02 | 127.0.0.1 SparkSlave02 localhost.localdomain localhost 192.168.1.101 SparkSlave01 192.168.1.102 SparkSlave02 192.168.1.103 SparkMaster ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts |
完成上述步骤后重新启动机器
6.安装SSH (三台主机执行相同命令)
sudo apt-get install openssh-server
然后确认sshserver是否启动了:
ps -e |grep ssh
7.设置无密码登录 (三台主机执行相同命令)
执行命令:ssh-keygen -t rsa
执行完这个命令后,会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥)
将公钥拷贝到要免登陆的机器上
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
或
ssh-copy-id -i SparkMaster
ssh-copy-id -i SparkSlave02
ssh-copy-id -i SparkSlave01
ZooKeeper集群搭建
本集群用的ZooKeeper版本是3.4.5,将/hadoopLearning/zookeeper-3.4.5/conf目录下的zoo_sample.cfg文件名重命名为zoo.cfg
vi conf/zoo.cfg,在文件中填入以下内容:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# ZK文件存放目录
dataDir=/hadoopLearning/zookeeper-3.4.5/zookeeper_data
# the port at which the clients will connect
clientPort=21