Centos环境配置
- 修改主机名和IP(shiyuan01:192.168.9.8, shiyuan02:192.168.9.9,shiyuan03:192.168.9.10)
- 关闭防火墙
- ssh免登录(ssh-keygen -t rsa, ssh-copy-id hostname)
- 安装JDK,配置环境变量等
Zookeeper集群搭建(先在一台节点上配置,然后copy到其他节点)
- 解压zookeeper-3.4.9.tar.gz到/usr/zookeeper:
tar -zxvf zookeeper-3.4.9.tar.gz -C /usr/zookeeper
- 进入/usr/zookeeper/zookeeper-3.4.9/conf添加一个zoo.cfg配置文件:
mv zoo_sample.cfg zoo.cfg
修改配置文件zoo.cfg
#修改 dataDir=/usr/zookeeper/zookeeper-3.4.9/data # 添加 server.1=shiyuan01:2888:3888 server.2=shiyuan02:2888:3888 server.3=shiyuan03:2888:3888
- 在(dataDir=/usr/zookeeper/zookeeper-3.4.9/data)创建一个myid文件,里面内容是server.N中的N(server.1里面内容为1) echo “1” > myid
- 将配置好的zookeeper拷贝到其他节点(shiyuan02,shiyuan03):
scp -r /usr/zookeeper/zookeeper-3.4.9/ shiyuan02:/usr/zookeeper/
scp -r /usr/zookeeper/zookeeper-3.4.9/ shiyuan03:/usr/zookeeper/
- 在其他节点上修改myid的内容,shiyuan02的改为2,shiyuan03的改为3
- 启动集群:
/usr/zookeeper/zookeeper-3.4.9/bin/zkServer.sh start
安装配置hadoop集群(先在一台节点上配置,然后copy到其他节点)
- 解压:
tar -zxvf hadoop-2.7.3.tar.gz -C /usr/hadoop/
- 添加到环境变量:vim /etc/profile,
export HADOOP_HOME=/usr/hadoop/hadoop-2.7.3
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
让配置文件生效:source /etc/profile - 修改配置:vim /usr/hadoop/hadoop-2.7.3/etc/hadoop/hadoop-env.sh,配置JAVA_HOME:export JAVA_HOME=/usr/java/jdk1.7.0_79
- 修改配置:vim /usr/hadoop/hadoop-2.7.3/etc/hadoop/core-site.xml:
<configuration>
<!-- 指定hdfs的nameservice为ns1 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<!-- 指定hadoop临时目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop/hadoop-2.7.3/tmp</value>
</property>
<!-- 指定zookeeper地址 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>shiyuan01:2181,shiyuan02:2181,shiyuan03:2181</value>
</property>
</configuration>
- 修改配置:vim /usr/hadoop/hadoop-2.7.3/etc/hadoop/hdfs-site.xml:
<configuration>
<!-- 指定hdfs的nameservice为ns1,需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!-- ns1下面有两个NameNode,分别是nn1,nn2 -->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>shiyuan01:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>shiyuan01:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>shiyuan02:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>shiyuan02:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://shiyuan01:8485;shiyuan02:8485;shiyuan03:8485/ns1</value>
</property>
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/hadoop/hadoop-2.7.3/journal</value>
</property>
<!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制方法,多个机制用换行分割,即每个机制占用一行 -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用sshfence隔离机制时需要ssh免登录 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔离机制超时时间 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
- 修改配置:vim /usr/hadoop/hadoop-2.7.3/etc/hadoop/mapred-site.xml:
<configuration>
<!-- 指定mr框架为yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
- 修改配置:vim /usr/hadoop/hadoop-2.7.3/etc/hadoop/yarn-site.xml:
<configuration>
<!-- 开启RM高可靠 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>shiyuan01</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>shiyuan02</value>
</property>
<!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>shiyuan01:2181,shiyuan02:2181,shiyuan03:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
- 修改配置:vim /usr/hadoop/hadoop-2.7.3/etc/hadoop/slaves,指定子节点(DataNode, NodeManager)的位置:
shiyuan01
shiyuan02
shiyuan03 将配置好的hadoop拷贝到其他节点
scp -r /usr/hadoop/ shiyuan02:/usr/
scp -r /usr/hadoop/ shiyuan03:/usr/
启动zookeeper集群(分别在shiyuan01, shiyuan02, shiyuan03上):
/usr/zookeeper/zookeeper-3.4.9/bin/zkServer.sh start
- 启动journalnode(分别在shiyuan01, shiyuan02, shiyuan03上):
/usr/hadoop/hadoop-2.7.3/sbin/hadoop-daemon.sh start journalnode
- 格式化HDFS(在shiyuan01上):
hdfs namenode -format
格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成文件,然后将其拷贝到shiyuan02相应目录下:
scp -r /usr/hadoop/hadoop-2.7.3/tmp/ shiyuan02:/usr/hadoop/hadoop-2.7.3/
- 格式化ZK(在shiyuan01上):
hdfs zkfc -formatZK
- 启动HDFS(在shiyuan01上):
/usr/hadoop/hadoop-2.7.3/sbin/start-dfs.sh
- 启动YARN(分别在shiyuan01, shiyuan02上,最好是将NameNode和ResourceManager分开放在不同的机器上,因为它们都要占用大量资源)
/usr/hadoop/hadoop-2.7.3/sbin/start-yarn.sh