配置节点
创建三个节点
Hostname分别更改为node-01,node-02,node-03
勾选Connect automatically
设置密码111111
时区选上海,安装完之后reboot
以node-01为例(三个都需配置)
vi /etc/hosts
加入三个节点的主机名和IP地址
vi /etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO改成static
后面加入四行内容
IP地址
网关
子网掩码
DNS
reboot
用户名root
密码111111
ping www.baidu.com
说明配置成功
换国内源
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
vi /etc/yum.repos.d/CentOS-Base.repo
复制以下内容
[base]
name=CentOS-6.10 - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos-vault/6.10/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos-vault/RPM-GPG-KEY-CentOS-6
#released updates
[updates]
name=CentOS-6.10 - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos-vault/6.10/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos-vault/RPM-GPG-KEY-CentOS-6
#additional packages that may be useful
[extras]
name=CentOS-6.10 - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos-vault/6.10/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos-vault/RPM-GPG-KEY-CentOS-6
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-6.10 - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos-vault/6.10/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos-vault/RPM-GPG-KEY-CentOS-6
#contrib - packages by Centos Users
[contrib]
name=CentOS-6.10 - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos-vault/6.10/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos-vault/RPM-GPG-KEY-CentOS-6
保存退出
yum clean all && yum makecache
yum repolist
换源成功
ssh免密登录
ssh-keygen -t rsa
连续Enter四次
密码111111
ssh-copy-id node-01
ssh-copy-id node-02
ssh-copy-id node-03
配置文件
创建文件夹
mkdir -p /export/data
mkdir -p /export/servers
mkdir -p /export/software
安装rz命令
cd /export/software
yum -y install lrzsz
rz
上床hadoop,jdk,zookeeper三个包,上传完之后解压
tar -zxvf hadoop-2.7.4.tar.gz -C /export/servers/
tar -zxvf zookeeper-3.4.10.tar.gz -C /export/servers/
tar -zxvf jdk-8u161-linux-x64.tar.gz -C /export/servers/
将jdk改名
cd /export/servers
mv jdk1.8.0_161 jdk
配置环境变量
vi /etc/profile
最后添加以下内容
export JAVA_HOME=/export/servers/jdk
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/export/servers/hadoop-2.7.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZK_HOME/bin
export ZK_HOME=/export/servers/zookeeper-3.4.10
保存退出
修改zoo.cfg文件
cd /export/servers/zookeeper-3.4.10/conf/
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
替换
dataDir=/export/data/zookeeper/zkdata
添加
server.1=node-01:2888:3888
server.2=node-02:2888:3888
server.3=node-03:2888:3888
保存退出
创建myid文件
mkdir -p /export/data/zookeeper/zkdata
cd /export/data/zookeeper/zkdata
echo 1 > myid
cat myid
使环境变量生效
source /etc/profile
查看java和hadoop版本
java -version
hadoop version
说明环境配置成功
内容配置
进入hadoop目录,修改文件内容
cd /export/servers/hadoop-2.7.4/etc/hadoop/
vi hadoop-env.sh
找到相应位置,替换
export JAVA_HOME=/export/servers/jdk
保存退出
修改core-site.xml文件
vi core-site.xml
添加内容
<configuration>
<!-- 指定hdfs的nameservice为ns1 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<!--指定hadoop数据临时存放目录-->
<property>
<name>hadoop.tmp.dir</name>
<value>/export/servers/hadoop-2.7.4/tmp</value>
</property>
<!--指定zookeeper地址-->
<property>
<name>ha.zookeeper.quorum</name>
<value>node-01:2181,node-02:2181,node-03:2181</value>
</property>
</configuration>
保存退出
修改hdfs-site.xml
vi hdfs-site.xml
<configuration>
<!-- 设置副本个数 -->
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<!-- 设置namenode.name目录 -->
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/export/data/hadoop/name</value>
</property>
<!-- 设置datanode.data目录 -->
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/export/data/hadoop/data</value>
</property>
<!-- 开启WebHDFS -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
<!-- 在NN和DN上开启WebHDFS (REST API)功能,不是必须 -->
</property>
<!-- 指定hdfs的nameservice为ns1,需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!-- ns1下面有两个NameNode,分别是nn1,nn2 -->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>node-01:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>node-01:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>node-02:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>node-02:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node-01:8485;node-02:8485;node-03:8485/ns1</value>
</property>
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/export/data/hadoop/journaldata</value>
</property>
<!-- 开启NameNode故障时自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行 -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔离机制超时时间 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>300000</value>
</property>
</configuration>
保存退出
修改mapred-site.xml
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
添加
<configuration>
<!--指定MapReduce框架为YARN方式-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
退出
修改yarn-site.xml
vi yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduluer.maximum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
<!--启用resourcemanager HA-->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property>
<!--指定resourcemanager的名称-->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!--配置rm1的主机-->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>node-01</value>
</property>
<!--配置rm2的主机-->
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>node-02</value>
</property>
<!--zookeeper集群地址-->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>node-01:2181,node-02:2181,node-03:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
退出
修改slaves
vi slaves
添加
node-01
node-02
node-03
文件分发
scp -r /export/servers/zookeeper-3.4.10 node-02:/export/servers
scp -r /export/servers/zookeeper-3.4.10 node-03:/export/servers
scp -r /export/data/zookeeper/ node-02:/export/data/
scp -r /export/data/zookeeper/ node-03:/export/data/
将profile文件分发至node-02和node-03服务器上
scp /etc/profile node-02:/etc/profile
scp /etc/profile node-03:/etc/profile
scp -r /export/ node-02:/
scp -r /export/ node-03:/
分别在node-02,node-03的zkdata目录下新建名称为myid,内容分别为2,3的文件
在node-02操作
cd /export/data/zookeeper/zkdata
echo 2 > myid
在node-03操作
cd /export/data/zookeeper/zkdata
echo 3 > myid
使环境变量生效,三个节点都执行
source /etc/profile
关闭防火墙 ,三个节点都执行
service iptables stop
chkconfig iptables off
打开window下的C:\Windows\System32\drivers\etc打开hosts文件,在文件末添加以下内容
192.168.121.131 node-01
192.168.121.132 node-02
192.168.121.133 node-03
##启动zookeeper服务
cd /export/servers/zookeeper-3.4.10/bin
./zkServer.sh start
./zkServer.sh status
启动集群各个节点监控namenode的管理日志的journalnode,(1,2,3都执行)
hadoop-daemon.sh start journalnode
格式化namenode,并将目录复制到2中(1执行)
hadoop namenode -format
scp -r /export/data/hadoop node-02:/export/data/
格式化ZKFC
hdfs zkfc -formatZK
start-dfs.sh
start-yarn.sh
集群启动成功