hadoop集群部署与安装(完全分布式)。
一:所需环境
CentOs-6.5
Centos链接:https://pan.baidu.com/s/1SKTYzryyjPBBti-r3TkPng
提取码:efuv
jdk-8u151-linux-x64.tar.gz
jdk链接:https://pan.baidu.com/s/1WPpbyHbSxaAarsIXTA3yLQ
提取码:mrks
hadoop-2.8.5.tar.gz 安装包。
hadoop链接:https://pan.baidu.com/s/1dPSqezvsSzC-s8Y9dw8xJg
提取码:i9nz
二:实验步骤。
1:建立hadoop集群,对集群进行安装jdk,本教程为root用户。(我的是3台)
先进行root:
[hadoop@localhost ~]$ su root
创建一个用于存放jdk的目录:
[root@localhost Desktop] # mkdir /usr/java
解压安装jdk:
[root@localhost Desktop] # tar -zxvf jdk-8u151-linux-x64.tar.gz
解压以后移动到刚刚你创建的java目录下:
[root@localhost Desktop] # mv jdk-8u151 /usr/java
配置环境变量:
[root@localhost Desktop] # vim /etc/profile
将一下内容添加到文件末尾:
export JAVA_HOME=/usr/java/jdk1.8.0_151
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin
export PATH=$PATH:${JAVA_PATH}
刷新环境变量(使配置环境变量生效):
[root@localhost Desktop] # source /etc/profile
测试是否正常 出现版本号则正确:
[root@localhost Desktop] # java -version
2:先查看ip,然后进行免密。
对你所需要的每一台进行查看)
[root@localhost Desktop] # ifconfig
对每一个节点进行修改(修改每一个节点中的hosts文件,3台都改,我的节点分别为h201,h202,h203。)
[root@localhost Desktop] # vi /etc/hosts
192.168.16.201 h201
192.168.16.202 h202
192.168.16.203 h203
保存以后建议重启一下。
[root@localhost Desktop] # reboot
安装ssh 证书,先回到用户家目录下;
[root@h201 Desktop] # cd ~
第一台主机(持续按回车4次)
[root@h201 ~]# ssh-keygen -t rsa
第二台主机(持续按回车4次)
[root@h202 ~]# ssh-keygen -t rsa
第三台主机(持续按回车4次)。
[root@h203 ~]# ssh-keygen -t rsa
将免密复制到其他节点(提示输入密码时,要输入密码)
第一台
[root@h201 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h201
[root@h201 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h202
[root@h201 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h203
第二台
[root@h202 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h201
[root@h202 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h202
[root@h202 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h203
第三台
[root@h203 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h201
[root@h203 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h202
[root@h203 ~]# ssh-copy-id -i ./.ssh/id_rsa.pub h203
可以使用ssh h202
登录到其他节点验证一下。
3:解压安装hadoop
[root@h201 ~]# tar -zxvf hadoop-2.8.5.tar.gz
将解压文件移到/usr/local下
[root@h201 ~]# mv hadoop-2.8.5 /usr/local
配置环境变量
[root@h201 ~]# vi /etc/profile
export HADOOP_HOME=/usr/local/hadoop-2.8.5
export PATH=${PATH}:${HADOOP_HOME}/bin
刷新环境变量(使配置环境变量生效):
[root@h201 ~]# source /etc/profile
测试是否成功,出现版本号成功
[root@h201 ~]# hadoop version
4:配置hadoop的文件内容需要配置core-site.xml,hdfs-site.xml ,yarn-site.xml,mapred-site.xml ,hadoop-env.sh这5分文件。(不讲解文件内容,直接复制粘贴即可使用)
进入hadoop的安装目录下的etc中hadoop
[root@h201 ~]# cd usr/local/hadoop/etc/hadoop
core-site.xml 文件
[root@h201 hadoop]# vim core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://h201:9000</value>
<description>NameNode URI.</description>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
<description>Size of read/write buffer used inSequenceFiles.</description>
</property>
</configuration>
hdfs-site.xml 文件
[root@h201 hadoop]# vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>h201:50090</value>
<description>The secondary namenode http server address andport.</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop-2.8.5/dfs/name</value>
<description>Path on the local filesystem where the NameNodestores the namespace and transactions logs persistently.</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/hadoop-2.8.5/dfs/data</value>
<description>Comma separated list of paths on the local filesystemof a DataNode where it should store its blocks.</description>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///usr/local/hadoop-2.8.5/dfs/namesecondary</value>
<description>Determines where on the local filesystem the DFSsecondary name node should store the temporary images to merge. If this is acomma-delimited list of directories then the image is replicated in all of thedirectories for redundancy.</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
yarn-site.xml 文件
[root@h201 hadoop]# vim yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>h201</value>
<description>The hostname of theRM.</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Shuffle service that needs to be set for Map Reduceapplications.</description>
</property>
</configuration>
mapred-site.xml 文件。(没有mapred-site.xml这份文件需要复制 mapred-site.xml.template为mapred-site.xml)
[root@h201 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[root@h201 hadoop]$ vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Theruntime framework for executing MapReduce jobs. Can be one of local, classic or yarn.</description>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>h201:10020</value>
<description>MapReduce JobHistoryServer IPC host:port</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>h201:19888</value>
<description>MapReduce JobHistoryServer Web UI host:port</description>
</property>
</configuration>
hadoop-env.sh 文件(找到JAVA_HOME配置java路径,一般在25~28行)
export JAVA_HOME=/usr/java/jdk1.8.0_151
做映射添加从节点h202,h203
[root@h201 hadoop] vim slaves
h202
h203
hadoop初始化
[root@h201 hadoop]# hadoop namenode -format
如果倒数第三行出现0初始化成功。
19/04/18 21:35:34 INFO util.ExitUtil: Exiting with status 0 #末尾这个0
将hadoop复制到其他节点。
[root@h201 hadoop]# scp -r /usr/local/hadoop-2.8.5/ @h202:/usr/local
[root@h201 hadoop]# scp -r /usr/local/hadoop-2.8.5/ @h203:/usr/local
回到sbin目录下启动hadoop
[root@h201 hadoop]# cd /usr/local/hadoop-2.8.5/sbin
[root@h201 sbin]# ./start-all.sh
查看服务
[root@h201 sbin]# jps
7054 SecondaryNameNode
7844 Jps
7318 NameNode
7598 ResourceManager