Hadoop集群安装day03

Hadoop集群安装

day03

Hadoop分布式环境搭建

上传压缩包并解压【上传的压缩包需要重新编译后支持snappy压缩的hadoop包】

cd /export/softwares/
tar -zxvf hadopp-2.6.0-cdh5.14.0.tar.gz -C ../servers/

三台机器在线安装openssl-devel

yum -y install openssl-devel

修改core-site.xml配置文件

cd /export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop
vim core-site.xml

** 需要修改**
<configuration>
<property>
	<name>fs.defaultFS</name>
	<value>hdfs://node01:8020</value>
</property>
<property>
	<name>hadoop.tmp.dir</name>
	<value>/export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/tempDatas</value>
</property>
<!--  缓冲区大小，实际工作中根据服务器性能动态调整 -->
<property>
	<name>io.file.buffer.size</name>
	<value>4096</value>
</property>

<!--  开启hdfs的垃圾桶机制，删除掉的数据可以从垃圾桶中回收，单位分钟 -->
<property>
	<name>fs.trash.interval</name>
	<value>10080</value>
</property>
</configuration>

修改hdfs-site.xml配置文件

cd /export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop
vim hdfs-site.xml

**需要修改**
<configuration>
 <property>
		<name>dfs.namenode.secondary.http-address</name>
		<value>node01:50090</value>
</property>

<property>
	<name>dfs.namenode.http-address</name>
	<value>node01:50070</value>
</property>
<property>
	<name>dfs.namenode.name.dir</name>
	<value>file:///export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/namenodeDatas</value>
</property>

<property>
	<name>dfs.datanode.data.dir</name>
	<value>file:///export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/datanodeDatas</value>
</property>

<property>
	<name>dfs.namenode.edits.dir</name>
	<value>file:///export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/dfs/nn/edits</value>
</property>
<property>
	<name>dfs.namenode.checkpoint.dir</name>
	<value>file:///export/servers/hadoop-2.6.0-					cdh5.14.0/hadoopDatas/dfs/snn/name</value>
</property>
<property>
	<name>dfs.namenode.checkpoint.edits.dir</name>
	<value>file:///export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/dfs/nn/snn/edits</value>
</property>
<property>
	<name>dfs.replication</name>
	<value>2</value>
</property>
<property>
	<name>dfs.permissions</name>
	<value>false</value>
</property>
<property>
	<name>dfs.blocksize</name>
	<value>134217728</value>
</property>
</configuration>

修改hadoop-env.sh

cd /export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop
vim hadoop-env.sh


**需要修改**
export JAVA_HOME=/export/servers/jdk1.8.0_141

修改mapred-site.xml

cd /export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop
vim mapred-site.xml

**需要修改**
<configuration>
<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
</property>

<property>
	<name>mapreduce.job.ubertask.enable</name>
	<value>true</value>
</property>

<property>
	<name>mapreduce.jobhistory.address</name>
	<value>node01:10020</value>
</property>

<property>
	<name>mapreduce.jobhistory.webapp.address</name>
	<value>node01:19888</value>
</property>
</configuration>

修改yarn-site.xml

cd /export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop
vim yarn-site.xml

**需要修改**
<configuration>
<property>
	<name>yarn.resourcemanager.hostname</name>
	<value>node01</value>
</property>
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>

<property>
	<name>yarn.log-aggregation-enable</name>
	<value>true</value>
</property>
<property>
	<name>yarn.log-aggregation.retain-seconds</name>
	<value>604800</value>
</property>
</configuration>

修改slaves文件
node01执行：

cd /export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop
vi slaves

**需要修改**
node01
node02
node03

创建文件存放目录

node01执行：

mkdir -p /export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/tempDatas
mkdir -p /export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/namenodeDatas
mkdir -p /export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/datanodeDatas 
mkdir -p /export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/dfs/nn/edits
mkdir -p /export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/dfs/snn/name
mkdir -p /export/servers/hadoop-2.6.0-cdh5.14.0/hadoopDatas/dfs/nn/snn/edits

分发安装包

node01执行：

cd /export/servers/
scp -r hadoop-2.6.0-cdh5.14.0/ node02:$PWD
scp -r hadoop-2.6.0-cdh5.14.0/ node03:$PWD

配置hadoop环境

三台机器执行：

vi /etc/profile

**需要修改**
export  HADOOP_HOME=/export/servers/hadoop-2.6.0-cdh5.14.0
export  PATH=:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

配置完成后

source /etc/profile

集群！启动！

首次启动需要格式化一下：

bin/hdfs namenode -format
or
bin/hadoop namenode -format

节点启动方式：

在主节点上使用以下命令启动 HDFS NameNode： 
hadoop-daemon.sh start namenode 
在每个从节点上使用以下命令启动 HDFS DataNode： 
hadoop-daemon.sh start datanode 
在主节点上使用以下命令启动 YARN ResourceManager： 
yarn-daemon.sh  start resourcemanager 
在每个从节点上使用以下命令启动 YARN nodemanager： 
yarn-daemon.sh start nodemanager