Hadoop集群环境安装（Ubuntu）

1 前言

        我是准备了三个虚拟机。

       192.168.58.133  hadoop1 （master）

        192.168.58.134  hadoop2

        192.168.58.135  hadoop3

两个前提：

①首先保证三个虚拟机上的jdk环境都已安装，此环节不在此赘述，可自行百度。

②保证三台服务器之间可以相互SSH免密码登录。

2 下载Hadoop安装包

Hadoop官网：http://hadoop.apache.org/
我这里用的Hadoop版本下载地址：http://archive.apache.org/dist/hadoop/core/hadoop-2.7.5/

3 解压

        新建目录 /opt/hadoop

        将下载好的包解压到 /opt/hadoop目录。

tar -zxvf hadoop-2.7.5.tar.gz

        修改文件夹名称

mv hadoop-2.7.5 hadoop

4 修改配置文件

        进入目录 /opt/hadoop/hadoop/etc/hadoop

core-site.xml：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://hadoop1:9000</value>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/opt/hadoop/hadoopdata</value>
	</property>
</configuration>

hdfs-site.xml：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
			<name>dfs.replication</name>
			<value>1</value>
	</property>
</configuration>

yarn-site.xml：

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->

	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.resourcemanager.address</name>
		<value>hadoop1:18040</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address</name>
		<value>hadoop1:18030</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address</name>
		<value>hadoop1:18025</value>
	</property>
	<property>
		<name>yarn.resourcemanager.admin.address</name>
		<value>hadoop1:18141</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.address</name>
		<value>hadoop1:18088</value>
	</property>


</configuration>

MapReduce：

        先复制一份

cp /opt/hadoop/hadoop/etc/hadoop/mapred-site.xml.template /opt/hadoop/hadoop/etc/hadoop/mapred-site.xml

        再对mapred-site.xml进行编辑：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

	<property>
			<name>mapreduce.framework.name</name>
			<value>yarn</value>
	</property>
</configuration>

配置salve文件

vi /opt/hadoop/hadoop/etc/hadoop/slaves

加入从服务器名称，如果文件中有localhost这行信息，直接删除掉

hadoop2
hadoop3

5 配置操作系统环境变量

        此步骤需要在三个节点中都配置，其他无特殊说明的只需要在主节点master上配置

cd /opt/hadoop；

vi ~/.bash_profile；

#HADOOP
export HADOOP_HOME=/opt/hadoop/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

source ~/.bash_profile

        在三个节点中执行以上命令。

6 创建数据存放目录

mkdir /opt/hadoop/hadoopdata

7 格式化文件系统

hadoop namenode -format

将配置文件复制到slave节点上

scp -r /opt/hadoop root@hadoop2:/opt
scp -r /opt/hadoop root@hadoop3:/opt

   

8 启动和关闭Hadoop集群

cd /opt/hadoop/hadoop/sbin

start-all.sh

关闭命令：

stop-all.sh

检查是否启动成功

jps

 主节点：

从节点