Hadoop 环境配置与安装

本文主要讲述在VMware虚拟机上配置Hadoop的运行环境，包含配置过程与配置代码，具体如下：

一、环境配置：
1.安装jdk

a)下载jdk-8u65-linux-x64.tar.gz
b)tar开
	$>su centos ; cd ~
	$>mkdir downloads
	$>cp /mnt/hdfs/downloads/bigdata/jdk-8u65-linux-x64.tar.gz ~/downlooads
	$>tar -xzvf jdk-8u65-linux-x64.tar.gz
c)创建/soft文件夹
	$>sudo mkdir /soft
	$>sudo chown centos:centos /soft
d)移动tar开的文件到/soft下
	$>mv ~/downloads/jdk-1.8.0_65 /soft/
e)创建符号连接
	$>ln -s /soft/jdk-1.8.0_65 /soft/jdk
f)验证jdk安装是否成功
	$>cd /soft/jdk/bin
	$>./java -version

2.centos下配置环境变量

1.编辑/etc/profile
	$>sudo nano /etc/profile
	...
	export JAVA_HOME=/soft/jdk
	exprot PATH=$PATH:$JAVA_HOME/bin
2.使环境变量即刻生效
	$>source /etc/profile

3.进入任意目录下,测试是否ok
	$>cd ~
	$>java -version

二、安装Hadoop

1.安装hadoop

a)下载hadoop-2.7.3.tar.gz
b)tar开
	$>su centos ; cd ~
	$>cp /mnt/hdfs/downloads/bigdata/hadoop-2.7.3.tar.gz ~/downloads
	$>tar -xzvf hadoop-2.7.3.tar.gz
c)无
d)移动tar开的文件到/soft下
	$>mv ~/downloads/hadoop-2.7.3 /soft/
e)创建符号连接
	$>ln -s /soft/hadoop-2.7.3 /soft/hadoop
f)验证jdk安装是否成功
	$>cd /soft/hadoop/bin
	$>./hadoop version

2.配置hadoop环境变量

$>sudo nano /etc/profile
...
export JAVA_HOME=/soft/jdk
exprot PATH=$PATH:$JAVA_HOME/bin

export HADOOP_HOME=/soft/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

3.生效

$>source /etc/profile

4.配置伪分布模式

a)进入${HADOOP_HOME}/etc/hadoop目录
b)编辑core-site.xml
	<?xml version="1.0"?>
	<configuration>
		<property>
			<name>fs.defaultFS</name>
			<value>hdfs://localhost/</value>
		</property>
	</configuration>
c)编辑hdfs-site.xml
	<?xml version="1.0"?>
	<configuration>
		<property>
			<name>dfs.replication</name>
			<value>1</value>
		</property>
	</configuration>
d)编辑mapred-site.xml
	注意:cp mapred-site.xml.template mapred-site.xml
	<?xml version="1.0"?>
	<configuration>
		<property>
			<name>mapreduce.framework.name</name>
			<value>yarn</value>
		</property>
	</configuration>
e)编辑yarn-site.xml
	<?xml version="1.0"?>
	<configuration>
		<property>
			<name>yarn.resourcemanager.hostname</name>
			<value>localhost</value>
		</property>
		<property>
			<name>yarn.nodemanager.aux-services</name>
			<value>mapreduce_shuffle</value>
		</property>
	</configuration>

f)配置SSH
	1)检查是否安装了ssh相关软件包(openssh-server + openssh-clients + openssh)
		$yum list installed | grep ssh

	2)检查是否启动了sshd进程
		$>ps -Af | grep sshd
	
	3)在client侧生成公私秘钥对。
		$>ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
	
	4)生成~/.ssh文件夹，里面有id_rsa(私钥) + id_rsa.pub(公钥)

	5)追加公钥到~/.ssh/authorized_keys文件中(文件名、位置固定)
		$>cd ~/.ssh
		$>cat id_rsa.pub >> authorized_keys
	
	6)修改authorized_keys的权限为644.
		$>chmod 644 authorized_keys
	
	7)测试
		$>ssh localhost