hadoop2.7.3集群搭建

JDK,环境变量在root下进行，hadoop安装都在hadoop账户下进行

Hadoop集群搭建完整攻略

1.创建虚拟机

2.创建centos6.6系统

3.编辑虚拟机--虚拟网络编辑器--还原默认设置--VMnet8-NAT模式--->外网连接成功

4.为三台主机添加同一用户，设置密码：

# 添加用户

useradd hadoop

# 修改密码

passwd hadoop

 

5.修改主机名Vim /etc/hosts

  92.168.60.128  master

  192.168.60.129  slave1

  192.168.60.130  slave2

6.修改vim /etc/sysconfig/network-scripts/ifcfg-eth0主机名

  DEVICE=eth0

  HWADDR=00:0C:29:F5:AF:5E

  TYPE=Ethernet

  UUID=5a1d8421-06f9-48c8-a155-28ed55f85fd4

  ONBOOT=no

  NM_CONTROLLED=yes

  BOOTPROTO=dhcp

7.JDK的安装与卸载（JDK安装在/usr/java 下，尝试安装别的目录，最终会提示报错找不到java）

# 检查当前安装的JDK（卸载JDK）

rpm -qa|grep gcj

rpm -qa|grep jdk

rpm -e  --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

rpm -e  --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

 

8.#查询当前系统中相关java 目录并删除

whereis java

java: /etc/java /usr/lib/java /usr/share/java

 

9.#删除查询出的结果目录

rm -fr /usr/share/java

rm -fr /usr/lib/java

rm -fr /etc/java

10.安装JDK

cd /usr

mkdir java

cd java

FileZilla上传JDK到/usr/java下

tar -zxvf jdk-8u102-linux-x64.tar.gz

11.配置环境变量

 编辑 vim /etc/profile 文件并且在末尾追加

JAVA_HOME=/usr/java/jdk1.8.0_102

PATH=$JAVA_HOME/bin:$PATH

CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar

export PATH JAVA_HOME CLASSPATH

12.生效环境变量

  source /etc/profile

13.检查java的安装状态

 [root@hadoop001 java]# javac

14.SSH 免密码登录安装，配置

保证由master主机能够免密码登录到datanodes节点机上

 1)、# ssh-keygen -t rsa  

会产生如下的信息：

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):      ->输入key文件的名称
Enter passphrase (empty for no passphrase):                       ->输入使用key时的密码，注意这里不是远处服务器的密码，

                                                                                                    只是你使用sshkey时需要输入的密码，

                                                                                                    建议不输入(如果要输入这个密码还不如直接输入

                                                                                                    远处主机的密码：）)
Enter same passphrase again:  
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
11:ae:00:bd:92:bc:70:f8:4b:cb:47:06:7f:e2:73:21 root@xxxxxx

 

2).#cd .ssh/

ls 

     scp id_rsa.pub 192.168.0.2:/root/.ssh/

 

3)、ssh 192.168.0.2    ->登录远程主机B

 

      root@192.168.0.2'spassword: ->输入密码 

 

4)、 # cd /root/.ssh    ->进入到key目录

      # cat id_rsa.pub >> authorized_keys

 

 

5）#cd ..

    chmod 700 .ssh

      #cd .ssh

    chmod 600  authorized_keys

 

6）.#exit    -> 推出远程B服务器

 

 

7）.#service sshd restart   ->重新启动ssh服务

 

8）.#ssh 192.168.0.2  ->远程登录到B测试下吧：）

15.安装 Hadoop2.7.3

1).cd /tmp

2),使用FileZilla将下载好的hadoop安装包上传到/tmp下

3）.解压 hadoop-2.7.3.tar.gz

tar -zxvf hadoop-2.7.3.tar.gz

4）.复制文件到 /usr 目录下

cp -R /tmp/hadoop-2.7.3  /usr/hadoop

5）.配置hadoop的环境变量，在/etc/profile下追加

vim /etc/profile

追加如下的环境变量设置

HADOOP_HOME=/usr/hadoop

export HADOOP_INSTALL=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

6）.修改 etc/hadoop/hadoop-env.sh 文件(地址待更改)

vim etc/hadoop/hadoop-env.sh

 

# The java implementation to use.

#export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/usr/java/jdk1.8.0_102

7）.修改 etc/hadoop/core-site.xml 文件

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

        <name>hadoop.tmp.dir</name>

        <value>file:/usr/hadoop/tmp</value>

        <description>Abase for other temporary directories.</description>

    </property>

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://Master:9000</value>

    </property>

 

</configuration>

8).修改 etc/hadoop/core-site.xml 文件

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

        <name>hadoop.tmp.dir</name>

        <value>file:/usr/hadoop/tmp</value>

        <description>Abase for other temporary directories.</description>

    </property>

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://Master:9000</value>

    </property>

 

</configuration>

9).修改 etc/hadoop/yarn-site.xml 文件

<configuration>

 

        <!-- Site specific YARN configuration properties -->

        <property>

                <name>yarn.resourcemanager.hostname</name>

                <value>Master</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

 

</configuration>

10).修改 etc/hadoop/yarn-site.xml 文件

<configuration>

 

        <!-- Site specific YARN configuration properties -->

        <property>

                <name>yarn.resourcemanager.hostname</name>

                <value>Master</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

 

</configuration>

11).修改 etc/hadoop/slaves 文件，添加

192.168.3.135

192.168.3.136

12）.打包文件夹 /home/hadoop ,复制到 datanode 节点机，保证节点机环境配置与master保持一致格式化文件系统

 scp -r /home/hadoop hadoop@192.168.3.135:/home/hadoop

hdfs namenode -format（格式化）

13）.启动文件服务

start-dfs.sh

Make the HDFS directories required to execute MapReduce jobs:

14）.创建用户文件系统文件夹

Hadoop fs -mkdir /user

hadoop fs -mkdir /user/hadoop

15).复制文件本地文件到分布式文件系统 input 下

hadoop fs -mkdir /input(不能少/)

Hadoop fs -put hadoop安装目录下/etc/hadoop/*.xml input

16）.至此全部配置成功，可使用jps查看各节点具体运行状况