《 Hadoop安装》实验流程

《 Hadoop安装》实验流程

一、准备虚拟机环境：

Master                  192.168.233.131                 master   30G/1G

 

Slave1                   192.168.233.129                 slave1    20G/512MB

 

Slave2                   192.168.233.130                 slave2    20G/512MB

说明：操作系统：oracle linux 4.8  ；Hadoop软件包：hadoop-0.20.2.tar.gz   ； Eclipse软件包：eclipse-SDK-3.6.2-linux-gtk.tar.gz  jdk版本：jdk1.6.0_35

[root@master ~]# cat /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1               localhost.localdomain localhost

192.168.233.129  slave1

192.168.233.130  slave2

192.168.233.131  master

二、所有节点配置java环境：

以master节点为例：

编辑、配置java环境变量：

[root@master ~]# vi /etc/profile

添加下面下面内容到：/etc/profile

export JAVA_HOME=/usr/java/jdk1.6.0_35

export JRE_HOME=/usr/java/jdk1.6.0_35/jre

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib

让环境变量生效：

[root@master ~]# source  /etc/profile

   测试java是否可用：

[root@master ~]# java   -version

java version "1.6.0_35"

Java(TM) SE Runtime Environment (build 1.6.0_35-b10)

Java HotSpot(TM) Client VM (build 20.10-b01, mixed mode, sharing)

三、所有节点创建hadoop用户和hadoop组：

以master为例：

[root@master ~]# ! /bin/bash

[root@master ~]# groupadd hadoop

[root@master ~]# useradd hadoop -g hadoop; echo hadoop | passwd --stdin hadoop

Changing password for user hadoop.

passwd: all authentication tokens updated successfully.

说明：同理在slave1节点和slave2建立hadoop用户和hadoop组

 

三、SSH配置：

1、所有节点上hadoop用户执行： ssh-keygen -t rsa

以slave1为例：

[root@slave1 ~]# su - hadoop

[hadoop@slave1 ~]$ ssh-keygen -t rsa   按Enter，按默认设置

 同理：其他节点master、slave2如此

2、各节点上执行：

Master节点：

 [hadoop@master ~]$ cd .ssh

 [hadoop@master .ssh]$ cat  id_rsa.pub>>authorized_keys

[hadoop@master .ssh]$ scp  authorized_keys  hadoop@slave1:/home/hadoop/.ssh/

The authenticity of host 'slave1 (192.168.233.129)' can't be established.

RSA key fingerprint is 55:95:47:1b:ba:9d:fa:68:dd:5f:b9:1a:1b:a3:d3:06.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave1,192.168.233.129' (RSA) to the list of known hosts.

hadoop@slave1's password:

authorized_keys                               100%  603     0.6KB/s   00:00

Slave1节点：

[hadoop@slave1 ~]$ cd  .ssh

[hadoop@slave1 .ssh]$ cat  id_rsa.pub >> authorized_keys

hadoop@slave1 .ssh]$ scp  authorized_keys  hadoop@slave2:/home/hadoop/.ssh/

The authenticity of host 'slave2 (192.168.233.130)' can't be established.

RSA key fingerprint is b3:d2:ad:95:79:78:45:6b:b0:49:d5:6e:86:31:83:43.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave2,192.168.233.130' (RSA) to the list of known hosts.

hadoop@slave2's password:

authorized_keys                                       100% 1206     1.2KB/s   00:00

Slave2节点：

[hadoop@slave2 ~]$ cd  .ssh

[hadoop@slave2 .ssh]$ cat  id_rsa.pub >> authorized_keys

[hadoop@slave2 .ssh]$ scp  authorized_keys  hadoop@master:/home/hadoop/.ssh/

The authenticity of host 'master (192.168.233.131)' can't be established.

RSA key fingerprint is 5b:d1:fe:0b:49:6c:ae:a7:62:ad:1e:d0:81:c5:01:3e.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'master,192.168.233.131' (RSA) to the list of known hosts.

hadoop@master's password:

authorized_keys                               100% 1809     1.8KB/s   00:00

Master节点：

[hadoop@master .ssh]$ scp  authorized_keys hadoop@slave1:/home/hadoop/.ssh/

hadoop@slave1's password:

authorized_keys                               100% 1809     1.8KB/s   00:00

[hadoop@master .ssh]$ scp  authorized_keys hadoop@slave2:/home/hadoop/.ssh/

The authenticity of host 'slave2 (192.168.233.130)' can't be established.

RSA key fingerprint is b3:d2:ad:95:79:78:45:6b:b0:49:d5:6e:86:31:83:43.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave2,192.168.233.130' (RSA) to the list of known hosts.

hadoop@slave2's password:

authorized_keys                               100% 1809     1.8KB/s   00:00

说明：对所有节点root用户执行：chmod  600  authorized_keys

[hadoop@master .ssh]$ chmod  600  authorized_keys

[hadoop@slave1 .ssh]$ chmod  600  authorized_keys

[hadoop@slave2 .ssh]$ chmod  600  authorized_keys

测试：ssh 各节点连接不需要密码，说明配置成功，否则，重新配置。

[hadoop@slave1 ~]$ ssh slave2

Last login: Sun Sep 30 12:03:38 2012 from master

[hadoop@slave2 ~]$ ssh master

Last login: Sun Sep 30 12:03:15 2012 from slave2

[hadoop@master ~]$ ssh slave1

Last login: Sun Sep 30 12:02:01 2012 from maste

四、安装hadoop软件：

 Master节点：

[root@master software]# tar –zxvf  hadoop-0.20.2.tar.gz

[root@master ~ ]# su  –  hadoop

[hadoop@master ~]$ mkdir  -p  /home/hadoop/hadoop

[root@master software]# cp  -r  hadoop-0.20.2/  /home/hadoop/hadoop

[root@master ~]# chown –R  hadoop:hadoop  /home/hadoop/hadoop/hadoop-0.20.2/

五、Hadoop配置：

Master节点：

 （一）配置namenode,修改 xml文件：

 [hadoop@master ~]$ cd /home/hadoop/hadoop/hadoop-0.20.2/conf

[hadoop@master conf]$ ls

capacity-scheduler.xml     hadoop-policy.xml  slaves

configuration.xsl          hdfs-site.xml      ssl-client.xml.example

core-site.xml              log4j.properties   ssl-server.xml.example

hadoop-env.sh              mapred-site.xml

hadoop-metrics.properties  masters

[hadoop@master conf]$ vi  core-site.xml  ----------- ------  修改#号里面的内容

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. --&gt

 

####################

        fs.default.name

        hdfs://master:9000

###################

[hadoop@master hadoop-0.20.2]$ mkdir data

[hadoop@master hadoop-0.20.2]$ cd conf/

[hadoop@master conf]$ vi  hdfs-site.xml   -----------------修改#号里面的内容

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. --&gt

 

##################

        dfs.data.dir

        /home/hadoop/hadoop/hadoop-0.20.2/data

        dfs.replication

        2

################

[hadoop@master conf]$ vi mapred-site.xml   ---------------------------------修改#号里的内容

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!-- Put site-specific property overrides in this file. --&gt

 

#####################

          mapred.job.tracker

          hdfs://master:9001

#################

（二）配置hadoop-env.sh：

[hadoop@master conf]$ vi  hadoop-env.sh

 设置JAVA_HOME如下：

# The java implementation to use.  Required.

export JAVA_HOME=/usr/java/jdk1.6.0_35

（三）配置masters文件

[hadoop@master conf]$ vi masters 

Master

（四）配置slaves文件

[hadoop@master conf]$ vi slaves

slave1

slave2

（五）master节点向个节点复制hadoop文件：

Slave1和slave2节点创建目录：mkdir  -p  /home/hadoop/hadoop/

[hadoop@master ~]$ scp -r /home/hadoop/hadoop/hadoop-0.20.2/ hadoop@slave1:/home/hadoop/hadoop/

[hadoop@master ~]$ scp -r /home/hadoop/hadoop/hadoop-0.20.2/ hadoop@slave2:/home/hadoop/hadoop

(六) 格式化namenode

[hadoop@master ~]$ cd /home/hadoop/hadoop/hadoop-0.20.2/

[hadoop@master hadoop-0.20.2]$ bin/hadoop namenode –format

(七)启动Hadoop

[hadoop@master hadoop-0.20.2]$ bin/start-all.sh

 六、检查各节点状态：

Master节点:

[hadoop@master hadoop-0.20.2]$ cd /usr/java/jdk1.6.0_35/bin/

[hadoop@master bin]$ ./jps

1972 JobTracker

1912 SecondaryNameNode

2080 Jps

1755 NameNode

Slave1节点：

[hadoop@slave1 ~]$ cd /usr/java/jdk1.6.0_35/bin

[hadoop@slave1 bin]$ ./jps

30457 DataNode

30605 Jps

30530 TaskTracker

16200 Jps

Slave2节点：

[hadoop@slave2 ~]$ cd /usr/java/jdk1.6.0_35/bin/

[hadoop@slave2 bin]$ ./jps

30555 TaskTracker

30483 DataNode

30655 Jps

七、测试hadoop:

 [hadoop@master hadoop-0.20.2]$ bin/hadoop fs -mkdir /Test

 [hadoop@master hadoop-0.20.2]$ vi test.txt

[hadoop@master hadoop-0.20.2]$ cat test.txt

I love you,hua!

[hadoop@master hadoop-0.20.2]$ bin/hadoop fs -copyFromLocal   test.txt   /Test

[hadoop@master hadoop-0.20.2]$ bin/hadoop fs -ls  /Test

Found 1 items

-rw-r--r--   2 hadoop supergroup         12 2012-10-10 11:49 /Test/test.txt

[hadoop@master hadoop-0.20.2]$ bin/hadoop fs -cat   /Test/test.txt

I love you,hua!

                                                     By  吴少杰   2012.10.11

Normal 0 7.8 磅 0 2 false false false EN-US ZH-CN X-NONE

转载于:http://blog.itpub.net/26857237/viewspace-746303/