Hadoop 2.7.1 集群模式安装【一】

Reference

Hadoop 2.7.1 Setting up a Single Node Cluster.
Hadoop2.7.1 Cluster Setup
Ubuntu 14 SSH Config

环境

• 操作系统 ubuntu 14 x64
• 硬盘/内存：45G/4G
• Hadoop 2.7 x64
• JDK 1.7.0.80 x64
• 三台集群

机器名称	IP地址	角色
master	10.10.76.50	master
slave-1	10.10.76.51	slave
slave-2	10.10.76.52	slave

准备环境

安装必要软件

$ sudo apt-get install ssh
$ sudo apt-get install rsync

ubuntu添加&配置用户

$ sudo groupadd hadoop                     #创建hadoop用户组
$ sudo useradd -g hadoop -m hadoop         #创建hadoop用户并添加到hadoop组
$ sudo passwd hadoop                       #修改密码：hadoop

hosts&主机名配置

• 修改hosts

$ sudo vim /etc/hosts
127.0.0.1       localhost
127.0.1.1       hadoop-master
10.10.76.50     hadoop-master
10.10.76.51     hadoop-slave-1
10.10.76.52     hadoop-slave-2

• 修改hostname

$ sudo hostname hadoop-master
$ sudo vim /etc/hostname
hadoop-master

#测试
hadoop-master

ssh免密码配置

#测试localhost是否有密码
$ ssh localhost
#如果有密码进行下面配置：
hadoop@hadoop-master:~$ ssh-keygen  -t rsa
hadoop@hadoop-master:~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
hadoop@hadoop-master:~$ chmod 600 .ssh/authorized_keys
#以上步骤在三台机器都重复执行
#拷贝秘钥到其他机器
hadoop@hadoop-master:~$ scp /home/hadoop/.ssh/id_rsa.pub hadoop@hadoop-slave-1:~/
hadoop@hadoop-slave-1:~$ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

至此系统准备环境以及配置好，以上步骤需要在其他两台slave节点机器上面同样配置。

配置环境变量

下载好Oracle jdk1.7.0_80拷贝到/usr/local路径下面，接着在/etc/profile中做如下配置：

$ sudo vim /etc/profile
JAVA_HOME=/usr/local/jdk1.7.0_80
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$JAVA_HOME/bin:$PATH

$ source /etc/profile

hadoop安装

编译hadoop源码（略）。我之前已经编译好了所以我就直接使用即可，这里编译过程参照hadoop-2.7.1-src/BUILDING.txt即可。

$ sudo mv hadoop-2.7.1 /usr/local/
$ cd /usr/local
$ sudo chown hadoop:hadoop -R hadoop-2.7.1

编辑etc/hadoop/hadoop-env.sh文件设置JAVA_HOME路径

$ cd /usr/locl/hadoop-2.7.1
$ vim etc/hadoop/hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/usr/local/jdk1.7.0_80

#测试
$ bin/hadoop

输出如下：

 Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
  CLASSNAME            run the class named CLASSNAME
 or
  where COMMAND is one of:
  fs                   run a generic filesystem user client
  version              print the version
  jar <jar>            run a jar file
                       note: please use "yarn jar" to launch
                             YARN applications, not this command.
  checknative [-a|-h]  check native hadoop and compression libraries availability
  distcp <srcurl> <desturl> copy file or directories recursively
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
  credential           interact with credential providers
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
  trace                view and modify Hadoop tracing settings

————————华丽丽的分割线—————————–
至此我们已经将全部的准备工作完成，你可以将系统做成一个备份，以后安装都可以基于这个
————————华丽丽的分割线—————————–