一、准备环境.
- centos6.5(minimal)下载地址
- java1.7,hadoop-2.5.2,
- date同步
- ssh免密码登录(无命令ssh)
-
- yum安装不了
二、安装虚拟机和虚拟镜像
使用vm12安装centos6.5,如何安装的步骤省略,大家自行百度,使用自定义的方式安装。
- yum安装不了
'选择10GB的空间,其余不选'
'移除声卡等没用的'
分别安装四个虚拟机并分别命名为node1,node2,node3,node4。
三、配置静态ip地址和dns
主机ip | 主机名 |
---|---|
192.168.146.100 | node1 |
192.168.146.101 | node2 |
192.168.146.102 | node3 |
192.168.146.103 | node4 |
配置静态ip
[root@node4 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:0C:29:AC:5B:B6
TYPE=Ethernet
UUID=ff9f551e-43d8-404f-bcf1-682436eb0281
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.146.103
NETMASK=255.255.255.0
GATEWAY=192.168.146.2
此处需要配置dns,否则ping不通外网
[root@node4 ~]# vi /etc/resolv.conf
nameserver 8.8.8.8
nameserver 1.2.4.8
service network restart:重启网络
测试
ping www.baidu.com
将软件包分别通过winscp等工具传到/home/software文件夹下
-关闭防火墙
暂时关闭:service iptables stop
永久关闭chkconfig iptables off
四、java配置(jdk-7u79)
一般企业使用的是最稳定的软件包:jdk-7u79-linux-x64.rpm
先查看是否已经安装:
[root@node4 software]# rpm -qa | grep java
补充:使用rpm命令卸载CentOS 6.4自带的OpenJDK
- 安装jdk
1.新建一个文件夹/opt/software,并将jdk-7u79-linux-x64.rpm移到该文件下。
2.修改文件的权限:
[root@node4 software]# cd /opt/software
[root@node4 software]# ll
total 134848
-rw-r--r--. 1 root root 138082565 Sep 5 09:00 jdk-7u79-linux-x64.rpm
[root@node4 software]# chmod 755 jdk-7u79-linux-x64.rpm
[root@node4 software]# ll
total 134848
-rwxr-xr-x. 1 root root 138082565 Sep 5 09:00 jdk-7u79-linux-x64.rpm
3.安装jdk:
[root@node4 software]# rpm -ivh jdk-7u79-linux-x64.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
- 设置环境变量
使用rpm的安装方式默认安装在/usr/java下面
export JAVA_HOME=/usr/java/jdk1.7.0_79
export PATH=$PATH:$JAVA_HOME/bin
- 检查是否安装成功
[root@node4 software]# java -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
[root@node4 software]# rpm -qa | grep jdk
jdk-1.7.0_79-fcs.x86_64
五、同步时间date
date命令 查看时间是否相同;相差几秒还是可以的,不同则需要同步时间,自行百度解决。
六、ssh免密码登录
- 本机的免密码登录
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@node1 software]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
9b:5c:5d:2f:67:9d:c0:48:09:45:72:f6:62:1b:27:95 root@node1
The key's randomart image is:
+--[ DSA 1024]----+
| o+*... |
| =.=E |
| * =. |
| ..*...o|
| S ... ..=|
| . + + |
| + |
| |
| |
+-----------------+
[root@node1 software]# cd ~/.ssh/
[root@node1 .ssh]# ls
id_dsa id_dsa.pub
[root@node1 .ssh]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@node1 .ssh]# ls
authorized_keys id_dsa id_dsa.pub
-测试本地的ssh免密码登录:
[root@node1 .ssh]# ssh node1
The authenticity of host 'node1 (192.168.146.100)' can't be established.
RSA key fingerprint is 94:04:c9:47:67:9e:8a:28:ce:05:3b:dd:92:5d:a8:b6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1,192.168.146.100' (RSA) to the list of known hosts.
Last login: Sat Oct 7 04:14:09 2017 from 192.168.146.1
[root@node1 ~]# exit
logout
Connection to node1 closed.
- 配置hosts,远程ssh免密码登录
[root@node1 .ssh]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.146.100 node1
192.168.146.101 node2
192.168.146.102 node3
192.168.146.103 node4
~
-其余node2/node3/node4的配置
将node1上的公证认证文件拷贝到其他的node节点上,并进行复制大文件下:
在node2,node3,node4上都进行如下的操作:
ssh-keygen -t dsa -P ” -f ~/.ssh/id_dsa
在node1上分别进行如下操作,将node1的公钥复制到node2,3,4上面。
scp ./id_dsa.pub root@node2:/opt/
scp ./id_dsa.pub root@node3:/opt/
scp ./id_dsa.pub root@node4:/opt/
在node2,node3,node4上都进行下面的操作
cat /opt/id_dsa.pub >> ~/.ssh/authorized_keys
- 测试
ssh node1
ssh node2
ssh node3
ssh node4
但是缺点是,node2,node3,node4不能ssh访问node的其他节点包括自己节点
[root@node1 .ssh]# ssh node1
ee^HLast login: Sat Oct 7 04:54:15 2017 from node1
[root@node1 ~]# exit
logout
Connection to node1 closed.
[root@node1 .ssh]# ssh node2
Last login: Sat Oct 7 04:54:41 2017 from node1
[root@node2 ~]# exit
logout
Connection to node2 closed.
[root@node1 .ssh]# ssh node3
Last login: Sat Oct 7 04:57:47 2017 from node1
[root@node3 ~]# exit
logout
Connection to node3 closed.
[root@node1 .ssh]# ssh node4
Last login: Sat Oct 7 04:47:24 2017 from node1
[root@node4 ~]# exit
logout
Connection to node4 closed.
七、hadoop的简单的搭建
节点的分配
节点 | 机器 |
---|---|
namenode | node1 |
secondnamenode | node2 |
datenode | node node node |
- 解压hadoop
解压到在/home/hadoop-2.5.1
[root@node3 home]# tar -zxvf hadoop-2.5.1_x64.tar.gz
[root@node3 software]# mv hadoop-2.5.1 /home/
- 配置环境
- hadoop-env.sh
寻找到该位置
export JAVA_HOME=/usr/java/jdk1.7.0_79
-配置core-site.xml文件
<property>
<name>fs.defaultFS</name>
<value>hdfs://node1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-2.5</value>
</property>
##配置临时目录,用于重启之后使用。
-配置hdfs-site.xml文件
##配置secondnamenode主机
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>node2:50090</value>
</property>
<property>
<name>dfs.namenode.secondary.https-address</name>
<value>node2:50091</value>
</property>
-配置slaves文件
##配置datanode,添加
node2
node3
node4
-配置masters文件,需要自己创建
##配置secondnamenode主机名,添加
node2
将node1的配置的文件copy到node2,node3,node4中
[root@node1 hadoop]# scp -r /home/hadoop-2.5.1/etc/hadoop/ root@node2:/home/hadoop-2.5.1/etc/hadoop/
[root@node1 hadoop]# scp -r /home/hadoop-2.5.1/etc/hadoop/ root@node3:/home/hadoop-2.5.1/etc/hadoop/
[root@node1 hadoop]# scp -r /home/hadoop-2.5.1/etc/hadoop/ root@node4:/home/hadoop-2.5.1/etc/hadoop/
##途中出现复制错误,使用rm -rf删除文件夹
-配置hadoop的环境变量
#编辑文件:[root@node1 hadoop]# vi ~/.bash_profile
export HADOOP_HOME=/home/hadoop-2.5.1
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
拷贝到其他节点上去
[root@node1 hadoop]# scp ~/.bash_profile root@node2:/root/
[root@node1 hadoop]# scp ~/.bash_profile root@node3:/root/
[root@node1 hadoop]# scp ~/.bash_profile root@node4:/root/
生效
[root@node1 hadoop]# source ~/.bash_profile
八、开始执行
- 格式化
##只能在namenode的node1上格式化,在/opt/hadoop-2.5/dfs、current上创建了文件
[root@node1 hadoop]# cd /opt/hadoop-2.5/dfs/name/current/
[root@node1 current]# ll
total 16
-rw-r--r--. 1 root root 351 Oct 7 06:01 fsimage_0000000000000000000
-rw-r--r--. 1 root root 62 Oct 7 06:01 fsimage_0000000000000000000.md5
-rw-r--r--. 1 root root 2 Oct 7 06:01 seen_txid
-rw-r--r--. 1 root root 207 Oct 7 06:01 VERSION
-启动
[root@node1 sbin]# start-dfs.sh
web界面:
打开不了的原因是需要关闭防火墙:service iptables stop
namenode
secondnamenode
精彩部分请看:hadoop-企业版环境搭建(二)