一、版本
组件名 | 版本 | 备注及下载地址 |
---|---|---|
Centos | 7.2 64bit | lsb_release -a查看操作系统版本 file /bin/ls查看操作系统位数 |
JRE | java version 1.8.45 | jdk-8u45-linux-x64.gz |
Hadoop | hadoop-2.6.0-cdh5.15.1 | hadoop-2.6.0-cdh5.15.1.tar.gz |
Zookeeper | zookeeper-3.4.6 | zookeeper-3.4.6.tar.gz |
二、主机规划
Host | 安装软件 | 进程 |
---|---|---|
ruozedata001 | Hadoop, Zookeeper | Name Node/DFSZKFailoverController/JournalNode/DataNode/ Resouce Manager/JobHistoryServer/NodeManager/QuorumPeerMain |
ruozedata002 | Hadoop, Zookeeper | Name Node/DFSZKFailoverController/JournalNode/DataNode/ Resouce Manager/JobHistoryServer/NodeManager/QuorumPeerMain |
ruozedata003 | Hadoop, Zookeeper | JournalNode/DataNode/NodeManager/QuorumPeerMain |
三、目录规划
名称 | 路径 | 备注 |
---|---|---|
$HADOOP_NAME | /opt/software/hadoop | |
Data | $HADOOP_NAME/data | |
Log | $HADOOP_NAME/logs | |
hadoop.tmp.dir | $HADOOP_NAME/tmp | 需要手工创建,权限777,root:root |
$ZOOKEEPER_HOME | /opt/software/zookeeper |
四、常用脚本及命令
1.启动集群
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh start
[root@ruozedata002 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh start
[root@ruozedata003 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh start
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/start-all.sh
[root@ruozedata002 ~]# $ZOOKEEPER_HOME/sbin/yarn-daemon.sh start resourcemananger
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
2.关闭集群
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/mr-jobhistory-daemon.sh stop historyserver
[root@ruozedata002 ~]# $ZOOKEEPER_HOME/sbin/yarn-daemon.sh stop resourcemananger
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/stop-all.sh
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh stop
[root@ruozedata002 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh stop
[root@ruozedata003 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh stop
3.监控集群
hdfs dfsadmin -report
###web界面
HDFS:http://120.24.190.115:50070
HDFS:http://47.106.109.17:50070
ResourceManager(Active):http://120.24.190.115:8088
ResourceManager(Standby):http://47.106.109.17:8088/cluster/cluster
JobHistory:http://120.24.190.115:19888/jobhistory
4.单个进程关闭/启动
hadoop-daemon.sh start|stop namenode|datanode|journalnode|zkfc
yarn-daemon.sh start|stop resourcemanager|nodemanager
五、环境配置
1.现在在阿里云上买三台云主机(2 vCPU 8 GiB (I/O优化) 1Mbps),按量付费的。
https://blog.csdn.net/weixin_41158277/article/details/99900916
2.每台机器上增加hadoop用户
[root@ruozedata001 ~]# useradd hadoop
[root@ruozedata001 ~]# su - hadoop
[hadoop@ruozedata001 ~]$ mkdir app software source data lib script tmp maven_repos
3.上传hadoop-2.6.0-cdh5.15.1.tar.gz,jdk-8u45-linux-x64.gz,zookeeper-3.4.6.tar.gz到/software
[hadoop@ruozedata001 software]$ rz
4.IP与hostname绑定(3台)
[root@ruozedata001 ~]# vi /etc/hosts
172.18.98.137 ruozedata001
172.18.98.138 ruozedata002
172.18.98.136 ruozedata003
5.设置3台machines,SSH互相通信
开启发送健输入到所有会话
[hadoop@ruozedata001 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
7d:4c:45:03:98:87:f8:aa:f3:d8:7f:c4:f9:ad:74:b2 hadoop@ruozedata001
The key's randomart image is:
+--[ RSA 2048]----+
| . +.o+ |
| . + .. . |
| . .. |
| ..o |
| S...o. |
| . .+ |
| . . .o..|
| oo ...+.|
| .oo... E. |
+-----------------+
[hadoop@ruozedata001 ~]$ cd .ssh
[hadoop@ruozedata001 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1679 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop 401 Aug 23 00:34 id_rsa.pub
选取ruozedata001作为主节点,其他两台作为分节点
#相当于ruozedata001机器的公钥放在这个可信任的文件里面,然后把ruozedata002、ruozedata003的公钥文件传给ruozedata001
hadoop@ruozedata001 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
#scp id_rsa.pub root@172.18.98.137:/home/hadoop/.ssh/id_rsa2
172.18.98.137这个是ruozedata001的内网ip
ruozedata003也是一样。
[hadoop@ruozedata002 ~]$ cd .ssh
[hadoop@ruozedata002 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1679 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop 401 Aug 23 00:34 id_rsa.pub
[hadoop@ruozedata002 .ssh]$ scp id_rsa.pub root@172.18.98.137:/home/hadoop/.ssh/id_rsa2
The authenticity of host '172.18.98.137 (172.18.98.137)' can't be established.
ECDSA key fingerprint is 56:84:2b:95:01:36:b7:11:18:54:7e:49:f0:e0:91:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.18.98.137' (ECDSA) to the list of known hosts.
root@172.18.98.137's password:
id_rsa.pub 100% 401 0.4KB/s 00:00
[hadoop@ruozedata002 .ssh]$
[hadoop@ruozedata003 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop 401 Aug 23 00:34 id_rsa.pub
[hadoop@ruozedata003 .ssh]$ scp id_rsa.pub root@172.18.98.137:/home/hadoop/.ssh/id_rsa3
The authenticity of host '172.18.98.137 (172.18.98.137)' can't be established.
ECDSA key fingerprint is 56:84:2b:95:01:36:b7:11:18:54:7e:49:f0:e0:91:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.18.98.137' (ECDSA) to the list of known hosts.
root@172.18.98.137's password:
id_rsa.pub 100% 401 0.4KB/s 00:00
[hadoop@ruozedata003 .ssh]$
[hadoop@ruozedata001 .ssh]$ ll
total 24
-rw------- 1 hadoop hadoop 401 Aug 23 00:41 authorized_keys
-rw------- 1 hadoop hadoop 1679 Aug 23 00:34 id_rsa
-rw-r--r-- 1 root root 401 Aug 23 00:50 id_rsa2
-rw-r--r-- 1 root root 401 Aug 23 00:54 id_rsa3
-rw-r--r-- 1 hadoop hadoop 401 Aug 23 00:34 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop 376 Aug 23 01:01 known_hosts
[hadoop@ruozedata001 .ssh]$ cat id_rsa2 >> authorized_keys
[hadoop@ruozedata001 .ssh]$ cat id_rsa3 >> authorized_keys
[hadoop@ruozedata001 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy64olNdPTq20DgFil2WcWp4FraJHoKEyapmSNyI1bzZH78VulQsqGJZMluInoEon5nuacN0AN638MGy0jOZJ1V1eTaoNEKjmuO8jtKIyuvl8LWF+tKo/Lv4ATU5vfTNg96BMdDxjX5pJmGvWaT55Pi+4C3tI/etPox1luqQkZyMKdIM853KIwmUgrjhhyAUaO3k9YIRoGXO5sDJeh35UpwLPoR2ysT4YWUmSd0gVxW210ZgwQJU5mdSr8JpIJx9G3KAYVaU0+mY8FCs1jm4G/IXQsCwIjZilbwbDFMLj6RZDH5E/npXRNeTY+JBhMExNPso+gEUx1qYlXJkY2Phzd hadoop@ruozedata001
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/SJOwDpturrrxszJFBrqHkXfS5vg6U0EtA+w8bPuuNQnkQKWGivX8Bnhe5vHKsQ95pejfHHF5weXRgGotxRXwahaB/1CpJBilvRlNIRO5uGrlYMv0SoppfNAQhCtoy3rBiiTyXr5IeXyK5C9UxeZ62F6E7tE+idcsPGxDjHMrtKd9nV9gkMPULe8M0Y62SH9M/BDIhZBAchB+KEDfWu4nbWDSz0oSa2nDaNsyi6VOoQMFLF5e1dXkYIKWOpGmroxR1D5R5bVYwepab5jtq+IxyVL39b6MUs0mcPSMpWOFc0TssExhfcDMyz12WW8Pdw3sN8TbywEfyLkncZdhIIjv hadoop@ruozedata002
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYLPhxaFKjymbH54uy2FQ8czEUr1ufGXWZU/j5QcSmB4223BEiaZ2HlK7/3JKiM9vMg14jCuDk9YyT/a+m4OfPj7e7QsIQLNCG9vdNAlwzr+DhNPywsABuoc+dnc7V3Pef9aHFLusiZgzbACxKy0OZ6nMGx38nbdRVxY76ATFQg3Wv/p4zXh56xh5xVmhIWgQTKOsSwK11JErOKtQvKGktDC+qg0pMc3md6PgDn3fiSym4YP6th16sz8YD5ZSC0mAoJM2tL7SFnly0uTIF9fWiPXGxNMLLN3l4S3MBpYp+DmXZMnTggab5vN1BNmYq5oZTpVcWsZ7gXRG7IPwDX6LZ hadoop@ruozedata003
[hadoop@ruozedata001 .ssh]$
scp authorized_keys root@ruozedata002:/home/hadoop/.ssh/
scp authorized_keys root@ruozedata003:/home/hadoop/.ssh/
[hadoop@ruozedata001 .ssh]$ scp authorized_keys root@ruozedata002:/home/hadoop/.ssh/
root@ruozedata002's password:
authorized_keys 100% 1203 1.2KB/s 00:00
[hadoop@ruozedata001 .ssh]$ scp authorized_keys root@ruozedata003:/home/hadoop/.ssh/
The authenticity of host 'ruozedata003 (172.18.98.136)' can't be established.
ECDSA key fingerprint is 4c:50:25:e7:47:46:a2:31:7e:84:f8:d2:6a:88:3d:09.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ruozedata003,172.18.98.136' (ECDSA) to the list of known hosts.
root@ruozedata003's password:
authorized_keys 100% 1203 1.2KB/s 00:00
[hadoop@ruozedata001 .ssh]$ chmod 600 authorized_keys
[hadoop@ruozedata002 .ssh]$ exit
logout
[root@ruozedata002 ~]# cd /home/hadoop/.ssh/
[root@ruozedata002 .ssh]# chmod 644 authorized_keys
[root@ruozedata002 .ssh]# ll
total 16
-rw-r--r-- 1 root root 1203 Aug 23 01:14 authorized_keys
-rw------- 1 hadoop hadoop 1675 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop 401 Aug 23 00:34 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop 175 Aug 23 00:54 known_hosts
[hadoop@ruozedata003 .ssh]$ exit
logout
[root@ruozedata003 ~]# cd /home/hadoop/.ssh/
[root@ruozedata003 .ssh]# chmod 644 authorized_keys
[root@ruozedata003 .ssh]# ll
total 16
-rw-r--r-- 1 root root 1203 Aug 23 01:14 authorized_keys
-rw------- 1 hadoop hadoop 1675 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop 401 Aug 23 00:34 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop 175 Aug 23 00:54 known_hosts
ssh ruozedata001 date
ssh ruozedata002 date
ssh ruozedata003 date
[hadoop@ruozedata001 .ssh]$ ssh ruozedata001 date
The authenticity of host 'ruozedata001 (172.18.98.137)' can't be established.
ECDSA key fingerpri