学习笔记:从0开始学习大数据-35.docker部署hadoop集群

上节学习安装使用docker,并运行了ubuntu16.04的镜像，本节就在docker的ubuntu16.04镜像容器中创建hadoop三机集群。

先启动一个ubuntu16.04的容器：

1.修改容器中的下载源：

nano sources.list
deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse

2. 下载java
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
下载到：/home/linbin/下载/jdk-8u191-linux-x64.tar.gz

2. docker run -ti ubuntu:16.04
root@5185cb8ca6ae:~# 

3. 在另外一个bash shell
docker cp sources.list 5185cb8ca6ae:/etc/apt/sources.list
docker cp /home/linbin/下载/jdk-8u191-linux-x64.tar.gz  5185cb8ca6ae:/opt

4.回到 root@5185cb8ca6ae:~# 

root@5185cb8ca6ae:~# apt-get update
root@5185cb8ca6ae:~# apt-get install nano
root@5185cb8ca6ae:~#cd /opt
root@5185cb8ca6ae:/opt# tar -zxvf jdk-8u191-linux-x64.tar.gz
root@5185cb8ca6ae:/opt# rm jdk-8u191-linux-x64.tar.gz

5.修改  /root/.basrc
root@5185cb8ca6ae:/opt# nano /root/.basrc
 
export JAVA_HOME=/opt/jdk1.8.0_191
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:JAVA_HOME/lib:JRE_HOME/lib:${CLASSPATH}
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

6.测试
root@5185cb8ca6ae:/opt# source /root/.basrc
root@5185cb8ca6ae:/opt# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
root@5185cb8ca6ae:/opt#exit

docker ps -al  //查看所有容器
docker ps  //查看运行中的容器

[root@centos7 ~]#docker start 5185cb8ca6ae  //可以再次启动
[root@centos7 ~]# docker attach 5185cb8ca6ae   //再次进入
root@5185cb8ca6ae:/#     

7.保存安装好java的镜像为新的镜像
root@5185cb8ca6ae:/# exit
exit
[root@centos7 ~]# docker commit 5185cb8ca6ae ubuntu-java
sha256:3f7331f968a4070f6f3cb2918aa2298b9eb684b23b76f9bde5bbc227c13b4478
[root@centos7 ~]# docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
ubuntu-java             latest              3f7331f968a4        8 seconds ago       539 MB
docker.io/ubuntu        16.04               a51debf7e1eb        4 weeks ago         116 MB
docker.io/hello-world   latest              4ab4c602aa5e        3 months ago        1.84 kB

8. 运行新的容器 
docker run -ti  ubuntu-java
root@af1c57038caf:/# exit

9.下载hadoop
下载：
http://mirrors.shu.edu.cn/apache/hadoop/common/hadoop-3.0.3/hadoop-3.0.3.tar.gz
复制下载的文件到容器：
docker cp hadoop-3.0.3.tar.gz af1c57038caf:/opt
在容器解压hadoop：
root@af1c57038caf:/# cd /opt
root@5185cb8ca6ae:/opt# tar -zxvf hadoop-3.0.3.tar.gz
root@5185cb8ca6ae:/opt# rm -zxvf hadoop-3.0.3.tar.gz

10.设置ssh免密码相互登录

root@af1c57038caf:/# apt-get install ssh
root@af1c57038caf:/# nano /etc/ssh/sshd_config
修改：PermitRootLogin yes

root@af1c57038caf:/etc/ssh# cd ~
root@af1c57038caf:~# ssh-keygen -t rsa -P '' -f ~/.ssh/id_dsa
Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
SHA256:+umoGC+j+Z139COSeWU5UZMKykt3fZHXrXwFsPakYlg root@af1c57038caf
The key's randomart image is:
+---[RSA 2048]----+
|            .o.oo|
|        .   +.o =|
|     . . .E+oo.+.|
|      + .o+..++ .|
|     . oS.oo.... |
|      ....=.     |
|  .   .+ + .     |
| .o= .=o+.o      |
|oooo=o.=+. .     |
+----[SHA256]-----+
root@af1c57038caf:~# cd .ssh
root@af1c57038caf:~/.ssh#  cat id_dsa.pub >> authorized_keys
root@af1c57038caf:~/.ssh# 

11.修改hadoop配置文件
在 /opt/hadoop-3.0.3/etc/hadoop目录

如下6个文件 

hadoop-env.sh
yarn-site.xml
mapred-site.xml
hdfs-site.xml
core-site.xml
workers

（1）hadoop-env.sh
export JAVA_HOME=/opt/jdk1.8.0_191

（2）yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

（3）mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

（4）hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/root/hadoop/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/root/hadoop/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>

(5) core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:8020</value>
</property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/root/hadoop/tmp</value>
 </property>
</configuration>

(6)workers

hadoop1
hadoop2
hadoop3

root@af1c57038caf:/opt/hadoop-3.0.3/etc/hadoop# nano workers
root@af1c57038caf:/opt/hadoop-3.0.3/etc/hadoop# nano hadoop-env.sh
root@af1c57038caf:/opt/hadoop-3.0.3/etc/hadoop# nano yarn-site.xml
root@af1c57038caf:/opt/hadoop-3.0.3/etc/hadoop# nano mapred-site.xml
root@af1c57038caf:/opt/hadoop-3.0.3/etc/hadoop# nano hdfs-site.xml
root@af1c57038caf:/opt/hadoop-3.0.3/etc/hadoop# nano core-site.xml

12.创建hadoop目录

root@af1c57038caf:~# cd ~ 
root@af1c57038caf:~# mkdir hadoop
root@af1c57038caf:~# cd hadoop
root@af1c57038caf:~/hadoop# mkdir namenode
root@af1c57038caf:~/hadoop# mkdir datanode
root@af1c57038caf:~/hadoop# mkdir tmp

13. 编辑 /root/.bashrc
export JAVA_HOME=/opt/jdk1.8.0_191
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:JAVA_HOME/lib:JRE_HOME/lib:${CLASSPATH}
export JAVA_HOME/bin:$JRE_HOME/bin:$PATH
export HADOOP_HOME=/opt/hadoop-3.0.3
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sb$

14.保存dacker容器为镜像

docker commit -m "hadoop-install" af1c57038caf ubuntu:hadoop

[root@centos7 linbin]# docker commit -m "hadoop-install" af1c57038caf ubuntu:hadoop
sha256:8dbec7cabd07d251f2ddabfa39508a0435194286b88a87849c2ae4d9836466df
[root@centos7 linbin]# docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
ubuntu                  hadoop              8dbec7cabd07        2 minutes ago       1.38 GB
ubuntu-java             latest              3f7331f968a4        5 hours ago         539 MB
docker.io/ubuntu        16.04               a51debf7e1eb        4 weeks ago         116 MB
docker.io/hello-world   latest              4ab4c602aa5e        3 months ago        1.84 kB
[root@centos7 linbin]# 

15.启动hadoop容器
分别启动三个，名为 hadoop1,hadoop2,hadoop3

docker network ls //查看网络
docker network create --subnet=192.168.10.0/24 mynetwork  //创建自定义网络，只有自定义网络才可指定docker ip

因为docker不能保存ip，hostname,hosts等，每次启动时指定

docker run -itd --name hadoop1 --hostname hadoop1 --net mynetwork --ip 192.168.10.11  --add-host hadoop2:192.168.10.12 --add-host hadoop3:192.168.10.13  -d -P -p 50070:50070 -p 8088:8088 -p:8020:8020 ubuntu:hadoop

其中：
--hostname ：指定hostname;
--net : 指定网络模式
--ip：指定IP
--add-host ：指定往/etc/hosts添加的host

如果已经存在同名的容器，先删除
docker stop hadoop1
docker rm hadoop1
然后再次运行上面命令创建容器

[root@centos7 linbin]# docker run -itd --name hadoop1 --hostname hadoop1 --net mynetwork --ip 192.168.10.11  --add-host hadoop2:192.168.10.12 --add-host hadoop3:192.168.10.13  -d -P -p 9870:9870 -p 8088:8088 -p:8020:8020 ubuntu:hadoop
a95330d0086d9f074b4032edafe9773e2a20c666b77be90cee0887caa763de61
[root@centos7 linbin]# docker attach hadoop1
root@hadoop1:/# 
root@hadoop1:/# cat /etc/hostname
hadoop1
root@hadoop1:/# cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
192.168.10.11    hadoop1
192.168.10.12    hadoop2
192.168.10.13    hadoop3

可见，按预期创建了docker容器
[root@centos7 ~]# ping 192.168.10.11
PING 192.168.10.11 (192.168.10.11) 56(84) bytes of data.
64 bytes from 192.168.10.11: icmp_seq=1 ttl=64 time=0.085 ms
64 bytes from 192.168.10.11: icmp_seq=2 ttl=64 time=0.127 ms
宿主机上ping容器，可以ping通

16.格式化namenode
root@hadoop1:/opt/hadoop-3.0.3/bin# ./hadoop namenode -format
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.

WARNING: /opt/hadoop-3.0.3/logs does not exist. Creating.
2018-12-22 13:46:19,132 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop1/192.168.10.11
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.0.3

17. 
编辑 /opt/hadoop-3.0.3/sbin/start-dfs.sh 和 stop-dfs.sh
前面加入：
HDFS_DATANODE_USER=root

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

编辑 /opt/hadoop-3.0.3/sbin/start-yarn.sh
 和stop-yarn.sh 
    
前面加入：
YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

18.启动hadoop
service ssh restart
start-dfs.sh
能正常启动，提示hadoop2,hadoop3 的port：22 拒绝连接，说明配置没有问题了，只是hadoop2,hadoop3还未创建

19. 创建容器hadoop2,hadoop3

创建：docker run -itd --name hadoop2 --hostname hadoop2 --net mynetwork --ip 192.168.10.12  --add-host hadoop1:192.168.10.11 --add-host hadoop3:192.168.10.13  -d -P ubuntu:hadoop

连接：docker attach hadoop2
启动ssh：service ssh start

创建：docker run -itd --name hadoop3 --hostname hadoop3 --net mynetwork --ip 192.168.10.13  --add-host hadoop1:192.168.10.11 --add-host hadoop2:192.168.10.12  -d -P ubuntu:hadoop

连接：docker attach hadoop3
启动ssh：service ssh restart

20.配置完成，测试hadoop集群
在hadoop1启动
start-dfs.sh
start-yarn.sh

opt/hadoop-3.0.3/sbin# jps
5861 DataNode
6101 SecondaryNameNode
6517 ResourceManager
6806 Jps
6638 NodeManager
5727 NameNode

root@hadoop2:/opt/hadoop-3.0.3/sbin# jps
611 Jps
500 NodeManager
366 DataNode

root@hadoop3:/opt/hadoop-3.0.3/sbin# jps
400 NodeManager
266 DataNode
511 Jps

http://192.168.10.11:9870  查看hdfs目录
http://192.168.10.11:8088/cluster  yarn mapreduce任务查看

参考：

https://www.cnblogs.com/linux-wangkun/p/5766562.html  Docker中自动化搭建Hadoop2.6完全分布式集群
https://www.cnblogs.com/onetwo/p/6419925.html  使用Docker在本地搭建Hadoop分布式集群