Docker安装hadoop-cdh

Docker安装hadoop-cdh

第1章 环境

1.1 操作系统及docker版本

Centos7
Docker服务器两台，overlay网络。私有docker registry

1.2 安装前准备

关掉防火墙和selinux

1.3 需要的安装包

jdk-8u181-linux-x64.tar.gz
mysql-connector-java-5.1.47.tar.gz（去mysql官网下载，不一定非得是这个版本，详细版本参考java和mysql的jdbc对应图）
consul_1.3.0_linux_amd64.zip（overlay用到的存储，如果用容器启动consul这个安装包可以不下载）
cloudera-manager-centos7-cm5.7.6_x86_64.tar.gz
manifest.json
CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel.sha1
CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel

1.4 安装包所在网址

consul安装包所在网址
https://releases.hashicorp.com/consul/
hadoop的cdh相关安装包网址
http://archive.cloudera.com//cm5/cm/5/
http://archive.cloudera.com/cdh5/parcels/5.7/

1.5 下载安装包

wget命令下载安装包到docker宿主机/opt目录
jdk和mysql-connector-java-5.1.47.tar.gz 到java和mysql官网下载

wget https://releases.hashicorp.com/consul/1.3.0/consul_1.3.0_linux_amd64.zip
wget http://archive.cloudera.com//cm5/cm/5/cloudera-manager-el7-cm5.7.6_x86_64.tar.gz
wget http://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parce1
wget http://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parce1.sha1
wget http://archive.cloudera.com/cdh5/parcels/5.7/manifest.json

说明:所有的包都已经保存在百度网盘，可以直接用。

1.6 java和mysql的jdbc对应图

第2章 搭建私有registry

说明：由于自己定制的镜像在官网没有，并且docker有两台主机，另一个主机要想用定制的镜像，就需要搭建个docker仓库。
官方文档：
https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry

Running a registry only accessible on localhost has limited usefulness. In order to make your registry accessible to external hosts, you must first secure it using TLS.

使用TLS认证registry容器时，必须有证书。一般情况下，是要去认证机构购买签名证书。这里使用openssl生成自签名的证书。

2.1 生成自签名证书

#选择一台docker服务器操作

mkdir -p /opt/docker/registry/certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout /opt/docker/registry/certs/domain.key -x509 -days 365 -out /opt/docker/registry/certs/domain.crt
Generating a 4096 bit RSA private key
...
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:BeiJing
Locality Name (eg, city) [Default City] BeiJing
Organization Name (eg, company) [Default Company Ltd]:ceicloud
Organizational Unit Name (eg, section) []:edu
Common Name (eg, your name or your server's hostname) []:registry.docker.com
Email Address []:zhaolidong@ceicloud.com

2.2 创建带有TLS认证的registry容器

docker run -d --name registry2 -p 443:443 -v /opt/docker/registry/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:2

2.3 在每一个docker客户端宿主机上配置/etc/hosts，以使客户端宿主机可以解析域名”registry.docker.com”。并创建与这个registry服务器域名一致的目录（因为我这里的域名是假的）

#两台docker服务器都操作

vim /etc/hosts
172.16.206.32 registry.docker.com
cd /etc/docker/certs.d/
mkdir -p /etc/docker/certs.d/registry.docker.com:443

2.4 将证书 domain.crt 复制到每一个docker客户端宿主机

scp -p /opt/docker/registry/certs/domain.crt root@192.168.229.113:/etc/docker/certs.d/registry.docker.com\:443/ca.crt

#不需要重启docker

2.5 配置insecure_registry

证书不识别
The push refers to a repository [registry.docker.com/my-ubuntu]
Get https://registry.docker.com/v1/_ping: x509: certificate signed by unknown authority
解决

vim /etc/sysconfig/docker
OPTIONS='--selinux-enabled=false --insecure-registry registry.docker.com:443'

还是报错，将我们生成的crt文件内容放入系统的CA bundle文件当中，使操作系统信任我们的自签名证书，docker自然也就没问题了。CentOS 6 / 7中bundle文件的位置在

/etc/pki/tls/certs/ca-bundle.crt
cd /etc/docker/certs.d/registry.docker.com\:443/
cat ca.crt >>/etc/pki/tls/certs/ca-bundle.crt

重启docker

systemctl restart docker.service

#如果是其他Linux发行版，该文件的位置可能是下面这些，视情况而定：

/etc/ssl/certs/ca-certificates.crt
/etc/ssl/ca-bundle.pem
/etc/ssl/cert.pem
/usr/local/share/certs/ca-root-nss.crt
/etc/init.d/docker restart

2.6 push镜像到registry

Docker客户端现在可以使用其外部地址从您的注册表中获取并推送到您的注册表。
docker pull ubuntu:16.04
docker tag ubuntu:16.04 registry.docker.com/my-ubuntu
docker push registry.docker.com/my-ubuntu
docker pull registry.docker.com /my-ubuntu

2.7 列出私有仓库中的所有镜像

curl -X GET https://registry.docker.com:443/v2/_catalog -k
{“repositories”:[“my-ubuntu”]}

2.8 查看存储在registry:2宿主机上的镜像

在registry:2创建的私有仓库中，上传的镜像保存在容器的/var/lib/registry目录下。创建registry:2的容器时，会自动创建一个数据卷(Data Volumes)，数据卷对应的宿主机下的目录一般为：/var/lib/docker/volumes/XXX/_data。

ls /var/lib/docker/volumes/91a0091963fa6d107dc988a60b61790bba843a115573e331db967921d5e83372/_data/docker/registry/v2/repositories/my-ubuntu/
_layers  _manifests  _uploads

可以在创建registry:2的容器时，通过-v参数，修改这种数据卷关系：

–v /opt/docker/registry/data:/var/lib/registry

除了可以将数据保存在当前主机的文件系统上，registry也支持其他基于云的存储系统，比如S3，Microsoft Azure, Ceph Rados, OpenStack Swift and Aliyun OSS等。可以在配置文件中进行配置：https://github.com/docker/distribution/blob/master/docs/configuration.md#storage

第3章 创建overlay网络

3.1 搭建consul服务

wget https://releases.hashicorp.com/consul/1.3.0/consul_1.3.0_linux_amd64.zip
mv consul_1.3.0_linux_amd64.zip /opt/
cd /opt/
yum -y install unzip
unzip consul_1.3.0_linux_amd64.zip
mv consul /usr/bin/
consul agent -dev -ui -node=consul-dev -client=ip地址 2>&1 > /tmp/consul.log&

3.2 配置docker存储

vim /usr/lib/systemd/system/docker.service
[Service]
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --cluster-store=consul://consul主机的ip地址:8500 --cluster-advertise=节点的ip地址:2376
systemctl daemon-reload
systemctl restart docker

第4章 定制hadoop-cdh镜像

4.1 启动centos容器

docker pull centos
docker run  -itd --privileged --name centos centos  /usr/sbin/init
docker exec -it centos /bin/bash

#改root密码

passwd root

4.2 复制安装包到centos容器

docker cp /opt/jdk-8u181-linux-x64.tar.gz centos:/root
docker cp /opt/cloudera-manager-centos7-cm5.7.6_x86_64.tar.gz centos:/root/
docker cp /opt/CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel.sha1 centos:/root/
docker cp /opt/CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel centos:/root/
docker cp /opt/manifest.json centos:/root/

4.3 centos容器安装依赖

docker exec -it centos yum -y install psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap rpcbind fuse-libs redhat-lsb net-tools

4.4 centos容器配置jdk

docker exec -it centos mkdir /usr/java
docker exec -it centos mv  /root/jdk-8u181-linux-x64.tar.gz /usr/java
docker exec -it centos tar zxvf  /usr/java/jdk-8u181-linux-x64.tar.gz
docker exec -it centos /bin/bash
cat /etc/profile.d/java.sh                     
export JAVA_HOME=/usr/java/jdk1.8.0_181
export CLASSPATH=.:$JAVA_HOME/jre/lib/*:$JAVA_HOME/lib/*
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile.d/java.sh

4.5 centos容器配置mysql

docker exec -it centos /bin/bash
yum -y install mariadb mariadb-server
cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

transaction-isolation = READ-COMMITTED
log_bin=/var/lib/mysql/mysql_binary_log
binlog_format=mixed
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT
key_buffer=16M
key_buffer_size=32M
thread_stack=256K
thread_cache_size=64
query_cache_limit=8M
query_cache_size=64M
query_cache_type=1
max_connections=550
read_buffer_size=2M
read_rnd_buffer_size=16M
sort_buffer_size=8M
join_buffer_size=8M
innodb_log_buffer_size=64M
#innodb_buffer_pool_size=4G
innodb_buffer_pool_size=512M
innodb_thread_concurrency=8
innodb_log_file_size=512M

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
sql_mode=STRICT_ALL_TABLES
!includedir /etc/my.cnf.d
systemctl enable mariadb
systemctl start mariadb

#建库

create database hive default character set utf8;
create database rman default character set utf8;
create database oozie default character set utf8;
grant all on *.* to 'root'@'%' identified by 'newton';
4.6 centos容器配置chrony
docker exec -it centos /bin/bash
yum -y install chrony
cat /etc/chrony.conf
server ntp1.aliyun.com iburst
systemctl enable chronyd
systemctl start chronyd

#验证

chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 120.25.115.20                 2  10   377   448   -581us[ -295us] +/-   21ms

4.7 centos容器配置Cloudera Manager

docker exec -it centos /bin/bash

#建立cm用户

useradd --system --home=/opt/cm-5.7.6/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment="Cloudera SCM User" cloudera-scm
usermod -a -G root cloudera-scm
echo USER=\"cloudera-scm\" >> /etc/default/cloudera-scm-agent
echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin" >> /etc/sudoers

#解压cm安装包

tar zxvf /root/cloudera-manager-centos7-cm5.7.6_x86_64.tar.gz -C /opt

#配置jdbc

tar zxvf /root/mysql-connector-java-5.1.47.tar.gz 
cd /root/mysql-connector-java-5.1.47/
/usr/share/java/
cp mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar

#填充数据库

/opt/cm-5.7.6/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p --scm-host localhost scm scm scm

#配置cm代理

vi /opt/cm-5.7.6/etc/cloudera-scm-agent/config.ini 


[General]
# Hostname of the CM server.
server_host=cdh1

#将parcel相关的三个文件复制到/opt/cloudera/parcel-repo/

cp /root/CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel /opt/cloudera/parcel-repo/
cp /root/CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel.sha1 /opt/cloudera/parcel-repo/
cp /root/manifest.json /opt/cloudera/parcel-repo/

#改名

mv /opt/cloudera/parcel-repo/CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.7.6-1.cdh5.7.6.p0.6-el7.parcel.sha

#修改属主

chown -R cloudera-scm:cloudera-scm /opt/cm-5.7.6/
chown -R cloudera-scm:cloudera-scm /opt/cloudera/

#建立/opt/cloudera/parcels目录，并修改属主

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels

#配置cm的环境变量

cat /etc/profile.d/cloudera-scm.sh 
export PATH=$PATH:/opt/cm-5.7.6/etc/init.d
source /etc/profile.d/cloudera-scm.sh

4.8 centos容器配置ssh免密登录

yum -y install openssh openssh-server openssh-clients
ssh-keygen #一路回车
ssh-copy-id localhost #一路回车

4.9 制作cdh镜像

exit退出容器

docker stop centos
docker commit -a "zld" -m "这是hadoop-CDH的镜像" centos hadoop-cdh:v1

4.10 将hadoop-cdh镜像推送到私有仓库

docker tag hadoop-cdh:v1 registry.docker.com/hadoop-cdh:v1
docker push registry.docker.com/hadoop-cdh:v1

查看镜像

curl -X GET https://registry.docker.com:443/v2/_catalog -k

第5章 启动cdh容器

5.1 在docker1上启动cdh1

docker run -itd –p 7180:7180 -h cdh1 --name cdh1 --network ov-net hadoop-cdh:v1
docker exec -it cdh1 /bin/bash

#启动cm server

cloudera-scm-server start

说明：此步骤需要运行一段时间，用下面的命令查看启动情况

cat /opt/cm-5.7.6/log/cloudera-scm-server/cloudera-scm-server.log

#启动cm-agent

mkdir /opt/cm-5.7.6/run/cloudera-scm-agent/
chown -R cloudera-scm:cloudera-scm /opt/cm-5.7.6/run/cloudera-scm-agent/
cloudera-scm-agent start

5.2 在docker2上启动cdh2

docker run -itd –p 7180:7180 -h cdh2 --name cdh2 --network ov-net hadoop-cdh:v1
docker exec -it cdh2 /bin/bash

#启动cm-agent

mkdir /opt/cm-5.7.6/run/cloudera-scm-agent/
chown -R cloudera-scm:cloudera-scm /opt/cm-5.7.6/run/cloudera-scm-agent/
cloudera-scm-agent start

第6章 web界面配置cdh

6.1 登录

浏览器输入启动cdh cm server的docker的主机的ip:7180
登录名和密码都是admin