mesos+zk+Marathon实现对docker容器的管理

1、系统环境:CentOS Linux release 7.6.1810 (Core)

五台机器

[root@m5 mesos]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.81  m1          #安装mesos,zk
192.168.137.82  m2          #安装mesos,zk
192.168.137.83  m3          #安装mesos,zk
192.168.137.85  m5          #安装mesos,docker
192.168.137.86  m6          #安装mesos,docker

 

2、配置必要环境，五台机器全部执行

关闭防火墙,selinux,ssh加速等，执行

[root@m5 init]# cat 1.sh
#!/bin/bash
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
sed -i 's/GSSAPIAuthentication yes /GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i '/#UseDNS yes/a\UseDNS no' /etc/ssh/sshd_config
service firewalld stop
chkconfig firewalld off

必要的文件存放在这个路径，方便脚本调用

[root@m5 mesos]# ls /shared/app/install/tar.gz/mesos/         
marathon-0.15.2.tgz  mesos-1.6.1.tar.gz  pip-1.5.4.tar.gz
mesos-0.25.0.tar.gz  others              zookeeper-3.4.12.tar.gz

配置JAVA环境，注意下载jdk-8u181-linux-x64.tar.gz文件的存放路径，执行

[root@m5 mesos]# cat java.sh 
#!/bin/bash
cd /shared/app
if [ ! -d /tmp/java ];then                                                      
  mkdir -p /tmp/java
fi 
tar -zxvf jdk-8u181-linux-x64.tar.gz -C /tmp/java/
if [ -d /usr/local/java ];then
  mv /usr/local/java /usr/local/javaold
fi 
mv /tmp/java/jdk1.8.0_181 /usr/local/java
cp -r /etc/profile /etc/profile.bak
echo "export JAVA_HOME=/usr/local/java" >> /etc/profile
echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile
echo "export CLASSPATH=\$JAVA_HOME/jre/lib/ext:\$JAVA_HOME/lib/tools.jar" >> /etc/profile
source /etc/profile
java -version

 

安装必要的软件

[root@m5 mesos]# cat yum.sh 
#!/bin/bash
yum -y groupinstall "Development Tools"
wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
cat >> /etc/yum.repos.d/wandisco-svn.repo <<EOF
[WANdiscoSVN]
name=WANdisco SVN Repo 1.9
enabled=1
baseurl=http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF
sleep 3
yum install -y apache-maven python-devel python-six python-virtualenv java-1.8.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel apr-devel subversion-devel apr-util-devel cyrus-sasl-md5

 

3、安装mesos，我这次是编译安装，事实证明非常慢，影响效率

[root@m5 mesos]# cat mesos.sh 
#!/bin/bash
cd /shared/app/install/tar.gz/mesos/
tar -zxvf mesos-0.25.0.tar.gz -C /tmp/
cd /tmp/mesos-0.25.0
mkdir build
cd build
../configure
make
make check
make install
echo "export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so" >> /etc/profile
echo "export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so" >> /etc/profile
source /etc/profile

 

4、安装zookeeper

[root@m5 mesos]# cat zk.sh 
#!/bin/bash
if [ ! -d /zk ];then                                                      
  mkdir -p /zk
fi 
cd /shared/app/install/tar.gz/mesos/
tar -zxvf zookeeper-3.4.12.tar.gz -C /zk
mv /zk/zookeeper-3.4.12 /zk/zookeeper
cp /zk/zookeeper/conf/zoo_sample.cfg /zk/zookeeper/conf/zoo.cfg
mkdir /zk/zookeeper/data
mkdir /zk/zookeeper/datalog
sed -i 's/dataDir\=\/tmp\/zookeeper/dataDir\=\/zk\/zookeeper\/data/' /zk/zookeeper/conf/zoo.cfg
echo "dataLogDir=/zk/zookeeper/datalog" >> /zk/zookeeper/conf/zoo.cfg
echo "server.1=192.168.137.81:2888:3888" >> /zk/zookeeper/conf/zoo.cfg 
echo "server.2=192.168.137.82:2888:3888" >> /zk/zookeeper/conf/zoo.cfg 
echo "server.3=192.168.137.83:2888:3888" >> /zk/zookeeper/conf/zoo.cfg 
echo $1 > /zk/zookeeper/data/myid
/zk/zookeeper/bin/zkServer.sh start /zk/zookeeper/conf/zoo.cfg

执行的时候带参数执行，m1,m2,m3三台使用1,2,3,的ID,sh -x zk.sh 1,  sh -x zk.sh 2,  sh -x zk.sh 3 这样执行

安装完以后查看状态，有一个为leader，两个为follower

[root@m3 zookeeper]# /zk/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /zk/zookeeper/bin/../conf/zoo.cfg
Mode: leader

[root@m2 ~]# /zk/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /zk/zookeeper/bin/../conf/zoo.cfg
Mode: follower

[root@m1 ~]# /zk/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /zk/zookeeper/bin/../conf/zoo.cfg
Mode: follower

 

5、m1,m2,m3都启动mesos-master服务，会占用一个终端

mesos-master --work_dir=/mesos/data --log_dir=/mesos/logs --no-hostname_lookup --ip=0.0.0.0 --zk=zk://192.168.137.81:2181/mesos --quorum=1

mesos-master --work_dir=/mesos/data --log_dir=/mesos/logs --no-hostname_lookup --ip=0.0.0.0 --zk=zk://192.168.137.82:2181/mesos --quorum=1

mesos-master --work_dir=/mesos/data --log_dir=/mesos/logs --no-hostname_lookup --ip=0.0.0.0 --zk=zk://192.168.137.83:2181/mesos --quorum=1

此时访问mesos会自动跳转到master的站点

 

 

 

到master的站点部署Marathon，本实验为m1机器192.168.137.81

Marathon版本得和mesos相兼容

cd /shared/app/install/tar.gz/mesos/

tar -zxvf marathon-1.4.6.tgz -C /tmp/

mv /tmp/marathon-1.4.6/ /marathon

cd /marathon/

./bin/start --hostname 192.168.137.81 --master zk://192.168.137.81:2181,192.168.137.82:2181,192.168.137.83:2181/mesos --http_address 0.0.0.0

 

访问界面如下：

 

 

6、m5,m6启动mesos-slave,两台机器都一样

先安装好docker

mkdir -p  /mesos/data  /mesos/logs

mesos-slave --containerizers="mesos,docker" --work_dir=/mesos/data --log_dir=/mesos/logs --master=zk://192.168.137.81:2181,192.168.137.82:2181,192.168.137.83:2181/mesos --no-hostname_lookup --ip=0.0.0.0

slave节点加进来以后mesos上面可以看得到

 

7、推送任务至marathon

写好docker启动的参数，保存为json格式，在任意机器上面post至marathon的api接口完成推送

[root@m6 tmp]# cat ng.json 
{
"id":"/nginx",
"container":{
"type":"DOCKER",
"docker":{
"image":"nginx",
"network":"HOST",
"parameters":[],
"privileged":false,
"forcePullImage":false
}
},
"cpus":0.5,
"mem":256,
"instances":1
}

 

[root@m6 tmp]# curl -X POST http://192.168.137.81:8080/v2/apps -d @ng.json -H "Content-type: application/json"          
{"id":"/nginx","cmd":null,"args":null,"user":null,"env":{},"instances":1,"cpus":0.5,"mem":256,"disk":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"ports":[0],"requirePorts":false,"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"nginx","network":"HOST","privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":1,"maximumOverCapacity":1},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2019-05-30T08:44:34.231Z","tasksStaged":0,"tasksRunning":0,"tasksHealthy":0,"tasksUnhealthy":0,"deployments":[{"id":"288543d7-3770-49bb-992f-8af88992ee6f"}],"tasks":[]}

 

实际操作的时候，失败了很多次，发现是slave服务器docker配置的原因

报错信息如下：

I0530 16:36:03.093997 46571 exec.cpp:134] Version: 0.25.0
I0530 16:36:03.130642 46578 exec.cpp:208] Executor registered on slave 2219633f-9257-4b0d-be8b-0f82ed10be18-S0
/usr/bin/docker-current: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/673b8393af503f0e5832ad052fee0c2b6a5cd341f0e7b8955b5cd96f525b95ee-init/merged: invalid argument.
See '/usr/bin/docker-current run --help'.
W0530 16:36:03.130642 46580 logging.cpp:81] RAW: Received signal SIGTERM from process 6373 of user 0; exiting

处理过程如下：

[root@m6 init.d]# service docker stop
Redirecting to /bin/systemctl stop docker.service
[root@m6 init.d]# rm -rf /var/lib/docker

vi /etc/sysconfig/docker-storage

改成：

DOCKER_STORAGE_OPTIONS="--storage-driver overlay"

vi /etc/sysconfig/docker

去掉option的–selinux-enabled

[root@m6 init.d]# service docker start
Redirecting to /bin/systemctl start docker.service

再次推送，其实通过WEB界面推送也是一样的

从安全的角度考虑，MESOS应做好用户认证和授权，实验中为测试效果，没有使用认证

 

8、验证任务的执行

 

在MESOS上面查看到任务在192.168.137.86机器执行

192.168.137.86机器上面容器已经启动

[root@m6 tmp]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
39d83df036de        nginx               "nginx -g 'daemon ..."   2 hours ago         Up 2 hours                              mesos-2219633f-9257-4b0d-be8b-0f82ed10be18-S0.83e4ace4-25ca-47ad-a98c-510851e8e8ed

验证访问