How to deploy CI test env (Jenkins+Docker)

1. Deploy jenkins-master

(1).  Add the Jenkins repository to the yum repos, and install Jenkins

[root@jenkinsmaster ~]# wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
[root@jenkinsmaster ~]# rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
[root@jenkinsmaster ~]# yum install jenkins

(2). Install Open

1. Check the java's version, if you get output similar to the following, it means you're using the default (GCJ) version of Java:

[root@jenkinsmaster ~]# java -version
java version "1.7.0_99"
OpenJDK Runtime Environment (rhel-2.6.5.1.el6-x86_64 u99-b00)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

2. Remove the GCJ version of Java and install OpenJDK

[root@jenkinsmaster ~]# yum remove java
[root@jenkinsmaster ~]# subscription-manager register --username=XXX --password=XXX --auto-attach
[root@jenkinsmaster ~]# yum install java-1.7.0-openjdk

(3). Start jenkins service

[root@jenkinsmaster ~]# service iptables stop
[root@jenkinsmaster ~]# service jenkins restart
[root@jenkinsmaster ~]# chkconfig jenkins on

(4). Open jenkins-master webUI

http://[jenkins-mster-ip]:8080

If see the following screenshot, it success to install and start jenkins

Input the admin password in /var/lib/jenkins/secrets/initialAdminPassword, then modify admin's password, you can bring up jenkins successfully.

(5). Install plugins

In jenkins master webUI,  go to Manage jenkins-->Manage plugins-->Available, choose the following plugins to install it.

++++++++++++++++++++++++++++++++++++++

AnsiColor

Build With Parameters

BuildResultTrigger Plug-in

Build Pipeline Plugin

Copy To Slave Plugin

Configure Job Column Plugin

Copy Artifact Plugin

docker plugin
docker slave plugin
docker step plugin

Environment File Plugin

Environment Injector Plugin

Kubernetes plugin

Multiple SCMs plugin
Multijob plugin
Nested View Plugin

Python Plugin

Rebuilder
URLTrigger Plug-in

xunit plugin

++++++++++++++++++++++++++++++++++++++

 

2. Deploy docker cloud on RHEL6(for server)

This docker cloud used to auto deploy Satellite/SAM and slave machine.

(1).  Add the docker repository to the yum repos, and install docker-engine

[root@dockercloud yum.repos.d]# cat docker.repo 
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/6/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

[server]
name=server
baseurl=http://download.eng.pek2.redhat.com//pub/rhel/released/RHEL-6/6.8/Server/x86_64/os/
enable=1
gpgcheck=0

(2).  Install docker-engine and start docker service

[root@dockercloud yum.repos.d]# yum install -y docker-engine

[root@dockercloud yum.repos.d]# service docker restart
[root@dockercloud yum.repos.d]# chkconfig docker on

(3).  Check docker engine run normally

[root@dockercloud yum.repos.d]# docker run hello-world

[root@dockercloud yum.repos.d]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
hello-world latest 7a5a2d73abce 3 weeks ago 1.84 kB

Since image "hello-world" has been downloaded successfully, docker engine run normally.

(4).   Configure network to bridge and intall pipework.

1. Configure a bridge as the following files:

[root@dockercloud ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 
DEVICE="eth1"
#BOOTPROTO="dhcp"
HWADDR="52:54:00:2C:27:82"
#NM_CONTROLLED="yes"
#ONBOOT="yes"
TYPE="Ethernet"
#UUID="9dbf0196-4c30-4751-a704-5d2f77c8fa86"
BRIDGE="br0"
[root@dockercloud ~]# cat /etc/sysconfig/network-scripts/ifcfg-br0 
DEVICE=br0
ONBOOT=yes
TYPE=Bridge
DELAY=0
BOOTPROTO=dhcp

[root@dockercloud ~]# reboot

2. Install pipework. 
[root@dockercloud ~]# git clone https://github.com/jpetazzo/pipework.git
[root@dockercloud ~]# cp pipework/pipework /usr/local/bin/

3. Update iproute package .

Check iproute package, if it show as the following version, it need to update

[root@dockercloud ~]# rpm -q iproute
iproute-2.6.32-54.el6.x86_64                            ====>It is wrong

[root@dockercloud ~]# rpm -Uvh https://repos.fedorapeople.org/openstack/EOL/openstack-grizzly/epel-6/iproute-2.6.32-130.el6ost.netns.2.x86_64.rpm

[root@dockercloud ~]# rpm -q iproute             ====>It is corret
iproute-2.6.32-130.el6ost.netns.2.x86_64

[NOTE]: If it has not updated to iproute-2.6.32-130.el6ost.netns, it can't get containter's ip and with error info after run

"pipework [BRIGE_NAME] [CONTERNER_NAME] dhclient "

eg: [root@hp-z220-14 ~]# pipework br0 ccc dhclient
Object "netns" is unknown, try "ip help".

(5).  Open docker remote api

Add "other_args='-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock'" to  /etc/sysconfig/docker, the same as the following:

[root@dockercloud ~]# cat /etc/sysconfig/docker
other_args='-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock'
[root@dockercloud ~]# service docker restart

(6).  Configure iptables

[root@dockercloud ~]# iptables -F

[root@dockercloud ~]# service docker restart

[NOTE]:

How to remove docker-engine?

[root@dockercloud yum.repos.d]# yum -y remove docker-engine

[root@dockercloud yum.repos.d]# rm -rf /var/lib/docker

 

3. Deploy docker cloud on RHEL7(for server)

This docker cloud used to auto deploy Satellite/SAM and slave machine.

(1).  Add the docker repository to the yum repos, and install docker-engine

[root@dockercloud yum.repos.d]# cat docker.repo 
[dockerrepo] 
name=Docker Repository 
baseurl=https://yum.dockerproject.org/repo/main/centos/7/ 
enabled=1 
gpgcheck=1 
gpgkey=https://yum.dockerproject.org/gpg 
[server] 
name=server
baseurl=http://download.eng.pek2.redhat.com//pub/rhel/released/RHEL-7/7.3/Server/x86_64/os/
enable=1
gpgcheck=0

(2).  Install docker-engine and start docker service

[root@dockercloud yum.repos.d]# yum install -y docker-engine

[root@dockercloud yum.repos.d]# systemctl restart docker
[root@dockercloud yum.repos.d]# chkconfig docker on

(3).  Check docker engine run normally

[root@dockerrhel7-2 network-scripts]# docker pull registry.access.redhat.com/rhel6.9
Using default tag: latest
latest: Pulling from rhel6.9
529f75f19b13: Pull complete 
d672225e8e59: Pull complete 
Digest: sha256:f48d52450ab17c39cdc009cd1aea3fa7017c680d75774318ac7058e44c7b8467
Status: Downloaded newer image for registry.access.redhat.com/rhel6.9:latest

[root@dockerrhel7-2 network-scripts]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.access.redhat.com/rhel6.9 latest 5ddc947ca220 11 days ago 199MB

(4).   Configure network to bridge and intall pipework.
1. Get HWADDR and network name

[root@dockerrhel7 yum.repos.d]# ifconfig

ens32: flags=4163 mtu 1500    ====>ens32 is network name
ether 00:0c:29:f1:67:dc txqueuelen 1000 (Ethernet)        ======>00:0c:29:f1:67:dc is HWADDR
RX packets 18342808 bytes 13383820065 (12.4 GiB)
RX errors 0 dropped 9466 overruns 0 frame 0
TX packets 3742629 bytes 1264573274 (1.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

2. Configure a bridge as the following files:

[root@dockerrhel7 yum.repos.d]# cat /etc/sysconfig/network-scripts/ifcfg-ens32 
# Generated by dracut initrd
NAME="ens32"
DEVICE="ens32"
HWADDR="00:0c:29:f1:67:dc"     =====>The same as above
TYPE="Ethernet"
BRIDGE="br0"
[root@dockerrhel7 yum.repos.d]# cat /etc/sysconfig/network-scripts/ifcfg-br0 
DEVICE=br0
ONBOOT=yes
TYPE=Bridge
DELAY=0
BOOTPROTO=dhcp

[root@dockercloud ~]# reboot

2. Install pipework. 
[root@dockercloud ~]# git clone https://github.com/jpetazzo/pipework.git
[root@dockercloud ~]# cp pipework/pipework /usr/local/bin/

3. Update iproute package .

Check iproute package, if it show as the following version, it need to update

[root@dockerrhel7-2 ~]# rpm -q iproute
iproute-3.10.0-74.el7.x86_64

(5).  Open docker remote api

• Configure "ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock" in  /usr/lib/systemd/system/docker.service, the same as the following:

[root@dockerrhel7 /]# cat /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock                =====>The updated part
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

 

• Reload and restart docker to make config take effect.
  [root@dockerrhel7-2 network-scripts]# systemctl daemon-reload && systemctl restart docker
  [root@dockerrhel7-2 network-scripts]# ps -ef|grep docker
  avahi 693 1 0 08:35 ? 00:00:00 avahi-daemon: running [dockerrhel7-2.local]
  root 4645 1 2 09:34 ? 00:00:00 /usr/bin/dockerd -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock    ====> It means the config is success.
  root 4651 4645 0 09:34 ? 00:00:00 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
  root 4745 2854 0 09:34 pts/0 00:00:00 grep --color=auto docker

(6).  Configure firewall

[root@dockercloud ~]# /usr/sbin/iptables -F

[root@dockercloud ~]# systemctl restart docker

 

[NOTE]:

How to remove docker-engine?

[root@dockercloud yum.repos.d]# yum -y remove docker-engine

[root@dockercloud yum.repos.d]# rm -rf /var/lib/docker

 

4. Deploy docker cloud (for RHEL host)

The same as it on above "Deploy docker cloud on RHEL6/RHEL7(for server)".

 

5. Config jenkins-master

(1).  Add the docker cloud to jenkins-master

In jenkins master webUI,  go to Manage jenkins-->Configuration system-->Cloud-->Add a new cloud-->Docker. add two docker clouds which configured in step2 and step3.  please config the following parameters:

Name: A name for this Docker Cloud, by define.

Docker URL: The URL to use to access your Docker server API. Eg: tcp://[cloud_ip]:4243

Docker Image: Image name  that you wish docker to run. Eg: rhel68

Labels: Labels (or tags) are used to group multiple agents into one logical group. by define.

Launch method: Choose "Docker SSH computer launcher"

Credential: Click add-->Jenkins, input username/password which used to ssh login docker cloud.After create successfully, choose this credential.

After setting sucessfully, it should the same as screenshot as the following:

(2).  Configure extended E-mail Notification

In jenkins master webUI,  go to Manage jenkins-->Configuration system-->Extended E-mail Notification.please config the following parameters:

SMTP Server: Name of the mail server. Eg: smtp.corp.redhat.com

Default user E-mail suffix: users' e-mail addresses can be computed automatically by simply adding a suffix. Eg: @redhat.com

Default Recipients: Customize the default recipient list of the email notifications Eg: xxx.redhat.com

Default subject: Customize the default subject line of the email notifications, by define. Eg: $PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS!

Default Content: Customize the default content of the email notifications, by define. Eg: $PROJECT_NAME - Build # $BUILD_NUMBER - $BUILD_STATUS:
Check console output at $BUILD_URL to view the results.

After setting sucessfully, it should the same as screenshot as the following:

(3).  Add Node.

In jenkins master webUI,  go to Manage jenkins-->Manage nodes-->New nodes,please config the following parameters:

Name: The uniquely identifies an agent, by define. eg: cloud-server
# of executors: The maximum number of concurrent builds that Jenkins may perform on this agent. eg: 10
Remote root directory: A directory dedicated to Jenkins, by define. eg:/home
Labels : Used to group multiple agents into one logical group, by define.
Usage: Choose "use this node as much as possible"
Launch method: choose "launch slave agent via SSH"
Host: Input the remote host ip
Credentials: credentials to be used for logging in to the remote host.eg: root/redhat

Host Key Verification Strategy: Manually trusted key Verification Strategy
Availablity: Choose "keep the agent online as much as possible"

After setting sucessfully,   it should the same as screenshot as the following:

 

6. Install and config Jenkins-job-builder

(1).  Install JJB

# git clone https://git.openstack.org/openstack-infra/jenkins-job-builder

# cd jenkins-job-builder
# pip install jenkins-job-builder

(2).  Configure JJB to make it connect with Jenkins master

# cat [config file]

[jenkins]
user=admin                               ----> Jenkins master's username
password=admin                      ----> Jenkins master's password
url=http://10.73.131.121:8080     ----> Jenkins master's url

(3). Test jobs with JJB

# jenkins-jobs --ignore --conf [config file] test [jobs file]

eg: # jenkins-jobs --ignore --conf config test satellite62-install.yaml

(4). Update jobs with JJB

# jenkins-jobs --ignore --conf [config file] update [jobs file]

# jenkins-jobs --ignore --conf config update satellite62-install.yaml

 

7. Q&A

Problem1:  Failed to bring up slave on jenkins master, it will show "Bad template 'rhel68' in cloud 'cloud1.server.redhat.com': 'java.net.NoRouteToHostException: No route to host (Host unreachable)'. Trying next template..."  or "iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 33567 -j DNAT --to-destination 172.17.0.80:22 ! -i docker0: iptables: No chain/target/match by that name"

Resolve:  

On jenkins: stop the job which used to run slave on docker.

On docker host :  

1. Check iptables service, make sure iptables service is running

2. Check iptables's rules, it must has DOCKER chain as the following:                                                 

[root@dhcp-128-227 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination 
DOCKER all -- 0.0.0.0/0 0.0.0.0/0                                                     ====> It should exist
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (1 references)                                                     ====> It should exist
target prot opt source destination 
ACCEPT tcp -- 0.0.0.0/0 172.17.0.1 tcp dpt:22

If docker chain is not exist, you can clean all config of iptables then restart docker service as the following:

2.1. Clean all config of iptables : [root@dockercloud ~]# iptables -F                              
2.2. Show all existed rules: [root@dockercloud ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination 
2.3. Restart docker service
[root@dockercloud ~]# service docker restart
Stopping docker: [ OK ]
Starting docker: [ OK ]

3. On jenkins, rerun the job which used to create slave on docker.

Problem2: Failed to transfer environment variables to another job

Resolve:

On jenkins machine: 

# cat /etc/sysconfig/jenkins

Update "JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true""