安装k8s+docker集群环境（基于centos7系统）

安装k8s+docker集群环境（基于centos7系统）

集群机器：
centos-master = 192.168.121.9
centos-minion-1 = 192.168.121.65
centos-minion-2 = 192.168.121.66
centos-minion-3 = 192.168.121.67
[centos-master]
配置yum源
vi /etc/yum.repos.d/virt7-docker-common-release.repo
[virt7-docker-common-release]
name=virt7-docker-common-release
baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/
gpgcheck=0
加载安装包
yum repolist virt7-docker-common-release virt7-docker-common-release 15
安装集群必要软件—-etcd/flannel/kubernetes
Etcd服务在k8s集群中用于配置共享和服务发现。
Flannel是针对k8s设计一个网络规划服务，让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址。

yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel
如果集群中没有使用DNS解析，那么需要在master节点的/etc/hosts中添加node的主机名信息，比如：
echo “192.168.121.9 centos-master
192.168.121.65 centos-minion-1
192.168.121.66 centos-minion-2
192.168.121.67 centos-minion-3” >> /etc/hosts
修改配置master节点的kubernetes配置文件
vi /etc/kubernetes/config
#表示错误日志记录到文件还是输出到stderr
KUBE_LOGTOSTDERR="–logtostderr=true"

#日志等级
KUBE_LOG_LEVEL="–v=0"

#允许运行特权容器
KUBE_ALLOW_PRIV="–allow-privileged=false"

#apiserver的服务地址，controller-manager、scheduler及kubelet都会用到这个配置，这里配置为192.168.121.9
KUBE_MASTER="–master=http://192.168.121.9:8080"
k8s集群中涉及的端口比较多，所以centos中的防火墙需要设置对应规则。
setenforce 0
systemctl stop firewalld.service
sed –i ‘/^SELINUX=/ s/enforcing/disabled /’ /etc/selinux/config
修改配置master节点的etcd配置文件
etcd服务的可调参数比较多，根据需求开启对应功能，此处我们大概调整如下几个功能：
vi /etc/etcd/etcd.conf

[member]

#etcd名称
ETCD_NAME=default

#etcd数据存储位置
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

#监听的端口
ETCD_LISTEN_CLIENT_URLS=“http://0.0.0.0:2379”

#[cluster]
#集群监听的端口
ETCD_ADVERTISE_CLIENT_URLS=“http://0.0.0.0:2379”
修改配置master节点的apiserver配置文件
vi /etc/kubernetes/apiserver
#监听的接口，如果配置为127.0.0.1则只监听localhost，配置为0.0.0.0会监听所有接口，这里配置为0.0.0.0
KUBE_API_ADDRESS="–insecure-bind-address=0.0.0.0"

#apiserver的监听端口，默认8080
KUBE_API_PORT="–port=8080"

#minion上kubelet监听的端口，默认10250
KUBELET_PORT="–kubelet-port=10250"

#etcd服务地址，前面已经启动了etcd服务,端口为2379
KUBE_ETCD_SERVERS="–etcd-servers=http://127.0.0.1:2379"

#kubernetes可以分配的ip的范围，kubernetes启动的每一个pod以及serveice都会分配一个ip地址，将从这个范围分配
KUBE_SERVICE_ADDRESSES="–service-cluster-ip-range=10.254.0.0/16"

#需要额外添加的配置项，简单地启用一个集群无需配置
KUBE_API_ARGS=""
启动并且配置etcd的网段，此网段一定是未被使用的
systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config “{ “Network”: “172.30.0.0/16”, “SubnetLen”: 24, “Backend”: { “Type”: “vxlan” } }”
修改配置master节点的flanneld配置文件/etc/sysconfig/flanneld
vi /etc/sysconfig/flanneld
#etcd的访问地址及端口
FLANNEL_ETCD_ENDPOINTS=“http://192.168.121.9:2379”

#服务范围
FLANNEL_ETCD_PREFIX="/kube-centos/network"

#其他
FLANNEL_OPTIONS=""
启动k8s集群
vi k8s.sh
for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
chmod +x k8s.sh
./k8s.sh
[centos-nodes]
修改配置nodes节点kubelet的配置文件
vi /etc/kubernetes/kubelet
#minion监听的地址，每个minion根据实际的ip配置,这里配置为0.0.0.0
KUBELET_ADDRESS="–address=0.0.0.0"

#监听的端口
KUBELET_PORT="–port=10250"

#apiserver的访问地址及端口
KUBELET_API_SERVER="–api-servers=http://192.168.121.9:8080"

#额外增加的参数
KUBELET_ARGS="–logtostderr=false --v=0 --log-dir=/data/logs/kubernetes"
修改配置nodes节点flanneld的配置文件
vi /etc/sysconfig/flanneld
#etcd的访问地址及端口
FLANNEL_ETCD=“http://192.168.121.9:2379”

#etcd服务范围
FLANNEL_ETCD_KEY="/kube-centos/network"

启动k8s集群服务
vi K8s.sh
for SERVICES in kube-proxy kubelet flanneld docker; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
chmod +x K8s.sh
./K8s.sh
设置kubectl的配置文件
kubectl config set-cluster default-cluster --server=http://192.168.121.9:8080
kubectl config set-context default-context --cluster=default-cluster --user=default-admin
kubectl config use-context default-context
 
[centos-master]
搭建私有库
下载并启动私有库
docker run --name registry -v /etc/localtime:/etc/localtime -v /opt/registry:/var/lib/registry -p 5000:5000 -itd docker.io/registry

#–name 表示启动的容器后名称，此处为registry
#-v 表示挂载路径 格式为宿主机路径:容器内路径
#-p 表示映射端口 格式为宿主机端口:容器内端口
#-itd docker的内部参数，此处声明后台运行容器并分配一个伪终端并绑定到容器的标准输入上，后跟镜像名称此处为docker.io/registry
创建一个secret服务，用于k8s调度私有库容器时的“令牌”。简单来说，secret服务就是一个存储密码的服务
kubectl create secret docker-registry registrykey --docker-server=registry.evehicle.cn --docker-username=docker --docker-password=docker --docker-email=lienhua@zhongchuangsanyou.com
kubectl get secret
此时登录时会提示认证错误: > /etc/hosts> /etc/hosts> /etc/hos> s
[centos-master]:docker login -u docker -p docker –e> /etc/h osts lienhua@zhongchuangsanyou.com registry.evehicle.cn> /etc/ho sts
Flag --email has been deprecated, will be removed in 1.13. > /
Error response from daemon: login attempt to> /etc/hosts> /e tc/hosts https://registry.evehicle.cn/v2/ failed with status: 401 Unauthorized
这是因为Docker官方是推荐采用Secure Registry的工作模式的，即transport采用> /tls。这样我们就需要为Registry配置tls所需的key和crt文件了> /etc/hosts /e

配置nginx反向代理
vi /etc/nginx/conf/registry.evehicle.cn.conf
#For versions of nginx > 1.3.9 that include chunked transfer encoding support
#Replace with appropriate values where necessary
upstream docker-registry {
server 192.168.121.9:5000;
#server 10.44.170.95:5000;
}
#uncomment if you want a 301 redirect for users attempting to connect
#on port 80
#NOTE: docker client will still fail. This is just for convenience
~#server {
~# listen *:80;
~# server_name my.docker.registry.com;
~# return 301 https://$serve{r}_{n}ame$request_uri;
~# }
server {
listen 443;
server_name registry.evehicle.cn;
ssl on;
ssl_certificate ssl/registry.evehicle.cn.crt;
ssl_certificate_key ssl/registry.evehicle.cn.key;
client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location / {
auth_basic “Restricted”;
auth_basic_user_file passwd;
add_header ‘Docker-Distribution-Api-Version’ ‘registry/2.0’ always;
proxy_pass http://docker-registry;
proxy_set_header Host $http_host; # required for docker client’s sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client’s IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
location /_ping {
auth_basic off;
include docker-registry.conf;
}
location /v1/_ping {
auth_basic off;
include docker-registry.conf;
}
location /v2/_ping {
auth_basic off;
include docker-registry.conf;
}
}

将key及crt证书文件放到…/ssl目录下。使用htpasswd生成密码放于./上一级目录
htpasswd -bcm passwd docker docker
#-c：创建一个加密文件
#-m：md5加密，默认可不填写
#-b：表示用户名密码在命令行中一并输入，不用分别填写
再次登录
docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn

Login Succeeded
表示成功，此时再pull\push既在私有库中进行
构建服务
docker的本意是将代码包含在容器内制作成镜像形成“产品”。但出于公司的（频繁修改代码及服务器资源受限）的特殊性，我们将代码以“外挂”的形式运行在宿主机上。下面以部署官网（apache）服务为例：
从docker的公有库里下载centos7的原生镜像
[centos-master]:docker pull centos

Using default tag: latest
Trying to pull repository docker.io/library/centos …
latest: Pulling from docker.io/library/centos
d9aaf4d82f24: Downloading [> ] 540 kB/73.39 MB
d9aaf4d82f24: Pulling fs layer
Digest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872e
Status: Downloaded newer image for centos:latest
编写Dockerfile制造apache基础镜像
######httpd####
FROM centos
MAINTAINER lienhua lienhua@zhongchuangsanyou.com
RUN yum -y install epel-release
RUN yum -y install httpd php php-mysql php-memcache* php-mbstring
ADD httpd.conf /etc/httpd/conf/httpd.conf

EXPOSE 80

CMD ["/usr/sbin/apachectl", “-D”, “FOREGROUND”]
其中httpd.conf文件需要在当前目录下真实存在，此处其内容为
ServerRoot “/etc/httpd”
Listen 80
Listen 8080
Include conf.modules.d/.conf
Include zcsy/.conf
User apache
Group apache
ServerAdmin root@localhost

AllowOverride none
Require all denied

DocumentRoot “/var/www/html”
<Directory “/var/www”>
AllowOverride None
Require all granted

<Directory “/var/www/html”>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted


DirectoryIndex index.html

<Files “.ht*”>
Require all denied

ErrorLog “logs/error_log”
LogLevel warn

LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combined
LogFormat “%h %l %u %t “%r” %>s %b” common

LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i” %I %O” combinedio

CustomLog “logs/access_log” combined


ScriptAlias /cgi-bin/ “/var/www/cgi-bin/”

<Directory “/var/www/cgi-bin”>
AllowOverride None
Options None
Require all granted


TypesConfig /etc/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType application/x-httpd-php .php
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

AddDefaultCharset UTF-8

MIMEMagicFile conf/magic

EnableSendfile off
EnableMMAP off
IncludeOptional conf.d/*.conf
执行[centos-master]:docker build -t registry.evehicle.cn/httpd . 命令制作名为”registry.evehicle.cn/httpd”的镜像（注意此处的点必须要有，并且其意义代表当前目录下的Dockerfile文件）
将制作好的镜像上传到私有库
docker push registry.evehicle.cn/httpd
编写启动apache服务的yaml文件
cat 13-rc-httpd.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: 13-rc-httpd
labels:
name: 13-rc-httpd
spec:
replicas: 2
selector:
name: 13-rc-httpd
template:
metadata:
labels:
name: 13-rc-httpd
spec:
containers:
- name: 13-rc-httpd
image: registry.evehicle.cn/httpd
env:
- name: LANG
value: en_US.UTF-8
ports:
- containerPort: 80
hostPort: 80
volumeMounts:
- name: time
mountPath: /etc/localtime
- name: zcsy
mountPath: /etc/httpd/zcsy
- name: deploy
mountPath: /docker/httpd/deploy
- name: log
mountPath: /var/log/httpd
volumes:
- name: time
hostPath:
path: /etc/localtime
- name: zcsy
hostPath:
path: /docker/httpd/zcsy
- name: deploy
hostPath:
path: /docker/httpd/deploy
- name: log
hostPath:
path: /docker/httpd/log
nodeSelector:
slave: “13”
imagePullSecrets:
- name: registrykey
给其中一个node加上标签为“13
kubectl label nodes centos-minion-1 slave=13
此时拥有标签“13”的nodes应具备的条件
/docker/httpd/zcsy下需要有官网的配置文件
<VirtualHost :80>
ServerName www.evehicle.cn
DocumentRoot /var/deploy/wordpress/
RewriteEngine on
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !^..(ico|pdf|flv|jpe?g|js|gif|png|html|shtml|zip|xml|gz|rar|swf|txt|apk|bmp|css|m4a|ogg|mp3|ipa|plist)$
RewriteCond %{REQUEST_URI} !^/server-status$
RewriteRule . /index.php [QSA,PT,L]

COPY startup.sh /startup.sh
RUN addgroup mysql &&
adduser -H -D -s /bin/false -G mysql mysql &&
apk add --update mysql mysql-client && rm -f /var/cache/apk/* &&
mkdir /data &&
chown -R mysql:mysql /data /etc/mysql &&
chmod 755 /startup.sh
;

WORKDIR /data
VOLUME /data
VOLUME /etc/mysql

EXPOSE 3306
CMD ["/startup.sh"]
启动mysql（建议mysql在宿主机启动）
[centos-master]: docker build -t registry.evehicle.cn/mysql
[centos-master]: docker push registry.evehicle.cn/mysql
[centos-master]: kubectl create -f rc-mysql.yaml
[centos-master]: cat rc-mysql.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: 13-rc-mysql
labels:
name: 13-rc-mysql
spec:
replicas: 2
selector:
name: 13-rc-mysql
template:
metadata:
labels:
name: 13-rc-mysql
spec:
containers:
- name: 13-rc-mysql
image: registry.evehicle.cn/mysql
env:
- name: MYSQL_DATABASE
value: admin
- name: MYSQL_USER
value: tony
- name: MYSQL_PASSWORD
value: 456
- name: MYSQL_ROOT_PASSWORD
value: 123
ports:
- containerPort: 3306
hostPort: 3306
volumeMounts:
- name: time
mountPath: /etc/localtime
- name: data
mountPath: /data
- name: etc
mountPath: /etc/mysql
- name: run
mountPath: /run/mysqld
volumes:
- name: time
hostPath:
path: /etc/localtime
- name: data
hostPath:
path: /docker/mysql/data
- name: etc
hostPath:
path: /docker/mysql/etc
- name: run
hostPath:
path: /docker/mysql/run
nodeSelector:
slave: “13”
imagePullSecrets:
- name: registrykey
为方便代码编写及统一管理，应提前做好内部DNS解析。将所负责的应用规整到对应的机器上。