Kubernetes集群配置高可用

高可用原理

kubelet的config参数可以监控特定目录下的yaml文件,确保指定目录下的pod可以稳定运行，我们可以在目录下创建apiserver,controller-manager,scheduler的对应yaml描述文件，通过启动kubelet来确保三者稳定运行。api-server可以使用nginx来进行负载均衡和健康状态监测，controller-manager,scheduler通过启动时设置–leader-elect=true进行选举确保同一时刻只有一台主机上的起作用，故障时standby会切换为master。

官方推荐master-standby-standby来配置高可用，一主两备。

kubelet的高可用：systemctl设置Restart=on-failure来实现，
Restart字段：定义了 kubelet 退出后，Systemd 的重启方式。
上面的例子中，Restart设为on-failure，表示任何意外的失败，就将重启kubelet。如果 kubelet 正常停止（比如执行systemctl stop命令），它就不会重启。

Restart字段可以设置的值如下。
- no（默认值）：退出后不会重启
- on-success：只有正常退出时（退出状态码为0），才会重启
- on-failure：非正常退出时（退出状态码非0），包括被信号终止和超时，才会重启
- on-abnormal：只有被信号终止和超时，才会重启
- on-abort：只有在收到没有捕捉到的信号终止时，才会重启
- on-watchdog：超时退出，才会重启
- always：不管是什么退出原因，总是重启

apiserver前端nginx高可用：keepalive+nginx不再赘述

etcd：集群形式

节点信息

主机IP	操作系统
172.20.1.99	centos7.2
172.20.1.100	centos7.2
172.20.1.101	centos7.2

配置过程（可直接保存为kubelet.sh执行）

#config配置文件
cat > /etc/kubernetes/config <<EOF
KUBE_LOGTOSTDERR="--logtostderr=true"

KUBE_LOG_LEVEL="--v=0"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER=""
EOF
#kubelet配置文件
cat > /etc/kubernetes/kubelet <<EOF
KUBELET_ADDRESS="--address=127.0.0.1"

KUBELET_HOSTNAME="--hostname-override=172.20.1.101"

KUBELET_API_SERVER=""

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=172.20.1.128:5000/pause"

KUBELET_ARGS="--config=/etc/kubernetes/manifests --register-node=false"
EOF

#启动文件

echo "
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
        \$KUBE_LOGTOSTDERR \
        \$KUBE_LOG_LEVEL \
        \$KUBELET_API_SERVER \
        \$KUBELET_ADDRESS \
        \$KUBELET_PORT \
        \$KUBELET_HOSTNAME \
        \$KUBE_ALLOW_PRIV \
        \$KUBELET_POD_INFRA_CONTAINER \
        \$KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target
" > /usr/lib/systemd/system/kubelet.service

#生成工作目录
mkdir -p /var/lib/kubelet/pods
mkdir -p /var/lib/kubelet/plugins

#设置开机启动
systemctl enable kubelet

#生成日志文件
touch /var/log/kube-apiserver.log
touch /var/log/kube-controller-manager.log
touch /var/log/kube-scheduler.log

#创建监听目录
mkdir -p /etc/kubernetes/manifests

#kube-apiserver.yaml
cat > /etc/kubernetes/manifests/kube-apiserver.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  hostNetwork: true
  containers:
  - name: kube-apiserver
    image: 172.20.1.128:5000/kube-apiserver:v1.3.0
    command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-apiserver --address=172.20.1.101 --etcd-servers=http://172.20.1.103:2379,http://172.20.1.99:2379,http://172.20.1.102:2379,http://172.20.1.100:2379,http://172.20.1.101:2379
      --admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota
      --v=2 --service-cluster-ip-range=10.254.0.0/16 
      --allow-privileged=False 1>>/var/log/kube-apiserver.log 2>&1
    ports:
    - containerPort: 8080
      hostPort: 8080
      name: local
    volumeMounts:
    - mountPath: /var/log/kube-apiserver.log
      name: logfile
  volumes:
  - hostPath:
      path: /var/log/kube-apiserver.log
    name: logfile
EOF

#kube-controller-manager
cat > /etc/kubernetes/manifests/kube-controller-manager.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-controller-manager --master=172.20.1.101:8080 --cluster-name=local
      --cluster-cidr=10.254.0.0/16
      --v=2 --leader-elect=true 1>>/var/log/kube-controller-manager.log 2>&1
    image: 172.20.1.128:5000/kube-controller-manager:v1.3.0
    livenessProbe:
      httpGet:
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    name: kube-controller-manager
    volumeMounts:
    - mountPath: /var/log/kube-controller-manager.log
      name: logfile
  hostNetwork: true
  volumes:
  - hostPath:
      path: /var/log/kube-controller-manager.log
    name: logfile
EOF


#kube-scheduler
cat > /etc/kubernetes/manifests/kube-scheduler.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: 172.20.1.128:5000/kube-scheduler:v1.3.0
    command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-scheduler --master=172.20.1.101:8080 --v=2 --leader-elect=true 1>>/var/log/kube-scheduler.log
      2>&1
    livenessProbe:
      httpGet:
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /var/log/kube-scheduler.log
      name: logfile
  volumes:
  - hostPath:
      path: /var/log/kube-scheduler.log
    name: logfile
EOF
#启动kueblet
systemctl start kubelet

nginx配置文件

upstream backend {
        ip_hash;
        server 172.20.1.99:8080  max_fails=1 fail_timeout=300s;
        server 172.20.1.100:8080 max_fails=1 fail_timeout=300s;
        server 172.20.1.101:8080 max_fails=1 fail_timeout=300s;
         }
    server {
        listen       0.0.0.0:8080;
        server_name  apiserver;
        location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_buffering off;
        proxy_pass http://backend;     
        }
    }

排错

• tailf /var/log/kube-*.log
• tailf /var/log/messages
• docker logs id

注意事项

• 其它节点上执行需要替换配置文件中的主机IP
• kubelet需要拷贝
• 启动文件$符号需要转义
• /etc/kubernetes目录下不要含有多余文件
• 脚本中的镜像需要替换，这里是自己部署的私有仓库

参考

Kubernetes - Building High-Availability Clusters
http://kubernetes.io/docs/admin/high-availability/#establishing-a-redundant-reliable-data-storage-layer