Kubernetes v1.8.4 集群各组件加密认证功能的配置参考

记录 v1.8.4 的组件配置如下，经测试能完美地工作。

这里 kube-apiserver 并没有做 HA ，先这个测试通过了，后面再做。ha 方面，只需要加上 haproxy+keepalived进行配置，使其以 <vip>:443的方式访问即可。haproxy负责负载转发, keepalived负责监控haproxy的ha。
haproxy和keepalived可以部署在三台master中的两台上，这样注意一下端口问题，不要使用以master的api-server相冲突的端口即可。

好了不多说了，我们直接看下如何配置组件的加密认证功能，具体都添加了注释。

其中的 ca 证书，需要先在 master 中按照 master_ssl.cnf 进行配置生成 root 证书，即 ca.crt, ca.key，然后以他们去生成 server.crt, server.key。最后面将 ca.crt, ca.key 放到node节点中，继续为kubelet生成证书。

master_ssl.cnf

编辑 master_ssl.cnf，添加相关的配置

[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.k8s.xxx.com # specify domain name.
DNS.5 = CDM1B12-209202200.wdds.com # apiserver hostname
IP.1 = 10.0.0.1 # kubernetes cluster ip
IP.2 = 10.209.202.200  # kubernetes apiserver ip

在master上创建证书的脚本

function create_master_ca() {
    echo "start create master ca ........."

    openssl genrsa -out ca.key 2048
    openssl req -x509 -new -nodes -key ca.key -subj "/CN=xxx.com" -days 5000 -out ca.crt
    openssl genrsa -out server.key 2048

    HN=`hostname`
    echo "hostname is :$HN"
    openssl req -new -key server.key -subj "/CN=$HN" -config master_ssl.cnf -out server.csr
    openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 5000 -extensions v3_req -extfile master_ssl.cnf -out server.crt


    openssl genrsa -out cs_client.key 2048
    openssl req -new -key cs_client.key -subj "/CN=$HN" -out cs_client.csr
    openssl x509 -req -in cs_client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out cs_client.crt -days 5000

    cp -f ca.* server.crt server.key cs_client.crt cs_client.key /etc/kubernetes/pki
    cp -f /root/kube_package/ssl/kubeconfig /etc/kubernetes/kubeconfig
    echo "end create master ca ........."
}

create_master_ca

在node上创建证书的脚本

#!/bin/bash

cd `dirname $0`
HN=`hostname`
openssl genrsa -out kubelet_client.key 2048
openssl req -new -key kubelet_client.key -subj "/CN=$HN" -out kubelet_client.csr
openssl x509 -req -in kubelet_client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kubelet_client.crt -days 5000

kube-apiserver

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  hostNetwork: true
  containers:
  - name: kube-apiserver
    image: 10.213.42.254:10500/root/hyperkube:v1.8.4-ceph
    command:
    - /bin/sh
    - -c
    - /hyperkube apiserver
      --apiserver-count=1 #注意：这里没有添加ha
      --allow-privileged=true
      --etcd-prefix=/cd-dev02
      --etcd-servers=http://10.209.202.200:2379,http://10.209.204.167:2379,http://10.209.204.199:2379
      --admission-control=SecurityContextDeny,ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota
      --insecure-bind-address=0.0.0.0
      --insecure-port=11080 # 这个留着主要是为了兼容之前非安全使用之需。
      --secure-port=443 # 在之前基础上添加这个安全端口即可。
      --advertise-address=10.209.202.200
      --service-cluster-ip-range=10.0.0.0/18
      --tls-cert-file=/etc/kubernetes/pki/server.crt # 注意：这些文件都是挂载上去的，所以后面的 VolumeMounts 中需要特别指定。
      --tls-private-key-file=/etc/kubernetes/pki/server.key
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --alsologtostderr=false
      --logtostderr=true
      --v=0
      --log-dir=/var/log/kubernetes
      --service-node-port-range=10000-12000
      --storage-backend=etcd3
      --storage-media-type=application/vnd.kubernetes.protobuf
      --runtime-config=v1,extensions/v1beta1=true,extensions/v1beta1/ingress=true >> /var/log/kubernetes/kube-apiserver.log 2>&1
    ports:
        - containerPort: 443
          hostPort: 443
          name: https
        - containerPort: 7080
          hostPort: 7080
          name: http
        - containerPort: 11080
          hostPort: 11080
          name: local
        - containerPort: 6443
          hostPort: 6443
          name: seport
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: pki
          readOnly: true
        - mountPath: /var/log
          name: logpath
        - mountPath: /etc/localtime
          name: localtime
      volumes:
      - hostPath:
          path: /etc/kubernetes
        name: pki
      - hostPath:
          path: /var/log
        name: logpath
      - hostPath:
          path: /etc/localtime
        name: localtime

kube-controller-manager

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
spec:
  hostNetwork: true
  containers:
  - name: kube-controller-manager
    image: 10.213.42.254:10500/root/hyperkube:v1.8.4-ceph
    command:
    - /bin/sh
    - -c
    - /hyperkube controller-manager
      --v=0
      --logtostderr=true
      --log-dir=/var/log/kubernetes
      --alsologtostderr=false
      --root-ca-file=/etc/kubernetes/pki/ca.crt
      --service-account-private-key-file=/etc/kubernetes/pki/server.key
      --kubeconfig=/etc/kubernetes/kubeconfig #这个yaml文件中本来要指定 --master=https://vip:443 的，但后来版本不支持了，都移到kubeconfig中进行配置了
      --leader-elect=true >> /var/log/kubernetes/kube-controller-manager.log 2>&1
    ports:
    - containerPort: 10252
      hostPort: 10252
      name: local
    volumeMounts:
    - mountPath: /etc/kubernetes
      name: pki
      readOnly: true
    - mountPath: /var/log
      name: logpath
    - mountPath: /sbin/modprobe
      name: modprobe
      readOnly: true
    - mountPath: /lib/modules
      name: modules
      readOnly: true
    - mountPath: /dev
      name: devices
  volumes:
  - hostPath:
      path: /etc/kubernetes
    name: pki
  - hostPath:
      path: /var/log
    name: logpath
  - hostPath:
      path: /sbin/modprobe
    name: modprobe
  - hostPath:
      path: /lib/modules
    name: modules
  - hostPath:
      path: /dev
    name: devices

kube-scheduler

apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: 10.213.42.254:10500/root/hyperkube:v1.8.4-ceph
    command:
    - /bin/sh
    - -c
    - /hyperkube scheduler
      --kubeconfig=/etc/kubernetes/kubeconfig #这个yaml文件中本来要指定 --master=https://vip:443 的，但后来版本不支持了，都移到kubeconfig中进行配置了
      --v=0
      --logtostderr=true
      --alsologtostderr=false
      --log-dir=/var/log/kubenetes
      --leader-elect=true >> /var/log/kubernetes/kube-scheduler.log 2>&1
    ports:
    - containerPort: 10251
      hostPort: 10251
      name: local
    volumeMounts:
    - mountPath: /etc/kubernetes
      name: pki
      readOnly: true
    - mountPath: /var/log
      name: logpath
    - mountPath: /etc/localtime
      name: localtime
  volumes:
    - hostPath:
        path: /etc/kubernetes
      name: pki
    - hostPath:
        path: /var/log
      name: logpath
    - hostPath:
        path: /etc/localtime
      name: localtime

master中的 /etc/kubernetes/kubeconfig

apiVersion: v1
kind: Config
users:
- name: controllermanager
  user:
    client-certificate: /etc/kubernetes/pki/cs_client.crt
    client-key: /etc/kubernetes/pki/cs_client.key
clusters:
- name: local
  cluster:
    server: https://10.209.202.200:443 #本来要在controller-manager和scheduler中的yaml文件中指定 --master=https://vip:443 的，但后来版本不支持了，都移到这里进行配置了，并且 https:// 这几个字不能去掉。
    certificate-authority: /etc/kubernetes/pki/ca.crt
contexts:
- context:
    cluster: local
    user: controllermanager
  name: my-context
current-context: my-context

---

kubelet.service

[Unit]
Description=Kubernetes Kubelet Server
Documentation=http://kubernetes.io/docs/admin/kubelet/
After=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/default/kube-default
EnvironmentFile=-/etc/default/kubelet
ExecStart=/bin/sh -c '/usr/local/bin/kubelet \
        --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin \
        --hostname-override=10.209.228.18 \
        --kubeconfig=/etc/kubernetes/kubeconfig \ #这个文件中本来要指定 --master=https://vip:443 的，但后来版本不支持了，都移到kubeconfig中进行配置了
        --pod-manifest-path=/etc/kubernetes/manifests \
        --require-kubeconfig=true \
        --logtostderr=true \
        --pod-infra-container-image=10.213.42.254:10500/pause:3.0 \
        --cluster-dns=10.0.0.10 \
        --cluster-domain=k8s.wanda.com \
        --max-pods=110 \
        --cgroup-driver=cgroupfs \
        --fail-swap-on=false \
        --runtime-cgroups=/systemd/system.slice \
        --kubelet-cgroups=/systemd/system.slice \
        --allow-privileged=true -v=0 >> /var/log/kubernetes/kubelet.log 2>&1'
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

node 节点中的 /etc/kubernetes/kubeconfig

apiVersion: v1
kind: Config
users:
- name: kubelet
  user:
    client-certificate: /etc/kubernetes/ssl/kubelet_client.crt
    client-key: /etc/kubernetes/ssl/kubelet_client.key
clusters:
- name: local
  cluster:
    server: https://10.209.202.200:443 #本来要在kubelet.service文件中指定 --api_servers=https://vip:443 的，但后来版本不支持了，都移到这里进行配置了，而且将--api-server 改成 server了，并且 https:// 这几个字不能去掉。
    certificate-authority: /etc/kubernetes/ssl/ca.crt
contexts:
- context:
    cluster: local
    user: kubelet
  name: my-context
current-context: my-context

kube-proxy.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-proxy
spec:
  hostNetwork: true
  containers:
  - name: kube-proxy
    image: 10.213.42.254:10500/root/hyperkube:v1.8.4-ceph
    command:
    - /bin/sh
    - -c
    - /hyperkube proxy
      --logtostderr=true
      --proxy-mode=iptables
      --master=https://10.209.202.200:443 #感觉这个可以不要，因为我已经在kubeconfig中用server指定了，即server来代替--api-servers。但这个server是否也能替代这里的--master？？后面要测试一下这个。
      --kubeconfig=/etc/kubernetes/kubeconfig
      -v=4
      --conntrack-tcp-timeout-established=1200s  >> /var/log/kubernetes/kube-proxy.log 2>&1
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /etc/kubernetes
      name: pki
      readOnly: true
    - mountPath: /var/log
      name: logpath
    - mountPath: /etc/localtime
      name: localtime
  volumes:
  - hostPath:
      path: /etc/kubernetes
    name: pki
  - hostPath:
      path: /var/log
    name: logpath
  - hostPath:
      path: /etc/localtime
    name: localtime