k8s1.12到1.17升级变更

1.17.2版本的部署和问题

部署环境：

• centos 7.5minimal
  • 192.168.2.137 master 密码: xxxx
    192.168.2.138 node 密码: xxxx
• k8s 1.17.2
• calico 3.7.5
• 集成了boc2.3 addon

部署过程的问题

1. k8s版本 < 1.13 是需要开启–allow-privileged=true 设置为true时，kubernetes允许在Pod中运行拥有系统特权的容器应用；当k8s版本 > 1.13 需要去掉这个参数，否则kubelet无法启动。

[root@node-137 addon]# cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
	$KUBELET_ADDRESS \
	$KUBELET_POD_INFRA_CONTAINER \
	$KUBELET_ARGS \
	$KUBE_LOGTOSTDERR \
	$KUBELET_NETWORK_ARGS \
	$KUBELET_DNS_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target
[root@node-137 addon]# 

2. k8s 摒弃了apiVersion：extensions/v1beta1。相关模块（calico，efk，prometheus，beyondlet，beyondac，ingress）修改apiVersion如下

apiVerion: apps/v1

3. deployment和daemonset的spec.selector标签缺失，之前1.12版本很多yaml文件没有加该配置，新版会报错。

error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false
## 解决方法 在spec加添加selector
spec:
  selector:
    matchLabels:
      app: xxxx

4. calico BGP not established 估计是没用发现实际的网卡

Warning Unhealthy 0s kubelet, 192.168.2.138 Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.2.1372020-01-21 06:20:51.501 INFO readiness.go 88: Number of node(s) with BGP peering established = 0

/*
调整calicao 网络插件的网卡发现机制，修改IP_AUTODETECTION_METHOD对应的value值。官方提供的yaml文件中，ip识别策略（IPDETECTMETHOD）没有配置，即默认为first-found，这会导致一个网络异常的ip作为nodeIP被注册，从而影响node-to-node mesh。我们可以修改成can-reach或者interface的策略，尝试连接某一个Ready的node的IP，以此选择出正确的IP。
*/

// calico.yaml 文件添加以下二行
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens.*"  # ens 根据实际网卡开头配置
 
 // 配置如下             
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens.*"
              #或者 value: "interface=ens160"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always" 

5. helm报错。

对于 Kubernetes v1.16.0 以上的版本，有可能会碰到 Error: error installing: the server could not find the requested resource 的错误。这是由于 extensions/v1beta1 已经被 apps/v1 替代。初始化命令如下修改

/usr/local/bin/helm init --service-account tiller --tiller-image deploy.bocloud/kubernetes-helm/tiller:v2.9.0 --skip-refresh --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -

6. etcd证书需要开启双向认证配置修改如下，server增加client auth,client增加server auth：

cat <<EOF > ca-config.json
{
    "signing": {
        "default": {
            "expiry": "876000h"
        },
        "profiles": {
            "server": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

7. etcd配置增加Type=notify

[root@node-216 k8s]# vi /etc/systemd/system/etcd.service 
[Unit]
Description=etcd server
After=network.target

[Service]
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
Type=notify
ExecStart=/usr/local/bin/etcd $ETCD_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

8. 修改cpu或者内存配置后：

rm -rf /var/lib/kubelet/cpu_manager_state && systemctl restart kubelet

9. ImageInspectError

请删除节点上所有使用损坏 image 的容器，然后删除 image，再重新 pull image

 Warning  InspectFailed  4m25s (x6 over 4m37s)  kubelet, node-217.dev  Failed to inspect image "deploy.bocloud/coreos/prometheus-config-reloader:v0.30.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
  Warning  Failed         4m25s (x6 over 4m37s)  kubelet, node-217.dev  Error: ImageInspectError

10. ingress-controller报错(删掉pod和镜像后apply重启)

didn't have free ports for the requested pod ports...

2.3升级的组件

组件	BOC-2.2	BOC-2.3
Kubernetes	1.12.6	1.17.2
Docker	18.09.2	19.03.5
CoreDNS	kubedns	1.6.6
Prometheus	2.4.3	2.11.0
Node-Exporter	0.16.0	0.18.1
OpenVswitch	2.10.1	2.11.2
beyondac	1.0.5	1.0.7
beyondlet	0.3.6	0.3.7