k8s1.12到1.17升级变更

1.17.2版本的部署和问题

部署环境:
  • centos 7.5minimal
    • 192.168.2.137 master 密码: xxxx
      192.168.2.138 node 密码: xxxx
  • k8s 1.17.2
  • calico 3.7.5
  • 集成了boc2.3 addon
部署过程的问题
  1. k8s版本 < 1.13 是需要开启–allow-privileged=true 设置为true时,kubernetes允许在Pod中运行拥有系统特权的容器应用;当k8s版本 > 1.13 需要去掉这个参数,否则kubelet无法启动。
[root@node-137 addon]# cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
	$KUBELET_ADDRESS \
	$KUBELET_POD_INFRA_CONTAINER \
	$KUBELET_ARGS \
	$KUBE_LOGTOSTDERR \
	$KUBELET_NETWORK_ARGS \
	$KUBELET_DNS_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target
[root@node-137 addon]# 
  1. k8s 摒弃了apiVersion:extensions/v1beta1。相关模块(calico,efk,prometheus,beyondlet,beyondac,ingress)修改apiVersion如下
apiVerion: apps/v1
  1. deployment和daemonset的spec.selector标签缺失,之前1.12版本很多yaml文件没有加该配置,新版会报错。
error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false
## 解决方法 在spec加添加selector
spec:
  selector:
    matchLabels:
      app: xxxx
  1. calico BGP not established 估计是没用发现实际的网卡

Warning Unhealthy 0s kubelet, 192.168.2.138 Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.2.1372020-01-21 06:20:51.501 INFO readiness.go 88: Number of node(s) with BGP peering established = 0

/*
调整calicao 网络插件的网卡发现机制,修改IP_AUTODETECTION_METHOD对应的value值。官方提供的yaml文件中,ip识别策略(IPDETECTMETHOD)没有配置,即默认为first-found,这会导致一个网络异常的ip作为nodeIP被注册,从而影响node-to-node mesh。我们可以修改成can-reach或者interface的策略,尝试连接某一个Ready的node的IP,以此选择出正确的IP。
*/

// calico.yaml 文件添加以下二行
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens.*"  # ens 根据实际网卡开头配置
 
 // 配置如下             
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens.*"
              #或者 value: "interface=ens160"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always" 
  1. helm报错。

对于 Kubernetes v1.16.0 以上的版本,有可能会碰到 Error: error installing: the server could not find the requested resource 的错误。这是由于 extensions/v1beta1 已经被 apps/v1 替代。初始化命令如下修改

/usr/local/bin/helm init --service-account tiller --tiller-image deploy.bocloud/kubernetes-helm/tiller:v2.9.0 --skip-refresh --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -
  1. etcd证书需要开启双向认证配置修改如下,server增加client auth,client增加server auth
cat <<EOF > ca-config.json
{
    "signing": {
        "default": {
            "expiry": "876000h"
        },
        "profiles": {
            "server": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "876000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
  1. etcd配置增加Type=notify
[root@node-216 k8s]# vi /etc/systemd/system/etcd.service 
[Unit]
Description=etcd server
After=network.target

[Service]
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
Type=notify
ExecStart=/usr/local/bin/etcd $ETCD_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  1. 修改cpu或者内存配置后:
rm -rf /var/lib/kubelet/cpu_manager_state && systemctl restart kubelet
  1. ImageInspectError

请删除节点上所有使用损坏 image 的容器,然后删除 image,再重新 pull image

 Warning  InspectFailed  4m25s (x6 over 4m37s)  kubelet, node-217.dev  Failed to inspect image "deploy.bocloud/coreos/prometheus-config-reloader:v0.30.0": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2: invalid argument
  Warning  Failed         4m25s (x6 over 4m37s)  kubelet, node-217.dev  Error: ImageInspectError
  1. ingress-controller报错(删掉pod和镜像后apply重启)
didn't have free ports for the requested pod ports...

2.3升级的组件

组件BOC-2.2BOC-2.3
Kubernetes1.12.61.17.2
Docker18.09.219.03.5
CoreDNSkubedns1.6.6
Prometheus2.4.32.11.0
Node-Exporter0.16.00.18.1
OpenVswitch2.10.12.11.2
beyondac1.0.51.0.7
beyondlet0.3.60.3.7
  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值