kubernetes 升级问题笔录

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/qq_21816375/article/details/82803201

本笔录主要是记录kubernetes从v1.7.6升级到v1.11.2的问题以及相关的解决办法

问题一:不能访问master的10255
日志

[root@test-master-113 qinzhao]# kubectl logs heapster-1967565778-v5wkr -n kube-system heapster 
I0918 10:37:05.893124       1 heapster.go:72] /heapster --source=kubernetes.summary_api:''
I0918 10:37:05.893235       1 heapster.go:73] Heapster version v1.4.1
I0918 10:37:05.893944       1 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version v1
I0918 10:37:05.893993       1 configs.go:62] Using kubelet port 10255
I0918 10:37:05.896697       1 heapster.go:196] Starting with Metric Sink
I0918 10:37:05.911591       1 heapster.go:106] Starting heapster on port 8082
E0918 10:38:05.026522       1 summary.go:97] error while getting metrics summary from Kubelet test-master-113(10.39.0.113:10255): Get http://10.39.0.113:10255/stats/summary/: dial tcp 10.39.0.113:10255: getsockopt: connection refused
[root@test-master-113 qinzhao]# kubectl logs heapster-1967565778-v5wkr -n kube-system heapster 
I0918 10:37:05.893124       1 heapster.go:72] /heapster --source=kubernetes.summary_api:''
I0918 10:37:05.893235       1 heapster.go:73] Heapster version v1.4.1
I0918 10:37:05.893944       1 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version v1
I0918 10:37:05.893993       1 configs.go:62] Using kubelet port 10255
I0918 10:37:05.896697       1 heapster.go:196] Starting with Metric Sink
I0918 10:37:05.911591       1 heapster.go:106] Starting heapster on port 8082
E0918 10:38:05.026522       1 summary.go:97] error while getting metrics summary from Kubelet test-master-113(10.39.0.113:10255): Get http://10.39.0.113:10255/stats/summary/: dial tcp 10.39.0.113:10255: getsockopt: connection refused

原因是没有打开10255这个端口

[root@test-master-113 qingcloud-csi]# netstat -tlnp 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      19995/kubelet       
tcp        0      0 127.0.0.1:40264         0.0.0.0:*               LISTEN      19995/kubelet       
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      32286/proxy         
tcp        0      0 10.39.0.113:10250       0.0.0.0:*               LISTEN      19995/kubelet       
tcp        0      0 127.0.0.1:9099          0.0.0.0:*               LISTEN      21392/calico-node   
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      22716/scheduler     
tcp        0      0 10.39.0.113:6443        0.0.0.0:*               LISTEN      22529/apiserver     
tcp        0      0 10.39.0.113:2379        0.0.0.0:*               LISTEN      30095/etcd          
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      30095/etcd          
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      21364/controller-ma 
tcp        0      0 10.39.0.113:2380        0.0.0.0:*               LISTEN      30095/etcd          
tcp        0      0 127.0.0.1:9101          0.0.0.0:*               LISTEN      1654/node_exporter  
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:179             0.0.0.0:*               LISTEN      21507/bird          
tcp        0      0 0.0.0.0:9120            0.0.0.0:*               LISTEN      795/sshd            
tcp6       0      0 :::9898                 :::*                    LISTEN      31707/kube-discover 
tcp6       0      0 :::9100                 :::*                    LISTEN      1725/kube-rbac-prox 
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::10256                :::*                    LISTEN      32286/proxy         
tcp6       0      0 :::8080                 :::*                    LISTEN      22529/apiserver     
tcp6       0      0 :::30004                :::*                    LISTEN      32286/proxy         
tcp6       0      0 :::30900                :::*                    LISTEN      32286/proxy         
tcp6       0      0 :::30902                :::*                    LISTEN      32286/proxy         
tcp6       0      0 :::31543                :::*                    LISTEN      32286/proxy         
tcp6       0      0 :::30903                :::*                    LISTEN      32286/proxy         
tcp6       0      0 :::9120                 :::*                    LISTEN      795/sshd           

问题二:装不上qingcloud-csi的存储插件
日志

Type     Reason   Age              From                     Message
  ----     ------   ----             ----                     -------
  Normal   Pulled   1m (x4 over 2m)  kubelet, test-slave-117  Container image "csiplugin/csi-qingcloud:v0.2.0" already present on machine
  Normal   Created  1m (x4 over 2m)  kubelet, test-slave-117  Created container
  Warning  Failed   1m (x4 over 2m)  kubelet, test-slave-117  Error: failed to start container "csi-qingcloud": Error response from daemon: linux mounts: Path /var/lib/kubelet is mounted on / but it is not a shared mount.
  Normal   Pulled   1m (x2 over 2m)  kubele
t, test-slave-117  Container image "quay.io/k8scsi/driver-registrar:v0.2.0" already present on machine
  Normal   Created  1m (x2 over 2m)  kubelet, test-slave-117  Created container
  Normal   Started  1m (x2 over 2m)  kubelet, test-slave-117  Started container
  Warning  BackOff  1m (x7 over 2m)  kubelet, test-slave-117  Back-off restarting failed container

解决办法

[root@test-slave-117 ~]# systemctl cat docker | grep -i mount
MountFlags=slave

把docker 的参数MountFlags=slave改为MountFlags=shared就可以搞定了

[root@test-slave-117 ~]# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target rhel-push-plugin.socket registries.service
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer

[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
          --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
          --default-runtime=docker-runc \
          --exec-opt native.cgroupdriver=systemd \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          --seccomp-profile=/etc/docker/seccomp.json \
          $OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY \
	  $REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=shared ##修改这个参数
KillMode=process

[Install]
WantedBy=multi-user.target

相关文档以及issue

https://kubernetes.io/docs/concepts/storage/volumes/#configuration
https://bugs.centos.org/view.php?id=14455
https://github.com/yunify/qingcloud-csi/blob/master/docs/static-provisioner-zh.md


问题三:在指定空间打开网络隔离通过kube-lb服务不能访问指定空间的服务,也ping 不通服务
关闭网路隔离


问题四:证书不对的问题
日志

[root@dev-master-105 ~]# journalctl -u kubelet -f 
-- Logs begin at Fri 2018-09-14 02:49:34 CST. --
Sep 14 18:21:19 dev-master-105 kubelet[17816]: I0914 18:21:19.863620   17816 feature_gate.go:230] feature gates: &{map[RotateKubeletClientCertificate:true RotateKubeletServerCertificate:true]}
Sep 14 18:21:19 dev-master-105 kubelet[17816]: I0914 18:21:19.863678   17816 feature_gate.go:230] feature gates: &{map[RotateKubeletServerCertificate:true RotateKubeletClientCertificate:true]}
Sep 14 18:21:19 dev-master-105 kubelet[17816]: I0914 18:21:19.863841   17816 plugins.go:97] No cloud provider specified.
Sep 14 18:21:19 dev-master-105 kubelet[17816]: I0914 18:21:19.863854   17816 server.go:524] No cloud provider specified: "" from the config file: ""
Sep 14 18:21:19 dev-master-105 kubelet[17816]: I0914 18:21:19.863875   17816 bootstrap.go:56] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
Sep 14 18:21:19 dev-master-105 kubelet[17816]: I0914 18:21:19.865699   17816 bootstrap.go:86] No valid private key and/or certificate found, reusing existing private key or creating a new one
Sep 14 18:21:19 dev-master-105 kubelet[17816]: F0914 18:21:19.889677   17816 server.go:262] failed to run Kubelet: cannot create certificate signing request: Post https://10.39.0.105:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: x509: certificate signed by unknown authority

对比kube-apiserver指定的证书就可以搞定


问题五:启用rbac模式下报的错误

  Normal   SandboxChanged          42s (x12 over 1m)  kubelet, dev-master-105  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  41s (x4 over 48s)  kubelet, dev-master-105  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "efb32cbd3303ed7761e84f6521566a2c6a38fce4cd289b0ff183a42002007f21" network for pod "kube-dns-58d745cd6d-gdgcp": NetworkPlugin cni failed to set up pod "kube-dns-58d745cd6d-gdgcp_kube-system" network: pods "kube-dns-58d745cd6d-gdgcp" is forbidden: User "system:serviceaccount:kube-system:default" cannot get pods in the namespace "kube-system"

解决方法:将system:serviceaccount:kube-system:default绑定cluster-admin clusterrole

[root@master-47-35 prome]# cat default.yaml 
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: default
  namespace: kube-system
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: ""


打node 或者master 节点的标签

kubectl label node dev-master-105 node-role.kubernetes.io/master='master'
kubectl label node dev-slave-107  node-role.kubernetes.io/node='node'
root@dev-master-105 ~]# kubectl get no
NAME             STATUS    ROLES     AGE       VERSION
dev-master-105   Ready     master    304d      v1.11.0-168+f47446a730ca03
dev-slave-107    Ready     node      282d      v1.11.0-168+f47446a730ca03
dev-slave-108    Ready     node      308d      v1.11.0-168+f47446a730ca03
dev-slave-110    Ready     node      308d      v1.11.0-168+f47446a730ca03

问题六:prometheus 报错

level=error ts=2018-09-16T23:33:54.76892054Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: Get https://10.0.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: no route to host"
level=error ts=2018-09-16T23:33:54.76892347Z caller=main.go:211 component=k8s_cli^Cent_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: Get https://10.0.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: no route to host"
level=error ts=2018-09-16T23:33:54.768960604Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: Get https://10.0.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: no route to host"

这个问题是dns挂掉了
重启dns就可以解决问题


问题七:通过kubectl logs pod-xxx -n namespace不能查看日志的解决办法

[root@test-master-113 kube-proxy]# kubectl logs prometheus-2159841327-pgfsv -n kube-system prometheus 
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log prometheus-2159841327-pgfsv)

执行以下命令

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

问题八 通过新的kubectl 命令获取pod时出现好多Evicted pod

[root@dev-master-105 prometheus]# kubectl get  pods -n kube-system 
NAME                                        READY     STATUS               RESTARTS   AGE
alertmanager-1589824335-hm74b               0/1       ContainerCannotRun   18         15h
calico-node-1tlfp                           2/2       Running              2          90d
calico-node-d0dz4                           2/2       Running              0          155d
calico-node-jjcgc                           2/2       Running              2          155d
calico-node-ntlqk                           2/2       Running              2          90d
calico-policy-controller-4081051551-4rc73   1/1       Running              1          146d
calico-policy-controller-4081051551-tgk92   0/1       Evicted              0          155d
dummy-3337601728-37zvf                      0/1       Evicted              0          155d
dummy-3337601728-h95z2                      0/1       Evicted              0          237d
dummy-3337601728-k7zlj                      1/1       Running              1          15h
elasticsearch-logging-v1-3910393438-k3cp8   1/1       Running              1          155d
etcd-dev-master-105                         1/1       Running              0          10h
filebeat-9k1ng                              1/1       Running              1          46d
filebeat-cx4q3                              1/1       Running              1          46d
filebeat-l4lhp                              1/1       Running              3          46d
filebeat-xxs7f                              1/1       Running              40         46d
fluentd-elk-jjnps                           1/1       Running              1          37d
grafana-1688250711-jx42d                    0/1       ContainerCannotRun   21         15h
grafana-1688250711-sk4p9                    0/1       Evicted              0          155d
heapster-4291925761-47kt1                   0/4       Evicted              0          155d
heapster-4291925761-r891j                   3/4       CrashLoopBackOff     21         146d
informer                                    0/1       Error                0          254d
kube-apiserver-dev-master-105               1/1       Running              0          10h
kube-controller-manager-dev-master-105      1/1       Running              0          10h
kube-dns-2744242050-8j83t                   0/3       CrashLoopBackOff     398        99d
kube-lb-bk738                               1/1       Running              1          99d
kube-lb-kx0wj                               1/1       Running              0          155d
kube-lb-qws2c                               1/1       Running              1          155d
kube-lb-vjbp7                               1/1       Running              1          155d
kube-proxy-54p8x                            1/1       Running              1          90d
kube-proxy-9t76j                            1/1       Running              1          90d
kube-proxy-nb1gl                            1/1       Running              6          90d
kube-proxy-v27hm                            1/1       Running              1          90d
kube-scheduler-dev-master-105               1/1       Running              0          10h
kubectl-dqjb8                               1/1       Running              1          99d
kubectl-kbfvb                               1/1       Running              1          155d
kubectl-ml3hb                               0/1       ContainerCreating    0          155d
kubectl-x08fl                               1/1       Running              1          155d
kubernetes-dashboard-1096786477-3xhb0       0/1       Evicted              0          155d
kubernetes-dashboard-1096786477-ngb71       0/1       CrashLoopBackOff     193        145d
prometheus-7343409-0wql3                    0/2       Evicted              0          264d
prometheus-7343409-2l1sr                    0/2       Evicted              0          265d
prometheus-7343409-401pp                    0/2       Evicted              0          264d
prometheus-7343409-731gp                    0/2       Evicted              0          266d
prometheus-7343409-bb6m9                    0/2       Evicted              0          264d
prometheus-7343409-cbbzj                    0/2       Evicted              0          277d
prometheus-7343409-g4j4z                    0/2       Evicted              0          266d
prometheus-7343409-hwk57                    0/2       Evicted              0          264d
prometheus-7343409-j4l2v                    0/2       Evicted              0          264d
prometheus-7343409-lsw54                    2/2       Running              2          155d
prometheus-7343409-mw3vv                    0/2       Evicted              0          264d
prometheus-7343409-n4l7z                    0/2       Evicted              0          266d
prometheus-7343409-sdmxd                    0/2       Evicted              0          266d
prometheus-7343409-spc5r                    0/2       Evicted              0          266d
qingcloud-volume-provisioner                1/1       Running              1          151d

解决办法:删除掉就行

kubectl get  pods -n kube-system |grep Evicted | awk '{print $1}' | xargs kubectl delete pod -nkube-system

问题九 通过新的calicoctl 命令获取查看不了node status

解决办法下载对应版本的calicoctl 配置上相应的命令就搞定


问题十
calico-node 起不来
kubectl describe pod pod-id的查看到以下信息

Events:
  Type     Reason            Age                From                    Message
  ----     ------            ----               ----                    -------
  Normal   Started           45s                kubelet, dev-slave-110  Started container
  Normal   Pulled            45s                kubelet, dev-slave-110  Container image "harbor.enncloud.cn/enncloud/node:v3.2.1" already present on machine
  Normal   Created           45s                kubelet, dev-slave-110  Created container
  Normal   Started           45s                kubelet, dev-slave-110  Started container
  Normal   Pulled            45s                kubelet, dev-slave-110  Container image "harbor.enncloud.cn/enncloud/cni:v3.2.1" already present on machine
  Normal   Created           45s                kubelet, dev-slave-110  Created container
  Warning  DNSConfigForming  43s (x4 over 46s)  kubelet, dev-slave-110  Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 127.0.0.1 10.38.240.8 10.36.8.40
  Warning  Unhealthy         10s (x3 over 30s)  kubelet, dev-slave-110  Readiness probe failed: calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

同时发现dns服务起不来

[root@dev-master-105 calico]# kubectl get pods -n kube-system  -owide 

kube-dns-58d745cd6d-gdgcp                   0/3       ContainerCreating   3          14d       <none>          dev-master-105   <none>

解决办法:
删除dns 的pod

或者删除 /etc/resolv.conf 多余的dns服务


etcd 服务重启出现publish error: etcdserver: request timed out, possibly due to connection lost

具体日志如下:

Sep 30 12:26:51 master-47-34 etcd[5342]: publish error: etcdserver: request timed out, possibly due to connection lost
Sep 30 12:26:58 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:05 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:12 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:19 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:26 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:33 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:40 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:47 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:27:54 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:28:01 master-47-34 etcd[5342]: publish error: etcdserver: request timed out
Sep 30 12:28:08 master-47-34 etcd[5342]: publish error: etcdserver: request timed out

排查思路
使用journalctl -u etcd -f 命令查看是否有权限的问题

cannot open database at /opt/etcd/member/snap/db (open /opt/etcd/member/snap/db: permission denied)

如果有使用

chown etcd:etcd /opt/etcd/member/snap/db

再次启动有问题,就直接通过执行systemctl status etcd 获取启动命令,手动执行看看是啥错误

[root@master-47-34 wal]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2018-09-30 12:36:06 CST; 11ms ago
  Process: 10892 ExecStart=/usr/bin/etcd --name=etcd-47-34 --cert-file=/etc/kubernetes/ssl/etcd.pem --key-file=/etc/kubernetes/ssl/etcd-key.pem --peer-cert-file=/etc/kubernetes/ssl/etcd.pem --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://10.39.47.34:2380 --listen-peer-urls=https://10.39.47.34:2380 --listen-client-urls=https://10.39.47.34:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.39.47.34:2379 --initial-cluster-token=k8s-etcd-cluster --initial-cluster=etcd-47-34=https://10.39.47.34:2380,etcd-47-35=https://10.39.47.35:2380,etcd-47-36=https://10.39.47.36:2380 --eletion-timeout=3000 --initial-cluster-state=new --data-dir=/opt/etcd/ (code=exited, status=2)
 Main PID: 10892 (code=exited, status=2)

Sep 30 12:36:06 master-47-34 systemd[1]: etcd.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 30 12:36:06 master-47-34 systemd[1]: Failed to start Etcd Server.
Sep 30 12:36:06 master-47-34 systemd[1]: Unit etcd.service entered failed state.
Sep 30 12:36:06 master-47-34 systemd[1]: etcd.service failed.

执行

/usr/bin/etcd --name=etcd-47-34 --cert-file=/etc/kubernetes/ssl/etcd.pem --key-file=/etc/kubernetes/ssl/etcd-key.pem --peer-cert-file=/etc/kubernetes/ssl/etcd.pem --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://10.39.47.34:2380 --listen-peer-urls=https://10.39.47.34:2380 --listen-client-urls=https://10.39.47.34:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.39.47.34:2379 --initial-cluster-token=k8s-etcd-cluster --initial-cluster=etcd-47-34=https://10.39.47.34:2380,etcd-47-35=https://10.39.47.35:2380,etcd-47-36=https://10.39.47.36:2380  --initial-cluster-state=new --data-dir=/opt/etcd/

执行看看是啥问题,确认没有问题后经systemctl resatrt etcd重启etcd 服务


CICD镜像pull不下来的原因
现象----> 日志

[2018/09/30 17:14:24][Warning] Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 127.0.0.1 10.38.240.8 10.36.8.40 
[2018/09/30 17:14:26][Warning] Failed to pull image "harbor.enncloud.cn/paas/clone-repo:v2.2": [rpc error: code = Unknown desc = Error: image paas/clone-repo:v2.2 not found, rpc error: code = Unknown desc = Error: image paas/clone-repo:v2.2 not found] 
[2018/09/30 17:14:26][Warning] Error: ErrImagePull 
[2018/09/30 17:14:26][Warning] Error: ImagePullBackOff

解决办法

[root@dev-slave-108 ~]# cd /var/lib/kubelet/
[root@dev-slave-108 kubelet]# ls -l 
total 196
lrwxrwxrwx  1 root root     20 Dec  8  2017 config -> /root/.docker/config
lrwxrwxrwx  1 root root     25 Jan  8  2018 config.json -> /root/.docker/config.json
-rw-r--r--  1 root root     40 Sep 15 08:57 cpu_manager_state
drwxr-xr-x  2 root root   4096 Sep 30 17:20 device-plugins
drwx------  2 root root   4096 Sep 15 08:57 plugin-containers
drwxr-x---  4 root root   4096 Sep 17 11:05 plugins
drwxr-x--- 51 root root 180224 Sep 30 17:20 pods
[root@dev-slave-108 kubelet]# rm -rf config
[root@dev-slave-108 kubelet]# ls -l 
total 196
lrwxrwxrwx  1 root root     25 Jan  8  2018 config.json -> /root/.docker/config.json
-rw-r--r--  1 root root     40 Sep 15 08:57 cpu_manager_state
drwxr-xr-x  2 root root   4096 Sep 30 17:20 device-plugins
drwx------  2 root root   4096 Sep 15 08:57 plugin-containers
drwxr-x---  4 root root   4096 Sep 17 11:05 plugins
drwxr-x--- 51 root root 180224 Sep 30 17:20 pods

Oct 12 11:27:41 test-slave-114 kubelet[27200]: E1012 11:27:41.992855   27200 kubelet_volumes.go:140] Orphaned pod "32828c49-a053-11e8-a528-5254eec04736" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them.
Oct 12 11:27:43 test-slave-114 kubelet[27200]: E1012 11:27:43.988378   27200 kubelet_volumes.go:140] Orphaned pod "32828c49-a053-11e8-a528-5254eec04736" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them.
Oct 12 11:27:45 test-slave-114 kubelet[27200]: E1012 11:27:45.992073   27200 kubelet_volumes.go:140] Orphaned pod "32828c49-a053-11e8-a528-5254eec04736" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them.
Oct 12 11:27:48 test-slave-114 kubelet[27200]: E1012 11:27:48.119069   27200 kubelet_volumes.go:140] Orphaned pod "32828c49-a053-11e8-a528-5254eec04736" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them.
Oct 12 11:27:49 test-slave-114 kubelet[27200]: E1012 11:27:49.994550   27200 kubelet_volumes.go:140] Orphaned pod "32828c49-a053-11e8-a528-5254eec04736" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them.

解决办法 手动删除对应的pod主机上的信息,还不行就重启kubelet
例如

错误

Oct 12 12:54:36 test-slave-114 kubelet[68577]: E1012 12:54:36.359113   68577 kubelet_volumes.go:140] Orphaned pod "80a01854-c1fe-11e8-8f5d-5254eec04736" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
[root@test-slave-114 pods]# ls -l 
total 32
drwxr-x--- 5 root root 4096 Oct  8 13:42 01bdc09f-cabd-11e8-afbb-5254eec04736
drwxr-x--- 4 root root 4096 Sep 27 10:39 80a01854-c1fe-11e8-8f5d-5254eec04736
drwxr-x--- 5 root root 4096 Oct  9 10:11 89620113-cb68-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct 12 11:12 9cf8d3b8-cdcc-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct  8 16:40 c07b96d6-cad5-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct  9 09:30 d71de50f-cb62-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct 12 11:07 e8fe1a5f-cdcb-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct  8 13:21 f800c26d-cab9-11e8-afbb-5254eec04736
[root@test-slave-114 pods]# rm -rf 80a01854-c1fe-11e8-8f5d-5254eec04736
[root@test-slave-114 pods]# ls -l 
total 28
drwxr-x--- 5 root root 4096 Oct  8 13:42 01bdc09f-cabd-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct  9 10:11 89620113-cb68-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct 12 11:12 9cf8d3b8-cdcc-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct  8 16:40 c07b96d6-cad5-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct  9 09:30 d71de50f-cb62-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct 12 11:07 e8fe1a5f-cdcb-11e8-afbb-5254eec04736
drwxr-x--- 5 root root 4096 Oct  8 13:21 f800c26d-cab9-11e8-afbb-5254eec04736

通过命令journalctl -u kubelet -f查看kubelet的日志,没有再看到相关的错误日志


问题kubelet: - exit status 1, rootInodeErr: cmd [find /var/lib/docker/overlay/fda10ef69b731eb6aa0dae3b0cb438a5d114ac1195ec45e6472ba26cd7f169b9 -xdev -printf .] failed. stderr: find

Oct 23 16:46:43 proxy-20-85 kubelet: - exit status 1, rootInodeErr: cmd [find /var/lib/docker/overlay/fda10ef69b731eb6aa0dae3b0cb438a5d114ac1195ec45e6472ba26cd7f169b9 -xdev -printf .] failed. stderr: find: ‘/var/lib/docker/overlay/fda10ef69b731eb6aa0dae3b0cb438a5d114ac1195ec45e6472ba26cd7f169b9’: No such file or directory
Oct 23 16:46:43 proxy-20-85 kubelet: ; err: exit status 1, extraDiskErr: du command failed on /var/lib/docker/containers/a073fea95af9ba1227bc2e91437fbc02f34a07dd3963042cd9d37a0f6086b365 with output stdout: , stderr: du: cannot access ‘/var/lib/docker/containers/a073fea95af9ba1227bc2e91437fbc02f34a07dd3963042cd9d37a0f6086b365’: No such file or directory
Oct 23 16:46:43 proxy-20-85 kubelet: - exit status 1
Oct 23 16:46:43 proxy-20-85 kubelet: - exit status 1, rootInodeErr: cmd [find /var/lib/docker/overlay/43887091ad63cbc9d83048b602d2968200e91dbe5465fe194470b3ecf1a8ee98 -xdev -printf .] failed. stderr: find: ‘/var/lib/docker/overlay/43887091ad63cbc9d83048b602d2968200e91dbe5465fe194470b3ecf1a8ee98’: No such file or directory
Oct 23 16:46:43 proxy-20-85 kubelet: ; err: exit status 1, extraDiskErr: du command failed on /var/lib/docker/containers/b7bc9aecb053743d5af528fd16ae814157c084c5be3f023f6ffd08022fa50969 with output stdout: , stderr: du: cannot access ‘/var/lib/docker/containers/b7bc9aecb053743d5af528fd16ae814157c084c5be3f023f6ffd08022fa50969’: No such file or directory
Oct 23 16:46:43 proxy-20-85 kubelet: - exit status 1
Oct 23 16:59:28 proxy-20-85 kubelet: ERROR:1023 16:59:28.852195   20189 kubelet_network.go:378] Failed to ensure marking rule for KUBE-MARK-DROP: error checking rule: exit status 4: iptables: Resource temporarily unavailable.
Oct 23 17:29:47 proxy-20-85 kubelet: ERROR:1023 17:29:47.538377   20189 kubelet_network.go:412] Failed to ensure marking rule for KUBE-MARK-MASQ: error checking rule: exit status 4: iptables: Resource temporarily unavailable.
Oct 23 18:24:40 proxy-20-85 kubelet: ERROR:1023 18:24:40.045560   20189 kubelet_network.go:412] Failed to ensure marking rule for KUBE-MARK-MASQ: error checking rule: exit status 4: iptables: Resource temporarily unavailable.

相关issues


Oct 24 16:05:15 node-23-40 kubelet[14309]: /usr/local/go/src/runtime/asm_amd64.s:2361
Oct 24 16:05:15 node-23-40 kubelet[14309]: E1024 16:05:15.438773   14309 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Oct 24 16:05:15 node-23-40 kubelet[14309]: /usr/local/go/src/runtime/asm_amd64.s:573
Oct 24 16:05:15 node-23-40 kubelet[14309]: /usr/local/go/src/runtime/panic.go:502
Oct 24 16:05:15 node-23-40 kubelet[14309]: /usr/local/go/src/runtime/panic.go:63
Oct 24 16:05:15 node-23-40 kubelet[14309]: /usr/local/go/src/runtime/signal_unix.go:388
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/plugin.go:85
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:130
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:88
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:88
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:77
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/plugins.go:593
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/plugins.go:540
Oct 24 16:05:15 node-23-40 kubelet[14309]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/cache/desired_state_of_world.go:192

这个问题是flex-volume的问题,删除这个目录就可以了

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/qingcloud~flex-volume/

nfs mount failed
相关日志

[root@node-23-41 ~]# mount -t nfs 10.39.35.39:/mnt/cid_4c5ba11442cf_tanyanliao_nfs8 /var/lib/kubelet/pods/0d522e1b-d8ce-11e8-b14e-5254416bb222/volumes/kubernetes.io~nfs/tanyanliao-nfs8
mount: wrong fs type, bad option, bad superblock on 10.39.35.39:/mnt/cid_4c5ba11442cf_tanyanliao_nfs8,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

解决办法nfs挂载错误wrong fs type, bad option, bad superblock

apt-get install nfs-common

或者

yum install nfs-utils

问题

相关日志

failed to open log file "/var/log/pods/7e4623a0-dda3-11e8-885a-5254c2cdf2fd/carrier-test-mask_0.log": open /var/log/pods/7e4623a0-dda3-11e8-885a-5254c2cdf2fd/carrier-test-mask_0.log: no such file or directory
Nov 01 15:05:32 slave-20-50 kubelet[5662]: ERROR:1101 15:05:32.221060    5662 rbd.go:415] rbd: failed to setup mount /var/lib/kubelet/pods/4901cdc2-d27e-11e8-885a-5254c2cdf2fd/volumes/kubernetes.io~rbd/ceres-study.datadir-redis01-2 rbd: image ceres-study.CID-516874818ed4.datadir-redis01-2 is locked by other nodes.   carrier-test-mask-3950755848-wgt4h   0/1       rpc error: code = 2 desc = failed to start container "9a78c1dceaa3ff06f13f94bdd9e4de618605a7e9cf4d8e150ca1d51fd6eaf658": Error response from daemon: {"message":"oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/var/lib/kubelet/pods/c8e98e2b-dda4-11e8-885a-5254c2cdf2fd/volumes/kubernetes.io~configmap/configmap-volume-1/default.conf\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/dc291cf0d7814a457fd5df2d4aeff54274a84d3265c706727d350f74691607da/rootfs\\\\\\\" at \\\\\\\"/etc/nginx/conf.d/default.conf/default.conf\\\\\\\" caused \\\\\\\"lstat /var/lib/docker/devicemapper/mnt/dc291cf0d7814a457fd5df2d4aeff54274a84d3265c706727d350f74691607da/rootfs/etc/nginx/conf.d/default.conf/default.conf: not a directory\\\\\\\"\\\"\"\n: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"

挂载了镜像里没有的目录

/rootfs/etc/nginx/conf.d/default.conf/default.conf: not a directory
阅读更多
换一批

没有更多推荐了,返回首页