kubelet上报的container相关指标总是莫名缺少相应的label值，如image, pod, name等

现象

无法获取到以下label的值：container, image, pod, name
集群刚搭建完的时候一切都是正常的，后来运行一段时间后就缺这缺那的。

原因排查

由于这些指标是kubelet上报的，因此看了对应节点上kubelet的日志，发现kubelet日志一直在报错。
centos系统

cat /var/log/messages  |grep "Failed"

Jul 19 17:37:51 m1 kubelet: E0719 17:37:51.724736     869 manager.go:1084] Failed to create existing container: /kubepods/besteffort/podcbdefb29-9b15-4be8-9b13-164b57549227/a1de8a3dc42f0ca1bbd965c9c7a3aaadc4a5accafa9959527e21acd12313db6c: failed to identify the read-write layer ID for container "a1de8a3dc42f0ca1bbd965c9c7a3aaadc4a5accafa9959527e21acd12313db6c". - open /var/lib/docker/image/overlay2/layerdb/mounts/a1de8a3dc42f0ca1bbd965c9c7a3aaadc4a5accafa9959527e21acd12313db6c/mount-id: no such file or directory
Jul 19 17:37:51 m1 kubelet: E0719 17:37:51.727490     869 manager.go:1084] Failed to create existing container: /kubepods/besteffort/podd48732a1-52f4-4a39-be7a-eaa9f174fbb9/582d3f2cfdcfa16797515991302c1509648d1e95946ea9a12fbf5f75cc2488ca: failed to identify the read-write layer ID for container "582d3f2cfdcfa16797515991302c1509648d1e95946ea9a12fbf5f75cc2488ca". - open /var/lib/docker/image/overlay2/layerdb/mounts/582d3f2cfdcfa16797515991302c1509648d1e95946ea9a12fbf5f75cc2488ca/mount-id: no such file or directory
Jul 19 17:37:51 m1 kubelet: E0719 17:37:51.729932     869 manager.go:1084] Failed to create existing container: /kubepods/besteffort/pode39a19ee6224e5d144913aec7b4d2615/619e8652e8228663b821e7a2babf22811931227868f3e0b9fe88cde146e07117: failed to identify the read-write layer ID for container "619e8652e8228663b821e7a2babf22811931227868f3e0b9fe88cde146e07117". - open /var/lib/docker/image/overlay2/layerdb/mounts/619e8652e8228663b821e7a2babf22811931227868f3e0b9fe88cde146e07117/mount-id: no such file or directory
Jul 19 17:37:51 m1 kubelet: E0719 17:37:51.732715     869 manager.go:1084] Failed to create existing container: /kubepods/burstable/pod869271af-b3b8-4188-9839-dd3eb41e892f/30b6f17363a39c0e2518b23ad2ad6eed673cc86dd444fcd51dcbe4c8e1d38b56: failed to identify the read-write layer ID for container "30b6f17363a39c0e2518b23ad2ad6eed673cc86dd444fcd51dcbe4c8e1d38b56". - open /var/lib/docker/image/overlay2/layerdb/mounts/30b6f17363a39c0e2518b23ad2ad6eed673cc86dd444fcd51dcbe4c8e1d38b56/mount-id: no such file or directory

由于kubelet的监控指标是从Cadvisor查询的，因此这个错跟Cadvisor有关。果然cadvisor项目下也存在类似issue
cAdvisor occasionally gets into a state where it has no container metadata

解决

按照cadvisor issue中的办法重启docker并不能彻底解决，问题还是会出现。
直接给kubelet增加了参数：–docker-root 问题解决。
由于docker-root默认值为/var/lib/docker，之所以在当前环境不行是因为公司的docker目录不在/var/lib/docker下，而是别的目录，修改为正确的目录后，重启kubelet问题解决。

--docker-root string                                                                                       DEPRECATED: docker root is read from docker info (this is a fallback, default: /var/lib/docker) (default "/var/lib/docker")

1.修改kubelet启动参数
k8s组件通过systemctl管理的，可以在/etc/systemd/system 下找到kubelet的配置。
在kubelet.service中的ExecStart后面新增–docker-root=/var/lib/it/docker
2. 修改完后reload kubelet即可

systemctl restart kubelet

修改后的kubelet.service

[root@n1 system]# cat kubelet.service 
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=http://kubernetes.io/docs/

[Service]
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.51.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests"
Environment="KUBELET_PRIVILEGED_ARGS="
Environment="KUBELET_RUNTIME_ARGS=--pod-infra-container-image=docker.io/kubernetes/pause:go"
Environment="KUBELET_CUSTOM_ARGS="

ExecStart=/usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --register-node $KUBELET_PRIVILEGED_ARGS $KUBELET_RUNTIME_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_DNS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_CUSTOM_ARGS --docker-root=/var/lib/it/docker
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target