《kubernetes-1.8.0》06-addon-calico

《kubernetes-1.8.0》06-addon-calico

《kubernetes 1.8.0 测试环境安装部署》

时间：2017-11-23

一、修改calico配置

在mritd.me/部署 Calico中提及：

官方文档中直接创建的 calico.yml 文件中，使用 DaemonSet 方式启动 calico-node，同时 calico-node 的 IP 设置和 NODENAME 设置均为空，此时 calico-node 会进行自动获取，网络复杂情况下获取会出现问题；比如 IP 拿到了 docker 网桥的 IP，NODENAME 获取不正确等，最终导致出现很奇怪的错误

经测试 2.6.1 calico-node镜像版本确实有这样的问题，漠然的方法是calico node采用systemd的方式控制，其他组件通过daemonset安装。后续calico node镜像升级成 2.6.2 该问题就没有再出现。

获取最新的calico.yaml：

$sudo mkdir ~/calico/
$cd ~/calico/
$wget https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/calico.yaml

查看calico-node所采用的镜像版本：

修改calico.yaml文件：

# 替换 Etcd 地址
sed -i 's@.*etcd_endpoints:.*@\ \ etcd_endpoints:\ \"https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379\"@gi' calico.yaml

# 替换 Etcd 证书
export ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`
export ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`
export ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d '\n'`

sed -i "s@.*etcd-cert:.*@\ \ etcd-cert:\ ${ETCD_CERT}@gi" calico.yaml
sed -i "s@.*etcd-key:.*@\ \ etcd-key:\ ${ETCD_KEY}@gi" calico.yaml
sed -i "s@.*etcd-ca:.*@\ \ etcd-ca:\ ${ETCD_CA}@gi" calico.yaml

sed -i 's@.*etcd_ca:.*@\ \ etcd_ca:\ "/calico-secrets/etcd-ca"@gi' calico.yaml
sed -i 's@.*etcd_cert:.*@\ \ etcd_cert:\ "/calico-secrets/etcd-cert"@gi' calico.yaml
sed -i 's@.*etcd_key:.*@\ \ etcd_key:\ "/calico-secrets/etcd-key"@gi' calico.yaml

二、修改kubelet配置

根据官方文档要求 kubelet 配置必须增加--network-plugin=cni选项，所以需要修改 kubelet 配置:

###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=172.18.169.131"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node.131"

# location of the api-server
# KUBELET_API_SERVER=""

# Add your own!
KUBELET_ARGS="--cgroup-driver=cgroupfs \
              --network-plugin=cni \
              --cluster-dns=10.254.0.2 \
              --resolv-conf=/etc/resolv.conf \
              --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
              --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
              --fail-swap-on=false \
              --cert-dir=/etc/kubernetes/ssl \
              --cluster-domain=cluster.local. \
              --hairpin-mode=promiscuous-bridge \
              --serialize-image-pulls=false \
              --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0"

分别重启4个节点的kubelet：

systemctl daemon-reload
systemctl restart kubelet

查看节点状态：

[root@node-131 calico]# kubectl get node
NAME       STATUS     ROLES     AGE       VERSION
node.131   NotReady   <none>    12h       v1.8.0
node.132   NotReady   <none>    12h       v1.8.0
node.133   NotReady   <none>    12h       v1.8.0
node.134   NotReady   <none>    12h       v1.8.0

此时执行 kubectl get node 会看到 Node 为 NotReady 状态，属于正常情况

三、创建calico Daemonset

# 先创建 RBAC
kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/rbac.yaml

# 再创建 Calico Daemonset
kubectl create -f calico.yaml

quay.io仓库的镜像还是拖的动的，这里就不docker load了，除了calico-node image，其他的镜像可以通过mritd提供的tarball进行load。

检查Daemonset和相应pod运行情况：

[root@node-131 images]# kubectl get pods -n kube-system
NAME                                      READY     STATUS    RESTARTS   AGE
calico-kube-controllers-94b7cb897-krckw   1/1       Running   0          29m
calico-node-5dc8z                         2/2       Running   0          29m
calico-node-gm9k8                         2/2       Running   0          29m
calico-node-kt5fk                         2/2       Running   0          29m
calico-node-xds45                         2/2       Running   0          29m
[root@node-131 images]# kubectl get ds -n kube-system
NAME          DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
calico-node   4         4         4         4            4           <none>          29m

重启kubelet、docker：

systemctl restart kubelet
systemctl restart docker

四、测试跨主机通讯

创建测试实例：

## 创建 deployment
$ mkdir ~/demo
$ cd ~/demo
$ cat << EOF >> demo.deploy.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: demo-deployment
spec:
  replicas: 4
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: demo
        image: mritd/demo
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
EOF

$ kubectl create -f demo.deploy.yml

验证通信：

[root@node-131 images]# kubectl get pod -o wide
NAME                               READY     STATUS    RESTARTS   AGE       IP               NODE
demo-deployment-5fc9c54fb4-5pgfk   1/1       Running   0          2m        192.168.177.65   node.132
demo-deployment-5fc9c54fb4-5svgl   1/1       Running   0          2m        192.168.33.193   node.131
demo-deployment-5fc9c54fb4-dfcfd   1/1       Running   0          2m        192.168.188.1    node.133
demo-deployment-5fc9c54fb4-dttvb   1/1       Running   0          2m        192.168.56.65    node.134

[root@node-131 images]# kubectl exec -ti demo-deployment-5fc9c54fb4-5svgl bash
bash-4.3# ping 192.168.56.66
PING 192.168.56.66 (192.168.56.66): 56 data bytes
64 bytes from 192.168.56.66: seq=0 ttl=62 time=0.407 ms
^C
--- 192.168.56.66 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.407/0.407/0.407 ms

至此，群集网络组件calico搭建完成

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考链接：

https://mritd.me/2017/10/09/set-up-kubernetes-1.8-ha-cluster/

https://docs.projectcalico.org/v2.6/getting-started/kubernetes/