部署k8s ssl集群实践5:部署flannel网络:-CSDN博客

参考文档:
https://github.com/opsnull/follow-me-install-kubernetes-cluster
感谢作者的无私分享。
集群环境已搭建成功跑起来。
文章是部署过程中遇到的错误和详细操作步骤记录。如有需要对比参考，请按照顺序阅读和测试。

5.1
下载和分发flannel软件包

[root@k8s-master kubernetes]# wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
[root@k8s-master kubernetes]# ls
etcd-v3.3.7-linux-amd64         flannel-v0.10.0-linux-amd64.tar.gz  kubernetes-client-linux-amd64.tar.gz
etcd-v3.3.7-linux-amd64.tar.gz  kubernetes                          kubernetes-src.tar.gz
[root@k8s-master kubernetes]#
[root@k8s-master kubernetes]# mkdir flannel
[root@k8s-master kubernetes]# tar -zxvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel
flanneld
mk-docker-opts.sh
README.md
[root@k8s-master kubernetes]# cd flannel/
[root@k8s-master flannel]# ll
总用量 35492
-rwxr-xr-x 1 k8s k8s 36327752 1月  24 2018 flanneld
-rwxr-xr-x 1 k8s k8s     2139 3月  18 2017 mk-docker-opts.sh
-rw-rw-r-- 1 k8s k8s     4298 12月 24 2017 README.md
[root@k8s-master flannel]#

分发

[root@k8s-master flannel]# cp flanneld mk-docker-opts.sh /opt/k8s/bin
[root@k8s-master flannel]# scp flanneld mk-docker-opts.sh root@k8s-node1:/opt/k8s/bin
flanneld                                                                              100%   35MB  91.9MB/s   00:00   
mk-docker-opts.sh                                                                     100% 2139     2.2MB/s   00:00   
[root@k8s-master flannel]# scp flanneld mk-docker-opts.sh root@k8s-node2:/opt/k8s/bin
flanneld                                                                              100%   35MB  90.8MB/s   00:00   
mk-docker-opts.sh                                                                     100% 2139     3.2MB/s   00:00   
[root@k8s-master flannel]#

查看下/opt/k8s/bin目录下的权限

[root@k8s-master flannel]# ll /opt/k8s/bin
总用量 141792
-rwxr-xr-x 1 root root 10376657 8月  16 15:33 cfssl
-rwxr-xr-x 1 root root  6595195 8月  16 15:33 cfssl-certinfo
-rwxr-xr-x 1 root root  2277873 8月  16 15:33 cfssljson
-rwxr-xr-x 1 k8s  root     1740 8月  16 15:23 environment.sh
-rwxr-xr-x 1 root root 19266976 8月  20 15:30 etcd
-rwxr-xr-x 1 root root 16018720 8月  20 15:30 etcdctl
-rwxr-xr-x 1 root root 36327752 8月  21 13:54 flanneld
-rwxr-xr-x 1 root root 54308597 8月  20 14:27 kubectl
-rwxr-xr-x 1 root root     2139 8月  21 13:54 mk-docker-opts.sh
[root@k8s-master flannel]#

把属主改成k8s，每个节点都改

[root@k8s-master flannel]# chown -R k8s /opt/k8s/bin
[root@k8s-master flannel]# ll /opt/k8s/bin
总用量 141792
-rwxr-xr-x 1 k8s root 10376657 8月  16 15:33 cfssl
-rwxr-xr-x 1 k8s root  6595195 8月  16 15:33 cfssl-certinfo
-rwxr-xr-x 1 k8s root  2277873 8月  16 15:33 cfssljson
-rwxr-xr-x 1 k8s root     1740 8月  16 15:23 environment.sh
-rwxr-xr-x 1 k8s root 19266976 8月  20 15:30 etcd
-rwxr-xr-x 1 k8s root 16018720 8月  20 15:30 etcdctl
-rwxr-xr-x 1 k8s root 36327752 8月  21 13:54 flanneld
-rwxr-xr-x 1 k8s root 54308597 8月  20 14:27 kubectl
-rwxr-xr-x 1 k8s root     2139 8月  21 13:54 mk-docker-opts.sh
[root@k8s-master flannel]#

5.2
创建flannel网络的证书和私钥
flannel 从 etcd 集群存取网段分配信息，而 etcd 集群启用了双向 x509 证书认证，所以
需要为 flanneld 生成证书和私钥。
创建证书签名请求：

[root@k8s-master flanneld]# cat flanneld-csr.json
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "SZ",
"O": "k8s",
"OU": "4Paradigm"
}
]
}

该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；
生成证书和私钥：

[root@k8s-master flanneld]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
[root@k8s-master flanneld]# ls
flanneld.csr  flanneld-csr.json  flanneld-key.pem  flanneld.pem
[root@k8s-master flanneld]#

分发证书
创建证书存放目录并修改属主

[root@k8s-master ~]# mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld/cert
[root@k8s-master ~]# ssh root@k8s-node1 "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld/cert"
[root@k8s-master ~]# ssh root@k8s-node2 "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld/cert"
[root@k8s-master ~]#

分发证书

[root@k8s-master flanneld]# cp flanneld* /etc/flanneld/cert/
[root@k8s-master flanneld]# scp flanneld* root@k8s-node1:/etc/flanneld/cert/
flanneld.csr                                                                          100%  989     1.4MB/s   00:00   
flanneld-csr.json                                                                     100%  156   297.3KB/s   00:00   
flanneld-key.pem                                                                      100% 1679     2.9MB/s   00:00   
flanneld.pem                                                                          100% 1371     2.8MB/s   00:00   
[root@k8s-master flanneld]# scp flanneld* root@k8s-node2:/etc/flanneld/cert/
flanneld.csr                                                                          100%  989     1.1MB/s   00:00   
flanneld-csr.json                                                                     100%  156   218.8KB/s   00:00   
flanneld-key.pem                                                                      100% 1679     2.5MB/s   00:00   
flanneld.pem                                                                          100% 1371     2.3MB/s   00:00   
[root@k8s-master flanneld]#

修改好文件属主

[root@k8s-master ~]# chown -R k8s /etc/flanneld/cert/
[root@k8s-master ~]# ssh root@k8s-node1 "chown -R k8s /etc/flanneld/cert/"
[root@k8s-master ~]# ssh root@k8s-node2 "chown -R k8s /etc/flanneld/cert/"
[root@k8s-master ~]#

5.3
pod网络信息写入etcd集群
##必须先source

[root@k8s-node1 cert]# source /opt/k8s/bin/environment.sh
[root@k8s-node1 cert]# echo ${ETCD_ENDPOINTS}
https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.95:2379
[root@k8s-node1 cert]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'",
"SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
{"Network":"172.30.0.0/16",
"SubnetLen": 24, "Backend": {"Type": "vxlan"}}

flanneld 当前版本 (v0.10.0) 不支持 etcd v3，故使用 etcd v2 API 写入配置 key 和
网段数据；
写入的 Pod 网段 ${CLUSTER_CIDR} 必须是 /16 段地址，必须与 kube-controllermanager
的 --cluster-cidr 参数值一致；

5.4
创建 flanneld 的 systemd unit 文件

[root@k8s-master ~]# source /opt/k8s/bin/environment.sh
[root@k8s-master ~]# export IFACE=ens192

[root@k8s-master flanneld]# cat flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \
-etcd-cafile=/etc/kubernetes/cert/ca.pem \
-etcd-certfile=/etc/flanneld/cert/flanneld.pem \
-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \
-etcd-endpoints=https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.95:2379 \
-etcd-prefix=/kubernetes/network \
-iface=ens192
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
[root@k8s-master flanneld]#

mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入
/run/flannel/docker 文件，后续 docker 启动时使用这个文件中的环境变量配
置 docker0 网桥；
flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如
内网和公网）的节点，可以用 -iface 参数指定通信接口，如上面的 ens192 接口;
flanneld 运行时需要 root 权限；

分发文件到节点

[root@k8s-master flanneld]# cp flanneld.service /etc/systemd/system/
[root@k8s-master flanneld]# scp flanneld.service root@k8s-node1:/etc/systemd/system/
flanneld.service                                                                      100%  640   873.5KB/s   00:00   
[root@k8s-master flanneld]# scp flanneld.service root@k8s-node2:/etc/systemd/system/
flanneld.service                                                                      100%  640   732.2KB/s   00:00   
[root@k8s-master flanneld]#

检查下文件有没有x权限
需要加上

[root@k8s-master flanneld]# chmod -R +x /etc/systemd/system

5.5
启动flanneld服务

[root@k8s-master ~]# systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /etc/systemd/system/flanneld.service.
Created symlink from /etc/systemd/system/docker.service.requires/flanneld.service to /etc/systemd/system/flanneld.service.

启动成功，检查分配给flanneld的各个pod网络信息

[root@k8s-master ~]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem ls kubernetes/network
/kubernetes/network/config
/kubernetes/network/subnets
[root@k8s-master ~]#
[root@k8s-master ~]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem ls kubernetes/network/subnets
/kubernetes/network/subnets/172.30.65.0-24
/kubernetes/network/subnets/172.30.76.0-24
/kubernetes/network/subnets/172.30.42.0-24
[root@k8s-master ~]#
[root@k8s-master ~]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem get kubernetes/network/subnets/172.30.42.0-24
{"PublicIP":"192.168.1.95","BackendType":"vxlan","BackendData":{"VtepMAC":"b6:44:59:11:bf:45"}}
[root@k8s-master ~]#

检查节点的flannel接口

4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 86:28:76:b1:11:24 brd ff:ff:ff:ff:ff:ff
    inet 172.30.65.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::8428:76ff:feb1:1124/64 scope link
       valid_lft forever preferred_lft forever
[root@k8s-master ~]#

分别ping了测试下

[root@k8s-node2 ~]# ping 172.30.76.0
PING 172.30.76.0 (172.30.76.0) 56(84) bytes of data.
64 bytes from 172.30.76.0: icmp_seq=1 ttl=64 time=0.676 ms
64 bytes from 172.30.76.0: icmp_seq=2 ttl=64 time=0.464 ms
^C
--- 172.30.76.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.464/0.570/0.676/0.106 ms
[root@k8s-node2 ~]# ping 172.30.65.0
PING 172.30.65.0 (172.30.65.0) 56(84) bytes of data.
64 bytes from 172.30.65.0: icmp_seq=1 ttl=64 time=0.554 ms
^C
--- 172.30.65.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.554/0.554/0.554/0.000 ms
[root@k8s-node2 ~]#

完成了

转载于:https://blog.51cto.com/goome/2164827