K8S-Demo集群实践03:准备集群各组件间HTTPS通讯需要的x509证书
- 这里一次准备好集群内部通讯需要的所有证书
- 你可以跳过本篇,在后续部署过程中需要相应证书的时候再创建
一、安装证书生产工具CFSSL
- 本案例使用CloudFlare的PKI工具集cfssl创建所有证书
- CSR:Certificate Signing Request 证书签名请求文件
[root@master1 ~]# mkdir -p /opt/install/soft/cfssl
[root@master1 ~]# cd /opt/soft/cfssl
[root@master1 cfssl]# wget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64
[root@master1 cfssl]# mv cfssl_1.4.1_linux_amd64 /opt/k8s/bin/cfssl
[root@master1 cfssl]# wget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64
[root@master1 cfssl]# mv cfssljson_1.4.1_linux_amd64 /opt/k8s/bin/cfssljson
[root@master1 cfssl]# wget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl-certinfo_1.4.1_linux_amd64
[root@master1 cfssl]# mv cfssl-certinfo_1.4.1_linux_amd64 /opt/k8s/bin/cfssl-certinfo
[root@master1 cfssl]# chmod +x /opt/k8s/bin/*
- 获取缺省配置,可以根据需要在这个基础上修改
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl print-defaults config > ca-config.json
[root@master1 cert]# cfssl print-defaults csr > ca-csr.json
二、创建根证书
- CA(Certificate Authority)是自签名的根证书,用来签名其它证书
- 如果没有特殊说明,后续操作均在master1节点上执行
- 证书临时存放目录 /opt/install/cert
1、创建根证书配置文件 ca-config.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"k8s-demo-server": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
},
"k8s-demo-client": {
"usages": [
"signing",
"key encipherment",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
- expiry: “87600h”:证书有效期设置为10年
- signing:表示该证书可用于签名其它证书(生成的 ca.pem 证书中 CA=TRUE)
- key encipherment:秘钥加密
- server auth:表示client可以用该该证书对server提供的证书进行验证
- client auth:表示server可以用该该证书对client提供的证书进行验证
2、创建证书签名请求文件 ca-csr.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > ca-csr.json <<EOF
{
"CN": "k8s-demo-ca",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "k8s-demo",
"OU": "jason@vip.qq.com"
}],
"ca": {
"expiry": "87600h"
}
}
EOF
- CN :Common Name,kube-apiserver从证书中提取该字段作为请求的用户名 (User Name),浏览器使用该字段验证网站是否合法
- C :国家
- ST :州,省
- L : 地区,城市
- O :Organization,组织名,公司名称,kube-apiserver从证书中提取该字段作为请求用户所属的组 (Group)
- OU :组织内部单位名称,部门名称
- 不同证书csr文件的CN、O、C、ST、L、OU组合必须不同,否则可能出现PEER’S CERTIFICATE HAS AN INVALID SIGNATURE错误
3、生成CA 根证书及私钥
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@master1 cert]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
4、分发证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# for node_ip in ${ALL_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /opt/k8s/etc/cert"
scp ca*.pem ca-config.json root@${node_ip}:/opt/k8s/etc/cert
done
三、生成集群管理员admin的证书(kubectl->kube-apiserver)
1、准备证书签名请求文件 kubectl-admin-csr.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > kubectl-admin-csr.json <<EOF
{
"CN": "k8s-demo-admin",
"hosts": [
"192.168.66.10",
"192.168.66.11",
"192.168.66.12"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "system:masters",
"OU": "jason@vip.qq.com"
}]
}
EOF
- kube-apiserver 将提取的 User、Group 作为RBAC授权的用户标识
- kubernetes使用RBAC进行角色权限控制,证书中的CN字段作为User,O字段作为Group
- 这里管理员名字设定为k8s-demo-admin,你可以根据实际情况修改
2、生成集群管理员证书和私钥
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -ca=/opt/install/cert/ca.pem \
-ca-key=/opt/install/cert/ca-key.pem \
-config=/opt/install/cert/ca-config.json \
-profile=k8s-demo-client kubectl-admin-csr.json | cfssljson -bare kubectl-admin
[root@master1 ~]# ls kubectl-admin*
- 如果授权部分操作权限给一个账户呢?证书如何生成?后面单独一篇文章来说RBAC
- RBAC:Role Based Access Control
- K8s还有其它五种的授权方式AlwaysDeny、AlwaysAllow、ABAC、Webhook、Node
3、分发证书到3个Master节点
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# for node_ip in ${MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp kubectl-admin*.pem root@${node_ip}:/opt/k8s/etc/cert
done
四、生成etcd节点之间通讯的证书
1、准备证书签名请求文件
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > etcd-csr.json <<EOF
{
"CN": "k8s-demo-etcd",
"hosts": [
"192.168.66.10",
"192.168.66.11",
"192.168.66.12"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "k8s-demo",
"OU": "jason@vip.qq.com"
}]
}
EOF
- UserName:k8s-demo-etcd
- Group:k8s-demo
2、生成etcd证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -ca=/opt/install/cert/ca.pem \
-ca-key=/opt/install/cert/ca-key.pem \
-config=/opt/install/cert/ca-config.json \
-profile=k8s-demo-server etcd-csr.json | cfssljson -bare etcd
ls etcd*pem
3、分发到3个master节点
[root@master1 ~]# cd /opt/install/cert
[root@master1 ~]# for node_ip in ${MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /opt/k8s/etcd/cert"
scp etcd*.pem root@${node_ip}:/opt/k8s/etcd/cert/
done
五、生成kube-apiserver证书
- kube-apiserver访问etcd、kubelet等接口时使用
1、准备证书签名请求文件 apiserver-csr.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 ~]# cat > apiserver-csr.json <<EOF
{
"CN": "k8s-demo-apiserver",
"hosts": [
"192.168.66.10",
"192.168.66.11",
"192.168.66.12",
"10.8.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.${CLUSTER_DNS_DOMAIN}."
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "k8s-demo",
"OU": "jason@vip.qq.com"
}]
}
EOF
- 请打开apiserver-csr.json文件检查使用的环境变量是否正确
2、生成kube-apiserver证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -ca=/opt/install/cert/ca.pem \
-ca-key=/opt/install/cert/ca-key.pem \
-config=/opt/install/cert/ca-config.json \
-profile=k8s-demo-server kube-apiserver-csr.json | cfssljson -bare kube-apiserver
[root@master1 ~]# ls kube-apiserver*pem
3、分发证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# for node_ip in ${MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /opt/k8s/etc/cert"
scp kube-apiserver*.pem root@${node_ip}:/opt/k8s/etc/cert/
done
六、生成kube-controller-manager证书
1、准备证书签名请求文件 controller-manager-csr.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > controller-manager-csr.json <<EOF
{
"CN": "k8s-demo-ctrl-mgr",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"192.168.66.10",
"192.168.66.11",
"192.168.66.12"
],
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "system:kube-controller-manager",
"OU": "jason@vip.qq.com"
}]
}
EOF
- 用户名 k8s-demo-ctrl-mgr 组名 system:kube-controller-manager
2、生成kube-controller-manager证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -ca=/opt/install/cert/ca.pem \
-ca-key=/opt/install/cert/ca-key.pem \
-config=/opt/install/cert/ca-config.json \
-profile=k8s-demo-server kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
[root@master1 ~]# ls kube-controller-manager*pem
3、分发证书到3个master节点
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# for node_ip in ${MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp kube-controller-manager*.pem root@${node_ip}:/opt/k8s/etc/cert/
done
七、生成kube-scheduler证书
1、准备证书签名请求文件 scheduler-csr.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > scheduler-csr.json <<EOF
{
"CN": "k8s-demo-scheduler",
"hosts": [
"192.168.66.10",
"192.168.66.11",
"192.168.66.12"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "system:kube-scheduler",
"OU": "jason@vip.qq.com"
}]
}
EOF
- 用户名 k8s-demo-scheduler 组 system:kube-scheduler
2、生成kube-scheduler证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -ca=/opt/install/cert/ca.pem \
-ca-key=/opt/install/cert/ca-key.pem \
-config=/opt/install/cert/ca-config.json \
-profile=k8s-demo-server kube-scheduler-csr.json | cfssljson -bare kube-scheduler
[root@master1 ~]# ls kube-scheduler*pem
3、分发证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# for node_ip in ${MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp kube-scheduler*.pem root@${node_ip}:/opt/k8s/etc/cert/
done
八、生成kube-proxy证书
1、准备证书签名请求文件 kube-proxy-csr.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > kube-proxy-csr.json <<EOF
{
"CN": "k8s-demo-kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "k8s-demo",
"OU": "jason@vip.qq.com"
}]
}
EOF
2、生成kube-proxy证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -ca=/opt/install/cert/ca.pem \
-ca-key=/opt/install/cert/ca-key.pem \
-config=/opt/install/cert/ca-config.json \
-profile=k8s-demo-client kube-proxy-csr.json | cfssljson -bare kube-proxy
[root@master1 ~]# ls kube-proxy*
3、分发证书到所有节点
[root@master1 ~]# cd /opt/cert
[root@master1 ~]# for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp kube-proxy*.pem root@${node_ip}:/etc/kubernetes/cert/
done
九、生成插件证书(aggregator.client)
1、准备证书签名请求文件 aggregator-client-csr.json
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cat > aggregator-client-csr.json <<EOF
{
"CN": "k8s-demo-aggregator",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C" : "CN",
"ST": "GuangDong",
"L" : "ShenZhen",
"O" : "k8s-demo",
"OU": "jason@vip.qq.com"
}]
}
EOF
- CN名称需要用于kube-apiserver的–requestheader-allowed-names参数中,否则后续访问metrics时会提示权限不足
2、生成aggregator-client证书
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
-ca-key=/opt/install/cert/ca-key.pem \
-config=/opt/install/cert/ca-config.json \
-profile=k8s-demo-client aggregator-client-csr.json | cfssljson -bare aggregator-client
[root@master1 ~]# ls aggregator-client*.pem
3、分发证书到3个master节点
[root@master1 ~]# cd /opt/install/cert
[root@master1 cert]# for node_ip in ${MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp aggregator-client*.pem root@${node_ip}:/etc/kubernetes/cert/
done
十、参考
- https://blog.csdn.net/molixuebeibi/article/details/91873432
- https://blog.csdn.net/lk2684753/article/details/100160856
附:K8s-Demo集群版本信息
组件 | 版本 | 命令 |
---|---|---|
kubernetes | 1.18.5 | kubectl version |
docker-ce | 19.03.11 | docker version 或者 rpm -qa | grep docker |
etcd | 3.4.3 | etcdctl version |
calico | 3.13.3 | calico -v |
coredns | 1.7.0 | coredns -version |
附:专栏链接
K8S-Demo集群实践00:搭建镜像仓库Harbor+安全扫描
K8S-Demo集群实践01:准备VMware虚拟机模板
K8S-Demo集群实践02:准备VMware虚拟机3台Master+3台Node
K8S-Demo集群实践03:准备集群各组件间HTTPS通讯需要的x509证书
K8S-Demo集群实践04:部署etcd三节点高可用集群
K8S-Demo集群实践05:安装kubectl并配置集群管理员账户
K8S-Demo集群实践06:部署kube-apiserver到master节点(3个无状态实例)
K8S-Demo集群实践07:kube-apiserver高可用方案
K8S-Demo集群实践08:部署高可用kube-controller-manager集群
K8S-Demo集群实践09:部署高可用kube-scheduler集群
K8S-Demo集群实践10:部署ipvs模式的kube-proxy组件
K8S-Demo集群实践11:部署ipvs模式的kube-kubelet组件
K8S-Demo集群实践12:部署Calico网络
K8S-Demo集群实践13:部署集群CoreDNS
K8S-Demo集群实践14:部署集群监控服务Metrics Server
K8S-Demo集群实践15:部署Kubernetes Dashboard
K8S-Demo集群实践16:部署Kube-Prometheus
K8S-Demo集群实践17:部署私有云盘owncloud(10.6版本)
K8S-Demo集群实践18:构建宇宙中第一个基础容器镜像
- 先用起来,通过操作实践认识k8s,积累多了自然就理解了
- 把理解的知识分享出来,自造福田,自得福缘
- 追求简单,容易使人理解,知识的上下文也是知识的一部分,例如版本,时间等
- 欢迎留言交流,也可以提出问题,一般在周末回复和完善文档
- Jason@vip.qq.com 2021-1-19。