一. 系统软件环境
软件 | 版本 |
---|---|
操作系统 | CentOS Linux release 7.8.2003 (Core) |
Docker | docker-20.10.6-ce |
Kubernetes | 1.20.6 |
ETCD | 3.4.15 |
节点组件
角色 | IP | 组件 |
---|---|---|
k8s-master1 | 3.1.101.49 | kube-apiserver, kube-controller-manager, kube-scheduler, docker, kubelet, kube-proxy,etcd,nginx,keepalived |
k8s-master2 | 3.1.101.50 | kube-apiserver, kube-controller-manager, kube-scheduler, docker, kubelet, kube-proxy,etcd,nginx,keepalived |
k8s-node1 | 3.1.101.51 | docker, kubelet, kube-proxy, etcd |
k8s-node2 | 3.1.101.52 | docker, kubelet, kube-proxy, etcd |
k8s-node1 | 3.1.101.53 | docker, kubelet, kube-proxy, etcd |
VIP | 3.1.101.45 |
master1和master2为基于Ningx+Keepalived的高可用
二. 基础环境配置
所有NODE节点
2.1 创建目录
## 创建目录结构
mkdir -pv /opt/etcd/{bin,cfg,ssl,logs}
mkdir -pv /opt/k8s/{bin,cfg,ssl,logs,yaml}
mkdir -pv /opt/cni/{bin,cfg,yaml}
mkdir -pv /etc/cni/
2.2 hosts配置
cat >> /etc/hosts << EOF
3.1.101.49 k8s-master1
3.1.101.50 k8s-master2
3.1.101.51 k8s-node1
3.1.101.52 k8s-node2
3.1.101.53 k8s-node3
3.1.101.49 etcd-1
3.1.101.50 etcd-2
3.1.101.51 etcd-3
3.1.101.52 etcd-4
3.1.101.53 etcd-5
EOF
## 2.3 主机名修改
在对应节点分别执行
hostnamectl set-hostname k8s-master1
hostnamectl set-hostname k8s-master2
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2
hostnamectl set-hostname k8s-node3
2.4 其他系统设置
## 启用IPVS模式相关配置
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
## 生效
sysctl --system
## 关闭缓存,配置/etc/fstab,永久关闭
# 临时关闭:
swapoff -a
## 关闭NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager
## 时间同步
/usr/bin/cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
ntpdate time.windows.com
##配置环境变量(根据节点情况,一般配置master节点即可)
echo 'export PATH=$PATH:/opt/k8s/bin/' >> /etc/profile
echo 'export PATH=$PATH:/opt/etcd/bin/' >> /etc/profile
source /etc/profile
## 为了便捷操作,在k8s-master1上创建免密登录其他节点
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@k8s-node1
ssh-copy-id -i /root/.ssh/id_rsa.pub root@k8s-node2
三. 安装cfssl证书工具
master节点
## 创建自签证书目录
mkdir -pv /data/TLS/{etcd,k8s}
## 下载地址
https://github.com/cloudflare/cfssl/releases/download
## 移动到/usr/bin目录下
mv cfssl_1.5.0_linux_amd64 /usr/bin/cfssl
mv cfssl-certinfo_1.5.0_linux_amd64 /usr/bin/cfssl-certinfo
mv cfssljson_1.5.0_linux_amd64 /usr/bin/cfssljson
## 添加可执行权限
chmod +x /usr/bin/cfssl*
## 生成配置模版命令
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json
四. 部署ETCD集群
节点名称 | IP |
---|---|
etcd-1 | 3.1.101.49 |
etcd-2 | 3.1.101.50 |
etcd-3 | 3.1.101.51 |
etcd-4 | 3.1.101.52 |
etcd-5 | 3.1.101.53 |
4.1 自签TLS证书
- ETCD-1操作,然后同步到其他节点
自签证书颁发机构(CA)
cd /data/TLS/etcd/
自签CA
cd /data/TLS/etcd/
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json << EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
生成证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
使用自签CA签发Etcd HTTPS证书
创建证书申请文件(hosts中要包含所有etcd节点ip,也可以多写几个预留)
cat > server-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"3.1.101.49",
"3.1.101.50",
"3.1.101.51",
"3.1.101.52",
"3.1.101.53",
"3.1.101.54",
"3.1.101.55"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare serverlsca-config.json ca-key.pem server-csr.jsonca.csr ca.pem server-key.pemca-csr.json server.csr server.pem
同步证书
cp /data/TLS/etcd/*.pem /opt/etcd/ssl/ls /opt/etcd/ssl/ca-key.pem ca.pem server-key.pem server.pemrsync -av /data/TLS/etcd/*.pem etcd-2:/opt/etcd/ssl/rsync -av /data/TLS/etcd/*.pem etcd-3:/opt/etcd/ssl/rsync -av /data/TLS/etcd/*.pem etcd-4:/opt/etcd/ssl/rsync -av /data/TLS/etcd/*.pem etcd-5:/opt/etcd/ssl/
4.2 ETCD安装
下载地址
https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz
解压部署
- ETCD-1操作,然后同步到其他节点
tar -zxf etcd-v3.4.15-linux-amd64.tar.gz
mv etcd-v3.4.15-linux-amd64/etcd* /opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-1:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-2:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-3:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-4:/opt/etcd/bin/
rsync -av /opt/etcd/bin/* etcd-5:/opt/etcd/bin/
4.3 创建ETCD配置文件
ETCD各节点配置基本相同, 注意修改如下配置, 修改成本机etcd-name或者IP
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://3.1.101.49:2380"
ETCD_LISTEN_CLIENT_URLS="https://3.1.101.49:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://3.1.101.49:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://3.1.101.49:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://3.1.101.49:2380,etcd-2=https://3.1.101.50:2380,etcd-3=https://3.1.101.51:2380,etcd-4=https://3.1.101.52:2380,etcd-5=https://3.1.101.53:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
- ETCD_NAME:节点名称,集群中唯一
- ETCD_DATA_DIR:数据目录
- ETCD_LISTEN_PEER_URLS:集群通信监听地址
- ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
- ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
- ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
- ETCD_INITIAL_CLUSTER:集群节点地址
- ETCD_INITIAL_CLUSTER_TOKEN:集群Token
- ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群
4.4 创建ETCD启动文件
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
4.5 启动ETCD
## 重载启动配置文件
systemctl daemon-reload
## 启动etcd
systemctl restart etcd
## 加入开机自启动
systemctl enable etcd
## 4.6 验证ETCD状态
/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379" endpoint health
/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379" member list
五. 二进制部署DOCKER
5.1 下载地址
https://download.docker.com/linux/static/stable/x86_64/
tar zxf docker-20.10.6.tgz
mv docker/* /usr/bin/
编辑docker配置文件
mkdir /etc/dockercat > /etc/docker/daemon.json << EOF{"registry-mirrors": ["https://gsm39obv.mirror.aliyuncs.com"]}EOF
5.2 创建systemd启动文件
cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
5.3 启动docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
六. kubenetes部署
二进制文件部署
下载地址
https://dl.k8s.io/v1.20.6/kubernetes-server-linux-amd64.tar.gz
解压
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
master节点
rsync -av kubectl kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy /opt/k8s/bin/
node节点
rsync -av kubelet kube-proxy root@k8s-node1:/opt/k8s/bin/
rsync -av kubelet kube-proxy root@k8s-node2:/opt/k8s/bin/
rsync -av kubelet kube-proxy root@k8s-node3:/opt/k8s/bin/
七. Master节点部署
7.1 部署kube-apiserver
生成kube-apiserver证书
- 自签证书颁发机构(CA)
cd /data/TLS/k8s/
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
- 使用自签CA签发kube-apiserver HTTPS证书
## 创建证书申请文件:
cat > server-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"3.1.101.48",
"3.1.101.49",
"3.1.101.50",
"3.1.101.51",
"3.1.101.52",
"3.1.101.53",
"3.1.101.45",
"3.1.101.46",
"3.1.101.57",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
注:上述文件hosts字段中IP为所有Master/LB/VIP IP,一个都不能少!为了方便后期扩容可以多写几个预留的IP。
生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
同步证书
# master 节点
cp /data/TLS/k8s/ca*pem /opt/k8s/ssl/
cp /data/TLS/k8s/server*pem /opt/k8s/ssl/
# 同步至node节点
scp /data/TLS/k8s/ca.pem root@k8s-node1:/opt/k8s/ssl
scp /data/TLS/k8s/ca.pem root@k8s-node2:/opt/k8s/ssl
scp /data/TLS/k8s/ca.pem root@k8s-node3:/opt/k8s/ssl
创建conf配置文件
cat > /opt/k8s/cfg/kube-apiserver.conf << EOFKUBE_APISERVER_OPTS="--logtostderr=false \\--feature-gates=RemoveSelfLink=false \\--v=2 \\--log-dir=/opt/k8s/logs \\--etcd-servers=https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379 \\--bind-address=3.1.101.49 \\--secure-port=6443 \\--advertise-address=3.1.101.49 \\--allow-privileged=true \\--service-cluster-ip-range=10.0.0.0/24 \\--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\--authorization-mode=RBAC,Node \\--enable-bootstrap-token-auth=true \\--token-auth-file=/opt/k8s/cfg/token.csv \\--service-node-port-range=30000-32767 \\--kubelet-client-certificate=/opt/k8s/ssl/server.pem \\--kubelet-client-key=/opt/k8s/ssl/server-key.pem \\--tls-cert-file=/opt/k8s/ssl/server.pem \\--tls-private-key-file=/opt/k8s/ssl/server-key.pem \\--client-ca-file=/opt/k8s/ssl/ca.pem \\--service-account-key-file=/opt/k8s/ssl/ca-key.pem \\--service-account-issuer=https://kubernetes.default.svc.cluster.local \\--service-account-signing-key-file=/opt/k8s/ssl/server-key.pem \\--etcd-cafile=/opt/etcd/ssl/ca.pem \\--etcd-certfile=/opt/etcd/ssl/server.pem \\--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\--requestheader-client-ca-file=/opt/k8s/ssl/ca.pem \\--proxy-client-cert-file=/opt/k8s/ssl/server.pem \\--proxy-client-key-file=/opt/k8s/ssl/server-key.pem \\--requestheader-allowed-names=kubernetes \\--requestheader-extra-headers-prefix=X-Remote-Extra- \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User \\--enable-aggregator-routing=true \\--audit-log-maxage=30 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-path=/opt/k8s/logs/k8s-audit.log"EOF
创建TLS机制所需TOKEN
- TLS Bootstraping:Master apiserver启用TLS认证后,Node节点kubelet和kube-proxy要与kube-apiserver进行通信,必须使用CA签发的有效证书才可以,当Node节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式,目前主要用于kubelet,kube-proxy还是由我们统一颁发一个证书。
## 创建kube-apiserver.conf中所需的token.csvecho "`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`,kubelet-bootstrap,10001,"system:node-bootstrapper"" > /opt/k8s/cfg/token.csv
创建systemd启动文件
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/kubernetes/kubernetesAfter=network.target[Service]EnvironmentFile=/opt/k8s/cfg/kube-apiserver.confExecStart=/opt/k8s/bin/kube-apiserver \$KUBE_APISERVER_OPTSRestart=on-failure[Install]WantedBy=multi-user.targetEOF
启动apiserver
systemctl daemon-reloadsystemctl restart kube-apiserversystemctl enable kube-apiserver
7.2 部署kube-controller-manager
创建conf配置文件
cat > /opt/k8s/cfg/kube-controller-manager.conf << EOFKUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\--v=2 \\--log-dir=/opt/k8s/logs \\--leader-elect=true \\--kubeconfig=/opt/k8s/cfg/kube-controller-manager.kubeconfig \\--bind-address=127.0.0.1 \\--allocate-node-cidrs=true \\--cluster-cidr=10.244.0.0/16 \\--service-cluster-ip-range=10.0.0.0/24 \\--cluster-signing-cert-file=/opt/k8s/ssl/ca.pem \\--cluster-signing-key-file=/opt/k8s/ssl/ca-key.pem \\--root-ca-file=/opt/k8s/ssl/ca.pem \\--service-account-private-key-file=/opt/k8s/ssl/ca-key.pem \\--cluster-signing-duration=87600h0m0s"EOF
- –kubeconfig:连接apiserver配置文件
- –leader-elect:当该组件启动多个时,自动选举(HA)
- –cluster-signing-cert-file/–cluster-signing-key-file:自动为kubelet颁发证书的CA,与apiserver保持一致
生成kubeconfig配置文件
生成证书
kube-controller-manager.kubeconfig
cd /data/TLS/k8s# 创建证书请求文件cat > kube-controller-manager-csr.json << EOF{ "CN": "system:kube-controller-manager", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ]}EOF# 生成证书cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
生成kubeconfig文件(在/data/TLS/k8s下执行)
KUBE_CONFIG="/opt/k8s/cfg/kube-controller-manager.kubeconfig"KUBE_APISERVER="https://3.1.101.49:6443"kubectl config set-cluster kubernetes \ --certificate-authority=/opt/k8s/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG}kubectl config set-credentials kube-controller-manager \ --client-certificate=./kube-controller-manager.pem \ --client-key=./kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG}kubectl config set-context default \ --cluster=kubernetes \ --user=kube-controller-manager \ --kubeconfig=${KUBE_CONFIG}kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
创建systemd启动文件
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF[Unit]Description=Kubernetes Controller ManagerDocumentation=https://github.com/kubernetes/kubernetesAfter=kube-apiserver.service[Service]EnvironmentFile=/opt/k8s/cfg/kube-controller-manager.confExecStart=/opt/k8s/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTSRestart=on-failure[Install]WantedBy=multi-user.targetEOF
启动kube-controller-manager
systemctl daemon-reload
systemctl restart kube-controller-manager
systemctl enable kube-controller-manager
7.3 部署kube-scheduler
创建conf配置文件
cat > /opt/k8s/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/k8s/logs \\
--leader-elect \\
--kubeconfig=/opt/k8s/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF
- –kubeconfig:连接apiserver配置文件
- –leader-elect:当该组件启动多个时,自动选举(HA)
生成kubeconfig配置文件
kube-scheduler.kubeconfig
cd /data/TLS/k8s
# 创建证书请求文件
cat > kube-scheduler-csr.json << EOF
{
"CN": "system:kube-scheduler",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
生成kubeconfig文件(在/data/TLS/k8s下执行)
KUBE_CONFIG="/opt/k8s/cfg/kube-scheduler.kubeconfig"KUBE_APISERVER="https://3.1.101.49:6443"kubectl config set-cluster kubernetes \ --certificate-authority=/opt/k8s/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG}kubectl config set-credentials kube-scheduler \ --client-certificate=./kube-scheduler.pem \ --client-key=./kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG}kubectl config set-context default \ --cluster=kubernetes \ --user=kube-scheduler \ --kubeconfig=${KUBE_CONFIG}kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
创建systemd启动文件
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF[Unit]Description=Kubernetes SchedulerDocumentation=https://github.com/kubernetes/kubernetesAfter=kube-apiserver.service[Service]EnvironmentFile=/opt/k8s/cfg/kube-scheduler.confExecStart=/opt/k8s/bin/kube-scheduler \$KUBE_SCHEDULER_OPTSRestart=on-failure[Install]WantedBy=multi-user.targetEOF
启动kube-scheduler
systemctl daemon-reloadsystemctl restart kube-schedulersystemctl enable kube-scheduler
7.4 查看集群状态
生成kubectl连接集群的证书
cd /data/TLS/k8scat > admin-csr.json <<EOF{ "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ]}EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
生成kubeconfig配置文件
mkdir -pv /root/.kubeKUBE_CONFIG="/root/.kube/config"KUBE_APISERVER="https://3.1.101.49:6443"kubectl config set-cluster kubernetes \ --certificate-authority=/opt/k8s/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG}kubectl config set-credentials cluster-admin \ --client-certificate=./admin.pem \ --client-key=./admin-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG}kubectl config set-context default \ --cluster=kubernetes \ --user=cluster-admin \ --kubeconfig=${KUBE_CONFIG}kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
查看集群状态
kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+NAME STATUS MESSAGE ERRORcontroller-manager Healthy okscheduler Healthy oketcd-3 Healthy {"health":"true"}etcd-0 Healthy {"health":"true"}etcd-1 Healthy {"health":"true"}etcd-2 Healthy {"health":"true"}etcd-4 Healthy {"health":"true"}
7.5 置到k8s-master2
- master1节点
同步k8s-master1配置到k8s-master2
## 同步k8s目录rsync -av /opt/k8s/* root@k8s-master2:/opt/k8s/## 同步连接集群所需的kube认证目录rsync -av /root/.kube root@k8s-master2:/root## 同步master组件启动文件rsync -av /usr/lib/systemd/system/kube-apiserver.service root@k8s-master2:/usr/lib/systemd/system/kube-apiserver.servicersync -av /usr/lib/systemd/system/kube-controller-manager.service root@k8s-master2:/usr/lib/systemd/system/kube-controller-manager.servicersync -av /usr/lib/systemd/system/kube-scheduler.service root@k8s-master2:/usr/lib/systemd/system/kube-scheduler.service
- master节点
- 修改对应组件配置中以及kube中的IP为master2本机IP
- 启动组件
systemctl daemon-reloadsystemctl restart kube-apiserversystemctl enable kube-apiserversystemctl daemon-reloadsystemctl restart kube-controller-managersystemctl enable kube-controller-managersystemctl daemon-reloadsystemctl restart kube-schedulersystemctl enable kube-scheduler
- 查看集群状态
kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+NAME STATUS MESSAGE ERRORscheduler Healthy okcontroller-manager Healthy oketcd-2 Healthy {"health":"true"}etcd-3 Healthy {"health":"true"}etcd-1 Healthy {"health":"true"}etcd-4 Healthy {"health":"true"}etcd-0 Healthy {"health":"true"}
八. NODE节点部署
- master1上执行
master也需要部署node节点相应组件: kubelet和kube-proxy
8.1 部署kubelet
创建conf配置文件
cat > /opt/k8s/cfg/kubelet.conf << EOFKUBELET_OPTS="--logtostderr=false \\--v=2 \\--log-dir=/opt/k8s/logs \\--hostname-override=k8s-master1 \\--network-plugin=cni \\--kubeconfig=/opt/k8s/cfg/kubelet.kubeconfig \\--bootstrap-kubeconfig=/opt/k8s/cfg/bootstrap.kubeconfig \\--config=/opt/k8s/cfg/kubelet-config.yml \\--cert-dir=/opt/k8s/ssl \\--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.1"EOF
- –hostname-override:显示名称,为节点hostname, 集群中唯一
- –network-plugin:启用CNI
- –kubeconfig:空路径,会自动生成,后面用于连接apiserver
- –bootstrap-kubeconfig:首次启动向apiserver申请证书
- –config:配置参数文件
- –cert-dir:kubelet证书生成目录
- –pod-infra-container-image:管理Pod网络容器的镜像
创建yml参数配置文件
kubelet-config.yml文件内容
cat > /opt/k8s/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/k8s/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
创建bootstrap.kubeconfig配置文件
- kubelet初次加入集群引导kubeconfig文件
master节点操作
KUBE_CONFIG="/opt/k8s/cfg/bootstrap.kubeconfig"
KUBE_APISERVER="https://3.1.101.49:6443"
TOKEN=`cat /opt/k8s/cfg/token.csv|awk -F',' '{print $1}'` # 与token.csv里保持一致
# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials "kubelet-bootstrap" \
--token=${TOKEN} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
创建systemd启动文件
cat > /usr/lib/systemd/system/kubelet.service << EOF[Unit]Description=Kubernetes KubeletAfter=docker.service[Service]EnvironmentFile=/opt/k8s/cfg/kubelet.confExecStart=/opt/k8s/bin/kubelet \$KUBELET_OPTSRestart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF
启动kubelet
systemctl daemon-reloadsystemctl restart kubeletsystemctl enable kubelet
同步kubelet配置到其余节点
同步kubelet.conf, kubelet-config.yml, bootstrap.kubeconfig, kubelet.service到所有节点, 修改对应masterIP
## 同步kubelet配置rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-master2:/opt/k8s/cfg/rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-node1:/opt/k8s/cfg/rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-node2:/opt/k8s/cfg/rsync -av /opt/k8s/cfg/{kubelet.conf,kubelet-config.yml,bootstrap.kubeconfig} root@k8s-node3:/opt/k8s/cfg/## 同步启动文件rsync -av /usr/lib/systemd/system/kubelet.service root@k8s-master2:/usr/lib/systemd/system/kubelet.servicersync -av /usr/lib/systemd/system/kubelet.service root@k8s-node1:/usr/lib/systemd/system/kubelet.servicersync -av /usr/lib/systemd/system/kubelet.service root@k8s-node2:/usr/lib/systemd/system/kubelet.servicersync -av /usr/lib/systemd/system/kubelet.service root@k8s-node3:/usr/lib/systemd/system/kubelet.service## 其余节点启动kubeletsystemctl daemon-reloadsystemctl restart kubeletsystemctl enable kubelet
连接集群暂时使用master1节点IP, 后面集群全部正常后, 再做master1和master2的高可用, 将连接集群IP改为VIP
kubelet-bootstrap授权
到这里, 启动kubelet时候会报错
failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "kubelet-bootstrap" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
这是因为kubelet-bootstrap没有权限申请证书,在master上查看证书申请列表也是空的
kubectl get csrNo resources found in default namespace.
这时候需要在master上操作,授权kubelet-bootstrap用户允许请求证书
kubectl create clusterrolebinding kubelet-bootstrap \--clusterrole=system:node-bootstrapper \--user=kubelet-bootstrap
或者执行yaml文件,效果相同
cat > /opt/k8s/yaml/kubelet-bootstrap-rbac.yaml << EOFkind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: kubelet-bootstrap subjects:- kind: User name: kubelet-bootstrap apiGroup: rbac.authorization.k8s.ioroleRef: kind: ClusterRole name: system:node-bootstrapper apiGroup: rbac.authorization.k8s.ioEOFkubectl apply -f kubelet-bootstrap-rbac.yaml
重新启动kubelet,然后在master上查看证书申请
kubectl get csrNAME AGE SIGNERNAME REQUESTOR CONDITIONnode-csr-dqVIp0rPbtw3PNeY25Z0V27I2wxANX8R29yjdXT9Q34 36s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
批准kubelet证书申请并加入集群
kubectl certificate approve node-csr-dqVIp0rPbtw3PNeY25Z0V27I2wxANX8R29yjdXT9Q34
再次查看证书申请
kubectl get csrNAME AGE SIGNERNAME REQUESTOR CONDITIONnode-csr-dqVIp0rPbtw3PNeY25Z0V27I2wxANX8R29yjdXT9Q34 2m9s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
查看节点状态
# kubectl get nodeNAME STATUS ROLES AGE VERSIONk8s-master1 NotReady <none> 11m v1.20.6k8s-master2 NotReady <none> 12m v1.20.6k8s-node1 NotReady <none> 3s v1.20.6k8s-node2 NotReady <none> 25s v1.20.6k8s-node3 NotReady <none> 14s v1.20.6
注:由于CNI网络插件还没有部署,节点会没有准备就绪 NotReady
8.2 部署kube-proxy
创建conf配置文件
cat > /opt/k8s/cfg/kube-proxy.conf << EOFKUBE_PROXY_OPTS="--logtostderr=false \\--v=2 \\--log-dir=/opt/k8s/logs \\--config=/opt/k8s/cfg/kube-proxy-config.yml"EOF
创建yml参数配置文件–IPVS模式
安装ipvs ipset
yum -y install ipvsadm ipset conntrack-tools
配置系统加载模块
cat > /etc/modules-load.d/ipvs.conf <<EOF ip_vsip_vs_rrip_vs_wrrip_vs_shnf_conntrack_ipv4ipipEOF## 配置开机启动systemctl restart systemd-modules-loadsystemctl enable systemd-modules-load
查看生效模块
lsmod |grep ip_vs
修改kube-proxy-config.yml
cat > /opt/k8s/cfg/kube-proxy-config.yml << EOFkind: KubeProxyConfigurationapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 0.0.0.0metricsBindAddress: 0.0.0.0:10249iptables: masqueradeAll: true masqueradeBit: null minSyncPeriod: 0s syncPeriod: 0sipvs: masqueradeAll: true excludeCIDRs: null minSyncPeriod: 0s scheduler: "rr" strictARP: false syncPeriod: 0s tcpFinTimeout: 0s tcpTimeout: 0s udpTimeout: 0smode: "ipvs"clientConnection: kubeconfig: /opt/k8s/cfg/kube-proxy.kubeconfighostnameOverride: k8s-master1clusterCIDR: 10.0.0.0/24EOF
注意:
修改hostnameOverride为节点hostnameclusterCIDR: kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量,指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNATclusterCIDR: 10.0.0.0/24这个是集群service段,和kube-apiserver.conf还有kube-controller-manager.conf中--service-cluster-ip-range=10.0.0.0/24参数保持一致
生成kube-proxy.kubeconfig文件
master节点操作
生成kube-proxy证书:
cd /data/TLS/k8s# 创建证书请求文件cat > kube-proxy-csr.json << EOF{ "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ]}EOF# 生成证书cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
生成kube-proxy.kubeconfig文件
KUBE_CONFIG="/opt/k8s/cfg/kube-proxy.kubeconfig"KUBE_APISERVER="https://3.1.101.49:6443"kubectl config set-cluster kubernetes \ --certificate-authority=/opt/k8s/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG}kubectl config set-credentials kube-proxy \ --client-certificate=./kube-proxy.pem \ --client-key=./kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG}kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=${KUBE_CONFIG}kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
创建systemd启动文件
cat > /usr/lib/systemd/system/kube-proxy.service << EOF[Unit]Description=Kubernetes ProxyAfter=docker.service[Service]EnvironmentFile=/opt/k8s/cfg/kube-proxy.confExecStart=/opt/k8s/bin/kube-proxy \$KUBE_PROXY_OPTSRestart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF
启动kube-proxy
systemctl daemon-reloadsystemctl restart kube-proxysystemctl enable kube-proxy
验证IPVS模式
## 验证ipvsadm -l
同步kube-proxy配置到其余节点
同步kube-proxy.conf, kube-proxy-config.yml, kube-proxy.kubeconfig, kube-proxy.service到所有节点, 修改对应masterIP
## 同步kube-proxy配置rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-master2:/opt/k8s/cfg/rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-node1:/opt/k8s/cfg/rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-node2:/opt/k8s/cfg/rsync -av /opt/k8s/cfg/{kube-proxy.conf,kube-proxy-config.yml,kube-proxy.kubeconfig} root@k8s-node3:/opt/k8s/cfg/## 同步启动文件rsync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-master2:/usr/lib/systemd/system/kube-proxy.servicersync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-node1:/usr/lib/systemd/system/kube-proxy.servicersync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-node2:/usr/lib/systemd/system/kube-proxy.servicersync -av /usr/lib/systemd/system/kube-proxy.service root@k8s-node3:/usr/lib/systemd/system/kube-proxy.service## 其余节点启动kubeletsystemctl daemon-reloadsystemctl restart kube-proxysystemctl enable kube-proxy
九. 授权apiserver访问kubelet
- 如果不进行授权, 将无法管理容器
cat > /opt/k8s/yaml/apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF
kubectl apply -f apiserver-to-kubelet-rbac.yaml
十. 部署相关插件
为master节点打污点, master节点不运行pod
kubectl taint nodes k8s-master1 node-role.kubernetes.io/master=:NoSchedule
kubectl taint nodes k8s-master2 node-role.kubernetes.io/master=:NoSchedule
master节点操作
10.1 部署cni网络-Calico
下载地址
https://docs.projectcalico.org/getting-started/kubernetes/installation/config-options
curl -k https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico-etcd.yaml
配置Secret
apiVersion: v1kind: Secrettype: Opaquemetadata: name: calico-etcd-secrets namespace: kube-systemdata: # Populate the following with etcd TLS configuration if desired, but leave blank if # not using TLS for etcd. # The keys below should be uncommented and the values populated with the base64 # encoded contents of each file that would be associated with the TLS data. # Example command for encoding a file contents: cat <file> | base64 -w 0 etcd-key: <server-key.pem转换内容> etcd-cert: <server.pem转换内容> etcd-ca: <ca.pem转换内容>
转换命令:cat <file> | base64 -w 0
配置ConfigMap
kind: ConfigMapapiVersion: v1metadata: name: calico-config namespace: kube-systemdata: # Configure this with the location of your etcd cluster. etcd_endpoints: "https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379" # If you're using TLS enabled etcd uncomment the following. # You must also populate the Secret below with these files. etcd_ca: "/calico-secrets/etcd-ca" etcd_cert: "/calico-secrets/etcd-cert" etcd_key: "/calico-secrets/etcd-key" # Typha is disabled. typha_service_name: "none" # Configure the backend to use. calico_backend: "bird"
etcd_endpoints: ETCD地址
修改Pod CIDR
查找关键字CALICO_IPV4POOL_CIDR; Pod CIDR要与控制器配置文件kube-controller-manager.conf中配置的对应,10.244.0.0/16
# The default IPv4 pool to create on startup if none exists. Pod IPs will be# chosen from this range. Changing this value after installation will have# no effect. This should fall within `--cluster-cidr`.- name: CALICO_IPV4POOL_CIDR value: "10.244.0.0/16"
配置calico工作模式
- 默认IPIP模式,如果关闭后,模式就变为BGP模式
# Enable IPIP- name: CALICO_IPV4POOL_IPIP value: "Always"
指定网卡
修改DaemonSet控制器下的containers.env加入# Specify interface- name: IP_AUTODETECTION_METHOD value: "interface=ens192"
ens.*根据实际环境修改
部署calico网络
kubectl apply -f calico-etcd.yaml
部署calico管理工具
下载
wget -O /usr/bin/calicoctl https://github.com/projectcalico/calicoctl/releases/download/v3.18.1/calicoctlchmod +x /usr/bin/calicoctl
calicoctl配置文件
mkdir -pv /etc/calico/cat > /etc/calico/calicoctl.cfg << EOFapiVersion: projectcalico.org/v3kind: CalicoAPIConfigmetadata:spec: datastoreType: "etcdv3" etcdEndpoints: "https://3.1.101.49:2379,https://3.1.101.50:2379,https://3.1.101.51:2379,https://3.1.101.52:2379,https://3.1.101.53:2379" etcdKeyFile: "/opt/etcd/ssl/server-key.pem" etcdCertFile: "/opt/etcd/ssl/server.pem" etcdCACertFile: "/opt/etcd/ssl/ca.pem"EOF
calicoctl常用命令
calicoctl node status //查看当前网络状态,不需要指定配置文件calicoctl get nodes -o wide //查看节点,需要指定配置文件calicoctl get ippool -o wide //查看 IPAM的IP地址池
10.2 部署Dashboard
下载yaml文件
curl https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml -o kubernetes-dashboard.yaml
替换镜像地址(二选一)
- 更换docker官方镜像更换为阿里云镜像地址(下载更快)
sed -i 's#kubernetesui#registry.cn-hangzhou.aliyuncs.com\/google_containers#g' kubernetes-dashboard.yaml
- 者将两个image地址更换为docker镜像仓库最新版本地址
kubernetesui/dashboard:v2.3.1kubernetesui/metrics-scraper:v1.0.7
配置dashboard-service
默认Dashboard只能集群内部访问,修改Service为NodePort类型,暴露到外部(kubernetes-dashboard部分), 如下:
cat >> kubernetes-dashboard.yaml << EOF---# ------------------- dashboard-service ------------------- #kind: ServiceapiVersion: v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboardspec: ports: - port: 443 targetPort: 8443 nodePort: 30001 type: NodePort selector: k8s-app: kubernetes-dashboardEOF
配置dashboard-admin帐号
cat >> kubernetes-dashboard.yaml << EOF---# ------------------- dashboard-admin ------------------- #apiVersion: v1kind: ServiceAccountmetadata: name: dashboard-admin namespace: kubernetes-dashboard---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: dashboard-adminsubjects:- kind: ServiceAccount name: dashboard-admin namespace: kubernetes-dashboardroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminEOF
如果不使用上面方法,也可以使用命令创建帐号并授权
// 先创建一个帐号kubectl create serviceaccount dashboard-admin-01 -n kubernetes-dashboard// 给账号授权角色kubectl create clusterrolebinding dashboard-admin-01 --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin-01// 获取角色帐号TOKEN令牌kubectl describe secrets -n kubernetes-dashboard $(kubectl -n kubernetes-dashboard get secret | awk '/dashboard-admin-01/{print $1}')
相关查询命令:kubectl -n kubernetes-dashboard get/describe serviceaccount/clusterrolebinding/secret dashboard-admin-01serviceaccount: 创建帐号clusterrolebinding: 绑定角色secret: token相关
部署kubernetes-dashboard
## 部署kubectl apply -f kubernetes-dashboard.yaml## 查看部署状态kubectl get all -n kubernetes-dashboard -o wide## 获取令牌kubectl describe secrets -n kubernetes-dashboard dashboard-admin## 访问https://NODE_IP:30001
10.3 部署coredns
下载yaml配置文件
https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/coredns/coredns.yaml.base
下载coredns.yaml.base,修改后保存为coredns.yaml
修改yaml配置文件
70行左右 kubernetes cluster.local { -->大写部分修改成自己的域 一般为 cluster.local
135行左右 image: coredns/coredns:1.8.0 -->image部分墙外的需要修改,coredns/coredns:1.8.0
140行左右 memory: 170Mi -->修改成自己适合的值,我这里修改为 170Mi
200行左右 clusterIP: 10.0.0.2 --> clusterIP 修改成kubelet-config.yml中设置的clusterDNS地址
PS: 结合官方模版修改,比如内存,image镜像地址,版本号等
https://github.com/coredns/deployment/blob/master/kubernetes/coredns.yaml.sed
部署coredns
## 部署
kubectl apply -f coredns.yaml
## 验证
kubectl get pod -n kube-system
## 测试
kubectl run busybox --image=busybox --command -- ping www.baidu.com
kubectl exec -it pod/busybox -- /bin/sh -il
或者直接
kubectl run -it --image=busybox:1.28.4 --rm test /bin/sh
------------------------------------------------------------------------------
执行nslookup:
# nslookup kubernetes
结果为如下,证明coredns生效
Server: 10.0.0.2
Address 1: 10.0.0.2
Name: kubernetes
Address 1: 10.0.0.1
------------------------------------------------------------------------------
执行ping命令
ping www.baidu.com
PING www.baidu.com (220.181.38.149): 56 data bytes
64 bytes from 220.181.38.149: seq=0 ttl=51 time=20.448 ms
64 bytes from 220.181.38.149: seq=1 ttl=51 time=22.957 ms
10.4 部署Ingress
下载yaml配置文件
https://github.com/kubernetes/ingress-nginx/blob/nginx-0.30.0/deploy/static/mandatory.yaml
内容保存为ingress-nginx.yaml
修改ingress-nginx.yaml配置文件
- 使用Deployment控制器, 配置replicas为2; 或者直接使用DaemonSet控制器
- 修改rbac.authorization.k8s.io/v1beta1为rbac.authorization.k8s.io/v1(多处)
- 加入default-http-backend(在Deployment.nginx-ingress-controller之前)
还需要再ingress-nginx.yaml中加入参数
–default-backend-service=$(POD_NAMESPACE)/default-http-backend
位置如下:
containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io
default-http-backend的yaml配置
---apiVersion: apps/v1kind: Deploymentmetadata: name: default-http-backend labels: app: default-http-backend namespace: ingress-nginxspec: replicas: 1 selector: matchLabels: app: default-http-backend template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend image: mirrorgooglecontainers/defaultbackend-amd64:1.5 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 100m memory: 100Mi requests: cpu: 100m memory: 100Mi---apiVersion: v1kind: Servicemetadata: name: default-http-backend # namespace: ingress-nginx namespace: ingress-nginx labels: app: default-http-backendspec: ports: - port: 80 targetPort: 8080 selector: app: default-http-backend---
配置为hostNetwork模式
需要在Ingress Controller的yaml配置文件中指定使用主机网络hostNetwork: true位置位于spec.tmplate.spec下
- 不需要配置Service,通过kubectl get ingress获取ingress信息时候,ADDRESS地址会为空
- 可以将控制器设置为DaemonSet,就可以将域名解析到任意节点进行访问了
- 部署ingress Controller节点端口(80,443)不能被占用
部署ingress-nginx
修改后nginx-ingress-controller部分配置如下
apiVersion: apps/v1kind: Deploymentmetadata: name: nginx-ingress-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginxspec: replicas: 2 selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: hostNetwork: true # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount nodeSelector: kubernetes.io/os: linux containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io
使用kubectl部署
kubectl apply -f ingress-nginx.yaml
查看部署状态
kubectl get all -n ingress-nginx
将ingress部署到指定节点
- 给节点打标签
kubectl label nodes k8s-node2 type=ingresskubectl label nodes k8s-node3 type=ingress
- 修改yaml文件, 添加pod选择器
spec: hostNetwork: true # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount nodeSelector: kubernetes.io/os: linux type: "ingress" containers: - name: nginx-ingress-controller image: devops/ingress/nginx-ingress-controller:0.30.0
更新配置,查看pod所在节点
kubectl apply -f ingress-nginx.yamlkubectl get all -n ingress-nginx -o wide
验证ingress-nginx
- 创建deployment和service
## 创建一个nginx deployment
kubectl create deployment nginx-deploy --image=nginx
## 创建发布nginx deployment的service
kubectl expose deployment/nginx-deploy --port=80 --type=NodePort --target-port=80 --name=nginx-svc
- 配置nginx-ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress-test
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: test.ingress.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: nginx-svc
port:
number: 80
配置hosts,域名访问,如果成功,说明ingress已生效
十一. 配置master高可用
NGINX四层负载配置
nginx.conf
- 与http平级
stream {
log_format proxy '$remote_addr - [$time_local] $status "$upstream_bytes_sent" "$upstream_bytes_received" - "$upstream_addr" "$upstream_connect_time"';
access_log /var/log/nginx/upstream-access.log proxy;
open_log_file_cache off;
include /etc/nginx/upstream/*.conf;
}
k8s.conf
- 在/etc/nginx/upstream/目录下
upstream k8s-server {
hash $remote_addr consistent;
server 3.1.101.49:6443;
server 3.1.101.50:6443;
}
server {
listen 6443;
proxy_pass k8s-server;
access_log /var/log/nginx/k8s-ha.log proxy;
}
重启NGINX生效配置
Keepalived配置
Master
global_defs {
notification_email {
ha.localgit.com
}
notification_email_from ha.localhost.com
router_id LB-M
}
vrrp_script check_k8s {
script "</dev/tcp/127.0.0.1/6443" #修改为自己需要监听的端口,理论上可以监听远程端口
interval 2 #检查脚本的频率,单位(秒)
weight -30 #端口检查失败,优先级减少30,weight的绝对值要大于两台priority的差值
}
vrrp_instance VI_1 {
state MASTER
interface ens192 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 100 # 优先级,备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 123456
}
# VIP
virtual_ipaddress {
3.1.101.45/24
}
track_script {
check_k8s
}
}
backup
global_defs {
notification_email {
ha.localgit.com
}
notification_email_from ha.localhost.com
router_id LB-S
}
vrrp_script check_k8s {
script "</dev/tcp/127.0.0.1/6443" #修改为自己需要监听的端口,理论上可以监听远程端口
interval 2 #检查脚本的频率,单位(秒)
weight -30 #端口检查失败,优先级减少30,weight的绝对值要大于两台priority的差值
}
vrrp_instance VI_1 {
state BACKUP
interface ens192 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 90 # 优先级,备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 123456
}
# VIP
virtual_ipaddress {
3.1.101.45/24
}
track_script {
check_k8s
}
}
启动keepalived
/usr/bin/keepalived -f /etc/keepalived/keepalived.conf
修改k8s配置文件
- 将配置中master节点IP修改为VIP
master节点
/root/.kube/config
/opt/k8s/cfg/bootstrap.kubeconfig
/opt/k8s/cfg/kube-controller-manager.kubeconfig
/opt/k8s/cfg/kube-scheduler.kubeconfig
node节点
/opt/k8s/cfg/bootstrap.kubeconfig
生效配置
## master节点,重启kube-controller-manager和kube-scheduler
systemctl restart kube-controller-manager
systemctl restart kube-scheduler
rm -f /opt/k8s/ssl/kubelet-client-*.pem
rm -f /opt/k8s/cfg/kubelet.kubeconfig
systemctl restart kubelet
## node节点,删除自申请成证书以及kubeconfig文件,重启kubelet重新申请(需要master节点重新批准)
rm -f /opt/k8s/ssl/kubelet-client-*.pem
rm -f /opt/k8s/cfg/kubelet.kubeconfig
systemctl restart kubelet
## 验证
kubectl get cs
kubectl get node
到此,k8s 1.20二进制版本高可用部署完成