目录
部署规划
名称 | 主机名称 | IP地址 |
---|---|---|
master节点 | master | 20.0.0.10/24 |
node节点 | node01 | 20.0.0.20/24 |
node节点 | node02 | 20.0.0.30/24 |
一、etcd数据库部署
master节点部署
1、创建目录
[root@master ~]# mkdir k8s
[root@master ~]# cd k8s/
[root@master k8s]# mkdir etcd-cert
[root@master k8s]# cd etcd-cert/
2、制作ca证书
上传制作证书的工具
[root@master etcd-cert]# ls
cfssl cfssl-certinfo cfssljson
[root@master etcd-cert]# chmod +x *
[root@master etcd-cert]# ll
总用量 18808
-rwxr-xr-x 1 root root 10376657 1月 16 2020 cfssl
-rwxr-xr-x 1 root root 6595195 1月 16 2020 cfssl-certinfo
-rwxr-xr-x 1 root root 2277873 1月 16 2020 cfssljson
[root@master etcd-cert]# mv cf* /usr/local/bin/
开始制作证书
1、定义ca证书
[root@master etcd-cert]# vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
2、设置ca证书签名
[root@master etcd-cert]# vim ca-csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
3、生成ca证书
[root@master etcd-cert]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/01/19 16:58:40 [INFO] generating a new CA key and certificate from CSR
2021/01/19 16:58:40 [INFO] generate received request
2021/01/19 16:58:40 [INFO] received CSR
2021/01/19 16:58:40 [INFO] generating key: rsa-2048
2021/01/19 16:58:40 [INFO] encoded CSR
2021/01/19 16:58:40 [INFO] signed certificate with serial number 80068717008280235134541270449892611068755928092
[root@master etcd-cert]# ll
总用量 20
-rw-r--r-- 1 root root 304 1月 19 16:55 ca-config.json
-rw-r--r-- 1 root root 956 1月 19 16:58 ca.csr
-rw-r--r-- 1 root root 212 1月 19 16:57 ca-csr.json
-rw------- 1 root root 1679 1月 19 16:58 ca-key.pem
-rw-r--r-- 1 root root 1265 1月 19 16:58 ca.pem #生成的ca证书
3、设置etcd通讯验证
[root@master etcd-cert]# vim server-csr.json
{
"CN": "etcd",
"hosts": [
"20.0.0.10",
"20.0.0.20",
"20.0.0.30"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
生成etcd证书
[root@master etcd-cert]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/01/19 17:05:14 [INFO] generate received request
2021/01/19 17:05:14 [INFO] received CSR
2021/01/19 17:05:14 [INFO] generating key: rsa-2048
2021/01/19 17:05:15 [INFO] encoded CSR
2021/01/19 17:05:15 [INFO] signed certificate with serial number 274602113254117039354472428397076843843685311146
2021/01/19 17:05:15 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master etcd-cert]# ll
总用量 36
-rw-r--r-- 1 root root 304 1月 19 16:55 ca-config.json
-rw-r--r-- 1 root root 956 1月 19 16:58 ca.csr
-rw-r--r-- 1 root root 212 1月 19 16:57 ca-csr.json
-rw------- 1 root root 1679 1月 19 16:58 ca-key.pem
-rw-r--r-- 1 root root 1265 1月 19 16:58 ca.pem
-rw-r--r-- 1 root root 1013 1月 19 17:05 server.csr
-rw-r--r-- 1 root root 278 1月 19 17:05 server-csr.json
-rw------- 1 root root 1679 1月 19 17:05 server-key.pem
-rw-r--r-- 1 root root 1338 1月 19 17:05 server.pem
[root@master etcd-cert]#
4、创建etcd的工作目录
[root@master etcd-cert]# mkdir /opt/etcd/{cfg,bin,ssl} -p
cfg:放tecd配置文件
bin:放etcd命令工具
ssl:放etcd证书
[root@master etcd-cert]# cp etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
[root@master etcd-cert]# cp *.pem /opt/etcd/ssl/
5、设置etcd配置文件和启动脚本
[root@master etcd-cert]# vim etcd.sh
#!/bin/bash# example: ./etcd.sh etcd01 20.0.0.10 etcd02=https://20.0.0.20:2380,etcd03=https://20.0.0.30:2380
ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3
WORK_DIR=/opt/etcd
cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd
执行脚本
[root@master etcd-cert]# bash etcd.sh etcd01 20.0.0.10 etcd02=https://20.0.0.20:2380,etcd03=https://20.0.0.30:2380
处于等待状态,应为node节点还没加入进来,此时应该生成etcd的配置文件和启动文件
#安装tree来查看etcd工作目录
[root@master etcd-cert]# yum -y install tree
[root@master etcd-cert]# tree /opt/etcd/
/opt/etcd/
├── bin
│ ├── etcd
│ └── etcdctl
├── cfg
│ └── etcd
└── ssl
├── ca-key.pem
├── ca.pem
├── server-key.pem
└── server.pem
node节点部署
node01节点部署
#在master节点将etcd的工作目录文件拷贝到node01的/opt目录下面
[root@master etcd]# scp -r /opt/etcd/ root@20.0.0.20:/opt/
The authenticity of host '20.0.0.20 (20.0.0.20)' can't be established.
ECDSA key fingerprint is SHA256:yELtpt+yAiWNtPQb5bPu3PyWman5X5xL5zwU607sqHE.
ECDSA key fingerprint is MD5:b1:3f:1c:fa:eb:42:fc:bf:02:40:80:ea:8e:01:0a:7a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '20.0.0.20' (ECDSA) to the list of known hosts.
root@20.0.0.20's password:
etcd 100% 481 152.4KB/s 00:00
etcd 100% 18MB 68.1MB/s 00:00
etcdctl 100% 15MB 74.6MB/s 00:00
ca-key.pem 100% 1679 1.4MB/s 00:00
ca.pem 100% 1265 1.1MB/s 00:00
server-key.pem 100% 1679 1.7MB/s 00:00
server.pem 100% 1338 1.1MB/s 00:00
#在master节点将etcd拷贝启动配置文件给node01
[root@master etcd]# scp /usr/lib/systemd/system/etcd.service root@20.0.0.20:/usr/lib/systemd/system/etcd.service
root@20.0.0.20's password:
etcd.service
#修改/opt/etcd/cfg/etcd文件里的IP地址和节点名称
[root@node01 ~]# vim /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02" #修改etcd02
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://20.0.0.20:2380" #修改为自己主机IP地址
ETCD_LISTEN_CLIENT_URLS="https://20.0.0.20:2379" #修改为自己主机IP地址
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://20.0.0.20:2380" #修改为自己主机IP地址
ETCD_ADVERTISE_CLIENT_URLS="https://20.0.0.20:2379" #修改为自己主机IP地址
ETCD_INITIAL_CLUSTER="etcd01=https://20.0.0.10:2380,etcd02=https://20.0.0.20:2380,etcd03=https://20.0.0.30:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
node02
#在master节点将etcd的工作目录文件拷贝到node02的/opt目录下面
[root@master etcd]# scp -r /opt/etcd/ root@20.0.0.30:/opt/
The authenticity of host '20.0.0.30 (20.0.0.30)' can't be established.
ECDSA key fingerprint is SHA256:S2ANKK3sAHs5II74zKqBXVTVfAQUFrImRm7pq/hOrpg.
ECDSA key fingerprint is MD5:bc:99:68:b2:2a:25:31:37:bc:2b:01:49:35:98:de:f0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '20.0.0.30' (ECDSA) to the list of known hosts.
root@20.0.0.30's password:
etcd 100% 481 194.1KB/s 00:00
etcd 100% 18MB 92.3MB/s 00:00
etcdctl 100% 15MB 93.0MB/s 00:00
ca-key.pem 100% 1679 745.9KB/s 00:00
ca.pem 100% 1265 416.2KB/s 00:00
server-key.pem 100% 1679 852.3KB/s 00:00
server.pem
#在master节点将etcd拷贝启动配置文件给node02
[root@master etcd]# scp /usr/lib/systemd/system/etcd.service root@20.0.0.30:/usr/lib/systemd/system/etcd.service
root@20.0.0.30's password:
etcd.service
#修改/opt/etcd/cfg/etcd文件里的IP地址和节点名称
[root@node01 ~]# vim /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd03" #修改etcd03
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://20.0.0.30:2380" #修改为自己主机IP地址
ETCD_LISTEN_CLIENT_URLS="https://20.0.0.30:2379" #修改为自己主机IP地址
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://20.0.0.30:2380" #修改为自己主机IP地址
ETCD_ADVERTISE_CLIENT_URLS="https://20.0.0.30:2379" #修改为自己主机IP地址
ETCD_INITIAL_CLUSTER="etcd01=https://20.0.0.10:2380,etcd02=https://20.0.0.20:2380,etcd03=https://20.0.0.30:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
node节点加入群集
#先启动master节点的etcd在启动node节点etcd
[root@master etcd]# systemctl start etcd
[root@node01 etcd]# systemctl start etcd
[root@node01 etcd]# systemctl start etcd
查看群集是否成功加入
[root@master etcd]# cd /opt/etcd/ssl/
[root@master ssl]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://20.0.0.10:2379,https://20.0.0.20:2379,https://20.0.0.30:2379" cluster-health
member 60bc7e36f63b965 is healthy: got healthy result from https://20.0.0.30:2379
member 4a9f1750486efa02 is healthy: got healthy result from https://20.0.0.10:2379
member e3197fd6a5933614 is healthy: got healthy result from https://20.0.0.20:2379
cluster is healthy
二、docker容器引擎部署
所有从节点部署docker
1、下载阿里源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
2、安装docker依赖环境
yum install -y yum-utils device-mapper-persistent-data lvm2
3、安装docker-ce
yum -y install docker-ce
4、启动docker
systemctl start docker.service
systemctl enable docker.service #设置开启自启动
三、flannel网络组件部署
master分配子网写入etcd
1、写入分配的子网段到tecd中,供flannel使用
cd /opt/etcd/ssl/
/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://20.0.0.10:2379,https://20.0.0.20:2379,https://20.0.0.30:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}} #出现这个表明成功
2、查看信息是否写入etcd
/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://20.0.0.10:2379,https://20.0.0.20:2379,https://20.0.0.30:2379" get /coreos.com/network/config
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}} #出现这个表面写入成功
node节点安装flannel
所有node节点安装flannel
1、上传flannel安装包
ll
-rw-r--r-- 1 root root 9706487 1月 13 2020 flannel-v0.10.0-linux-amd64.tar.gz
tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
ll
-rwxr-xr-x 1 1001 1001 36327752 1月 24 2018 flanneld
-rwxr-xr-x 1 1001 1001 2139 3月 18 2017 mk-docker-opts.sh
-rw-rw-r-- 1 1001 1001 4298 12月 24 2017 README.md
2、创建flannel工作目录
mkdir /opt/kubernetes/{cfg,bin,ssl} -p
3、flannel工具移动到创建的工作目录中
mv flanneld mk-docker-opts.sh /opt/kubernetes/bin/
4、创建flannel配置、启动脚本
vim flannel.sh
#!/bin/bash
ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}
cat <<EOF >/opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF
cat <<EOF >/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld
5、开启flannel
bash flannel.sh https://20.0.0.10:2379,https://20.0.0.20:2379,https://20.0.0.30:2379
此时node节点都会出现一个flannel网卡,而且每个节点的flannel网卡的网段都不相同
6、配置docker连接flannel
vim /usr/lib/systemd/system/docker.service
在[Service]模块下
EnvironmentFile=/run/flannel/subnet.env #增加
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock 在-H前面增加 $DOCKER_NETWORK_OPTIONS
#查看$DOCKER_NETWORK_OPTIONS是什么
cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.100.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.100.1/24 --ip-masq=false --mtu=1450" #--bip指定docker启动时的子网段,每个node节点都不相同
7、重启docker
systemctl daemon-reload #重新加载系统参数
systemctl restart docker.service
8、测试不同节点之间docker网络是否可以正常通讯
1、node01创建一个容器
[root@node01 ~]# docker run -itd centos:7
2、node01创建一个容器
[root@node02 ~]# docker run -itd centos:7
3、查看node01节点容器IP地址
[root@node01 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b74642711c2f centos:7 "/bin/bash" 3 minutes ago Up 3 minutes elastic_leakey
[root@node01 ~]# docker inspect --format={{.NetworkSettings.IPAddress}} b74642711c2f
172.17.100.2
4、查看node02节点容器IP地址
[root@node2 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
43a46bd8868f centos:7 "/bin/bash" About a minute ago Up About a minute nice_driscoll
[root@node2 ~]# docker inspect --format={{.NetworkSettings.IPAddress}} 43a46bd8868f
172.17.75.2
5、node01节点容器ping node02节点容器
[root@node01 ~]# docker exec -it b74642711c2f ping 172.17.75.2
PING 172.17.75.2 (172.17.75.2) 56(84) bytes of data.
64 bytes from 172.17.75.2: icmp_seq=1 ttl=62 time=0.618 ms
64 bytes from 172.17.75.2: icmp_seq=2 ttl=62 time=1.33 ms
64 bytes from 172.17.75.2: icmp_seq=3 ttl=62 time=0.758 ms
6、node02节点容器ping node01节点容器
[root@node2 ~]# docker exec -it 43a46bd8868f ping 172.17.100.2
PING 172.17.100.2 (172.17.100.2) 56(84) bytes of data.
64 bytes from 172.17.100.2: icmp_seq=1 ttl=62 time=0.469 ms
64 bytes from 172.17.100.2: icmp_seq=2 ttl=62 time=0.505 ms
64 bytes from 172.17.100.2: icmp_seq=3 ttl=62 time=1.31 ms
四、master节点部署
1、创建kubernetes工作目录
[root@master ~]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p
2、制作api-server证书
[root@master ~]# mkdir k8s-cert
[root@master ~]# cd k8s-cert/
#######################制作ca证书############################
[root@master k8s-cert]#vim ca-config.json #配置ca证书
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
[root@master k8s-cert]#vim ca-csr.json #ca证书签名
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
[root@master k8s-cert]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - 生成ca证书
#######################api-server############################
[root@master k8s-cert]# vim server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"20.0.0.10", #master01 节点IP,就是本机IP地址
"20.0.0.40", #master02 节点IP,为k8s多节点部署提前设置
"20.0.0.50", #nginx-master
"20.0.0.60", #nginx-back
"20.0.0.100", #VIP地址
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
[root@master k8s-cert]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server #生成api-server证书
#######################管理员角色证书############################
[root@master k8s-cert]# vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
[root@master k8s-cert]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin #生成管理员证书
#######################kube-proxy证书############################
[root@master k8s-cert]# vim kube-proxy.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
[root@master k8s-cert]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy 生成kube-proxy证书
[root@master k8s-cert]# ls *.pem
admin-key.pem admin.pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem server-key.pem server.pem
3、将证书移动到制定位置
[root@master k8s-cert]# cp *.pem /opt/kubernetes/ssl/
4、上传k8s master核心组件包
[root@master k8s-cert# cd /root/k8s/
[root@master k8s]# ll
总用量 412700
drwxr-xr-x 3 root root 260 1月 19 17:49 etcd-cert
-rw-r--r-- 1 root root 422603860 1月 12 2020 kubernetes-server-linux-amd64.tar.gz
[root@master k8s]# tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@master k8s]# cd kubernetes/
[root@master kubernetes]# cd server/
[root@master server]# cd bin/
[root@master bin]# ls
apiextensions-apiserver kubeadm kube-controller-manager.docker_tag kube-proxy.docker_tag mounter
cloud-controller-manager kube-apiserver kube-controller-manager.tar kube-proxy.tar
cloud-controller-manager.docker_tag kube-apiserver.docker_tag kubectl kube-scheduler
cloud-controller-manager.tar kube-apiserver.tar kubelet kube-scheduler.docker_tag
hyperkube kube-controller-manager kube-proxy kube-scheduler.tar
5、将核心组件移动到制定位置
[root@master bin]# cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/
6、创建token令牌
[root@master bin]# cd /root/k8s
[root@master k8s]# head -c 16 /dev/urandom | od -An -t x | tr -d ' ' #生成随机序列号
5ff18641b25d7e18e62722ff4b119546
[root@master k8s]# vim /opt/kubernetes/cfg/token.csv
5ff18641b25d7e18e62722ff4b119546,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
7、创建apiserver脚本生成apiserver的配置文件和启动脚本
[root@master k8s]# vim apiserver.sh
#!/bin/bash
MASTER_ADDRESS=$1
ETCD_SERVERS=$2
cat <<EOF >/opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \\
--v=4 \\
--etcd-servers=${ETCD_SERVERS} \\
--bind-address=${MASTER_ADDRESS} \\
--secure-port=6443 \\
--advertise-address=${MASTER_ADDRESS} \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--kubelet-https=true \\
--enable-bootstrap-token-auth \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-50000 \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF
cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver
8、执行脚本生成apiserver的配置文件和启动文件
[root@master k8s]# bash apiserver.sh 20.0.0.10 https://20.0.0.10:2379,https://20.0.0.20:2379,https://20.0.0.30:2379
五、启动scheduler
1、创建scheduler脚本生成scheduler配置文件启动文件
[root@master k8s]# vim scheduler.sh
#!/bin/bash
MASTER_ADDRESS=$1
cat <<EOF >/opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true \\
--v=4 \\
--master=${MASTER_ADDRESS}:8080 \\
--leader-elect"
EOF
cat <<EOF >/usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler
2、执行脚本生成scheduler配置文件启动文件
[root@master k8s]# sh scheduler.sh 127.0.0.1
3、查看进程
[root@master k8s]# ps aux | grep ku
六、启动controller-manager
1、创建controller-manager脚本生成controller-manager配置文件启动文件
[root@master k8s]# vim controller-manager
#!/bin/bash
MASTER_ADDRESS=$1
cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\
--v=4 \\
--master=${MASTER_ADDRESS}:8080 \\
--leader-elect=true \\
--address=127.0.0.1 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF
cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager
2、执行脚本生成scheduler配置文件启动文件
[root@master k8s]# sh controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
3、查看进程
[root@master k8s]# ps aux | grep ku
4、查看master节点状态
[root@master k8s]# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
[root@master k8s]#
七、kubelet部署
master节点操作
1、拷贝kubelet kube-proxy工具到 node01和node02上
[root@master bin]# cd /root/k8s/kubernetes/server/bin
[root@master bin]# scp kubelet kube-proxy root@20.0.0.20:/opt/kubernetes/bin/
root@20.0.0.20's password:
kubelet 100% 168MB 131.2MB/s 00:01
kube-proxy 100% 48MB 93.7MB/s 00:00
[root@master bin]# scp kubelet kube-proxy root@20.0.0.30:/opt/kubernetes/bin/
root@20.0.0.30's password:
kubelet 100% 168MB 139.0MB/s 00:01
kube-proxy
2、master节点创建配置文件目录
[root@master bin]# cd /root/k8s
[root@master k8s]# mkdir kubeconfig
[root@master k8s]# cd kubeconfig/
3、查看token信息
[root@master kubeconfig]# cat /opt/kubernetes/cfg/token.csv
5ff18641b25d7e18e62722ff4b119546,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
#需要复制5ff18641b25d7e18e62722ff4b119546到后面创建的kubeconfig文件里
4、创建群集通信配置
[root@master kubeconfig]# vim kubeconfig.sh
APISERVER=$1
SSL_DIR=$2
# 创建kubelet bootstrapping kubeconfig
export KUBE_APISERVER="https://$APISERVER:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=5ff18641b25d7e18e62722ff4b119546 \
--kubeconfig=bootstrap.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------
# 创建kube-proxy kubeconfig文件
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------
# 创建kube-proxy kubeconfig文件
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--kubeconfig=bootstrap.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------
# 创建kube-proxy kubeconfig文件
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=$SSL_DIR/kube-proxy.pem \
--client-key=$SSL_DIR/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
5、重命名
[root@master kubeconfig]# mv kubeconfig.sh kubeconfig
6、将kubectl命令加入系统环境变量
[root@master kubeconfig]# vim /etc/profile
export PATH=$PATH:/opt/kubernetes/bin/ #在最后面增加
7、加载系统变量
[root@master kubeconfig]# source /etc/profile
8、查看maser节点组件是否正常
[root@master kubeconfig]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
9、执行群集通信配置
[root@master kubeconfig]# bash kubeconfig 20.0.0.10 /opt/kubernetes/ssl/
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".
[root@master kubeconfig]# ls
bootstrap.kubeconfig kubeconfig kube-proxy.kubeconfig
10、将生成的配置文件拷贝到node01上面
[root@master kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@20.0.0.20:/opt/kubernetes/cfg/
root@20.0.0.20's password:
bootstrap.kubeconfig 100% 2163 1.4MB/s 00:00
kube-proxy.kubeconfig 100% 6269 4.3MB/s 00:00
11、master节点创建bootstrap角色赋予权限用于连接apiserver请求签名(关键)
[root@master kubeconfig]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
node节点
1、创建kubelet配置kubelet配置文件和启动文件
[root@node01 ~]# vim kubelet.sh
#!/bin/bash
NODE_ADDRESS=$1
DNS_SERVER_IP=${2:-"10.0.0.2"}
cat <<EOF >/opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=${NODE_ADDRESS} \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet.config \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
EOF
cat <<EOF >/opt/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: ${NODE_ADDRESS}
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- ${DNS_SERVER_IP}
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
EOF
cat <<EOF >/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet
2、执行脚本创建kubelet配置文件、启动文件
[root@node01 ~]# bash kubelet.sh 20.0.0.20
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service
#kubelet负责向master节点申请证书,启动时候就会去申请
3、master节点申请node01节点的申请
[root@master kubeconfig]# kubectl get csr #查看节点申请
NAME AGE REQUESTOR CONDITION
node-csr-wiQUMex6280Dk5g5s-2mjWHEKF-EpVHcxtH4pv7cj6Y 2m kubelet-bootstrap Pending
[root@master kubeconfig]# kubectl certificate approve node-csr-wiQUMex6280Dk5g5s-2mjWHEKF-EpVHcxtH4pv7cj6Ycertificatesigningrequest.certificates.k8s.io/node-csr-wiQUMex6280Dk5g5s-2mjWHEKF-EpVHcxtH4pv7cj6Y approved
[root@master kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-wiQUMex6280Dk5g5s-2mjWHEKF-EpVHcxtH4pv7cj6Y 4m56s kubelet-bootstrap Approved,Issued
八、kube-proxy部署
1、创建kube-proxy脚本创建kube-proxy配置文件和启动文件
[root@node01 ~]# vim proxy.sh
#!/bin/bash
NODE_ADDRESS=$1
cat <<EOF >/opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=${NODE_ADDRESS} \\
--cluster-cidr=10.0.0.0/24 \\
--proxy-mode=ipvs \\
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
EOF
cat <<EOF >/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy
5、执行脚本
[root@node01 ~]# bash proxy.sh 20.0.0.20
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
6、master节点查看node01节点是否加入
root@master kubeconfig]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
20.0.0.20 Ready <none> 3m56s v1.12.3
7、将node02节点加入群集
#将node01节点的配置文件拷贝到node02节点上面
[root@node01 cfg]# scp -r /opt/kubernetes/ root@20.0.0.30:/opt/
root@20.0.0.30's password:
flanneld 100% 223 329.7KB/s 00:00
bootstrap.kubeconfig 100% 2163 380.7KB/s 00:00
kube-proxy.kubeconfig 100% 6269 8.2MB/s 00:00
kubelet 100% 373 715.3KB/s 00:00
kubelet.config 100% 263 431.4KB/s 00:00
kubelet.kubeconfig 100% 2292 2.5MB/s 00:00
kube-proxy 100% 185 515.0KB/s 00:00
scp: /opt//kubernetes/bin/flanneld: Text file busy
mk-docker-opts.sh 100% 2139 7.7MB/s 00:00
kubelet 100% 168MB 155.5MB/s 00:01
kube-proxy 100% 48MB 153.6MB/s 00:00
kubelet.crt 100% 2165 264.1KB/s 00:00
kubelet.key 100% 1679 2.5MB/s 00:00
kubelet-client-2021-01-21-10-31-18.pem 100% 1269 2.6MB/s 00:00
kubelet-client-current.pem 100% 1269 3.4MB/s 00:00
#将node01节点的启动配置文件拷贝到node02节点上面
[root@node01 cfg]# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@20.0.0.30:/usr/lib/systemd/system/
root@20.0.0.30's password:
kubelet.service 100% 264 234.4KB/s 00:00
kube-proxy.service
8、删除node01拷贝node02上的ssl证书
[root@node2 ~]# cd /opt/kubernetes/ssl/
[root@node2 ssl]# ls
kubelet-client-2021-01-21-10-31-18.pem kubelet-client-current.pem kubelet.crt kubelet.key
[root@node2 ssl]# rm -fr *
9、修改kubelet IP地址指向自己
[root@node2 ssl]# vim /opt/kubernetes/cfg/kubelet
--hostname-override=20.0.0.30 \ #修改为自己主机IP地址
10、修改kubelet.config 的IP地址指向自己
[root@node2 ssl]# vim /opt/kubernetes/cfg/kubelet.config
address: 20.0.0.30 #修改为自己主机IP地址
11、修改kube-proxy IP地址指向自己
[root@node2 ssl]# vim /opt/kubernetes/cfg/kube-proxy
--hostname-override=20.0.0.30 \ #修改为自己主机IP地址
12、启动kubelet和kube-proxy
[root@node2 ssl]# systemctl start kubelet.service
[root@node2 ssl]# systemctl enable kubelet.service
[root@node2 ssl]# systemctl start kube-proxy.service
[root@node2 ssl]# systemctl enable kube-proxy.service
13、master节点查看是否有申请证书的
[root@master kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-4qcbVkszkvwHo0Ml4QK2wgV-VH3VHylecyGujC8d28M 2m29s kubelet-bootstrap Pending
node-csr-wiQUMex6280Dk5g5s-2mjWHEKF-EpVHcxtH4pv7cj6Y 53m kubelet-bootstrap Approved,Issued
#授权node02节点证书
[root@master kubeconfig]# kubectl certificate approve node-csr-4qcbVkszkvwHo0Ml4QK2wgV-VH3VHylecyGujC8d28M
certificatesigningrequest.certificates.k8s.io/node-csr-4qcbVkszkvwHo0Ml4QK2wgV-VH3VHylecyGujC8d28M approved
#查看证书是否授权成功
[root@master kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-4qcbVkszkvwHo0Ml4QK2wgV-VH3VHylecyGujC8d28M 3m15s kubelet-bootstrap Approved,Issued
node-csr-wiQUMex6280Dk5g5s-2mjWHEKF-EpVHcxtH4pv7cj6Y 54m kubelet-bootstrap Approved,Issued
#查看节点是否加入成功
[root@master kubeconfig]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
20.0.0.20 Ready <none> 50m v1.12.3
20.0.0.30 Ready <none> 35s v1.12.3