二进制安装 kubernetes 1.15.2
环境
主机名 | 角色 | ip | 内存 | 硬盘 |
---|---|---|---|---|
hdss7-11.host.com | k8s代理节点1 | 10.4.7.11 | 2g | 200G |
hdss7-12.host.com | k8s代理节点2 | 10.4.7.12 | 2g | 200G |
hdss7-21.host.com | k8s运算节点1 | 10.4.7.21 | 4g | 200G |
hdss7-22.host.com | k8s运算节点2 | 10.4.7.22 | 4g | 200G |
hdss7-200.host.com | k8s运维节点(docker仓库) | 10.4.7.200 | 8g | 200G |
软件环境
-
OS: CentOS Linux release 7.6.1810 (Core)
-
docker: Docker version 20.10.4, build d3cb89e
-
kubernetes: v1.13.2
-
etcd: v3.1.18
-
flannel: v0.10.0
-
bind9: v9.9.4
-
harbor: v1.7.1
-
证书签发工具CFSSL: R1.2
-
其他
其他可能用到的软件,均使用操作系统自带的yum源和epel源进行安装
前置准备工作
调整操作系统,所有主机都要操作
-
调整yum源
-
安装epel-release
# yum install epel-release -y
-
关闭SElinux和firewalld
# setenforce 0 # systemctl stop firewalld
-
关闭SElinux和firewalld
安装必要工具
# yum install vim wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils -y
DNS服务初始化
- 创建主机域host.com
- 创建业务域od.com
- 主辅同步(10.4.7.11主、10.4.7.12辅)
- 客户端配置指向自建DNS
安装bind9软件 (HDSS7-11.host.com
)
[root@hdss7-11 ~]# yum install bind -y
修改主配置文件
[root@hdss7-11 ~]# vim /etc/named.conf #找出相关配置,进行修改
listen-on port 53 { 10.4.7.11; };
allow-query { any; };
forwarders { 10.4.7.1; }; # 指向网关
dnssec-enable no;
dnssec-validation no;
[root@hdss7-11 ~]# named-checkconf #检查主配置文件是否正确,无报错为正常
修改区域配置文件
[root@hdss7-11 ~]# vim /etc/named.rfc1912.zones #将以下内容,追加至文件尾部
zone "host.com" IN {
type master;
file "host.com.zone";
allow-update { 10.4.7.11; };
};
zone "od.com" IN {
type master;
file "od.com.zone";
allow-update { 10.4.7.11; };
};
配置区域数据文件
-
配置主机域数据文件
[root@hdss7-11 ~]# vim /var/named/host.com.zone $ORIGIN host.com. $TTL 600 ; 10 minutes @ IN SOA dns.host.com. dnsadmin.host.com. ( 2019111001 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.host.com. $TTL 60 ; 1 minute dns A 10.4.7.11 HDSS7-11 A 10.4.7.11 HDSS7-12 A 10.4.7.12 HDSS7-21 A 10.4.7.21 HDSS7-22 A 10.4.7.22 HDSS7-200 A 10.4.7.200
-
配置业务域数据文件
[root@hdss7-11 ~]# vim /var/named/od.com.zone $ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2019111001 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11
启动bind9
[root@hdss7-11 ~]# named-checkzone host.com host.com.zone
zone host.com/IN: loaded serial 2021031101
OK
[root@hdss7-11 ~]# named-checkzone od.com od.com.zone
zone od.com/IN: loaded serial 2021031101
OK
[root@hdss7-11 ~]# systemctl start named
[root@hdss7-11 ~]# systemctl enable named
检查
[root@hdss7-11 ~]# netstat -luntp|grep 53
[root@hdss7-11 ~]# dig -t A hdss7-12.host.com @10.4.7.11 +short
10.4.7.12
[root@hdss7-11 ~]# dig -t A dns.od.com @10.4.7.11 +short
10.4.7.11
配置DNS客户端
-
Linux主机上
# vim /etc/resolv.conf # Generated by NetworkManager search host.com nameserver 10.4.7.11
检查
- Linux主机上
[root@hdss7-11 ~]# ping dns.od.com
PING dns.od.com (10.4.7.11) 56(84) bytes of data.
64 bytes from hdss7-11.host.com (10.4.7.11): icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from hdss7-11.host.com (10.4.7.11): icmp_seq=2 ttl=64 time=0.059 ms
64 bytes from hdss7-11.host.com (10.4.7.11): icmp_seq=3 ttl=64 time=0.048 ms
^C
--- dns.od.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.023/0.043/0.059/0.016 ms
[root@hdss7-11 ~]# ping hdss7-12.host.com
PING HDSS7-12.host.com (10.4.7.12) 56(84) bytes of data.
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=1 ttl=64 time=3.59 ms
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=2 ttl=64 time=10.2 ms
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=3 ttl=64 time=11.5 ms
^C
--- HDSS7-12.host.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 3.597/8.480/11.592/3.496 ms
准备签发证书环境(HDSS7-200.host.com
)
安装CFSSL
- 证书签发工具CFSSL: R1.2
[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json
[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo
[root@hdss7-200 ~]# chmod +x /usr/bin/cfssl*
创建生成CA证书签名请求(csr)的JSON配置文件
[root@hdss7-200 ~]# mkdir /opt/certs && cd /opt/certs
[root@hdss7-200 certs]# vim /opt/certs/ca-csr.json
{
"CN": "kubernetes",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
],
"ca": {
"expiry": "175200h"
}
}
CN: Common Name,浏览器使用该字段验证网站是否合法,一般写的是域名。非常重要。
C: Country, 国家
ST: State,州,省
L: Locality,地区,城市
O: Organization Name,组织名称,公司名称
OU: Organization Unit Name,组织单位名称,公司部门“expiry”: “175200h” #ca证书的有效期为20年
生成CA证书和私钥
[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
[root@hdss7-200 certs]# ll
total 16
-rw-r--r-- 1 root root 997 Dec 23 01:46 ca.csr
-rw-r--r-- 1 root root 331 Dec 23 01:45 ca-csr.json
-rw------- 1 root root 1675 Dec 23 01:46 ca-key.pem
-rw-r--r-- 1 root root 1354 Dec 23 01:46 ca.pem
部署docker环境
(HDSS7-21.host.com,HDSS7-22.host.com,HDSS7-200.host.com
)
~]# curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
配置
# vim /etc/docker/daemon.json
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.21.1/24", #根据宿主机地址填写
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
注意:
这里bip要根据宿主机ip变化
启动
systemctl start docker
systemctl enable docker
systemctl status docker
部署docker镜像私有仓库harbor(HDSS7-200.host.com
)
下载软件二进制包并解压
[harbor下载地址]
https://github.com/goharbor/harbor/releases/download/v2.0.5/harbor-offline-installer-v2.0.5.tgz
[root@hdss7-200 ~]# mkdir /opt/src && cd /opt/src
[root@hdss7-200 src]# wget https://github.com/goharbor/harbor/releases/download/v2.0.5/harbor-offline-installer-v2.0.5.tgz
[root@hdss7-200 src]# tar xf harbor-offline-installer-v2.0.5.tgz -C /opt/
[root@hdss7-200 src]# cd /opt/
[root@hdss7-200 opt]# mv harbor harbor-v2.0.5
[root@hdss7-200 opt]# ln -s harbor-v2.0.5 harbor
配置
[root@hdss7-200 ~]# cd /opt/harbor
[root@hdss7-200 harbor]# vim harbor.yml #修改以下内容,把https相关的注释掉
hostname: harbor.od.com
http:
port: 180
data_volume: /data/harbor
location: /data/harbor/logs
#创建对应目录
[root@hdss7-200 harbor]# mkdir -p /data/harbor/logs
安装docker-compose
[root@hdss7-200 harbor]# yum install docker-compose -y
安装harbor
[root@hdss7-200 harbor]# ./install.sh
检查harbor启动情况
[root@hdss7-200 harbor]# docker-compose ps
配置harbor的dns内网解析
#在hdss7-11.host.com上
[root@hdss7-11 ~]# vi /var/named/od.com.zone
2019111002 ; serial #每增加一条,序列号前滚一次
harbor 60 IN A 10.4.7.200
#重启named
[root@hdss7-11 ~]# systemctl restart named
#检查
[root@hdss7-11 ~]# dig -t A harbor.od.com @10.4.7.11 +short
10.4.7.200
安装nginx并配置
#安装nginx
[root@hdss7-200 ~]# yum install nginx -y
#配置
[root@hdss7-200 ~]# vi /etc/nginx/conf.d/harbor.od.com.conf
server {
listen 80;
server_name harbor.od.com;
client_max_body_size 1000m;
location / {
proxy_pass http://127.0.0.1:180;
}
}
#检查并启动nginx
[root@hdss7-200 ~]# nginx -t
[root@hdss7-200 ~]# systemctl start nginx
[root@hdss7-200 ~]# systemctl enable nginx
浏览器打开http://harbor.od.com
- 用户名:admin
- 密码: Harbor12345
新建项目
测试仓库可用性
#将从公网上pull下来的镜像,打tag,并推送到刚才新建的仓库
[root@hdss7-200 ~]# docker pull nginx:1.7.9
#查看下载下来的镜像
[root@hdss7-200 ~]# docker images
#重新打标签
[root@hdss7-200 ~]# docker tag 84581e99d807 harbor.od.com/public/nginx:1.7.9
#需要进行登录认证,否则无法推送镜像
[root@hdss7-200 ~]# docker login harbor.od.com
[root@hdss7-200 ~]# docker push harbor.od.com/public/nginx:1.7.9
部署Master节点服务
部署etcd集群
集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-12.host.com | etcd lead | 10.4.7.12 |
HDSS7-21.host.com | etcd follow | 10.4.7.21 |
HDSS7-22.host.com | etcd follow | 10.4.7.22 |
注意:
这里部署文档以
HDSS7-12.host.com
主机为例,另外两台主机安装部署方法类似
创建基于根证书的config配置文件(HDSS7-200.host.com
)
[root@hdss7-200 ~]# mkdir /opt/certs/ && vim /opt/certs/ca-config.json
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
证书类型:
client certificate: 客户端使用,用于服务端认证客户端,例如etcdctl、etcd proxy、fleetctl、docker客户端
server certificate: 服务端使用,客户端以此验证服务端身份,例如docker服务端、 kube-apiserverpeer certificate: 双向证书,用于etcd集群成员间通信
创建生成自签证书签名请求(csr)的JSON配置文件
[root@hdss7-200 ~]# vim /opt/certs/etcd-peer-csr.json
{
"CN": "k8s-etcd",
"hosts": [
"10.4.7.11",
"10.4.7.12",
"10.4.7.21",
"10.4.7.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成etcd证书和私钥
[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
[root@hdss7-200 certs]# ll
total 36
-rw-r--r-- 1 root root 836 Dec 24 11:17 ca-config.json
-rw-r--r-- 1 root root 997 Dec 23 01:46 ca.csr
-rw-r--r-- 1 root root 331 Dec 23 01:45 ca-csr.json
-rw------- 1 root root 1675 Dec 23 01:46 ca-key.pem
-rw-r--r-- 1 root root 1354 Dec 23 01:46 ca.pem
-rw-r--r-- 1 root root 1066 Dec 24 11:31 etcd-peer.csr
-rw-r--r-- 1 root root 377 Dec 24 11:31 etcd-peer-csr.json
-rw------- 1 root root 1675 Dec 24 11:31 etcd-peer-key.pem
-rw-r--r-- 1 root root 1436 Dec 24 11:31 etcd-peer.pem
下载软件,解压,做软连接
[etcd下载地址]
https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz
(HDSS7-12.host.com
)
[root@hdss7-12 ~]# mkdir -p /opt/src && cd /opt/src
[root@hdss7-12 src]# wget https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz
[root@hdss7-12 src]# tar xf /opt/src/etcd-v3.1.20-linux-amd64.tar.gz -C /opt/
[root@hdss7-12 src]# cd /opt
[root@hdss7-12 opt]# mv /opt/etcd-v3.1.20-linux-amd64/ /opt/etcd-v3.1.20
[root@hdss7-12 opt]# ln -s /opt/etcd-v3.1.20/ /opt/etcd
创建etcd用户
HDSS7-12.host.com
上:
[root@hdss7-12 ~]# useradd -s /sbin/nologin -M etcd
创建目录,拷贝证书、私钥(HDSS7-12.host.com
)
[root@hdss7-12 ~]# mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
[root@hdss7-12 ~]# cd /opt/etcd/certs/
[root@hdss7-12 certs]# scp -rp hdss7-200:/opt/certs/{ca.pem,etcd-peer.pem,etcd-peer-key.pem} /opt/etcd/certs/
[root@hdss7-12 certs]# chmod 600 /opt/etcd/certs/etcd-peer-key.pem #注意私钥文件权限600
[root@hdss7-12 certs]# chown -R etcd.etcd /opt/etcd/certs/
创建etcd服务启动脚本(HDSS7-12.host.com
)
[root@hdss7-12 ~]# vim /opt/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.12:2380 \
--listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.12:2380 \
--advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
注意:
etcd集群各主机的启动脚本略有不同,部署其他节点时注意修改!!!
调整权限和目录(HDSS7-12.host.com
)
[root@hdss7-12 ~]# chmod +x /opt/etcd/etcd-server-startup.sh
[root@hdss7-12 ~]# chown -R etcd.etcd /opt/etcd-v3.1.20/
[root@hdss7-12 ~]# chown -R etcd.etcd /data/etcd/
[root@hdss7-12 ~]# chown -R etcd.etcd /data/logs/etcd-server/
安装supervisor软件(HDSS7-12.host.com
)
[root@hdss7-12 ~]# yum install supervisor -y
[root@hdss7-12 ~]# systemctl start supervisord
[root@hdss7-12 ~]# systemctl enable supervisord
创建etcd-server的启动配置(HDSS7-12.host.com
)
[root@hdss7-12 ~]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-12]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
启动etcd服务并检查
HDSS7-12.host.com
上:
[root@hdss7-12 certs]# supervisorctl update
etcd-server-7-12: added process group
[root@hdss7-12 certs]# supervisorctl status
etcd-server-7-12 STARTING
[root@hdss7-12 certs]# supervisorctl status
etcd-server-7-12 RUNNING pid 8200, uptime 0:00:31
安装部署启动检查所有集群规划主机上的etcd服务
步骤同上
检查集群状态
3台均启动后,检查集群状态
[root@hdss7-12 certs]# /opt/etcd/etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy
[root@hdss7-12 certs]#
[root@hdss7-12 certs]#
[root@hdss7-12 certs]# /opt/etcd/etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
部署kube-apiserver集群
集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-11.host.com | 4层负载均衡 | 10.4.7.11 |
HDSS7-12.host.com | 4层负载均衡 | 10.4.7.12 |
HDSS7-21.host.com | kube-apiserver | 10.4.7.21 |
HDSS7-22.host.com | kube-apiserver | 10.4.7.22 |
**注意:**这里10.4.7.11 和10.4.7.12 使用nginx做4层负载均衡器,用keepalived跑一个vip:10.4.7.10,代理两个kube-apiserver,实现高可用 |
这里部署文档以HDSS7-21.host.com
主机为例,另外一台运算节点安装部署方法类似
下载软件,解压,做软连接(HDSS7-21.host.com
)
[kubernetes下载地址]
https://dl.k8s.io/v1.15.4/kubernetes-server-linux-amd64.tar.gz
[root@hdss7-21 ~]# cd /opt/src/
[root@hdss7-21 src]# wget https://dl.k8s.io/v1.15.4/kubernetes-server-linux-amd64.tar.gz
[root@hdss7-21 src]# tar xf kubernetes-server-linux-amd64-v1.15.4.tar.gz -C /opt/
[root@hdss7-21 src]# cd /opt && mv kubernetes/ kubernetes-v1.15.4
[root@hdss7-21 opt]# ln -s /opt/kubernetes-v1.15.4/ /opt/kubernetes
[root@hdss7-21 opt]# cd kubernetes
[root@hdss7-21 kubernetes]# rm -rf kubernetes-src.tar.gz
[root@hdss7-21 kubernetes]# rm -rf server/bin/*.tar
[root@hdss7-21 kubernetes]# rm -rf server/bin/*_tag
签发client证书(HDSS7-200.host.com
)
创建生成证书签名请求(csr)的JSON配置文件
[root@hdss7-200 ~]# vi /opt/certs/client-csr.json
{
"CN": "k8s-node",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成client证书和私钥
[root@hdss7-200 ~]# cd /opt/certs
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssl-json -bare client
签发kube-apiserver证书(HDSS7-200.host.com
)
创建生成证书签名请求(csr)的JSON配置文件
[root@hdss7-200 ~]# vi /opt/certs/apiserver-csr.json
{
"CN": "k8s-apiserver",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成kube-apiserver证书和私钥
[root@hdss7-200 ~]# cd /opt/certs
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json |cfssl-json -bare apiserver
拷贝证书至各运算节点,并创建配置(HDSS7-21.host.com
)
拷贝证书、私钥,注意私钥文件属性600
[root@hdss7-21 ~]# mkdir /opt/kubernetes/server/bin/certs -p && cd /opt/kubernetes/server/bin/certs
[root@hdss7-21 certs]# scp -rp hdss7-200:/opt/certs/{apiserver-key.pem,apiserver.pem,ca-key.pem,ca.pem,client-key.pem,client.pem} ./
[root@hdss7-21 certs]# ll
total 24
-rw------- 1 root root 1679 Dec 25 01:05 apiserver-key.pem
-rw-r--r-- 1 root root 1606 Dec 25 01:05 apiserver.pem
-rw------- 1 root root 1675 Dec 23 01:46 ca-key.pem
-rw-r--r-- 1 root root 1354 Dec 23 01:46 ca.pem
-rw------- 1 root root 1675 Dec 25 00:56 client-key.pem
-rw-r--r-- 1 root root 1371 Dec 25 00:56 client.pem
创建apiserver启动的配置文件
[root@hdss7-21 ~]# mkdir /opt/kubernetes/server/bin/conf -p && cd /opt/kubernetes/server/bin/conf
[root@hdss7-21 conf]# vi audit.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
创建启动脚本
[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kube-apiserver.sh
#!/bin/bash
./kube-apiserver \
--apiserver-count 2 \
--audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \
--audit-policy-file ./conf/audit.yaml \
--authorization-mode RBAC \
--client-ca-file ./certs/ca.pem \
--requestheader-client-ca-file ./certs/ca.pem \
--enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
--etcd-cafile ./certs/ca.pem \
--etcd-certfile ./certs/client.pem \
--etcd-keyfile ./certs/client-key.pem \
--etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
--service-account-key-file ./certs/ca-key.pem \
--service-cluster-ip-range 192.168.0.0/16 \
--service-node-port-range 3000-29999 \
--target-ram-mb=1024 \
--kubelet-client-certificate ./certs/client.pem \
--kubelet-client-key ./certs/client-key.pem \
--log-dir /data/logs/kubernetes/kube-apiserver \
--tls-cert-file ./certs/apiserver.pem \
--tls-private-key-file ./certs/apiserver-key.pem \
--v 2
调整权限和目录(HDSS7-21.host.com
)
[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-apiserver
创建supervisor配置(HDSS7-21.host.com
)
[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-apiserver.ini
[program:kube-apiserver-7-21]
command=/opt/kubernetes/server/bin/kube-apiserver.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
启动服务并检查(HDSS7-21.host.com
)
[root@hdss7-21 ~]# supervisorctl update
kube-apiserver-7-21: added process group
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 8881, uptime 1:44:02
kube-apiserver-7-21 RUNNING pid 9024, uptime 0:00:34
安装部署启动检查所有集群规划主机上的kube-apiserver
同上
配4层反向代理(HDSS7-11.host.com
,HDSS7-12.host.com
)
nginx配置
# yum install nginx -y
# vim /etc/nginx/nginx.conf
stream {
upstream kube-apiserver {
server 10.4.7.21:6443 max_fails=3 fail_timeout=30s;
server 10.4.7.22:6443 max_fails=3 fail_timeout=30s;
}
server {
listen 7443;
proxy_connect_timeout 2s;
proxy_timeout 900s;
proxy_pass kube-apiserver;
}
}
#检查并启动nginx
# nginx -t
# systemctl start nginx
# systemctl enable nginx
keepalived配置(HDSS7-11.host.com
,HDSS7-12.host.com
)
#安装keepalived
# yum install keepalived -y
#编写端口监听脚本
# vim /etc/keepalived/check_port.sh
#!/bin/bash
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
if [ $PORT_PROCESS -eq 0 ];then
echo "Port $CHK_PORT Is Not Used,End."
exit 1
fi
else
echo "Check Port Cant Be Empty!"
fi
# chmod +x /etc/keepalived/check_port.sh
#使用方法:
#在keepalived的配置文件中
#vrrp_script check_port {#创建一个vrrp_script脚本,检查配置
# script "/etc/keepalived/check_port.sh 6379" #配置监听的端口
# interval 2 #检查脚本的频率,单位(秒)
#}
keepalived 主节点配置:
#keepalived 主节点配置:
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 10.4.7.11
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 251
priority 100
advert_int 1
mcast_src_ip 10.4.7.11
nopreempt
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.4.7.10
}
}
keepalived 从节点配置:
#keepalived 从节点配置:
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 10.4.7.12
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 251
mcast_src_ip 10.4.7.12
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.4.7.10
}
}
启动代理并检查(HDSS7-11.host.com
,HDSS7-12.host.com
)
- 启动
# systemctl start keepalived
# systemctl enable keepalived
# nginx -s reload
- 检查
[root@hdss7-11 ~]# netstat -luntp|grep 7443
tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 8621/nginx: master
[root@hdss7-11 ~]# ip addr|grep 10.4.7.10
inet 10.4.7.10/32 scope global eth0
部署controller-manager
集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-21.host.com | controller-manager | 10.4.7.21 |
HDSS7-22.host.com | controller-manager | 10.4.7.22 |
注意:
这里部署文档以
HDSS7-21.host.com
主机为例,另外一台运算节点安装部署方法类似
创建启动脚本(HDSS7-21.host.com
)
# vim /opt/kubernetes/server/bin/kube-controller-manager.sh
#!/bin/sh
./kube-controller-manager \
--cluster-cidr 172.7.0.0/16 \
--leader-elect true \
--log-dir /data/logs/kubernetes/kube-controller-manager \
--master http://127.0.0.1:8080 \
--service-account-private-key-file ./certs/ca-key.pem \
--service-cluster-ip-range 192.168.0.0/16 \
--root-ca-file ./certs/ca.pem \
--v 2
调整文件权限,创建目录(HDSS7-21.host.com
)
[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-controller-manager
创建supervisor配置(HDSS7-21.host.com
)
# vim /etc/supervisord.d/kube-conntroller-manager.ini
[program:kube-controller-manager-7-21]
command=/opt/kubernetes/server/bin/kube-controller-manager.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
启动服务并检查(HDSS7-21.host.com
)
[root@hdss7-21 ~]# supervisorctl update
kube-controller-manager-7-21: added process group
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 8881, uptime 2:00:18
kube-apiserver-7-21 RUNNING pid 9024, uptime 0:16:50
kube-controller-manager-7-21 RUNNING pid 9062, uptime 0:00:35
安装部署启动检查所有集群规划主机上的kube-controller-manager服务
同上
部署kube-scheduler
集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-21.host.com | kube-scheduler | 10.4.7.21 |
HDSS7-22.host.com | kube-scheduler | 10.4.7.22 |
注意:
这里部署文档以
HDSS7-21.host.com
主机为例,另外一台运算节点安装部署方法类似
创建启动脚本(HDSS7-21.host.com
)
# vim /opt/kubernetes/server/bin/kube-scheduler.sh
#!/bin/sh
./kube-scheduler \
--leader-elect \
--log-dir /data/logs/kubernetes/kube-scheduler \
--master http://127.0.0.1:8080 \
--v 2
调整文件权限,创建目录(HDSS7-21.host.com
)
[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-scheduler
创建supervisor配置(HDSS7-21.host.com
)
# vim /etc/supervisord.d/kube-scheduler.ini
[program:kube-scheduler]
command=/opt/kubernetes/server/bin/kube-scheduler.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=22 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=false ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stderr.log ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4 ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false ; emit events on stderr writes (default false)
启动服务并检查(HDSS7-21.host.com
)
[root@hdss7-21 ~]# supervisorctl update
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 8881, uptime 2:07:19
kube-apiserver-7-21 RUNNING pid 9024, uptime 0:23:51
kube-controller-manager-7-21 RUNNING pid 9062, uptime 0:07:36
kube-scheduler-7-21 RUNNING pid 9099, uptime 0:02:35
检查集群健康状态
[root@hdss7-21 conf]# ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
[root@hdss7-21 conf]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
安装部署启动检查所有集群规划主机上的kube-scheduler服务
同上
部署Node节点服务
部署kubelet
集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-21.host.com | kube-scheduler | 10.4.7.21 |
HDSS7-22.host.com | kube-scheduler | 10.4.7.22 |
**注意:**这里部署文档以HDSS7-21.host.com
主机为例,另外一台运算节点安装部署方法类似
签发kubelet证书(HDSS7-200.host.com
)
创建生成证书签名请求(csr)的JSON配置文件
[root@hdss7-200 certs]# vim /opt/certs/kubelet-csr.json
{
"CN": "k8s-kubelet",
"hosts": [
"127.0.0.1",
"10.4.7.10",
"10.4.7.11",
"10.4.7.12",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23",
"10.4.7.24",
"10.4.7.25",
"10.4.7.26",
"10.4.7.27",
"10.4.7.28"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成kubelet证书和私钥
[root@hdss7-200 ~]# cd /opt/certs
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
拷贝证书至各运算节点,并创建配置(HDSS7-21.host.com
,HDSS7-22.host.com
)
拷贝证书、私钥,注意私钥文件属性600
# cd /opt/kubernetes/server/bin/certs && scp -rp hdss7-200:/opt/certs/kubelet.pem /opt/kubernetes/server/bin/certs/
# scp -rp hdss7-200:/opt/certs/kubelet-key.pem /opt/kubernetes/server/bin/certs/
以下操作只在HDSS7-21
一台主机上操作一次!
set-cluster, 在kubeconfig配置文件中设置一个集群项
注意:
在conf目录下
[root@hdss7-21 ~]# cd /opt/kubernetes/server/conf
[root@hdss7-21 conf]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=kubelet.kubeconfig
# 正常返回 Cluster "myk8s" set.
set-credentials, 在kubeconfig配置文件中设置一个用户项
**注意:**在conf目录下
[root@hdss7-21 conf]# kubectl config set-credentials k8s-node \
--client-certificate=/opt/kubernetes/server/bin/certs/client.pem \
--client-key=/opt/kubernetes/server/bin/certs/client-key.pem \
--embed-certs=true \
--kubeconfig=kubelet.kubeconfig
# 正常返回 User "k8s-node" set.
set-context, 在kubeconfig配置文件中设置一个环境项
注意:
在conf目录下
[root@hdss7-21 conf]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=k8s-node \
--kubeconfig=kubelet.kubeconfig
# 正常返回 Context "myk8s-context" created.
use-context, 使用kubeconfig中的一个环境项作为当前配置
注意:
在conf目录下
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig
# 正常返回 Switched to context "myk8s-context".
k8s-node.yaml
- 创建资源配置文件
[root@hdss7-21 conf]# vim k8s-node.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8s-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
- 应用资源配置文件
[root@hdss7-21 conf]# kubectl apply -f k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
- 检查
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node
NAME AGE
k8s-node 2m57s
拷贝配置文件(HDSS7-22.host.com
)
[root@hdss7-22 ~]# cd /opt/kubernetes/server/bin/conf/
[root@hdss7-22 conf]# scp -rp hdss7-21:/opt/kubernetes/server/bin/conf/kubelet.kubeconfig .
准备pause基础镜像(HDSS7-200.host.com
)
[root@hdss7-200 ~]# docker pull kubernetes/pause
[root@hdss7-200 ~]# docker tag f9d5de079539 harbor.od.com/public/pause:latest
[root@hdss7-200 ~]# docker push harbor.od.com/public/pause:latest
创建kubelet启动脚本(HDSS7-21.host.com
、HDSS7-22.host.com
)
# vim /opt/kubernetes/server/bin/kubelet.sh
#!/bin/sh
./kubelet \
--anonymous-auth=false \
--cgroup-driver systemd \
--cluster-dns 192.168.0.2 \
--cluster-domain cluster.local \
--runtime-cgroups=/systemd/system.slice \
--kubelet-cgroups=/systemd/system.slice \
--fail-swap-on="false" \
--client-ca-file ./certs/ca.pem \
--tls-cert-file ./certs/kubelet.pem \
--tls-private-key-file ./certs/kubelet-key.pem \
--hostname-override hdss7-21.host.com \ #这里写主机名
--image-gc-high-threshold 20 \
--image-gc-low-threshold 10 \
--kubeconfig ./conf/kubelet.kubeconfig \
--log-dir /data/logs/kubernetes/kube-kubelet \
--pod-infra-container-image harbor.od.com/public/pause:latest \
--root-dir /data/kubelet
**注意:**kubelet集群各主机的启动脚本略有不同,部署其他节点时注意修改。
检查配置,权限,创建日志目录(HDSS7-21.host.com
、HDSS7-22.host.com
)
# chmod +x /opt/kubernetes/server/bin/kubelet.sh
# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
创建supervisor配置(HDSS7-21.host.com
、HDSS7-22.host.com
)
# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-21]
command=/opt/kubernetes/server/bin/kubelet.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
启动服务并检查(HDSS7-21.host.com
、HDSS7-22.host.com
)
# supervisorctl update
kube-kubelet-7-21: added process group
# supervisorctl status
etcd-server-7-22 RUNNING pid 8855, uptime 18:12:12
kube-apiserver-7-22 RUNNING pid 9005, uptime 16:31:31
kube-controller-manager-7-22 RUNNING pid 9037, uptime 16:15:20
kube-kubelet-7-22 RUNNING pid 15077, uptime 0:01:03
kube-scheduler-7-22 RUNNING pid 9072, uptime 16:10:31
安装部署启动检查所有集群规划主机上的kubelet服务
同上
检查运算节点(HDSS7-21.host.com
、HDSS7-22.host.com
)
[root@hdss7-21 cert]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 97s v1.15.4
hdss7-22.host.com Ready <none> 87s v1.15.4
部署kube-proxy
集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-21.host.com | kube-scheduler | 10.4.7.21 |
HDSS7-22.host.com | kube-scheduler | 10.4.7.22 |
**注意:**这里部署文档以HDSS7-21.host.com
主机为例,另外一台运算节点安装部署方法类似
签发kube-proxy证书(HDSS7-200.host.com
)
创建生成证书签名请求(csr)的JSON配置文件
[root@hdss7-200 ~]# vim /opt/certs/kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成kube-proxy证书和私钥
[root@hdss7-200 ~]# cd /opt/certs
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client
拷贝证书至各运算节点,并创建配置(HDSS7-21.host.com
、HDSS7-22.host.com
)
# cd /opt/kubernetes/server/bin/certs
# scp -rp hdss7-200:/opt/certs/kube-proxy-client-key.pem ./
# scp -rp hdss7-200:/opt/certs/kube-proxy-client.pem ./
拷贝证书、私钥,注意私钥文件属性600
创建配置,只在HDSS7-21.host.com
这一台主机上操作一次
set-cluster,在kubeconfig配置文件中设置一个集群项
注意:
在/opt/kubernetes/server/bin/conf目录下
[root@hdss7-21 ~]# cd /opt/kubernetes/server/bin/conf
[root@hdss7-21 conf]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=kube-proxy.kubeconfig
Cluster "myk8s" set.
set-credentials,在kubeconfig配置文件中设置一个用户项
注意:
在/opt/kubernetes/server/bin/conf目录下
[root@hdss7-21 ~]# cd /opt/kubernetes/server/bin/conf
[root@hdss7-21 conf]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/server/bin/certs/kube-proxy-client.pem \
--client-key=/opt/kubernetes/server/bin/certs/kube-proxy-client-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.
set-context,在kubeconfig配置文件中设置一个环境项
注意:
在/opt/kubernetes/server/bin/conf目录下
[root@hdss7-21 ~]# cd /opt/kubernetes/server/bin/conf
[root@hdss7-21 conf]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
Context "myk8s-context" created.
use-context,使用kubeconfig中的一个环境项作为当前配置
注意:
在/opt/kubernetes/server/bin/conf目录下
[root@hdss7-21 ~]# cd /opt/kubernetes/server/bin/conf
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=kube-proxy.kubeconfig
Switched to context "myk8s-context".
创建kube-proxy启动脚本(HDSS7-21.host.com
、HDSS7-22.host.com
)
-
加载ipvs模块
# vim /root/ipvs.sh #!/bin/bash ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs" for i in $(ls $ipvs_mods_dir|grep -o "^[^.]*") do /sbin/modinfo -F filename $i &>/dev/null if [ $? -eq 0 ];then /sbin/modprobe $i fi done # 执行脚本 # sh /root/ipvs.sh #检查ipvs模块是否加载 # lsmod |grep ip_vs
-
创建启动脚本
# vim /opt/kubernetes/server/bin/kube-proxy.sh
#!/bin/sh
./kube-proxy \
--cluster-cidr 172.7.0.0/16 \
--hostname-override hdss7-21.host.com \ # 这里写主机名
--proxy-mode=ipvs \
--ipvs-scheduler=nq \
--kubeconfig ./conf/kube-proxy.kubeconfig
**注意:**kube-proxy集群各主机的启动脚本略有不同,部署其他节点时注意修改。
检查配置,权限,创建日志目录(HDSS7-21.host.com
、HDSS7-22.host.com
)
# chmod +x /opt/kubernetes/server/bin/kube-proxy.sh
# mkdir -p /data/logs/kubernetes/kube-proxy
# 只在`hdss7-22.host.com`这一台主机上操作
[root@hdss7-22 ~]# cd /opt/kubernetes/server/bin/conf/
[root@hdss7-22 conf]# scp -rp hdss7-21:/opt/kubernetes/server/bin/conf/kube-proxy.kubeconfig ./
创建supervisor配置(HDSS7-21.host.com
、HDSS7-22.host.com
)
# vim /etc/supervisord.d/kube-proxy.ini
[program:kube-proxy-7-21]
command=/opt/kubernetes/server/bin/kube-proxy.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
安装部署启动检查所有集群规划主机上的kube-proxy服务
同上
启动服务并检查(HDSS7-21.host.com
、HDSS7-22.host.com
)
# supervisorctl update
kube-proxy-91-4: added process group
# supervisorctl status
etcd-server-7-21 RUNNING pid 8881, uptime 18:38:22
kube-apiserver-7-21 RUNNING pid 9024, uptime 16:54:54
kube-controller-manager-7-21 RUNNING pid 9062, uptime 16:38:39
kube-kubelet-7-21 RUNNING pid 15100, uptime 0:24:32
kube-proxy-7-21 RUNNING pid 20892, uptime 0:01:54
kube-scheduler-7-21 RUNNING pid 9099, uptime 16:33:38
验证kubernetes集群
在任意一个运算节点,创建一个资源配置清单(HDSS7-21.host.com
)
# vim /root/nginx-ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ds
spec:
template:
metadata:
labels:
app: nginx-ds
spec:
containers:
- name: my-nginx
image: harbor.od.com/public/nginx:1.7.9
ports:
- containerPort: 80
应用资源配置,并检查
[root@hdss7-21 ~]# kubectl apply -f /root/nginx-ds.yaml
daemonset.extensions/nginx-ds created
[root@hdss7-21 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-ds-bql4m 1/1 Running 0 29s
nginx-ds-f8fkf 1/1 Running 0 29s
部署flannel
Flannel的工作模型
1、host-gw模型:
注意:
Flannel的host-gw模型,有一个非常重要的前提条件:一个集群里的所有宿主机要处在同一个二层网络下,简单来说,就是所有宿主机要指向同一个网关,这样才能采用Flannel的host-gw模型,通过维护静态路由表的方式,让docker做到跨宿主机通信。
2、VxLAN模型
注意:
当集群内的宿主机处在不同的二层网络时,就需要用到VxLAN模型,VxLAN模型相当于在不同的宿主机上,给你实例化出一个虚拟的网络设备,这个设备就叫flannel.1,再通过flannel的网络隧道,来实现docker的跨主机通信。
集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-21.host.com | flannel | 10.4.7.21 |
HDSS7-22.host.com | flannel | 10.4.7.22 |
注意:
这里部署文档以
HDSS7-21.host.com
主机为例,另外一台运算节点安装部署方法类似
下载软件,解压,做软连接(HDSS7-21.host.com
、HDSS7-22.host.com
)
flannel官方下载地址
https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@hdss7-21 ~]# cd /opt/src/
[root@hdss7-21 src]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@hdss7-21 src]# mkdir /opt/flannel-v0.11.0
[root@hdss7-21 src]# tar xf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0/
[root@hdss7-21 src]# cd /opt/
[root@hdss7-21 opt]# ln -s flannel-v0.11.0/ flannel
[root@hdss7-21 opt]# cd flannel && mkdir certs && cd certs
#拷贝证书及私钥
[root@hdss7-21 certs]# scp hdss7-200:/opt/certs/{ca.pem,client.pem,client-key.pem} .
创建配置文件
[root@sh91-4 ~]# vi /opt/flannel/subnet.env
FLANNEL_NETWORK=172.7.0.0/16
FLANNEL_SUBNET=172.7.21.1/24 # 这里的SUBNET随主机IP变化,如主机IP:10.4.7.21
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
注意:
flannel集群各主机的配置略有不同,部署其他节点时注意修改。
创建启动脚本(hdss7-21.host.com
)
[root@hdss7-21 ~]# vi /opt/flannel/flanneld.sh
#!/bin/sh
./flanneld \
--public-ip=10.4.7.21 \ # 注意修改
--etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
--etcd-keyfile=./certs/client-key.pem \
--etcd-certfile=./certs/client.pem \
--etcd-cafile=./certs/ca.pem \
--iface=eth0 \
--subnet-file=./subnet.env \
--healthz-port=2401
检查配置,权限,创建日志目录(HDSS7-21.host.com
、HDSS7-22.host.com
)
# mkdir -p /data/logs/flanneld
# chmod +x /opt/flannel/flanneld.sh
将Flannel网络的相关信息写入到etcd,仅在某一台ETCD机器操作一次!
[root@hdss7-21 ~]# cd /opt/etcd
[root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
创建supervisor配置(HDSS7-21.host.com
、HDSS7-22.host.com
)
# vi /etc/supervisord.d/flannel.ini
[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/flannel ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
启动服务并检查(HDSS7-21.host.com
、HDSS7-22.host.com
)
# supervisorctl update
flanneld-7-21: added process group
# supervisorctl status
etcd-server-7-21 RUNNING pid 8881, uptime 19:20:42
flanneld-7-21 RUNNING pid 32124, uptime 0:01:06
kube-apiserver-7-21 RUNNING pid 9024, uptime 17:37:14
kube-controller-manager-7-21 RUNNING pid 9062, uptime 17:20:59
kube-kubelet-7-21 RUNNING pid 15100, uptime 1:06:52
kube-proxy-7-21 RUNNING pid 20892, uptime 0:44:14
kube-scheduler-7-21 RUNNING pid 9099, uptime 17:15:58
安装部署启动检查所有集群规划主机上的flannel服务
略
在各运算节点上增加iptables规则(HDSS7-21.host.com
、HDSS7-22.host.com
)
注意:
iptables规则各主机的略有不同,其他运算节点上执行时注意修改。
- 优化SNAT规则,各运算节点之间的各POD之间的网络通信不再出网
[root@hdss7-21 ~]# yum install iptables-services -y
[root@hdss7-21 ~]# systemctl start iptables.service
[root@hdss7-21 ~]# systemctl enable iptables.service
[root@hdss7-21 ~]# iptables-save |grep -i postrouting
[root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
[root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
[root@hdss7-21 ~]# iptables-save |grep -i reject
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables-save > /etc/sysconfig/iptables
#hdss7-22.host.com
[root@hdss7-22 ~]# yum install iptables-services -y
[root@hdss7-22 ~]# systemctl start iptables.service
[root@hdss7-22 ~]# systemctl enable iptables.service
[root@hdss7-22 ~]# iptables -t nat -D POSTROUTING -s 172.7.22.0/24 ! -o docker0 -j MASQUERADE
[root@hdss7-22 ~]# iptables -t nat -I POSTROUTING -s 172.7.22.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
[root@hdss7-22 ~]# iptables-save |grep -i reject
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-22 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-22 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-22 ~]# iptables-save > /etc/sysconfig/iptables
10.4.7.21主机上的,来源是172.7.21.0/24段的docker的ip,目标ip不是172.7.0.0/16段,网络发包不从docker0桥设备出站的,才进行SNAT转换
各运算节点保存iptables规则
# iptables-save > /etc/sysconfig/iptables
# 修改iptable之后,必须重启docker: systemctl restart docker!!!
部署k8s资源配置清单的内网http服务(HDSS7-200.host.com)
配置一个nginx虚拟主机,用以提供k8s统一的资源配置清单访问入口
[root@hdss7-200 ~]# vim /etc/nginx/conf.d/k8s-yaml.od.com.conf
server {
listen 80;
server_name k8s-yaml.od.com;
location / {
autoindex on;
default_type text/plain;
root /data/k8s-yaml;
}
}
配置内网DNS解析
HDSS7-11.host.com
上
[root@hdss7-11 ~]# vim /var/named/od.com.zone #追加一条解析
k8s-yaml A 10.4.7.200
[root@hdss7-11 ~]# systemctl restart named
所有的资源配置清单统一放置在运维主机
HDSS7-200.host.com
的/data/k8s-yaml
目录下即可
[root@hdss7-200 ~]# mkdir /data/k8s-yaml
[root@hdss7-200 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@hdss7-200 ~]# nginx -s reload
部署k8s的服务发现插件—CoreDNS
准备coredns-v1.6.1镜像(HDSS7-200.host.com
)
[root@hdss7-200 ~]# docker pull coredns/coredns:1.6.1
[root@hdss7-200 ~]# docker tag c0f6e815079e harbor.od.com/public/coredns:v1.6.1
[root@hdss7-200 ~]# docker push harbor.od.com/public/coredns:v1.6.1
准备资源配置清单(HDSS7-200.host.com
)
[root@hdss7-200 ~]# mkdir -p /data/k8s-yaml/coredns && cd /data/k8s-yaml/coredns
[root@hdss7-200 ~]# vim /data/k8s-yaml/coredns/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: Reconcile
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: EnsureExists
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
[root@hdss7-200 ~]# vim /data/k8s-yaml/coredns/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
log
health
ready
kubernetes cluster.local 192.168.0.0/16
forward . 10.4.7.11
cache 30
loop
reload
loadbalance
}
[root@hdss7-200 ~]# vim /data/k8s-yaml/coredns/dp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: coredns
kubernetes.io/name: "CoreDNS"
spec:
replicas: 1
selector:
matchLabels:
k8s-app: coredns
template:
metadata:
labels:
k8s-app: coredns
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
containers:
- name: coredns
image: harbor.od.com/public/coredns:v1.6.1
args:
- -conf
- /etc/coredns/Corefile
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
[root@hdss7-200 ~]# vim /data/k8s-yaml/coredns/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: coredns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: coredns
clusterIP: 192.168.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
- name: metrics
port: 9153
protocol: TCP
依次执行创建
浏览器打开:http://k8s-yaml.od.com/coredns 检查资源配置清单文件是否正确创建
在任意运算节点上应用资源配置清单
# kubectl apply -f http://k8s-yaml.od.com/coredns/rbac.yaml
# kubectl apply -f http://k8s-yaml.od.com/coredns/cm.yaml
# kubectl apply -f http://k8s-yaml.od.com/coredns/svc.yaml
# kubectl apply -f http://k8s-yaml.od.com/coredns/dp.yaml
检查
[root@hdss7-21 ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-6b6c4f9648-ztll5 1/1 Running 0 7m5s 172.7.22.3 hdss7-22.host.com <none> <none>
#创建一个deployment资源,并创建对应的svc,测试是否可以用svc查到对应的CLUSTER-IP
[root@hdss7-21 ~]# kubectl expose deployment nginx-ds --port=80 -n default
[root@hdss7-21 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 24h
nginx-ds ClusterIP 192.168.249.118 <none> 80/TCP 52s
[root@hdss7-21 ~]# dig -t A nginx-ds.default.svc.cluster.local. @192.168.0.2 +short
192.168.249.118
部署traefik(ingress)
准备traefik镜像(HDSS7-200.host.com
)
[root@hdss7-200 ~]# docker pull traefik:v1.7.2-alpine
[root@hdss7-200 ~]# docker tag add5fac61ae5 harbor.od.com/public/traefik:v1.7.2
[root@hdss7-200 ~]# docker push harbor.od.com/public/traefik:v1.7.2
准备资源配置清单(HDSS7-200.host.com
)
[root@hdss7-200 ~]# mkdir -p /data/k8s-yaml/traefik && cd /data/k8s-yaml/traefik
[root@hdss7-200 traefik]# vim rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system
[root@hdss7-200 traefik]# vim ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: traefik-ingress
namespace: kube-system
labels:
k8s-app: traefik-ingress
spec:
template:
metadata:
labels:
k8s-app: traefik-ingress
name: traefik-ingress
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
containers:
- image: harbor.od.com/public/traefik:v1.7.2
name: traefik-ingress
ports:
- name: controller
containerPort: 80
hostPort: 81
- name: admin-web
containerPort: 8080
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
args:
- --api
- --kubernetes
- --logLevel=INFO
- --insecureskipverify=true
- --kubernetes.endpoint=https://10.4.7.10:7443
- --accesslog
- --accesslog.filepath=/var/log/traefik_access.log
- --traefiklog
- --traefiklog.filepath=/var/log/traefik.log
- --metrics.prometheus
[root@hdss7-200 traefik]# vim svc.yaml
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress
ports:
- protocol: TCP
port: 80
name: controller
- protocol: TCP
port: 8080
name: admin-web
[root@hdss7-200 traefik]# vim ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: kube-system
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: traefik.od.com
http:
paths:
- path: /
backend:
serviceName: traefik-ingress-service
servicePort: 8080
配置内网DNS解析(HDSS7-11.host.com
)
[root@hdss7-11 ~]# vim /var/named/od.com.zone #追加一条解析
traefik A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
依次执行创建
浏览器打开:http://k8s-yaml.od.com/traefik 检查资源配置清单文件是否正确创建
在任意运算节点应用资源配置清单
# kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml
# kubectl apply -f http://k8s-yaml.od.com/traefik/ds.yaml
# kubectl apply -f http://k8s-yaml.od.com/traefik/svc.yaml
# kubectl apply -f http://k8s-yaml.od.com/traefik/ingress.yaml
配置反代(HDSS7-11.host.com
和HDSS7-12.host.com
)
两台主机上的nginx均需要配置,这里可以考虑使用ansible或者saltstack进行统一配置管理
# vim /etc/nginx/conf.d/od.com.conf
upstream default_backend_traefik {
server 10.4.7.21:81 max_fails=3 fail_timeout=10s;
server 10.4.7.22:81 max_fails=3 fail_timeout=10s;
}
server {
server_name *.od.com;
location / {
proxy_pass http://default_backend_traefik;
proxy_set_header Host $http_host;
proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
}
}
检查并重启nginx
# nginx -t
# systemctl restart nginx
浏览器访问
部署dashboard
准备dashboard镜像(HDSS7-200.host.com
)
[root@hdss7-200 ~]# docker pull k8scn/kubernetes-dashboard-amd64:latest
[root@hdss7-200 ~]# docker tag f9aed6605b81 harbor.od.com/public/dashboard:v1.10.1
[root@hdss7-200 ~]# docker push harbor.od.com/public/dashboard:v1.10.1
准备资源配置清单(HDSS7-200.host.com
)
[root@hdss7-200 ~]# mkdir /data/k8s-yaml/dashboard && cd /data/k8s-yaml/dashboard
[root@hdss7-200 dashboard]# vim rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
addonmanager.kubernetes.io/mode: Reconcile
name: kubernetes-dashboard-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard-admin
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard-admin
namespace: kube-system
[root@hdss7-200 dashboard]# vi dp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
priorityClassName: system-cluster-critical
containers:
- name: kubernetes-dashboard
image: harbor.od.com/public/dashboard:v1.10.1
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 50m
memory: 100Mi
ports:
- containerPort: 8443
protocol: TCP
args:
# PLATFORM-SPECIFIC ARGS HERE
- --auto-generate-certificates
volumeMounts:
- name: tmp-volume
mountPath: /tmp
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard-admin
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
[root@hdss7-200 dashboard]# vi svc.yaml
apiVersion: v1
kind: Service
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
k8s-app: kubernetes-dashboard
ports:
- port: 443
targetPort: 8443
[root@hdss7-200 dashboard]# vi ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kubernetes-dashboard
namespace: kube-system
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: dashboard.od.com
http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 443
解析域名(HDSS7-11.host.com
)
# vi /var/named/od.com.zone
dashboard 60 IN A 10.4.7.10
# 重启named
systemctl restart named
依次执行创建
浏览器打开:http://k8s-yaml.od.com/dashboard 检查资源配置清单文件是否正确创建
在任意运算节点应用资源配置清单
# kubectl apply -f http://k8s-yaml.od.com/dashboard/rbac.yaml
# kubectl apply -f http://k8s-yaml.od.com/dashboard/svc.yaml
# kubectl apply -f http://k8s-yaml.od.com/dashboard/ingress.yaml
# kubectl apply -f http://k8s-yaml.od.com/dashboard/dp.yaml
浏览器访问
配置dashboard-https认证(HDSS7-200.host.com
)
创建私钥
[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# (umask 077; openssl genrsa -out dashboard.od.com.key 2048)
创建证书签发的请求文件
[root@hdss7-200 certs]# openssl req -new -key dashboard.od.com.key -out dashboard.od.com.csr -subj "/CN=dashboard.od.com/C=CN/ST=BJ/L=Beijing/O=demon/OU=ops"
签发证书
[root@hdss7-200 certs]# openssl x509 -req -in dashboard.od.com.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out dashboard.od.com.crt -days 3650
拷贝证书到nginx服务器(HDSS7-11.host.com
、HDSS7-12.host.com
)
# mkdir /etc/nginx/certs && cd /etc/nginx/certs
# scp hdss7-200:/opt/certs/dashboard.od.com.crt ./
# scp hdss7-200:/opt/certs/dashboard.od.com.key ./
修改nginx配置,走https(HDSS7-11.host.com
、HDSS7-12.host.com
)
# vim /etc/nginx/conf.d/dashboard.od.com.conf
server {
listen 80;
server_name dashboard.od.com;
rewrite ^(.*)$ https://${server_name}$1 permanent;
}
server {
listen 443 ssl;
server_name dashboard.od.com;
ssl_certificate "certs/dashboard.od.com.crt";
ssl_certificate_key "certs/dashboard.od.com.key";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
proxy_pass http://default_backend_traefik;
proxy_set_header Host $http_host;
proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
}
}
重启nginx(HDSS7-11.host.com
、HDSS7-12.host.com
)
# nginx -s reload
浏览器访问,查看https是否生效
https://dashboard.od.com
登录dashboard界面
获取token
[root@hdss7-21 ~]# kubectl get secrets -n kube-system
NAME TYPE DATA AGE
coredns-token-zlkln kubernetes.io/service-account-token 3 42h
default-token-rc5n2 kubernetes.io/service-account-token 3 2d18h
kubernetes-dashboard-admin-token-bf7gs kubernetes.io/service-account-token 3 111m
kubernetes-dashboard-key-holder Opaque 2 110m
traefik-ingress-controller-token-vkkt6 kubernetes.io/service-account-token 3 152m
[root@hdss7-21 ~]# kubectl describe secret kubernetes-dashboard-admin-token-bf7gs -n kube-system
Name: kubernetes-dashboard-admin-token-bf7gs
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: kubernetes-dashboard-admin
kubernetes.io/service-account.uid: 435d6a94-3e28-4e43-aea3-41bb07fa6aa3
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1346 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1iZjdncyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQzNWQ2YTk0LTNlMjgtNGU0My1hZWEzLTQxYmIwN2ZhNmFhMyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.GgmJaZTmTWcx9lKlC7DYPTG0cLQrQa4jQqNbeDKi0qamfc9JddAWzflG7o3KuDk_ufKQcqw8kQEc0Wip7j7CncOPhe74keVFtLkPp53HzpO5Jw968IjC5AYkT-F60zxCKrI2GPCqTEqJFzJIxk-2_mJF0XoRWUu5UBkJ-nH7pbaC0IZcIqXwvf53M3dZDqjI60ILK3Etr-pOxEVM6RLFuboFDrErfWbrIOL6ZVwFKAmYG0xpRkWSYlc_a0Myf_cdHEzA36NaUvqMJDbHXc0OnP2oOr9DmpwcXbJIzpknA47G_t3nyaFt3TnxSiJiLP8c5-pu-CGKZv7cX1uRf6QFxQ
# 将token这个字段里的内容复制到登录界面