Docker容器化封装的好处
-
Docker引擎同意了基础设施环境 -docker环境
- 硬件的配置
- 操作系统的配置
- 运行时环境的异构
-
Docker引擎同意了程序打包(装箱)方式 - docker镜像
- java程序
- python程序
- nodejs程序
- …
-
Docker引擎统一了程序部署(运行)的方式 -docker容器
- java -jar … ====> docker run
- python manage.py runserver … ====> docker run
- npm run dev … ====> docker run
Docker容器化封装的坏处
- 单机使用,无法有效集群
- 随着容器数量上升,管理成本攀升
- 没有有效的容灾、自愈机制
- 没有预设编排模板,无法实现快速、大规模容器调度
- 没有统一的配置管理工具
- 没有容器生命周期的管理工具
- 没有图形化运维管理工具
- …
之所以在这里引入Docker的特性,就是要让我们看到Docker在实际应用中还是需要一套容器编排工具的。
- 基于Docker容器引擎的开源容器编排工具目前市场上主要有:
- docker compose 、docker swarm
- Mesosphere + Marathon
Kubernetes(K8S)
Kubernetes(K8S)
- 一、Kubernetes概述
- 二、Kubernetes快速入门
- 2.6 **`核心组件`**
- 2.7 **`核心附件`**
- 2.8 **K8S逻辑架构图**
- 2.9 **网络架构图**
- 三、实验部署集群架构详解
- 四、部署K8S集群前准备
- 五、部署Master节点服务
- 六、部署运算节点服务
- 七、完成部署并验证更新
- 八、资源需求说明
一、Kubernetes概述
1.1 Kubernetes概述
- 官网:https://kubernetes.io
- GitHub:https://github.com/kubernetes/kubernetes
- 由来:谷歌的Borg系统,后经G语言重写并捐献给CNCF基金会开源
- 含义:词根源于希腊语:舵手/飞行员,、简称K8s,是用8代替8个字符“ubernete”而成的缩写
- 重要作用:开源的容器编排框架工具(生态极其丰富)
- 学习的意义:解决上述裸跑docker的若干痛点
1.2 Kubernetes优势
- 自动装箱,水平拓展,自我修复
- 服务发现和负载均衡
- 自动发布(默认滚动发布模式)和回滚
- 灰度发布
- 蓝绿发布
- 滚动发布
- 集中化配置管理和密匙管理
- 存储编排
- 任务批处理运行
- …
二、Kubernetes快速入门
- 四组基本概念:
- Pod/Pod控制器
- Name/NameSpace
- Label/Lable选择器
- Service/Ingress
2.1Pod/Pod控制器
-
Pod
- Pod是K8S里能够被运行的最小单元(原子单元)
- 一个Pod里面可以运行多个容器,他们共享UTS+NET+IPC名称空间
- 可以把Pod理解成豌豆荚,而同一 Pod 内的每个容器是一颗颗豌豆
- 一个Pod里运行多个容器,又叫:边车(SidCar)模式
-
Pod 控制器
- Pod控制器是Pod启动的一种模板,用来保证在K8S里启动Pod应始终按照我们的预期运行(副本数、生命周期、健康状态检查…)
- K8S内提供了众多的Pod控制器,常用的有一下几种:
Deployment
DaemonSet
- ReplicaSet
- StatefulSet
- Job
- CronJob
2.2 Name/NameSpace
- Name
- 由于K8S内部,使用
“资源”
来定义每一种逻辑概念(功能),故每种 “资源” 都应该有自己的“名称”
- “资源” 有以下几个配置信息:
- api版本(apiVersion)
- 类别(kind)
元数据(metadate)
- 定义清单(spec)
- 状态(status)
- …
“名称”
通常定义在“资源”
的“元数据”
信息里
- 由于K8S内部,使用
- NameSpace
- 随着项目增多、人员增加、集群规模的扩大,需要一种能够隔离K8S内各种
资源
的方法,这就是名称空间 - 名称空间可以理解为K8S内部的虚拟群组
- 不同的名称空间内的
资源
,名称可以相同,相同名称空间内的同种资源
、名称
不能相同 - 合理的使用K8S的名称空间,使得集群管理员能够更好的对交付到K8S里的服务进行分类管理和浏览
- K8S里默认存在的名称空间有:
- default
- kube-system
- kube-public
- 查询K8S里特点
资源
要带上相应的名称空间
- 随着项目增多、人员增加、集群规模的扩大,需要一种能够隔离K8S内各种
2.3 Label/Lable选择器 **
-
Label
- 标签是K8S特色的管理方式,便于分类管理资源对象
- 一个标签可以对应多个资源,一个资源也可以有多个标签,多对多的关系
- 一个资源拥有多个标签,可以实现不同维度的管理
- 标签的组成:key=value
- 与标签类似的还有一种
注解
(annotations)
-
Lable选择器
- 给资源打上标签后,可以使用标签选择器过滤指定的标签
- 标签选择器目前有两个:
- 基于等职关系(等于、不等于)
- 基于集合的关系(属于、不属于、存在)
- 许多资源支持内嵌标签选择器字段
- matchLabels
- matchExpressions
2.4 Service/Ingress ***
-
Service
**
- 在K8S中,虽然每个Pod都会被分配一个单独的IP地址,但这个IP地址会随着Pod的销毁而消失
- Service (服务)就是用来解决这个问题的核心概念
- 一个Service可以看作一组提供相同服务的Pod的对外访问接口
- Service作用于哪些Pod是通过标签选择器来定义的
- Service是K8S集群里工作在OSI网络参考模型下,
第4层
的应用,
-
Ingress
***
- Ingress是K8S集群里工作在OSI网络参考模型下,
第7层
的应用,对外暴露的接口 - Service 只能进行L4流量调度,表现形式是ip+port
- Ingress则可以调度不同业务域、不同URL访问路径的业务流量
- Ingress是K8S集群里工作在OSI网络参考模型下,
2.5 OSI网络参考模型
详解链接: https://blog.csdn.net/u014581901/article/details/50733888
2.6 核心组件
-
配置存储中心 ==> etcd服务(相当于非关系型数据库)
-
主控(master)节点
-
kube-apiserver服务
- 提供了集群管理的RESTAPI接口(包括鉴权、数据校验及集群状态变更)
- 负责其他模块之间的数据交互,承担通信枢纽功能
- 是资源配额控制的入口
- 提供完备的集群安全机制
-
kube-controller-manager服务
- 由一系列控制器组成,通过apiserver监控整个集群的状态,并确保集群处于预期的工作状态
- Node Controller
- Deployment Controller
- NameSpace Controller
- …
-
kube-scheduler服务
- 主要功能是接收调度Pod到适合的运算节点上
- 预算策略(perdict)
- 优选策略(priorities)
-
-
运算(node)节点
- kube-kubelet服务
- 定时从某个地方获取节点上Pod的期望状态(要运行的容器,运行的副本数量,网络或者存储如何配置等),并调用对应的容器平台接口达到这个状态
- 定时汇报当前节点的状态给apiserver,以供调度的时候使用
- 镜像和容器的清理工作,保证节点上的镜像不会占满磁盘空间,退出的容器不会占用太多的资源
- kueb-proxy服务
- 是K8S在每个节点上运行网络代理,service资源的载体
- 建立了Pod网络和集群网络的关系(clusterip==>podip)
- 常用三种流量调度模式
- Userspace(废弃)
- Iptables(濒临废弃)
- Ipvs(推荐)
***
- 负责建立和删除包括更新调度规则、通知apiserver自己的更新,或者从apiserver哪里获取其他kube-proxy的调度规则变化来更新自己
- CLI客户端
- kubectl
- kube-kubelet服务
2.7 核心附件
- CNI网络插件==>flannel/calico
- 服务发现插件==>coredns
- 服务暴露插件==>traefik
- GUI管理插件==>Dashboard
2.8 K8S逻辑架构图
2.9 网络架构图
- 网络配置建议(三种不同网段的地址好区分)
Service网络:192.168.0.0/16
Pod网络:172.7.0.0/16
节点网络:10.4.7.0/24
Node1:10.4.7.21
Node1:10.4.7.22
pod:172.7.21.0/24
pod:172.7.22.0/24
三、实验部署集群架构详解
四、部署K8S集群前准备
- 常见的K8S安装部署方式:
- Minikube 单节点微型K8S (仅供学习、预览使用)
二进制安装部署(生产首选,新手推荐)
- 使用kubeadmin进行部署,K8S的部署工具,泡在K8S里(相对简单,老手推荐)
4.1 K8S前置准备工作–准备虚拟机
- 准备5台2c/2g/50g虚机,使用192.168.43.11/24 网络
- 预装CentOS7.6操作系统,做好基础优化
- 安装部署bin9,部署自建DNS系统
- 准备自签证书
- 安装部署Docker环境,部署Harbor仓库
根据集群架构图准备5台虚拟机:
192.168.43.11
192.168.43.12
192.168.43.21
192.168.43.22
192.168.43.200
以下操截至到 4.8 需要在5台虚拟机都进行一次
4.2 配置网络:
vi /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=none
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.43.11
NETMASK=255.255.255.0
GATEWAY=192.168.43.254
DNS1=192.168.43.11
4.3 修改SELINUX:
vi /etc/selinux/config
SELINUX=disabled
4.4 修改hostname:
hostnamectl set-hostname HDSS7-11.host
4.5 关闭防火墙:
systemctl stop firewalld
sudo systemctl disable firewalld.service
4.6 安装base源:
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
4.7 安装epel源:
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
4.8 安装必要工具:
yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils -y
4.9 安装bin9软件:(HDSS-11.host)
yum install bind -y
4.9.1 配置bind
主配置文件格式很重要,该有的空格得有
vi /etc/named.conf
/*配置本机ip*/
listen-on port 53 { 192.168.43.11; };
/*删除ipv6*/
listen-on-v6 prot 53 {::1;};
/*为所有的ip提供DNS解析*/
allow-query { any; };
/*增加--上级DNS(直接指向网关)*/
forwarders { 192.168.43.254; };
/*采用递归算法查询ip*/
recursion yes;
/*节省资源*/
dnssec-enable no;
dnssec-validation no;
- 检查:
named-checkconf
区域配置文件:
vi /etc/named.rfc1912.zones
zone "host.com" IN {
type master;
file "host.com.zone";
allow-update{ 192.168.43.11; };
};
zone "al.com" IN {
type master;
file "al.com.zone";
allow-update{ 192.168.43.11; };
};
配置区域数据文件
- 主机域数据文件
vi /var/named/host.com.zone
$ORIGIN host.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.host.com. dnsadmin.host.com. (
2021033001 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.host.com.
$TTL 60 ; 1 minute
dns A 192.168.43.11
HDSS7-11 A 192.168.43.11
HDSS7-12 A 192.168.43.12
HDSS7-21 A 192.168.43.21
HDSS7-22 A 192.168.43.22
HDSS7-200 A 192.168.43.200
- 业务域数据文件
vi /var/named/al.com.zone
$ORIGIN al.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.al.com. dnsadmin.al.com. (
2021033001 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.al.com.
$TTL 60 ; 1 minute
dns A 192.168.43.11
- 插入知识点:
- 查看域名对应的ip地址
lookup www.qq.com
- 查看软件版本:
rpm -qa bind
4.9.2 启动bin9:
named-checkconf
systemctl start named
[root@ip-11 ~]# netstat -luntp|grep 53
tcp 0 0 192.168.43.11:53 0.0.0.0:* LISTEN 1498/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 1498/named
tcp6 0 0 ::1:953 :::* LISTEN 1498/named
udp 0 0 192.168.43.11:53 0.0.0.0:*
//这一步别忘了
[root@ip-11 ~]# dig -t A HDSS7-21.host.com @192.168.43.11 +short
192.168.43.21
- 更改
vi /etc/sysconfig/network-scripts/ifcfg-ens33
DNS1=192.168.43.11
- 执行
systemctl restart network
[root@ip-11 ~]# ping HDSS7-21.host.com
PING HDSS7-21.host.com.host (136.243.78.216) 56(84) bytes of data.
64 bytes from static.216.78.243.136.clients.your-server.de (136.243.78.216): icmp_seq=1 ttl=128 time=308 ms
64 bytes from static.216.78.243.136.clients.your-server.de (136.243.78.216): icmp_seq=2 ttl=128 time=298 ms
64 bytes from static.216.78.243.136.clients.your-server.de (136.243.78.216): icmp_seq=3 ttl=128 time=301 ms
**重写所有虚拟机的 DNS1 指向**
vi /etc/sysconfig/network-scripts/ifcfg-ens33
DNS1=192.168.43.11
systemctl restart network
**重写所有虚拟机的 DNS1 指向**
vi /etc/sysconfig/network-scripts/ifcfg-ens33
DNS1=192.168.43.11
systemctl restart network
//设置主机域名:
vi /etc/resolv.conf
//在DNS1上添加
search host.com
- 执行
[root@ip-11 ~]# ping HDSS7-21.host
PING HDSS7-21.host.com.host (136.243.78.216) 56(84) bytes of data.
64 bytes from static.216.78.243.136.clients.your-server.de (136.243.78.216): icmp_seq=1 ttl=128 time=308 ms
64 bytes from static.216.78.243.136.clients.your-server.de (136.243.78.216): icmp_seq=2 ttl=128 time=298 ms
64 bytes from static.216.78.243.136.clients.your-server.de (136.243.78.216): icmp_seq=3 ttl=128 time=301 ms
- 配置windows:
4.10 准备签发证书环境
运维主机:HDSS-200.host.com
:
4.10.1 安装CFSSL
- 证书签发工具CFSSL:R1.22
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo
chmod +x /usr/bin/cfssl*
//查询软件
which cfssl
4.10.2 创建生成CA证书(根证书)签名请求(csr)的JSON配置文件
cd /opt/
mkdir certs
cd certs/
touch ca-csr.json
vi /opt/cers/ca-csr.json
{
"CN": "Albatross7",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "al",
"OU": "test"
}
],
"ca": {
"expiry": "175200h"
}
}
CN:名称
algo:加密方法
C:国家
ST:省市
L :城市
O:组织名称
OU:组织内项目组名称
ca:证书期限
4.10.3 生成CA证书和私钥
cd /opt/certs
cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
2021/04/01 14:19:34 [INFO] generating a new CA key and certificate from CSR
2021/04/01 14:19:34 [INFO] generate received request
2021/04/01 14:19:34 [INFO] received CSR
2021/04/01 14:19:34 [INFO] generating key: rsa-2048
2021/04/01 14:19:35 [INFO] encoded CSR
2021/04/01 14:19:35 [INFO] signed certificate with serial number 193446578194154447583873515532209860518857409421
certs]# ll
total 16
-rw-r–r-- 1 root root 997 Apr 1 14:19 ca.csr
-rw-r–r-- 1 root root 331 Apr 1 14:13 ca-csr.json
-rw------- 1 root root 1675 Apr 1 14:19 ca-key.pem
-rw-r–r-- 1 root root 1350 Apr 1 14:19 ca.pem
4.11 部署docker
HDSS-21.host.com HDSS-22.host.com HDSS-200.host.com
安装
rm -f /etc/yum.repos.d/local.repo
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
配置docker
mkdir /etc/docker
vi /etc/docker/daemon.json
"bip":"172.43.与当前机器IP一致.1/24",
{
"graph":"/data/docker",
"storage-driver":"overlay2",
"insecure-registries":["registry.access.redhat.com","quay.io","harbor.al.com"],
"registry-mirrors":["https://q2gr04ke.mirror.aliyuncs.com"],
"bip":"172.43.21.1/24",
"exec-opts":["native.cgroupdriver=systemd"],
"live-restore":true
}
mkdir -p /data/docker
systemctl daemon-reload
systemctl start docker
systemctl enable docker
4.11 部署docker镜像私有仓库harbor
HDSS-200.HOST.COM
harbor下载地址: GitHub.
harbor1.8.5版本下载地址: 1.8.5
mkdir -p /opt/src
cd /opt/src
4.11.1 下载解压
wget https://github.com/goharbor/harbor/releases/tag/v1.8.5/harbor-offline-installer-v1.8.5.tgz
tar xf harbor-offline-installer-v1.8.5.tgz -C /opt/
cd /opt/
//加标签
mv harbor harbor-v1.8.5
//加软链接
ln -s /opt/harbor-v1.8.5/ /opt/harbor
4.11.2 下载解压配置
vi /opt/harbor/harbor.yml
hostname: harbor.al.com
port: 180
harbor_admin_password: Harbor12345
data_volume: /data/harbor
location: /data/harbor/logs
- 创建日志输出文件夹
mkdir -p /data/harbor/logs
- 安装单机依赖
yum install docker-compose -y
4.11.3 启动harbor
sh /opt/harbor/install.sh
- 查看
docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
harbor-core /harbor/start.sh Up
harbor-db /entrypoint.sh postgres Up 5432/tcp
harbor-jobservice /harbor/start.sh Up
harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up 80/tcp
nginx nginx -g daemon off; Up 0.0.0.0:180->80/tcp
redis docker-entrypoint.sh redis ... Up 6379/tcp
registry /entrypoint.sh /etc/regist ... Up 5000/tcp
registryctl /harbor/start.sh Up
4.11.4 安装nginx(代理harbor的端口)
yum install nginx -y
4.11.4.1 配置nginx
vi /etc/nginx/conf.d/harbor.al.com.conf
server {
listen 80;
server_name harbor.al.com;
#配置镜像大小
client_max_body_size 1000m;
location / {
proxy_pass http://127.0.0.1:180;
}
}
- 启动Nginx
systemctl start nginx
systemctl enable nginx
4.11.4.2 配置DNS解析**
HDSS-11.host.com
vi /var/named//al.com.zone
$ORIGIN al.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.al.com. dnsadmin.al.com. (
2021033002 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.al.com.
$TTL 60 ; 1 minute
dns A 192.168.43.11
harbor A 192.168.43.200
systemctl restart named
dig -t A harbor.al.com +short
192.168.43.200
4.11.5 访问harbor仓库
HDSS-200.host.com
curl harbor.al.com
- 浏览器访问
harbor.al.com
至此harbor仓库搭建成功!
4.12
- 下载Nginx镜像
docker pull nginx:1.7.9
- 加标签
docker tag 84581e99d807 harbor.al.com/public/nginx:v1.7.9
//登录
docker login harbor.al.com
//推送
docker push harbor.al.com/public/nginx:v1.7.9
五、部署Master节点服务
5.1 部署etcd集群
5.1.1 集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS-7-12 | etcd lead | 192.168.43.12 |
HDSS-7-21 | etcd follow | 192.168.43.21 |
HDSS-7-22 | etcd follow | 192.168.43.22 |
**注意:**这里部署文档以HDSS-12.host.com主机为例,另外两台主机安装部署方法类似
5.1.2创建基于根证书的config配置文件
HDSS7-200.host.com
vi /opt/certs/ca-config.json
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
- 证书类型
client certificate:客户端使用,用于服务端认证客户端,例如:etcdctl、etcd proxy、fleetctl、docker客户端等
server certificate:服务端使用,客户端以此验证服务端身份,例如:docker服务端、kube-apiserver
peer certificate:双向证书,用于etcd集群成员间通信
5.1.3 生成自签证书签名请求(csr)的JSON配置文件
HDSS7-200.host.com
vi /opt/certs/etcd-peer-csr.json
{
"CN": "k8s-etcd",
"hosts": [
"192.168.43.11",
"192.168.43.12",
"192.168.43.21",
"192.168.43.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O":"al",
"OU":"ops"
}
]
}
5.1.4 生成etcd的证书和私钥
HDSS7-200.host.com
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
5.1.5 创建新用户
HDSS7-12.host.com
useradd -s /sbin/nologin -M etcd
id etcd
uid=1000(etcd) gid=1000(etcd) groups=1000(etcd)
5.1.6 下载软件、解压、做软连接
HDSS7-12.host.com
etcd下载地址,建议版本3.1.20
mkdir /opt/src
cd /opt/src
- 下载
wget https://github.com/etcd-io/etcd/releases/tag/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz
- 解压
tar -zxvf etcd-v3.1.20-linux-amd64.tar.gz -C /opt
mv /opt/etcd-v3.1.20-linux-amd64/ etcd-v3.1.20
ln -s etcd-v3.1.20/ /opt/etcd
5.1.7 创建目录、拷贝证书、私钥
HDSS7-12.host.com
cd /opt/etcd
- 创建目录
mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
- 拷贝证书、私钥
cd /opt/etcd/certs/
scp 192.168.43.200:/opt/certs/ca.pem .
yes
输入密码
scp 192.168.43.200:/opt/certs/etcd-peer.pem .
输入密码
scp 192.168.43.200:/opt/certs/etcd-peer-key.pem .
输入密码
- 修改权限
chown -R etcd.etcd /opt/etcd/certs /data/etcd /data/logs/etcd-server
- 插入知识点
改密:
(无需操作此步骤) -
passwd --stdin root 为root用户设置密码。
5.1.8 创建etcd服务启动脚本
HDSS7-12.host.com
vi /opt/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://192.168.43.12:2380 \
--listen-client-urls https://192.168.43.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.43.12:2380 \
--advertise-client-urls https://192.168.43.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://192.168.43.12:2380,etcd-server-7-21=https://192.168.43.21:2380,etcd-server-7-22=https://192.168.43.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
5.1.9 调整权限
HDSS7-12.host.com
chmod +x /opt/etcd/etcd-server-startup.sh
chown -R etcd.etcd /opt/etcd-v3.1.20/
chown -R etcd.etcd /data/etcd/
chown -R etcd.etcd /data/logs/etcd-server/
5.1.10 安装supervisor软件
HDSS7-12.host.com
yum install supervisor -y
systemctl start supervisord
systemctl enable supervisord
5.1.11 创建etcd-server的启动配置
HDSS7-12.host.com
vi /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-12]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
注意:
etcd集群各主机启动配置略有不同,配置其他其他节点时注意修改。
5.1.12 启动etcd服务并检查
HDSS7-12.host.com
supervisorctl update
supervisorctl status
etcd-server-7-12 RUNNING pid 3682, uptime 0:00:09
报错
参考:https://blog.csdn.net/weixin_44439488/article/details/115378194
netstat -luntp|grep etcd
tcp 0 0 192.168.43.12:2379 0.0.0.0:* LISTEN 3683/./etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 3683/./etcd
tcp 0 0 192.168.43.12:2380 0.0.0.0:* LISTEN 3683/./etcd
tail -fn 200 /data/logs/etcd-server/etcd.stdout.log
5.1.13 安装部署其余两台 21 22
etcd集群
- 从5.1.5 创建新用户 —— 到5.1.12 启动etcd服务并检查
5.1.14 检查集群状态
- 第一种方式:
./etcdctl cluster-health
member 5e5f961100a3e3f1 is healthy: got healthy result from http://127.0.0.1:2379
member c49a941744a96143 is healthy: got healthy result from http://127.0.0.1:2379
member ca0ad05a831d1c10 is healthy: got healthy result from http://127.0.0.1:2379
- 第二种方式:
- who is Leader?
isLeader=true
./etcdctl member list
5e5f961100a3e3f1: name=etcd-server-7-12 peerURLs=https://192.168.43.12:2380 clientURLs=http://127.0.0.1:2379,https://192.168.43.12:2379 isLeader=true
c49a941744a96143: name=etcd-server-7-22 peerURLs=https://192.168.43.22:2380 clientURLs=http://127.0.0.1:2379,https://192.168.43.22:2379 isLeader=false
ca0ad05a831d1c10: name=etcd-server-7-21 peerURLs=https://192.168.43.21:2380 clientURLs=http://127.0.0.1:2379,https://192.168.43.21:2379 isLeader=false
5.2 部署kube-apiserver集群
5.2.1 集群规划
主机名 | 角色 | ip |
---|---|---|
HDSS7-21.host.com | kube-apiserver | 192.168.43.21 |
HDSS7-22.host.com | kube-apiserver | 192.168.43.22 |
HDSS7-11.host.com | 4层负载均衡 | 192.168.43.11 |
HDSS7-12.host.com | 4层负载均衡 | 192.168.43.12 |
注意: 这里192.168.43.11
和 192.168.43.12
使用nginx做4层负载均衡,用keepalived跑一个vip:192.168.43.10
代理两个 kube-apiserver ,实现高可用
5.2.2 下载软件、解压、做软连接
HDSS7-21.host.com
kebernetes官方Github地址
直接点击下载:Kubernetes v1.20.5
拷贝到 21
下的 /opt/src/
cd /opt/src/
tar zxvf kubernetes-server-linux-amd64.tar.gz -C /opt
mv /opt/kubernetes/ kubernetes-v1.20.5
ln -s /opt/kubernetes-v1.20.5/ /opt/kubernetes
删除k8s go语言源码包和镜像
cd kubernetes
rm -rf kubernetes-src.tar.gz
cd server/bin
rm -f *.tar
rm -f *tag
此时项目包内结构已经优化完毕。
5.2.3 签发client证书
HDSS7-200.host.com
创建生成证书签名请求(csr)的JSON配置文件
vi /opt/certs/client-csr.json
{
"CN": "k8s-node",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "al",
"OU": "ops"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssl-json -bare client
5.2.4 签发kube-apiserver证书
HDSS7-200.host.com
创建生成证书签名请求(csr)的JSON配置文件
vi /opt/certs/apiserver-csr.json
{
"CN": "k8s-apiserver",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"192.168.43.10",
"192.168.43.21",
"192.168.43.22",
"192.168.43.23"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "al",
"OU": "ops"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json |cfssl-json -bare apiserver
5.2.5 拷贝证书至各运算节点,并创建
HDSS7-21.host.com
5.2.5.1 拷贝证书、私钥,注意私钥文件属性600
mkdir /opt/kubernetes/server/bin/cert
cd /opt/kubernetes/server/bin/cert
scp 192.168.43.200:/opt/certs/ca.pem .
scp 192.168.43.200:/opt/certs/ca-key.pem .
scp 192.168.43.200:/opt/certs/apiserver.pem .
scp 192.168.43.200:/opt/certs/apiserver-key.pem .
scp 192.168.43.200:/opt/certs/client.pem .
scp 192.168.43.200:/opt/certs/client-key.pem .
5.2.5.2 创建配置
mkdir /opt/kubernetes/server/bin/conf
cd /opt/kubernetes/server/bin/conf
vi /opt/kubernetes/server/bin/conf/audit.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
5.2.6 创建启动脚本
HDSS7-21.host.com
vi /opt/kubernetes/server/bin/kube-apiserver.sh
#!/bin/bash
./kube-apiserver \
--apiserver-count 2 \
--audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \
--audit-policy-file ./conf/audit.yaml \
--authorization-mode RBAC \
--client-ca-file ./cert/ca.pem \
--requestheader-client-ca-file ./cert/ca.pem \
--enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
--etcd-cafile ./cert/ca.pem \
--etcd-certfile ./cert/client.pem \
--etcd-keyfile ./cert/client-key.pem \
--etcd-servers https://192.168.43.12:2379,https://192.168.43.21:2379,https://192.168.43.22:2379 \
--service-account-key-file ./cert/ca-key.pem \
--service-cluster-ip-range 192.168.0.0/16 \
--service-node-port-range 3000-29999 \
--target-ram-mb=1024 \
--kubelet-client-certificate ./cert/client.pem \
--kubelet-client-key ./cert/client-key.pem \
--log-dir /data/logs/kubernetes/kube-apiserver \
--tls-cert-file ./cert/apiserver.pem \
--tls-private-key-file ./cert/apiserver-key.pem \
--v 2
5.2.7 调整权限和目录
HDSS7-21.host.com
chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
mkdir -p /data/logs/kubernetes/kube-apiserver
5.2.8 创建supervisor配置
HDSS7-21.host.com
vi /etc/supervisord.d/kube-apiserver.ini
[program:kube-apiserver-7-21]
command=/opt/kubernetes/server/bin/kube-apiserver.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
5.2.9 启动代理并检查
HDSS7-21.host.com
supervisorctl update
supervisorctl status
5.2.10 安装部署启动检查所有集群规范
HDSS7-21.host.com
netstat -luntp|grep kube-api
5.2.11 配4层反向代理
HDSS7-11.host.com
HDSS7-12.host.com
yum install nginx -y
vi /etc/nginx/nginx.conf
stream {
upstream kube-apiserver {
server 192.168.43.21:6443 max_fails=3 fail_timeout=30s;
server 192.168.43.22:6443 max_fails=3 fail_timeout=30s;
}
server {
listen 7443;
proxy_connect_timeout 2s;
proxy_timeout 900s;
proxy_pass kube-apiserver;
}
}
5.2.12 启动代理并检查
HDSS7-11.host.com
HDSS7-12.host.com
5.2.12.1 安装nginx
nginx -t
systemctl start nginx
systemctl enable nginx
5.2.12.2 安装keepalived
yum install keepalived -y
5.2.12.2.1 check_port.sh
vi /etc/keepalived/check_port.sh
#!/bin/bash
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
if [ $PORT_PROCESS -eq 0 ];then
echo "Port $CHK_PORT Is Not Used,End."
exit 1
fi
else
echo "Check Port Cant Be Empty!"
fi
chmod +x /etc/keepalived/check_port.sh
5.2.12.2.1 keepalived主
HDSS7-11.host.com
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 192.168.43.11
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 251
priority 100
advert_int 1
mcast_src_ip 192.168.43.11
nopreempt
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
192.168.43.10
}
}
systemctl start keepalived
systemctl enable keepalived
5.2.12.2.1 keepalived从
HDSS7-12.host.com
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 192.168.43.12
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 251
priority 90
advert_int 1
mcast_src_ip 192.168.43.12
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
192.168.43.10
}
}
systemctl start keepalived
systemctl enable keepalived
- 查看日志
tail -200 /var/log/messages
报错 :
解决办法:
interface ens33
,这是网卡信息配置文件名,每个人可能不一样
未完待续。。。。