第四周记录

最新推荐文章于 2024-11-14 17:45:18 发布

神奇工坊

最新推荐文章于 2024-11-14 17:45:18 发布

阅读量977

点赞数

分类专栏：学习记录文章标签： linux

本文链接：https://blog.csdn.net/weixin_43451568/article/details/128177195

版权

本文详细记录了使用Ansible自动化部署k8s集群的过程，包括Harbor证书签发、安装、部署节点环境配置、K8s集群搭建与伸缩管理。此外，还涉及DNS服务配置、etcd的备份和恢复，以及Kubernetes Dashboard的安装和用户管理。整个流程覆盖了从环境规划到集群维护的多个环节。

摘要由CSDN通过智能技术生成

环境规划

类型	服务器IP地址
k8s-deploy	172.21.90.211
k8s-harbor	172.21.90.219
k8s-master（3台）	172.21.90.212/213/220
k8s-node（3台）	172.21.90.214/215/221
k8s-etcd（3台）	172.21.90.216/217/218
阿里云SLB	47.122.7.18

一、harbor证书签发

将harbor离线安装包解压后，进入harbor/目录，创建certs/目录并进入

# pwd
/apps/harbor/certs

步骤参考官网Harbor docs | Configure HTTPS Access to Harbor (goharbor.io)

1、自签名CA机构

# openssl genrsa -out ca.key 4096       #私有CA key
Generating RSA private key, 4096 bit long modulus (2 primes)
..................++++
..........................................................................................................................................................................................................................................++++
e is 65537 (0x010001)

# openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=y73.harbor.com" \
 -key ca.key \
 -out ca.crt      #自签发CA crt证书
# ll
total 20
drwxr-xr-x 2 root root 4096 Nov 20 16:52 ./
drwxr-xr-x 3 root root 4096 Nov 20 16:41 ../
-rw-r--r-- 1 root root 2053 Nov 20 16:51 ca.crt
-rw------- 1 root root 3243 Nov 20 16:45 ca.key

# touch /root/.rnd   #记录证书签发信息

2、客户端域名证书申请

# openssl genrsa -out y73.harbor.com.key 4096  #harbor服务器私钥
# ll
total 20
drwxr-xr-x 2 root root 4096 Nov 20 16:52 ./
drwxr-xr-x 3 root root 4096 Nov 20 16:41 ../
-rw-r--r-- 1 root root 2053 Nov 20 16:51 ca.crt
-rw------- 1 root root 3243 Nov 20 16:45 ca.key
-rw------- 1 root root 3243 Nov 20 16:52 y73.harbor.com.key   #harbor私钥key


# openssl req -sha512 -new \
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=y73.harbor.com" \
    -key y73.harbor.com.key \
    -out y73.harbor.com.csr      #harbor服务器csr文件
# ll
total 24
drwxr-xr-x 2 root root 4096 Nov 20 16:55 ./
drwxr-xr-x 3 root root 4096 Nov 20 16:41 ../
-rw-r--r-- 1 root root 2053 Nov 20 16:51 ca.crt
-rw------- 1 root root 3243 Nov 20 16:45 ca.key
-rw-r--r-- 1 root root 1708 Nov 20 16:55 y73.harbor.com.csr
-rw------- 1 root root 3243 Nov 20 16:52 y73.harbor.com.key

3、准备签发环境

# cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=y73.harbor.com
DNS.2=y73.harbor
DNS.3=k8s-harbor1
EOF     #证书签发SAN文件

# ll
total 28
drwxr-xr-x 2 root root 4096 Nov 20 16:58 ./
drwxr-xr-x 3 root root 4096 Nov 20 16:41 ../
-rw-r--r-- 1 root root 2053 Nov 20 16:51 ca.crt
-rw------- 1 root root 3243 Nov 20 16:45 ca.key
-rw-r--r-- 1 root root  268 Nov 20 16:58 v3.ext
-rw-r--r-- 1 root root 1708 Nov 20 16:55 y73.harbor.com.csr
-rw------- 1 root root 3243 Nov 20 16:52 y73.harbor.com.key

4、使用自签名CA签发证书

# openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in y73.harbor.com.csr \
    -out y73.harbor.com.crt    #自签发harbor证书
    
# ll
total 36
drwxr-xr-x 2 root root 4096 Nov 20 17:01 ./
drwxr-xr-x 3 root root 4096 Nov 20 16:41 ../
-rw-r--r-- 1 root root 2053 Nov 20 16:51 ca.crt
-rw------- 1 root root 3243 Nov 20 16:45 ca.key
-rw-r--r-- 1 root root   41 Nov 20 17:01 ca.srl
-rw-r--r-- 1 root root  268 Nov 20 16:58 v3.ext
-rw-r--r-- 1 root root 2122 Nov 20 17:01 y73.harbor.com.crt
-rw-r--r-- 1 root root 1708 Nov 20 16:55 y73.harbor.com.csr
-rw------- 1 root root 3243 Nov 20 16:52 y73.harbor.com.key

5、安装harbor

修改配置文件引用证书

/apps/harbor/certs/y73.harbor.com.crt

/apps/harbor/certs/y73.harbor.com.key

# ./install.sh --help

Note: Please set hostname and other necessary attributes in harbor.yml first. DO NOT use localhost or 127.0.0.1 for hostname, because Harbor needs to be accessed by external clients.
Please set --with-notary if needs enable Notary in Harbor, and set ui_url_protocol/ssl_cert/ssl_cert_key in harbor.yml bacause notary must run under https. 
Please set --with-trivy if needs enable Trivy in Harbor
Please set --with-chartmuseum if needs enable Chartmuseum in Harbor

# ./install.sh  --with-trivy --with-chartmuseum

查看网站

6、部署节点安装docker并同步harbor crt证书

在部署节点k8s-deploy创建目录

# mkdir /etc/docker/certs.d/y73.harbor.com -p

在服务器把公钥拷贝到客户端

# pwd
/apps/harbor/certs

# scp y73.harbor.com.crt 172.21.90.189:/etc/docker/certs.d/y73.harbor.com


# systemctl restart docker    重启docker

之后在客户端才能成功登录harbor仓库

# docker login y73.harbor.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

测试push镜像到harbor

# docker pull alpine
# docker tag alpine:latest y73.harbor.com/baseimages/alpine:latest     #要先创建baseimages项目
# docker push y73.harbor.com/baseimages/alpine:latest
The push refers to repository [y73.harbor.com/baseimages/alpine]
8d3ac3489996: Pushed 
latest: digest: sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 size: 528

二、ansible部署k8s集群

1、基础环境准备

root@k8s-deploy:~# apt install ansible -y
root@k8s-deploy:~# ssh-keygen
root@k8s-deploy:~# apt install sshpass   #安装sshpass命令用于同步公钥到各k8s服务器
root@k8s-deploy:~# vim key.sh

#!/bin/bash
IP="
172.21.90.204
172.21.90.203
172.21.90.202
172.21.90.206
172.21.90.207
172.21.90.209
172.21.90.208
"

for node in ${IP};do
        sshpass -p Abcd1234 ssh-copy-id ${node} -o StrictHostKeyChecking=no
          echo "${node} 密钥copy完成"
        ssh ${node} ln -sv /usr/bin/python3 /usr/bin/python    #ansible需要python环境，为每个节点创建软连接
          echo "${node} /usr/bin/python3 软连接创建完成"
done      

root@k8s-deploy:~# bash key.sh  #执行脚本同步

root@k8s-deploy:~# ln -sv /usr/bin/python3 /usr/bin/python     #ansible需要python环境，为每个节点创建软连接
'/usr/bin/python' -> '/usr/bin/python3'

验证可以免密钥登录其它服务器

2、下载kubeasz项目及组件

步骤参考githubkubeasz/00-planning_and_overall_intro.md at master · easzlab/kubeasz (github.com)

root@k8s-deploy:~# apt install git
root@k8s-deploy:~# export release=3.3.1
root@k8s-deploy:~# wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown
root@k8s-deploy:~# chmod +x ./ezdown
root@k8s-deploy:~# ./ezdown -D    #下载kubeasz代码、二进制、默认容器镜像

上述脚本运行成功后，所有文件（kubeasz代码、二进制、离线镜像）均已整理好放入目录/etc/kubeasz

3、生产并自定义hosts文件

root@k8s-deploy:~# cd /etc/kubeasz/
root@k8s-deploy:/etc/kubeasz# ./ezctl new k8s-cluster01
2022-11-25 15:48:10 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-cluster01
2022-11-25 15:48:10 DEBUG set versions
2022-11-25 15:48:11 DEBUG cluster k8s-cluster01: files successfully created.
2022-11-25 15:48:11 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-cluster01/hosts'
2022-11-25 15:48:11 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-cluster01/config.yml'

然后根据提示配置’/etc/kubeasz/clusters/k8s-01/hosts’ 和 ‘/etc/kubeasz/clusters/k8s-01/config.yml’：根据前面节点规划修改hosts 文件和其他集群层面的主要配置选项；其他集群组件等配置项可以在config.yml 文件中修改。

1、编辑hosts文件

指定etcd节点、master节点、node节点、VIP、运行时、网络组件类型、service IP和pod IP范围等配置信息

root@k8s-deploy:~# cd /etc/kubeasz/clusters/k8s-cluster01
root@k8s-deploy:/etc/kubeasz/clusters/k8s-cluster01# pwd
/etc/kubeasz/clusters/k8s-cluster01

root@k8s-deploy:/etc/kubeasz/clusters/k8s-cluster01# vim hosts

以下为hosts文件

# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
172.21.90.204
172.21.90.203
172.21.90.202

# master node(s)
[kube_master]
172.21.90.209
172.21.90.208

# work node(s)
[kube_node]
172.21.90.206
172.21.90.207

# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#192.168.1.8 NEW_INSTALL=false

# [optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
#192.168.1.1

[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"

# Cluster container-runtime supported: docker, containerd
# if k8s version >= 1.24, docker is not supported
CONTAINER_RUNTIME="containerd"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.100.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.200.0.0/16/16"

# NodePort Range
NODE_PORT_RANGE="30000-60000"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="y73.local"

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/usr/local/bin"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"

# Directory for a specific cluster
cluster_dir="{
   { base_dir }}/clusters/k8s-cluster01"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

2、编辑config.yml文件

root@k8s-deploy:~# vim /etc/kubeasz/clusters/k8s-cluster01/config.yml

############################
# prepare
############################
# 可选离线安装系统软件包 (offline|online)
INSTALL_SOURCE: "online"

# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening
OS_HARDEN: false


############################
# role:deploy
############################
# default: ca will expire in 100 years
# default: certs issued by the ca will expire in 50 years
CA_EXPIRY: "876000h"
CERT_EXPIRY: "438000h"

# kubeconfig 配置参数
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{
   { CLUSTER_NAME }}"

# k8s version
K8S_VER: "1.24.2"

############################
# role:etcd
############################
# 设置不同的wal目录，可以避免磁盘io竞争，提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""


############################
# role:runtime [containerd,docker]
############################
# ------------------------------------------- containerd
# [.]启用容器仓库镜像
ENABLE_MIRROR_REGISTRY: true

# [containerd]基础容器镜像
#SANDBOX_IMAGE: "easzlab.io.local:5000/easzlab/pause:3.7"
SANDBOX_IMAGE: "y73.harbor.com/baseimages/pause:3.7"
# [containerd]容器持久化存储目录
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"

# ------------------------------------------- docker
# [docker]容器存储目录
DOCKER_STORAGE_DIR: "/var/lib/docker"

# [docker]开启Restful API
ENABLE_REMOTE_API: false

# [docker]信任的HTTP仓库
INSECURE_REG: '["http://easzlab.io.local:5000"]'


############################
# role:kube-master
############################
# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）
MASTER_CERT_HOSTS:
  - "10.1.1.1"
  - "k8s.easzlab.io"
  #- "www.test.com"

# node 节点上 pod 网段掩码长度（决定每个节点最多能分配的pod ip地址）
# 如果flannel 使用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点分配pod网段
# https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24


############################
# role:kube-node
############################
# Kubelet 根目录
KUBELET_ROOT_DIR: "/var/lib/kubelet"

# node节点最大pod 数
MAX_PODS: 500

# 配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量
# 数值设置详见templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"

# k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控，了解系统的资源占用状况；
# 并且随着系统运行时间，需要适当增加资源预留，数值设置详见templates/kubelet-config.yaml.j2
# 系统预留设置基于 4c/8g 虚机，最小化安装系统服务，如果使用高性能物理机可以适当增加预留
# 另外，集群安装时候apiserver等资源占用会短时较大，建议至少预留1g内存
SYS_RESERVED_ENABLED: "no"


############################
# role:network [flannel,calico,cilium,kube-ovn,kube-router]