CKA 04_部署 harbor 仓库 containerd 连接 harbor 仓库 kubeadm 引导集群

最新推荐文章于 2024-04-06 22:44:41 发布
最新推荐文章于 2024-04-06 22:44:41 发布
阅读量2.5k 收藏 2
点赞数
分类专栏： kubernetes 文章标签： kubernetes 容器 云原生
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_47133613/article/details/124569758
版权

文章目录

1. 清空之前的策略

1.1 kubeadm 重置

[root@k8s1 ~]# kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
The 'update-cluster-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

[root@k8s2 ~]# kubeadm reset

[root@k8s3 ~]# kubeadm reset

1.2 刷新 IPtables 表

[root@k8s1 ~]# iptables -F
[root@k8s1 ~]# iptables -F -t nat

[root@k8s2 ~]# iptables -F
[root@k8s2 ~]# iptables -F -t nat

[root@k8s3 ~]# iptables -F
[root@k8s3 ~]# iptables -F -t nat

2. 查看 Kubernetes 集群使用的镜像

  • 先打开网络
[root@foundation21 ~]# systemctl start firewalld.service 

[root@k8s1 ~]# crictl img
IMAGE                                                             TAG                 IMAGE ID            SIZE
docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin          v1.0.1              ac40ce6257406       3.82MB
docker.io/rancher/mirrored-flannelcni-flannel                     v0.17.0             9247abf086779       19.9MB
registry.aliyuncs.com/google_containers/coredns                   v1.8.4              8d147537fb7d1       13.7MB
registry.aliyuncs.com/google_containers/coredns                   v1.8.6              a4ca41631cc7a       13.6MB
registry.aliyuncs.com/google_containers/etcd                      3.5.0-0             0048118155842       99.9MB
registry.aliyuncs.com/google_containers/etcd                      3.5.1-0             25f8c7f3da61c       98.9MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.22.1             f30469a2491a5       31.3MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.22.2             e64579b7d8862       31.3MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.22.1             6e002eb89a881       29.8MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.22.2             5425bcbd23c54       29.8MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.22.1             36c4ebbc9d979       35.9MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.22.2             873127efbc8a7       35.9MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.22.1             aca5ededae9c8       15MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.22.2             b51ddc1014b04       15MB
registry.aliyuncs.com/google_containers/pause                     3.5                 ed210e3e4a5ba       301kB
registry.aliyuncs.com/google_containers/pause                     3.6                 6270bb605e12e       302kB

3. 搭建 harbor 仓库

3.1 部署 docker

3.1.1 准备镜像源

[root@k8s4 ~]# cd /etc/yum.repos.d/
[root@k8s4 yum.repos.d]# vim docker.repo
[docker]
name=docker-ce
baseurl=http://172.25.21.250/docker-ce
gpgcheck=0

3.1.2 安装 docker

[root@k8s4 yum.repos.d]# yum install -y docker-ce

3.1.3 开机自启 docker

[root@k8s4 yum.repos.d]# systemctl enable --now docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

3.1.4 修改内核参数，打开桥接

[root@k8s4 ~]# cd /etc/sysctl.d/
[root@k8s4 sysctl.d]# vim docker.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
[root@k8s4 sysctl.d]# sysctl --system

3.1.5 验证生效

[root@k8s4 sysctl.d]# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.15
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-957.el7.x86_64
 Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 991MiB
 Name: k8s4
 ID: RMUV:W2WV:GCXQ:QCE7:IXW4:ZGEY:JT55:DEOM:VPY5:UIU3:MXBO:EZ2S
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

3.2 准备 harbor 仓库的压缩包

官方网址：harbor仓库官方网站

[root@k8s4 ~]# ls
harbor-offline-installer-v1.10.1.tgz
[root@k8s4 ~]# tar zxf harbor-offline-installer-v1.10.1.tgz

3.3 配置 harbor 仓库

  • 修改 harbor 仓库的域名
  • 修改证书的位置
  • 修改证书密钥的位置
  • 修改 harbor 仓库的登陆密码
[root@k8s4 ~]# cd harbor/
[root@k8s4 harbor]# ls
common.sh  harbor.v1.10.1.tar.gz  harbor.yml  install.sh  LICENSE  prepare
[root@k8s4 harbor]# vim harbor.yml 
hostname: reg.westos.org
  certificate: /data/certs/westos.org.crt
  private_key: /data/certs/westos.org.key
harbor_admin_password: westos

在这里插入图片描述

3.4 准备 harbor 仓库的证书

  • 当前 k8s4 主机的系统版本是 RedHat 7.6，其 openssl 工具的版本较低，没有 --addext 命令（RedHat 8.2 版本上的 openssl 工具有）
[westos@foundation21 Desktop]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)
[westos@foundation21 Desktop]$ openssl req --help
......
 -addext val         Additional cert extension key=value pair (may be given more than once)

在这里插入图片描述

  • RedHat 7.6 没有
[root@k8s4 harbor]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)
[root@k8s4 harbor]# openssl req --help

【截图】

  • 去 openssl 官方下载 1.1.1 版本（也可以去 阿里云镜像站 下载）

下载地址：openssl官方网站
阿里云镜像站下载地址：阿里云 | openssl 

[root@k8s4 ~]# ls
harbor                                openssl11-1.1.1k-3.el7.x86_64.rpm
harbor-offline-installer-v1.10.1.tgz  openssl11-libs-1.1.1k-3.el7.x86_64.rpm
[root@k8s4 ~]# yum install -y openssl11*
  • 生成证书
    （证书的域名要和 harbor 仓库的域名一致）
[root@k8s4 ~]# mkdir /data/certs -p
[root@k8s4 ~]# cd /data/
[root@k8s4 data]# ls
certs

[root@k8s4 data]# openssl11 req -newkey rsa:4096 -nodes -sha256 -keyout /data/certs/westos.org.key -addext "subjectAltName = DNS:reg.westos.org" -x509 -days 365 -out /data/certs/westos.org.crt
Can't load /root/.rnd into RNG
139809964558144:error:2406F079:random number generator:RAND_load_file:Cannot open file:crypto/rand/randfile.c:98:Filename=/root/.rnd
Generating a RSA private key
....................................................++++
.................................++++
writing new private key to '/data/certs/westos.org.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:cn
State or Province Name (full name) []:shannxi
Locality Name (eg, city) [Default City]:xi'an
Organization Name (eg, company) [Default Company Ltd]:westos
Organizational Unit Name (eg, section) []:cka
Common Name (eg, your name or your server's hostname) []:reg.westos.org
Email Address []:root@westos.org

[root@k8s4 data]# ls certs/
westos.org.crt  westos.org.key

3.5 准备解析文件

[root@k8s4 data]# vim /etc/hosts
172.25.21.4     k8s4 reg.westos.org

3.6 部署 docker-compose 工具

[root@k8s4 ~]# mv docker-compose-Linux-x86_64-1.27.0 /usr/local/bin/docker-compose
[root@k8s4 ~]# chmod +x /usr/local/bin/docker-compose
[root@k8s4 ~]# docker-compose 			//命令可以正常执行

3.7 安装 harbor 仓库

[root@k8s4 ~]# cd harbor/
[root@k8s4 harbor]# ./install.sh

在这里插入图片描述

[root@k8s4 harbor]# docker-compose ps
      Name                     Command                  State                      Ports               
-------------------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)                                      
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp                           
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)                                      
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp          
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp                           
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp,              
                                                                    0.0.0.0:443->8443/tcp              
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp                           
registry            /home/harbor/entrypoint.sh       Up (healthy)   5000/tcp                           
registryctl         /home/harbor/start.sh            Up (healthy)

3.8 验证部署

验证仓库是否部署成功，访问 harbor 控制面板
在这里插入图片描述在这里插入图片描述

3.9 将 Kubernetes 所有镜像放入harbor 仓库

  • 使用 ctr 命令导出镜像（ ctr 命令很长）
[root@k8s1 ~]# ctr ns ls
NAME   LABELS 
k8s.io        
moby  
[root@k8s1 ~]# ctr -n k8s.io i export k8s-1.22.2.tar registry.aliyuncs.com/google_containers/coredns:v1.8.6 registry.aliyuncs.com/google_containers/etcd:3.5.1-0 registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.2 registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.2 registry.aliyuncs.com/google_containers/kube-proxy:v1.22.2 registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.2 registry.aliyuncs.com/google_containers/pause:3.6
  • 将 镜像包 发送给 harbor 主机
[root@k8s1 ~]# scp k8s-1.22.2.tar k8s4:
  • docker load 命令上传镜像包
[root@k8s4 ~]# docker load -i k8s-1.22.2.tar 
16679402dc20: Loading layer [==================================================>]  657.7kB/657.7kB
2d10b4260c78: Loading layer [==================================================>]  718.9kB/718.9kB
287b21f53b5a: Loading layer [==================================================>]  29.92MB/29.92MB
Loaded image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.2
529dc7750f69: Loading layer [==================================================>]   28.4MB/28.4MB
Loaded image: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.2
48b90c7688a2: Loading layer [==================================================>]   23.8MB/23.8MB
9e94f35577fb: Loading layer [==================================================>]  12.14MB/12.14MB
Loaded image: registry.aliyuncs.com/google_containers/kube-proxy:v1.22.2
701b343e4ffd: Loading layer [==================================================>]  13.64MB/13.64MB
Loaded image: registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.2
1021ef88c797: Loading layer [==================================================>]  296.5kB/296.5kB
Loaded image: registry.aliyuncs.com/google_containers/pause:3.6
256bc5c338a6: Loading layer [==================================================>]  122.2kB/122.2kB
80e4a2390030: Loading layer [==================================================>]  13.46MB/13.46MB
Loaded image: registry.aliyuncs.com/google_containers/coredns:v1.8.6
6d75f23be3dd: Loading layer [==================================================>]  803.8kB/803.8kB
b6e8c573c18d: Loading layer [==================================================>]  1.076MB/1.076MB
d80003ff5706: Loading layer [==================================================>]  87.39MB/87.39MB
664dd6f2834b: Loading layer [==================================================>]  1.188MB/1.188MB
62ae031121b1: Loading layer [==================================================>]  8.426MB/8.426MB
Loaded image: registry.aliyuncs.com/google_containers/etcd:3.5.1-0

[root@k8s4 ~]# docker images
REPOSITORY                                                        TAG                              IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/etcd                      3.5.1-0                          25f8c7f3da61        5 months ago        293MB
registry.aliyuncs.com/google_containers/coredns                   v1.8.6                           a4ca41631cc7        6 months ago        46.8MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.22.2                          e64579b7d886        7 months ago        128MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.22.2                          5425bcbd23c5        7 months ago        122MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.22.2                          873127efbc8a        7 months ago        104MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.22.2                          b51ddc1014b0        7 months ago        52.7MB
registry.aliyuncs.com/google_containers/pause                     3.6                              6270bb605e12        7 months ago        683kB
goharbor/chartmuseum-photon                                       v0.9.0-v1.10.1                   0245d66323de        2 years ago         128MB
goharbor/harbor-migrator                                          v1.10.1                          a4f99495e0b0        2 years ago         364MB
goharbor/redis-photon                                             v1.10.1                          550a58b0a311        2 years ago         111MB
goharbor/clair-adapter-photon                                     v1.0.1-v1.10.1                   2ec99537693f        2 years ago         61.6MB
goharbor/clair-photon                                             v2.1.1-v1.10.1                   622624e16994        2 years ago         171MB
goharbor/notary-server-photon                                     v0.6.1-v1.10.1                   e4ff6d1f71f9        2 years ago         143MB
goharbor/notary-signer-photon                                     v0.6.1-v1.10.1                   d3aae2fc17c6        2 years ago         140MB
goharbor/harbor-registryctl                                       v1.10.1                          ddef86de6480        2 years ago         104MB
goharbor/registry-photon                                          v2.7.1-patch-2819-2553-v1.10.1   1a0c5f22cfa7        2 years ago         86.5MB
goharbor/nginx-photon                                             v1.10.1                          01276d086ad6        2 years ago         44MB
goharbor/harbor-log                                               v1.10.1                          1f5c9ea164bf        2 years ago         82.3MB
goharbor/harbor-jobservice                                        v1.10.1                          689368d30108        2 years ago         143MB
goharbor/harbor-core                                              v1.10.1                          14151d58ac3f        2 years ago         130MB
goharbor/harbor-portal                                            v1.10.1                          8a9856c37798        2 years ago         52.1MB
goharbor/harbor-db                                                v1.10.1                          18548720d8ad        2 years ago         148MB
goharbor/prepare                                                  v1.10.1                          897a4d535ced        2 years ago         192MB

  • 在 harbor 仓库中创建一个公开项目，存放 k8s 的镜像包
    在这里插入图片描述

  • docker push 命令上传镜像到 harbor 仓库

[root@k8s4 ~]# docker images | grep google_containers | awk '{print $1":"$2}' | awk -F/ '{system("docker tag "$0" reg.westos.org/k8s/"$3"")}'

[root@k8s4 ~]# docker images | grep reg.westos.org/k8s/
reg.westos.org/k8s/etcd                                           3.5.1-0                          25f8c7f3da61        5 months ago        293MB
reg.westos.org/k8s/coredns                                        v1.8.6                           a4ca41631cc7        6 months ago        46.8MB
reg.westos.org/k8s/kube-apiserver                                 v1.22.2                          e64579b7d886        7 months ago        128MB
reg.westos.org/k8s/kube-controller-manager                        v1.22.2                          5425bcbd23c5        7 months ago        122MB
reg.westos.org/k8s/kube-scheduler                                 v1.22.2                          b51ddc1014b0        7 months ago        52.7MB
reg.westos.org/k8s/kube-proxy                                     v1.22.2                          873127efbc8a        7 months ago        104MB
reg.westos.org/k8s/pause                                          3.6                              6270bb605e12        7 months ago        683kB

第一个 error：Get https://reg.westos.org/v2/: x509: certificate signed by unknown authority

  • docker 无 harbor 的证书，无访问 harbor 的权限
[root@k8s4 ~]# docker push reg.westos.org/k8s/etcd:3.5.1-0 
The push refers to repository [reg.westos.org/k8s/etcd]
Get https://reg.westos.org/v2/: x509: certificate signed by unknown authority

解决方法：将 harbor 的 证书 拷贝一份给 docker

[root@k8s4 ~]# cd /etc/docker/
[root@k8s4 docker]# mkdir certs.d
[root@k8s4 docker]# cd certs.d/
[root@k8s4 certs.d]# mkdir reg.westos.org
[root@k8s4 certs.d]# cd reg.westos.org/
[root@k8s4 reg.westos.org]# cp /data/certs/westos.org.crt .
[root@k8s4 reg.westos.org]# ls
westos.org.crt
[root@k8s4 reg.westos.org]# pwd
/etc/docker/certs.d/reg.westos.org

第二个 error：denied: requested access to the resource is denied

  • 下载可以匿名，上传需要认证。需要在终端登陆 harbor 仓库
[root@k8s4 reg.westos.org]# docker push reg.westos.org/k8s/etcd:3.5.1-0 
The push refers to repository [reg.westos.org/k8s/etcd]
62ae031121b1: Preparing 
664dd6f2834b: Preparing 
d80003ff5706: Preparing 
b6e8c573c18d: Preparing 
6d75f23be3dd: Preparing 
denied: requested access to the resource is denied

解决方法：docker login 登陆 harbor 仓库

[root@k8s4 reg.westos.org]# docker login reg.westos.org
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@k8s4 reg.westos.org]# cat /root/.docker/config.json 
{
	"auths": {
		"reg.westos.org": {
			"auth": "YWRtaW46d2VzdG9z"
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.15 (linux)"
	}

3.9 将 Kubernetes 所有镜像放入harbor 仓库

[root@k8s4 ~]# docker images | grep westos | awk '{system("docker push "$1":"$2"")}'
  • 上传成功！！！
    在这里插入图片描述

4. containerd 连接 harbor 仓库

4.1 修改 Kubernetes 集群中 containerd 的配置文件

  • Kubernetes 修改 containerd 的配置文件 config.toml
  • 修改拉取镜像的地址
  • 修改仓库的证书路径
[root@k8s1 ~]# cd /etc/containerd/
[root@k8s1 containerd]# ls
config.toml
[root@k8s1 containerd]# vim config.toml
     sandbox_image = "reg.westos.org/k8s/pause:3.6"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

[root@k8s2 ~]# cd /etc/containerd/
[root@k8s2 containerd]# ls
certs.d  config.toml
[root@k8s2 containerd]# vim config.toml 
    sandbox_image = "reg.westos.org/k8s/pause:3.6"
      config_path = "/etc/containerd/certs.d"

[root@k8s3 ~]# vim /etc/containerd/config.toml 
    sandbox_image = "reg.westos.org/k8s/pause:3.6"
      config_path = "/etc/containerd/certs.d"

4.2 将 harbor 仓库的证书发送给 Kubernetes 集群

[root@k8s1 containerd]# pwd
/etc/containerd
[root@k8s1 containerd]# mkdir certs.d
[root@k8s1 containerd]# cd certs.d/
[root@k8s1 certs.d]# mkdir reg.westos.org
[root@k8s1 certs.d]# cd reg.westos.org/
[root@k8s1 reg.westos.org]# scp k8s4:/data/certs/westos.org.crt .
root@k8s4's password: 
westos.org.crt                                                                    100% 2155     2.1MB/s   00:00    
[root@k8s1 reg.westos.org]# ls
westos.org.crt

[root@k8s1 containerd]# pwd
/etc/containerd
[root@k8s1 containerd]# scp -r certs.d/ k8s2:/etc/containerd
westos.org.crt                                                                    100% 2155     2.8MB/s   00:00    
[root@k8s1 containerd]# scp -r certs.d/ k8s3:/etc/containerd
westos.org.crt                                                                    100% 2155     2.4MB/s   00:00

4.3 准备集群的解析文件

[root@k8s1 reg.westos.org]# vim /etc/hosts
172.25.21.4     k8s4 reg.westos.org

[root@k8s2 reg.westos.org]# vim /etc/hosts
172.25.21.4     k8s4 reg.westos.org

[root@k8s3 reg.westos.org]# vim /etc/hosts
172.25.21.4     k8s4 reg.westos.org

4.4 重启 containerd

[root@k8s1 ~]# systemctl restart containerd.service 

[root@k8s2 ~]# systemctl restart containerd.service 

[root@k8s3 ~]# systemctl restart containerd.service

4.5 验证连接

[root@k8s1 containerd]# crictl pull reg.westos.org/k8s/pause:3.6
Image is up to date for sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee
  • harbor 仓库中记录了下载的经过

在这里插入图片描述

5. 使用 kubeadm 引导集群

5.1 修改初始化文件

[root@k8s1 ~]# vim kubeadm-init.yaml 
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "172.25.21.100:6443"		// 可以忽略这一点
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: reg.westos.org/k8s			//指向私有仓库的位置
kind: ClusterConfiguration
kubernetesVersion: 1.22.1
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

5.2 kubeadm 拉取需要的镜像

第三个 error：私有仓库中没有合适的版本

[root@k8s1 ~]# kubeadm config images pull --config kubeadm-init.yaml
failed to pull image "reg.westos.org/k8s/kube-apiserver:v1.22.1": output: time="2022-04-21T20:24:15+08:00" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"reg.westos.org/k8s/kube-apiserver:v1.22.1\": failed to resolve reference \"reg.westos.org/k8s/kube-apiserver:v1.22.1\": reg.westos.org/k8s/kube-apiserver:v1.22.1: not found"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

解决方法

  • docker 根据报错的信息去 阿里云镜像站 拉取镜像
[root@k8s4 ~]# docker pull registry.aliyuncs.com/google_containers/pause:3.5
[root@k8s4 ~]# docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.1
[root@k8s4 ~]# docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.1
[root@k8s4 ~]# docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.1
[root@k8s4 ~]# docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.22.1
[root@k8s4 ~]# docker pull registry.aliyuncs.com/google_containers/etcd:3.5.0-0
[root@k8s4 ~]# docker pull registry.aliyuncs.com/google_containers/coredns:v1.8.4
  • 再改名，分组
[root@k8s4 ~]# docker tag registry.aliyuncs.com/google_containers/pause:3.5 reg.westos.org/k8s/pause:3.5
[root@k8s4 ~]# docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.1 reg.westos.org/k8s/kube-apiserver:v1.22.1
[root@k8s4 ~]# docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.1 reg.westos.org/k8s/kube-controller-manager:v1.22.1
[root@k8s4 ~]# docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.1 reg.westos.org/k8s/kube-scheduler:v1.22.1
[root@k8s4 ~]# docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.22.1 reg.westos.org/k8s/kube-proxy:v1.22.1
[root@k8s4 ~]# docker tag registry.aliyuncs.com/google_containers/etcd:3.5.0-0 reg.westos.org/k8s/etcd:3.5.0-0
[root@k8s4 ~]# docker tag registry.aliyuncs.com/google_containers/coredns:v1.8.4 reg.westos.org/k8s/coredns:v1.8.4
  • 上传到 harbor
[root@k8s4 ~]# docker push reg.westos.org/k8s/pause:3.5
The push refers to repository [reg.westos.org/k8s/pause]
dee215ffc666: Pushed 
3.5: digest: sha256:2f4b437353f90e646504ec8317dacd6123e931152674628289c990a7a05790b0 size: 526
[root@k8s4 ~]# docker push reg.westos.org/k8s/kube-apiserver:v1.22.1
The push refers to repository [reg.westos.org/k8s/kube-apiserver]
09a0fcc34bc8: Pushed 
71204d686e50: Pushed 
07363fa84210: Pushed 
v1.22.1: digest: sha256:d61567706f42ef70e6351e2fd5637e69bcef6d487442fbfa9d1fee15e694faa8 size: 949
[root@k8s4 ~]# docker push reg.westos.org/k8s/kube-controller-manager:v1.22.1
[root@k8s4 ~]# docker push reg.westos.org/k8s/kube-scheduler:v1.22.1
[root@k8s4 ~]# docker push reg.westos.org/k8s/kube-proxy:v1.22.1
[root@k8s4 ~]# docker push reg.westos.org/k8s/etcd:3.5.0-0
[root@k8s4 ~]# docker push reg.westos.org/k8s/coredns:v1.8.4

5.2 kubeadm 拉取需要的镜像

再次拉取成功！！！

[root@k8s1 ~]# kubeadm config images pull --config kubeadm-init.yaml
[config/images] Pulled reg.westos.org/k8s/kube-apiserver:v1.22.1
[config/images] Pulled reg.westos.org/k8s/kube-controller-manager:v1.22.1
[config/images] Pulled reg.westos.org/k8s/kube-scheduler:v1.22.1
[config/images] Pulled reg.westos.org/k8s/kube-proxy:v1.22.1
[config/images] Pulled reg.westos.org/k8s/pause:3.5
[config/images] Pulled reg.westos.org/k8s/etcd:3.5.0-0
[config/images] Pulled reg.westos.org/k8s/coredns:v1.8.4

5.3 再次初始化

  • 保留 join 命令
[root@k8s1 ~]# kubeadm init --config kubeadm-init.yaml --upload-certs

在这里插入图片描述

5.4 集群加入节点

[root@k8s2 ~]# kubeadm join 172.25.21.100:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:31b8b3c4fda3fb790bd7d6bb3cca2dd0e6b6870eec9842a201493b0a78e45aee --control-plane --certificate-key 03fc498033a4f8e66cc8a74dbfdac7d8f40908d3fa7dd6b8f3a3014ac5aee3c4

[root@k8s3 ~]# kubeadm join 172.25.21.100:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:31b8b3c4fda3fb790bd7d6bb3cca2dd0e6b6870eec9842a201493b0a78e45aee --control-plane --certificate-key 03fc498033a4f8e66cc8a74dbfdac7d8f40908d3fa7dd6b8f3a3014ac5aee3c4

5.5 验证集群

[root@k8s1 ~]# kubectl get pod -A
NAMESPACE     NAME                           READY   STATUS    RESTARTS        AGE
kube-system   coredns-bdc44d9f-fgm8h         1/1     Running   0               8m15s
kube-system   coredns-bdc44d9f-zr4mf         1/1     Running   0               8m15s
kube-system   etcd-k8s1                      1/1     Running   0               8m35s
kube-system   etcd-k8s2                      1/1     Running   0               5m49s
kube-system   etcd-k8s3                      1/1     Running   0               3m44s
kube-system   kube-apiserver-k8s1            1/1     Running   0               8m21s
kube-system   kube-apiserver-k8s2            1/1     Running   0               5m51s
kube-system   kube-apiserver-k8s3            1/1     Running   0               3m45s
kube-system   kube-controller-manager-k8s1   1/1     Running   1 (5m40s ago)   8m34s
kube-system   kube-controller-manager-k8s2   1/1     Running   0               5m51s
kube-system   kube-controller-manager-k8s3   1/1     Running   0               3m45s
kube-system   kube-proxy-bnhvg               1/1     Running   0               5m51s
kube-system   kube-proxy-gkgxt               1/1     Running   0               8m15s
kube-system   kube-proxy-vnhjh               1/1     Running   0               3m45s
kube-system   kube-scheduler-k8s1            1/1     Running   1 (5m40s ago)   8m32s
kube-system   kube-scheduler-k8s2            1/1     Running   0               5m51s
kube-system   kube-scheduler-k8s3            1/1     Running   0               3m45s
Outlook(^_^)
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
cka考试环境部署脚本 - 参数传递.zip
07-13
cka考试环境部署yaml文件，可以配合说明文章使用
Ku(MDO500LKA)Ka(MDO500CKA)_datasheet_antenna_
10-02
antenna datasheet APMA Ka Ku band
K8S基于containerd做容器从harbor拉取镜
最新发布
weixin_40364776的博客
04-06 786
当K8S使用containerd做容器运行时，实现创建pod资源时 从harbor上拉取镜像
Ku(MDC750LKA)Ka(MDC750CKA)_datasheet_antenna_
10-04
Datasheet for KA
Ku(MDC500LKA)Ka(MDC500CKA)_datasheet_antenna_kuband_
10-02
Datasheet for antenna Ku and Ka band
Ku(MDC600LKA)Ka(MDC600CKA)_datasheet_antenna_kuband_
10-01
Datasheet for antenna Ku and Ka band #2
containerd对接harbor
魏志标的博客
10-18 2745
containerd 对接harbor
Containerd接入Harbor仓库
关于代码的那些事
11-13 956
在使用容器时，避免不了会使用到私有仓库，一般都是采用harbor作为私有仓库，docker 对接harbor仓库非常简单，哪containerd如何对接harbor呢？在内网使用harbor根据个人习惯，一般都是非 http 并且是通过IP 直接访问，如下：containerd如何上传或者下载镜像呢？
《Kubernetes部署篇：Containerd配置Harbor基于https协议私有镜像仓库》
东城绝神
03-31 1892
《Kubernetes部署篇：Containerd配置Harbor私有镜像仓库》
Harbor安装对接Containerd
谭宇
01-28 1282
使用docker-compose安装harbor 先决条件： 安装docker 安装docker-compose 安装参考 下载并解压 wget -c https://github.com/goharbor/harbor/releases/download/v2.3.5/harbor-offline-installer-v2.3.5.tgz tar -zxvf harbor-offline-...
如何令containerd连接私有harbor
kanganrui的专栏
01-22 1152
尝试用kubeadmin安装k8s 1.27.6，并使用containerd来运行容器，结果containerd连harbor这个问题搞了我一天，终于搞通了。发现情况跟我一样，我的harbor也是只有IP，虽然使用的是https，但端口号改了3443，参考做法，居然成功了，总结步骤如下。搞了一天，终于搞通了，无论做科研，搞研究，只要不放弃，肯努力一定会有收获的。心灰意冷，准备换回docker，后来找到这篇文章。接着参考网上很多方法，搞了一天，结果都失败。harbor使用自签证书安装。
Containerd对接harbor（设置 Containerd 容器的 insecure-registries）
justlpf的专栏
08-10 1244
请注意，设置 insecure 注册表可能会降低容器的安全性，因此请谨慎操作，并确保只将信任的注册表设置为 insecure。要设置 containerd 容器的 insecure-registries，您需要编辑 containerd 的配置文件。在配置文件中找到 [plugins."io.containerd.grpc.v1.cri".registry.mirrors]打开 containerd 的配置文件。在该部分下方添加一个新的行，指定您要设置为 insecure 的注册表。保存并关闭配置文件。
containerd对接自签名harbor
weixin_44742630的博客
11-11 1669
containerd版本: v1.5.5 生成并配置文件 # kb01为harbor域名 $ containerd config default > /etc/containerd/config.toml # containerd支持docker的证书文件模式 $ mkdir /etc/containerd/certs.d/kb01.com -p # 将ca.crt、kb01.com.key、kb01.com.cert目录放在此目录下 $ mkdir /etc/docker/certs.d/kb01.
【K8S】containerd配置Harbor镜像仓库
zhilianxian2212的博客
05-16 1053
containerd配置Harbor镜像仓库
Kubernetes ≥ 1.25 Containerd配置Harbor私有镜像仓库
qq_35925862的博客
01-11 5500
Kubernetes ≥ 1.25 Containerd配置Harbor私有镜像仓库 containerd 实现了 kubernetes 的 Container Runtime Interface (CRI) 接口，提供容器运行时核心功能，如镜像管理、容器管理等，相比 dockerd 更加简单、健壮和可移植。 从docker过度还是需要一点时间慢慢习惯的，今天来探讨containerd 如何从无域名与权威证书的私有仓库harbor，下载镜像！
Containerd对接Harbor拉取与上传镜像
m0_47116305的博客
06-28 5510
Containerd对接Harbor拉取与上传镜像
【Harbor Https】Harbor部署https并配置containerd拉取镜像
你的博客
12-08 1889
新项目，要接手另一个k8s集群，那个集群是kubernates1.24的rke2版本，里面已经不用docker容器了，使用了containerd容器来生成pod，因此需要变化我们以前的docker使用方式。containerd的镜像仓库配置地址在 /etc/rancher/rke2/registries.yaml，修改该文件，将harbor地址，证书配置进去，具体的配置参数和方法可以参考官方文档。修改harbor.yml的https配置，这里我修改了默认端口为1443，并引入服务端证书。
k8s containerd对接harbor私有仓库
南风喃的博客
08-26 2943
containerd对接https的私有harbor仓库
Harbor 安装 && Containerd 拉取 harbor私有镜像实验
qq_46141154的博客
11-23 459
containerd 下载私有仓库镜像
以下参数有误吗 CK_KEY_TYPE input_keyType = CKK_GENERIC_SECRET; CK_UTF8CHAR label[] = "An rsa secret key object "; CK_BYTE value[] = "12345678123456781234567812345678"; CK_BBOOL true = CK_TRUE; CK_BBOOL false = CK_FALSE; CK_BYTE cka_ecid[NVPKCS11_MAX_KEY_ID_SIZE]; rv = string_to_pkcs11_byte_array(cka_ecid_string_ptr,cka_ecid_string_len,cka_ecid,sizeof(cka_ecid)); GO_TO_LABEL_ON_ERROR(rv,"string_to_pkcs11_byte_array",eddsa_key_object_create_end,CKR_OK); CK_MECHANISM_TYPE rsaMechanismList[1] = {CKM_RSA_PKCS_PSS}; CK_MECHANISM_TYPE input_key_mech = CKM_SP800_108_COUNTER_KDF; //CK_BYTE cka_input_key_string[32]="INPUT_KEY" CK_BYTE input_vector[32U]={ 0x12,0x34,0x56,0x78,0x90,0x12,0x34,0x56, 0x78,0x90,0x12,0x34,0x56,0x78,0x90,0x12, 0x34,0x56,0x78,0x90,0x12,0x34,0x56,0x78, 0x90,0x12,0x34,0x56,0x78,0x90,0x12,0x34};
07-22
根据提供的代码片段，以下参数可能存在问题： 1. CK_KEY_TYPE input_keyType = CKK_GENERIC_SECRET;：CKK_GENERIC_SECRET是一个通用的密钥类型，但具体使用时需要根据需求选择合适的密钥类型。您需要确认CKK_GENERIC_SECRET是否适用于您的具体情况。 2. CK_UTF8CHAR label[] = "An rsa secret key object ";：label是用于标识密钥对象的字符串，需要使用UTF-8编码。请确保label的内容符合UTF-8编码规范。 3. CK_BYTE value[] = "12345678123456781234567812345678";：value是密钥的字节值，但提供的值是一个字符串。如果您想使用字符串作为密钥值，您需要将其转换为字节形式。 4. CK_BBOOL true = CK_TRUE;和CK_BBOOL false = CK_FALSE;：这些变量定义了布尔类型的取值，但布尔类型在PKCS#11中已经有定义，无需重新定义。您可以直接使用CK_TRUE和CK_FALSE。 5. CK_MECHANISM_TYPE input_key_mech = CKM_SP800_108_COUNTER_KDF;：CKM_SP800_108_COUNTER_KDF是一种密钥派生函数机制，但是否适用于您的情况需要根据具体需求进行确认。 6. CK_BYTE input_vector[32U]：input_vector是输入向量，长度为32字节。请确保提供的字节数组符合您所使用的密码机制和需求。 请注意，以上问题只是根据代码片段可能存在的问题进行的分析，具体是否有误还需要根据您的具体需求和使用情况进行判断和调整。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值