k8s containerd对接harbor私有仓库

聪明勇敢有力气...

已于 2023-07-27 10:43:21 修改

阅读量3.1k

点赞数 6

分类专栏： k8s Linux运维文章标签： kubernetes linux 运维 containerd harbor

于 2022-08-26 10:47:43 首次发布

本文链接：https://blog.csdn.net/liang_operations/article/details/126538237

版权

Linux运维同时被 2 个专栏收录

16 篇文章 1 订阅

订阅专栏

k8s

3 篇文章 1 订阅

订阅专栏

文章目录

1.harbor自签证书配置https
2.containerd对接harbor(在所有节点上操作)
3.配置k8s执行yaml自动拉取
4.测试
5.私有harbor对接docker
6.私有harbor对接podman

1.harbor自签证书配置https

harbor建议还是配置https,没有配置的https的情况测试了很多方法都没成功过0.0
(这里配置是参考这位大佬的文章)

这里开始生成证书,觉得我这里步骤乱的可以直接去看大佬的原文章
root@compute02_16:39:13_/data/server/harbor/certs $openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
............................................................................................................................................................................................................................................++
.......................................................................................++
e is 65537 (0x10001)


root@compute02_16:39:46_/data/server/harbor/certs $openssl req -x509 -new -nodes -sha512 -days 3650  -subj "/CN=harbor.yh.com" \
> -key ca.key  -out ca.crt
root@compute02_16:40:05_/data/server/harbor/certs $ll
总用量 8
-rw-r--r-- 1 root root 1797 8月  25 16:40 ca.crt
-rw-r--r-- 1 root root 3247 8月  25 16:39 ca.key
root@compute02_16:40:07_/data/server/harbor/certs $openssl genrsa -out server.key 4096
Generating RSA private key, 4096 bit long modulus
...............................................++
...................................................................................++
e is 65537 (0x10001)


root@compute02_16:40:14_/data/server/harbor/certs $openssl req  -new -sha512  -subj "/CN=harbor.yh.com"  -key server.key  -out \
> server.csr


root@compute02_16:40:43_/data/server/harbor/certs $ll
总用量 16
-rw-r--r-- 1 root root 1797 8月  25 16:40 ca.crt
-rw-r--r-- 1 root root 3247 8月  25 16:39 ca.key
-rw-r--r-- 1 root root 1590 8月  25 16:40 server.csr
-rw-r--r-- 1 root root 3243 8月  25 16:40 server.key


root@compute02_17:25:33_/data/server/harbor $cat certs/v3.ext 
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment,dataEncipherment
extendedKeyUsage = serverAuth 
subjectAltName = @alt_names
[alt_names]
DNS.1=harbor.yh.com


root@compute02_16:41:33_/data/server/harbor/certs $openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in server.csr -out server.crt 
Signature ok
subject=/CN=harbor.yh.com
Getting CA Private Key


root@compute02_16:44:31_/data/server/harbor $vim harbor.yml 
hostname: harbor.yh.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 8080

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/server/harbor/certs/server.crt
  private_key: /data/server/harbor/certs/server.key
  
  

root@compute02_16:43:04_/data/server/harbor $./prepare 
prepare base dir is set to /data/server/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir


root@compute02_16:45:45_/data/server/harbor $docker-compose down
Stopping nginx             ... done
Stopping harbor-jobservice ... done
Stopping harbor-core       ... done
Stopping redis             ... done
Stopping registry          ... done
Stopping harbor-db         ... done
Stopping registryctl       ... done
Stopping harbor-portal     ... done
Stopping harbor-log        ... done
Removing nginx             ... done
Removing harbor-jobservice ... done
Removing harbor-core       ... done
Removing redis             ... done
Removing registry          ... done
Removing harbor-db         ... done
Removing registryctl       ... done
Removing harbor-portal     ... done
Removing harbor-log        ... done
Removing network harbor_harbor


root@compute02_16:45:45_/data/server/harbor $docker-compose up -d
root@compute02_17:21:13_/data/server/harbor $netstat -lntup|egrep '8080|443'
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      32071/docker-proxy  
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      32057/docker-proxy 


[root@k8s-node04 ~]# ctr -n k8s.io image pull harbor.yh.com:8080/test/nginx:v1
INFO[0000] trying next host                              error="failed to do request: Head \"https://harbor.yh.com:8080/v2/test/nginx/manifests/v1\": http: server gave HTTP response to HTTPS client" host="harbor.yh.com:8080"
ctr: failed to resolve reference "harbor.yh.com:8080/test/nginx:v1": failed to do request: Head "https://harbor.yh.com:8080/v2/test/nginx/manifests/v1": http: server gave HTTP response to HTTPS client

2.containerd对接harbor(在所有节点上操作)

 ####配置了https直接拉取镜像是不成功的
 [root@k8s-master ~]#  ctr -n k8s.io image pull harbor.yh.com/test/nginx:v1     
INFO[0000] trying next host                              error="failed to do request: Head \"https://harbor.yh.com/v2/test/nginx/manifests/v1\": x509: certificate signed by unknown authority" host=harbor.yh.com
ctr: failed to resolve reference "harbor.yh.com/test/nginx:v1": failed to do request: Head "https://harbor.yh.com/v2/test/nginx/manifests/v1": x509: certificate signed by unknown authority
####配置证书
[root@k8s-master ~]# yum install -y ca-certificates
[root@k8s-master ~]# cp -a server.crt /etc/pki/ca-trust/source/anchors/
[root@k8s-master ~]# ln -s /etc/pki/ca-trust/source/anchors/server.crt /etc/ssl/certs/
[root@k8s-master ~]# update-ca-trust    
####登陆测试
[root@k8s-master ~]# nerdctl login -u admin harbor.yh.com
Enter Password: Login Succeeded
####拉取镜像测试
[root@k8s-master ~]#  ctr -n k8s.io image pull harbor.yh.com/test/nginx:v1
harbor.yh.com/test/nginx:v1:                                                      resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:48d56bae87c65ca642b0a1d13c3dc97c4430994991e5531ff123f77cdf975fae: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6f28985ad1843afd6fd4fe0b42a30bfab63c27d302362e7341e3316e8ba25ced:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:6084105296a952523c36eea261af38885f41e9d1d0001b4916fa426e45377ffe:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:29f7ebf60efda2064ed8f3ca5f748b757c9eb4194e8db766ee370067d2c72210:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:158aac73782cb5bf2f03cc3b3f9afa49ce582c26a546f6dba65994d1c7ddd43d:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:879a7c160ac652fea0b56d0d28a9fe5a4dfb9716fe0147c5d163a841c8d83fae:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:de58cd48a671f1121ff5cc9f04cb93916d1a71f25c378f3048975421d87f5a05:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:be704f37b5f442aeb0ee33e0a86d08263d23e7343321b4fc96b68d71e869e3b4:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.7 s                                                                    total:  51.2 M (30.1 MiB/s)                                      
unpacking linux/amd64 sha256:48d56bae87c65ca642b0a1d13c3dc97c4430994991e5531ff123f77cdf975fae...
done: 11.646226ms

3.配置k8s执行yaml自动拉取

k8s执行yaml使用的命令是这个,现在还是拉取失败的
[root@k8s-node02 ~]# crictl pull harbor.yh.cn/yw/centos:v7
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and
unpack image "harbor.yh.cn/yw/centos:v7": failed to resolve reference
"harbor.yh.cn/yw/centos:v7": failed to do request: Head
"https://harbor.yh.cn/v2/yw/centos/manifests/v7": x509: certificate signed by
unknown authority 

增加(150-154行;159-160行;)
 cat -n /etc/containerd/config.toml  
   144      [plugins."io.containerd.grpc.v1.cri".registry]
   145        config_path = ""
   146
   147        [plugins."io.containerd.grpc.v1.cri".registry.auths]
   148
   149        [plugins."io.containerd.grpc.v1.cri".registry.configs]
   150          [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.yh.cn".tls]
   151            ca_file = "/etc/ssl/certs/server.crt"
   152          [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.yh.cn".auth]
   153            username = "admin"
   154            password = "Harbor12345"
   155
   156        [plugins."io.containerd.grpc.v1.cri".registry.headers]
   157
   158        [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
   159          [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.yh.cn"]
   160            endpoint = ["https://harbor.yh.cn"]
   161
   162      [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
   163        tls_cert_file = ""
   164        tls_key_file = ""

重启测试
[root@k8s-node02 ~]# systemctl restart containerd
[root@k8s-node02 ~]# crictl pull harbor.yh.cn/yw/centos:v7
Image is up to date for
sha256:eeb6ee3f44bd0b5103bb561b4c16bcb82328cfe5809ab675bb17ab3a16c517c9

4.测试

删除刚刚所有节点上测试拉取的镜像
ctr -n k8s.io image rm harbor.yh.com/test/nginx:v1
通过yaml去拉取测试
[root@k8s-master ~]# cat nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: sa
spec:
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      nodeSelector:
        kubernetes.io/hostname: k8s-node02
      containers:
      - image: harbor.yh.com/test/nginx:v1
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP

我这里目前就一个master跟两个node节点 都分别测试是不是都能执行yaml后拉取成功
[root@k8s-master ~]#  kubect apply -f nginx.yaml
[root@k8s-master ~]# kubectl -n sa get pod -o wide()                  
NAME                    READY   STATUS        RESTARTS   AGE     IP
NODE         NOMINATED NODE   READINESS GATES
demo-56768bf96f-l7fl7   1/1     Running       0          3s      10.16.0.131
k8s-node02   <none>           <none>
jenkins-0               1/1     Running       0          2d16h   10.16.0.140
k8s-master   <none>           <none>

测试node1
[root@k8s-master ~]# egrep 'nodeSelector|hostname' nginx.yaml 
      nodeSelector:
        kubernetes.io/hostname: k8s-node01
[root@k8s-master ~]# kubectl apply -f nginx.yaml  
deployment.apps/demo configured
[root@k8s-master ~]# kubectl -n sa get pod -o wide                  
NAME                    READY   STATUS        RESTARTS   AGE     IP
NODE         NOMINATED NODE   READINESS GATES
demo-56768bf96f-9ptq2   0/1     Terminating   0          7m1s    10.16.0.16
k8s-node02   <none>           <none>
demo-56768bf96f-l7fl7   1/1     Running       0          3s      10.16.0.131
k8s-node01   <none>           <none>
jenkins-0               1/1     Running       0          2d16h   10.16.0.140
k8s-master   <none>           <none>
测试master
[root@k8s-master ~]# kubectl -n sa get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP
NODE         NOMINATED NODE   READINESS GATES
demo-5dfdfc7bbc-dnr2c   1/1     Running   0          3s      10.16.0.150
k8s-master   <none>           <none>
jenkins-0               1/1     Running   0          2d16h   10.16.0.140
k8s-master   <none>           <none>

5.私有harbor对接docker

docker配置比较简单,直接在指定的目录创建域名命名的目录,再把证书传上去就好
root@kube-node2_11:15:30_~ $ll /etc/docker/certs.d/harbor.yh.cn/server.crt 
-rw-r--r-- 1 root root 1830 9月  14 18:21 /etc/docker/certs.d/harbor.yh.cn/server.crt

6.私有harbor对接podman

配置方法跟docker的差不多,创建域名命名的目录
root@m3db_node01_11:32:30_~ $mkdir  /etc/containers/certs.d/harbor.yh.cn
root@m3db_node01_11:32:41_~ $cd /etc/containers/certs.d/harbor.yh.cn
把证书上传上来
root@m3db_node01_11:34:04_/etc/containers/certs.d/harbor.yh.cn $ll
总用量 4
-rw-r--r-- 1 root root 1830 5月  31 16:02 server.crt
登陆测试
root@m3db_node01_11:32:51_/etc/containers/certs.d/harbor.yh.cn $podman login -u admin harbor.yh.cn         
Password: 
Login Succeeded!
登陆成功拉取镜像测试
root@m3db_node01_11:33:31_/etc/containers/certs.d/harbor.yh.cn $podman  images
REPOSITORY   TAG   IMAGE ID   CREATED   SIZE
root@m3db_node01_11:33:39_/etc/containers/certs.d/harbor.yh.cn $podman pull  harbor.yh.cn/test/alpine:latest
Trying to pull harbor.yh.cn/test/alpine:latest...
Getting image source signatures
Copying blob 540db60ca938 done  
Copying config 6dbb9cc540 done  
Writing manifest to image destination
Storing signatures
6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec
root@m3db_node01_11:34:02_/etc/containers/certs.d/harbor.yh.cn $podman  images
REPOSITORY                 TAG      IMAGE ID       CREATED       SIZE
harbor.yh.cn/test/alpine   latest   6dbb9cc54074   2 years ago   5.88 MB