文章目录
1.harbor自签证书配置https
harbor建议还是配置https,没有配置的https的情况测试了很多方法都没成功过0.0
(这里配置是参考这位大佬的文章)
这里开始生成证书,觉得我这里步骤乱的可以直接去看大佬的原文章
root@compute02_16:39:13_/data/server/harbor/certs $openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
............................................................................................................................................................................................................................................++
.......................................................................................++
e is 65537 (0x10001)
root@compute02_16:39:46_/data/server/harbor/certs $openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/CN=harbor.yh.com" \
> -key ca.key -out ca.crt
root@compute02_16:40:05_/data/server/harbor/certs $ll
总用量 8
-rw-r--r-- 1 root root 1797 8月 25 16:40 ca.crt
-rw-r--r-- 1 root root 3247 8月 25 16:39 ca.key
root@compute02_16:40:07_/data/server/harbor/certs $openssl genrsa -out server.key 4096
Generating RSA private key, 4096 bit long modulus
...............................................++
...................................................................................++
e is 65537 (0x10001)
root@compute02_16:40:14_/data/server/harbor/certs $openssl req -new -sha512 -subj "/CN=harbor.yh.com" -key server.key -out \
> server.csr
root@compute02_16:40:43_/data/server/harbor/certs $ll
总用量 16
-rw-r--r-- 1 root root 1797 8月 25 16:40 ca.crt
-rw-r--r-- 1 root root 3247 8月 25 16:39 ca.key
-rw-r--r-- 1 root root 1590 8月 25 16:40 server.csr
-rw-r--r-- 1 root root 3243 8月 25 16:40 server.key
root@compute02_17:25:33_/data/server/harbor $cat certs/v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment,dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=harbor.yh.com
root@compute02_16:41:33_/data/server/harbor/certs $openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in server.csr -out server.crt
Signature ok
subject=/CN=harbor.yh.com
Getting CA Private Key
root@compute02_16:44:31_/data/server/harbor $vim harbor.yml
hostname: harbor.yh.com
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 8080
# https related config
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
certificate: /data/server/harbor/certs/server.crt
private_key: /data/server/harbor/certs/server.key
root@compute02_16:43:04_/data/server/harbor $./prepare
prepare base dir is set to /data/server/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
root@compute02_16:45:45_/data/server/harbor $docker-compose down
Stopping nginx ... done
Stopping harbor-jobservice ... done
Stopping harbor-core ... done
Stopping redis ... done
Stopping registry ... done
Stopping harbor-db ... done
Stopping registryctl ... done
Stopping harbor-portal ... done
Stopping harbor-log ... done
Removing nginx ... done
Removing harbor-jobservice ... done
Removing harbor-core ... done
Removing redis ... done
Removing registry ... done
Removing harbor-db ... done
Removing registryctl ... done
Removing harbor-portal ... done
Removing harbor-log ... done
Removing network harbor_harbor
root@compute02_16:45:45_/data/server/harbor $docker-compose up -d
root@compute02_17:21:13_/data/server/harbor $netstat -lntup|egrep '8080|443'
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 32071/docker-proxy
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 32057/docker-proxy
[root@k8s-node04 ~]# ctr -n k8s.io image pull harbor.yh.com:8080/test/nginx:v1
INFO[0000] trying next host error="failed to do request: Head \"https://harbor.yh.com:8080/v2/test/nginx/manifests/v1\": http: server gave HTTP response to HTTPS client" host="harbor.yh.com:8080"
ctr: failed to resolve reference "harbor.yh.com:8080/test/nginx:v1": failed to do request: Head "https://harbor.yh.com:8080/v2/test/nginx/manifests/v1": http: server gave HTTP response to HTTPS client
2.containerd对接harbor(在所有节点上操作)
####配置了https直接拉取镜像是不成功的
[root@k8s-master ~]# ctr -n k8s.io image pull harbor.yh.com/test/nginx:v1
INFO[0000] trying next host error="failed to do request: Head \"https://harbor.yh.com/v2/test/nginx/manifests/v1\": x509: certificate signed by unknown authority" host=harbor.yh.com
ctr: failed to resolve reference "harbor.yh.com/test/nginx:v1": failed to do request: Head "https://harbor.yh.com/v2/test/nginx/manifests/v1": x509: certificate signed by unknown authority
####配置证书
[root@k8s-master ~]# yum install -y ca-certificates
[root@k8s-master ~]# cp -a server.crt /etc/pki/ca-trust/source/anchors/
[root@k8s-master ~]# ln -s /etc/pki/ca-trust/source/anchors/server.crt /etc/ssl/certs/
[root@k8s-master ~]# update-ca-trust
####登陆测试
[root@k8s-master ~]# nerdctl login -u admin harbor.yh.com
Enter Password: Login Succeeded
####拉取镜像测试
[root@k8s-master ~]# ctr -n k8s.io image pull harbor.yh.com/test/nginx:v1
harbor.yh.com/test/nginx:v1: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:48d56bae87c65ca642b0a1d13c3dc97c4430994991e5531ff123f77cdf975fae: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6f28985ad1843afd6fd4fe0b42a30bfab63c27d302362e7341e3316e8ba25ced: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:6084105296a952523c36eea261af38885f41e9d1d0001b4916fa426e45377ffe: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:29f7ebf60efda2064ed8f3ca5f748b757c9eb4194e8db766ee370067d2c72210: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:158aac73782cb5bf2f03cc3b3f9afa49ce582c26a546f6dba65994d1c7ddd43d: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:879a7c160ac652fea0b56d0d28a9fe5a4dfb9716fe0147c5d163a841c8d83fae: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:de58cd48a671f1121ff5cc9f04cb93916d1a71f25c378f3048975421d87f5a05: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:be704f37b5f442aeb0ee33e0a86d08263d23e7343321b4fc96b68d71e869e3b4: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.7 s total: 51.2 M (30.1 MiB/s)
unpacking linux/amd64 sha256:48d56bae87c65ca642b0a1d13c3dc97c4430994991e5531ff123f77cdf975fae...
done: 11.646226ms
3.配置k8s执行yaml自动拉取
k8s执行yaml使用的命令是这个,现在还是拉取失败的
[root@k8s-node02 ~]# crictl pull harbor.yh.cn/yw/centos:v7
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and
unpack image "harbor.yh.cn/yw/centos:v7": failed to resolve reference
"harbor.yh.cn/yw/centos:v7": failed to do request: Head
"https://harbor.yh.cn/v2/yw/centos/manifests/v7": x509: certificate signed by
unknown authority
增加(150-154行;159-160行;)
cat -n /etc/containerd/config.toml
144 [plugins."io.containerd.grpc.v1.cri".registry]
145 config_path = ""
146
147 [plugins."io.containerd.grpc.v1.cri".registry.auths]
148
149 [plugins."io.containerd.grpc.v1.cri".registry.configs]
150 [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.yh.cn".tls]
151 ca_file = "/etc/ssl/certs/server.crt"
152 [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.yh.cn".auth]
153 username = "admin"
154 password = "Harbor12345"
155
156 [plugins."io.containerd.grpc.v1.cri".registry.headers]
157
158 [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
159 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.yh.cn"]
160 endpoint = ["https://harbor.yh.cn"]
161
162 [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
163 tls_cert_file = ""
164 tls_key_file = ""
重启测试
[root@k8s-node02 ~]# systemctl restart containerd
[root@k8s-node02 ~]# crictl pull harbor.yh.cn/yw/centos:v7
Image is up to date for
sha256:eeb6ee3f44bd0b5103bb561b4c16bcb82328cfe5809ab675bb17ab3a16c517c9
4.测试
删除刚刚所有节点上测试拉取的镜像
ctr -n k8s.io image rm harbor.yh.com/test/nginx:v1
通过yaml去拉取测试
[root@k8s-master ~]# cat nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo
namespace: sa
spec:
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
nodeSelector:
kubernetes.io/hostname: k8s-node02
containers:
- image: harbor.yh.com/test/nginx:v1
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
我这里目前就一个master跟两个node节点 都分别测试是不是都能执行yaml后拉取成功
[root@k8s-master ~]# kubect apply -f nginx.yaml
[root@k8s-master ~]# kubectl -n sa get pod -o wide()
NAME READY STATUS RESTARTS AGE IP
NODE NOMINATED NODE READINESS GATES
demo-56768bf96f-l7fl7 1/1 Running 0 3s 10.16.0.131
k8s-node02 <none> <none>
jenkins-0 1/1 Running 0 2d16h 10.16.0.140
k8s-master <none> <none>
测试node1
[root@k8s-master ~]# egrep 'nodeSelector|hostname' nginx.yaml
nodeSelector:
kubernetes.io/hostname: k8s-node01
[root@k8s-master ~]# kubectl apply -f nginx.yaml
deployment.apps/demo configured
[root@k8s-master ~]# kubectl -n sa get pod -o wide
NAME READY STATUS RESTARTS AGE IP
NODE NOMINATED NODE READINESS GATES
demo-56768bf96f-9ptq2 0/1 Terminating 0 7m1s 10.16.0.16
k8s-node02 <none> <none>
demo-56768bf96f-l7fl7 1/1 Running 0 3s 10.16.0.131
k8s-node01 <none> <none>
jenkins-0 1/1 Running 0 2d16h 10.16.0.140
k8s-master <none> <none>
测试master
[root@k8s-master ~]# kubectl -n sa get pod -o wide
NAME READY STATUS RESTARTS AGE IP
NODE NOMINATED NODE READINESS GATES
demo-5dfdfc7bbc-dnr2c 1/1 Running 0 3s 10.16.0.150
k8s-master <none> <none>
jenkins-0 1/1 Running 0 2d16h 10.16.0.140
k8s-master <none> <none>
5.私有harbor对接docker
docker配置比较简单,直接在指定的目录创建域名命名的目录,再把证书传上去就好
root@kube-node2_11:15:30_~ $ll /etc/docker/certs.d/harbor.yh.cn/server.crt
-rw-r--r-- 1 root root 1830 9月 14 18:21 /etc/docker/certs.d/harbor.yh.cn/server.crt
6.私有harbor对接podman
配置方法跟docker的差不多,创建域名命名的目录
root@m3db_node01_11:32:30_~ $mkdir /etc/containers/certs.d/harbor.yh.cn
root@m3db_node01_11:32:41_~ $cd /etc/containers/certs.d/harbor.yh.cn
把证书上传上来
root@m3db_node01_11:34:04_/etc/containers/certs.d/harbor.yh.cn $ll
总用量 4
-rw-r--r-- 1 root root 1830 5月 31 16:02 server.crt
登陆测试
root@m3db_node01_11:32:51_/etc/containers/certs.d/harbor.yh.cn $podman login -u admin harbor.yh.cn
Password:
Login Succeeded!
登陆成功拉取镜像测试
root@m3db_node01_11:33:31_/etc/containers/certs.d/harbor.yh.cn $podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
root@m3db_node01_11:33:39_/etc/containers/certs.d/harbor.yh.cn $podman pull harbor.yh.cn/test/alpine:latest
Trying to pull harbor.yh.cn/test/alpine:latest...
Getting image source signatures
Copying blob 540db60ca938 done
Copying config 6dbb9cc540 done
Writing manifest to image destination
Storing signatures
6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec
root@m3db_node01_11:34:02_/etc/containers/certs.d/harbor.yh.cn $podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
harbor.yh.cn/test/alpine latest 6dbb9cc54074 2 years ago 5.88 MB