最近半个多月我被vim编译器坑死了,自己挖了个大坑,耗时20多天才爬出来。不管怎么样,爬出来了,总该庆贺一下。
我前不久安装了harbor要建立私有仓库。要创建自己的私有仓库首先要配置harbor 同docker通信,要让docker 可以通过login登录harbor,并且可以通过push上传镜像到harbor里面。
我的harbor是通过helm安装,ingress控制器使用的traefik,并将traefik的服务通过noteport暴露出来,使之所有https服务都有了统一的访问入口,端口号为32042。
参考官网Harbor docs | Configure HTTPS Access to Harbor
我做了使用自己创建的密钥访问https,密钥制作方式和官网一致,我重点说一下helm安装的harbor配置。
首先,创建自定义的secret,参考Harbor docs | Customize the Harbor Token Service 这里使用的域名和Common Name都使用 *.harbor.shell.com
将用于的docker pull 和 push images 的token密钥也做进去。
kubectl create secret generic harbor-ingress-custom \
--from-file=ca.crt \
--from-file=tls.crt \
--from-file=tls.key \
--from-file=private_key.pem \
--from-file=root.crt \
--type=kubernetes.io/tls -n harbor
修改values.yaml文件
1、certSource: secret //这里修改为从自定义secret取密钥
2、 secret:
# The name of secret which contains keys named:
# "tls.crt" - the certificate
# "tls.key" - the private key
secretName: "harbor-ingress-custom" //修改为刚才创建的secret名
# The name of secret which contains keys named:
# "tls.crt" - the certificate
# "tls.key" - the private key
# Only needed when the "expose.type" is "ingress".
notarySecretName: "harbor-ingress-custom" //这里同上
ingress:
hosts:
3、 ingress:
annotations:
//这里加一条
kubernetes.io/ingress.class: "traefik"
4、 externalURL: https://core.harbor.shell.com:32042
5、然后就是设置存储storageClass ,安装的时候说过,这里不做赘述
6、然后在这里找到
# The initial password of Harbor admin. Change it from portal after launching Harbor
harborAdminPassword: "Harbor12345"
# The name of the secret which contains key named "ca.crt". Setting this enables the
# download link on portal to download the certificate of CA when the certificate isn't
# generated automatically
caSecretName: "harbor-ingress-custom" // 设置这里
7、然后在这里找到
core:
image:
repository: goharbor/harbor-core
tag: v2.3.1
# set the service
...
secretName: "harbor-ingress-custom" //设置这里
# The XSRF key. Will be generated automatically if it isn't specified
xsrfKey: "root.crt" //设置这里
如果现在着急使用sudo docker login -u admin core.harbor.shell.com:32042登录话,会报类似错误
Error getting v2 registry: Get https://harbor.shell.com:32042/v2/: dial tcp 10.12.70.132:32042:
connect: no route to host"
需要配置一下docker
首先在要使用docker的客户机上配置一下/etc/hosts 将core.harbor.shell.com与ip地址绑定
然后创建mkdir -p /etc/docker/certs.d/core.harbor.shell.com:32042
将密钥拷贝进去。
如果这里拷贝的密钥和harbor加载的密钥不一致,或者你这里用自建的密钥,harbor中用helm创建的密钥,也就是certSource使用auto,会报一下类似错误
Error response from daemon: Get https://harbor.shell.com:32042/v2/: x509: certificate signed by unknown authority
最后配置/etc/docker/daemon.json
添加 "registry-mirrors": ["https://core.harbor.shell.com:32042"],
"insecure-registries": ["10.12.70.130:32042"]
其实这里如果不用https使用http也可以通过docker login登陆的,不过再往下走,使用docker push上传image会报错如下:
error parsing HTTP 405 response body: invalid character '<' looking for beginning of value: "<html>\r
\n<head><title>405 Not Allowed</title></head>\r\n<body>\r\n<center><h1>405 Not Allowed</h1></center>\r
\n<hr><center>nginx/1.19.3</center>\r\n</body>\r\n</html>
这说明push使用post方法传输是要通过https通信的。
然后 sudo systemctl daemon-reload
sudo systemctl restart docker
配置到如此,按理来说,应该一切正常了。不过我却遇到了莫名其妙的错误:
Error response from daemon: Get https://core.harbor.shell.com:32042/v2/: Get /service/token?
account=admin&client_id=docker&offline_token=true&service=harbor-registry: unsupported protocol scheme ""
但通过curl测试也可以正常通过
//sudo cp tls.crt /etc/ssl/certs/ca-certificates.crt
curl -u "admin:Harbor12345" -k -i -v https://core.harbor.shell.com:32042
Trying 10.12.70.130:32042...
* TCP_NODELAY set
* Connected to core.harbor.shell.com (10.12.70.130) port 32042 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=CN; ST=TianJin; L=TianJin; O=dzj; OU=dzj; CN=*.harbor.shell.com
* start date: Sep 6 09:21:52 2021 GMT
* expire date: Sep 4 09:21:52 2031 GMT
* issuer: C=CN; ST=TianJin; L=TianJin; O=dzj; OU=dzj; CN=*.harbor.shell.com
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Basic with user 'admin'
* Using Stream ID: 1 (easy handle 0x565310e86c80)
> GET / HTTP/2
> Host: core.harbor.shell.com:32042
> authorization: Basic YWRtaW46SGFyYm9yMTIzNDU=
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
HTTP/2 200
< cache-control: no-store, no-cache, must-revalidate
cache-control: no-store, no-cache, must-revalidate
< content-type: text/html
content-type: text/html
< date: Tue, 07 Sep 2021 03:38:32 GMT
date: Tue, 07 Sep 2021 03:38:32 GMT
< etag: W/"60f55533-48f"
etag: W/"60f55533-48f"
< last-modified: Mon, 19 Jul 2021 10:34:27 GMT
last-modified: Mon, 19 Jul 2021 10:34:27 GMT
< server: nginx/1.19.3
server: nginx/1.19.3
<
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Harbor</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
<link rel="preload" as="style" href="./light-theme.css?buildTimestamp=1626690627398">
<link rel="preload" as="style" href="./dark-theme.css?buildTimestamp=1626690627398">
<link rel="stylesheet" href="styles.0588221ae7a21e1ad681.css"></head>
<body>
<harbor-app>
<div class="spinner spinner-lg app-loading app-loading-fixed">
Loading...
</div>
</harbor-app>
<script src="runtime-es2015.3dddbb4251aaabd9f502.js" type="module"></script><script src="runtime-es5.3dddbb4251aaabd9f502.js" nomodule defer></script><script src="polyfills-es5.174f74044ee744816e0d.js" nomodule defer></script><script src="polyfills-es2015.746a3b159a979d256b42.js" type="module"></script><script src="scripts.cd39b4a7ab651b82e180.js" defer></script><script src="main-es2015.96bbe18d2039f20d8d41.js" type="module"></script><script src="main-es5.96bbe18d2039f20d8d41.js" nomodule defer></script></body>
* Connection #0 to host core.harbor.shell.com left intact
网上找了很久都没有答案,可能没人遇到过类似错误,于是我就在坑里被困了很久。
后来,翻阅资料,发现notary组件是用来提供各组件注册信任通信的,于是我到helm项目的安装目录harbor/conf/notary-server.json这个文件看到了
"{{ .Values.externalURL }}/service/token"
然后看了看harbor/conf/notary-signer.json文件
"level": "{{ .Values.logLevel }}"
回过头来又去查看配置文件values.yaml
发现logLevel这个配置项是顶格书写的,而externalURL配置项却空了两格,按照上面两个文件信息,可以知道这两个配置项应该是同级的,要顶格同时顶格,要空格同时空格,所以我猜测应该是externalURL的配置项前面空两格导致的错误,于是我将其顶格更新当前harbor
helm upgrade harbor harbor -n harbor
发现成功了,可以正常登录,并且正常push镜像了。
后来又尝试了一下,发现是因为vim在修改externalURL配置项时,当输入:时vim自动将此配置项向后空两格,当时没有发现,所以直至导致错误。现在终于明白了,所以以后书写yaml文件时一定要谨慎。要是遇到莫名其妙的错误,应多来检查一下yaml文件的书写规范。
后记——注意事项:
因为使用helm安装harbor,在helm配置文件values.yaml中使用自己定义的证书实现https访问,secret默认是tls.crt和tls.key,在导入docker支持的时候参考Harbor docs | Configure HTTPS Access to Harbor文章内容
中的自定义域名的
cp yourdomain.com.cert /etc/docker/certs.d/yourdomain.com/
cp yourdomain.com.key /etc/docker/certs.d/yourdomain.com/
cp ca.crt /etc/docker/certs.d/yourdomain.com/
需全部改成tls.cert、tls.key和ca. crt拷贝到
/etc/docker/certs.d/
└── yourdomain.com:port
├── yourdomain.com.cert <-- Server certificate signed by CA
├── yourdomain.com.key <-- Server key signed by CA
└── ca.crt <-- Certificate authority that signed the registry certificate
在重新启动docker,这样再使用docker login登录,自定义的证书才可以生效。