k8s学习笔记2-搭建harbor私有仓库

一.介绍

本次安装时的机器系统为ubuntu 22.04,harbor的版本为v2.5.3，docker已经安装，docker的版本为v20.10.12，本次安装直接在上一篇博客的基础上做的，harbor直接安装到k8s-master1上。

二.环境准备

1.docker-compose安装

apt install pip -y
pip install docker-compose

检查安装情况

docker-compose --version

2.下载和解压harbor

官网地址：https://github.com/goharbor/harbor/releases
目前最新的版本为:v2.5.3
下载harbor版本

wget https://storage.googleapis.com/harbor-releases/release-2.5.0/harbor-online-installer-v2.5.3.tgz

解压harbor文件

root@k8s-master1:~# tar -xvf harbor-online-installer-v2.5.3.tgz 
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl

三.harbor部署

1.http方式部署

修改harbor配置文件，如果使用非https方式部署，直接修改hostname字段，并注释https部分然后执行部署即可

root@k8s-master1:~# cd harbor/
root@k8s-master1:~/harbor# cp harbor.yml.tmpl harbor.yml
root@k8s-master1:~/harbor# nano harbor.yml
root@k8s-master1:~/harbor# more harbor.yml
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 192.168.100.240

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related confighttps
#https:
  # https port for harbor, default is 443
#  port: 443
  # The path of cert and key files for nginx
 # certificate: /your/certificate/path
 # private_key: /your/private/key/path
.......

开始安装（下载镜像，估计需要6分钟左右，这个完全看网速）

./install.sh

安装好之后，登录界面（帐号默认为：admin，密码为：Harbor12345）

登录进去后

到目前为止，http方式部署已经完成

2.https方式部署

官方配置https文档：https://goharbor.io/docs/2.0.0/install-config/configure-https/
使用命令，将当前http部署的harbor删除，如果没有部署http，那就无需执行如下命令

root@k8s-master1:~/harbor#docker-compose down
root@k8s-master1:~/harbor#rm docker-compose.yml

a.自制证书部署

1.修改harbor.yml配置文件

root@k8s-master1:~/harbor# cat harbor.yml
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: registry.harbor.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  #特别说明：该harbor服务器的证书和私钥的路径，与自制证书或者第三方签名的证书的位置对应，目前我的路径就是这个位置
  certificate: /data/cert/registry.harbor.com.crt
  private_key: /data/cert/registry.harbor.com.key
......

2.生成自制证书

#!/bin/bash
############################生成证书颁发机构证书############################
#1、生成CA证书私钥
mkdir -p /root/harbor/ssl
cd /root/harbor/ssl
openssl genrsa -out ca.key 4096
#2、生成CA证书
openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.harbor.com" \
 -key ca.key \
 -out ca.crt
############################生成服务器证书############################
#1、生成harbor服务器私钥
openssl genrsa -out registry.harbor.com.key 4096
#2、生成证书签名请求（CSR）
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.harbor.com" -key registry.harbor.com.key -out registry.harbor.com.csr
#3、生成一个x509 v3扩展文件
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=registry.harbor.com
DNS.2=registry.harbor
DNS.3=harbor
EOF
#4、使用该v3.ext文件为您的Harbor主机生成证书
openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in registry.harbor.com.csr \
    -out registry.harbor.com.crt
############################提供证书给Harbor和Docker############################
#1、将服务器证书和密钥复制到Harbor主机上的certficates文件夹中
mkdir -p /data/cert
cp registry.harbor.com.crt /data/cert/
cp registry.harbor.com.key /data/cert/
#2、转换yourdomain.com.crt为yourdomain.com.cert，供Docker使用
openssl x509 -inform PEM -in registry.harbor.com.crt -out registry.harbor.com.cert
#3、将服务器证书，密钥和CA文件复制到Harbor主机上的Docker证书文件夹中
mkdir -p /etc/docker/certs.d/registry.harbor.com/
cp registry.harbor.com.cert /etc/docker/certs.d/registry.harbor.com/
cp registry.harbor.com.key /etc/docker/certs.d/registry.harbor.com/
cp ca.crt /etc/docker/certs.d/registry.harbor.com/
#4、重新启动Docker Engine
systemctl restart docker

3.开始安装

./install.sh

4.在k8s-master1机器，配置ip和域名对应

echo 192.168.100.240 registry.harbor.com >> /etc/hosts

5.安装成功后，使用https://registry.harbor.com进行登录，会提示有风险

提示有风险的原因，是因为这是我们自制的证书，系统中存在CA机构不能够对这个harbor的证书进行验证，这样的话，系统就认为这个网站是有风险的，不安全的。
我们直接在浏览器中，将我们生成的ca.crt安装到浏览器上，就可以啦，这样就可以进行正常访问了
在浏览器上，安装此ca.crt
先进入证书导入界面

点击import按钮导入ca.crt证书机构，全部勾选

6.导入成功后，进入https://registry.harbor.com,发现不会出现风险提示，并且小钥匙也正常了

b.第三方签名证书部署

部署第三放签名的证书，相对来说，比自制证书简单
比如说，我们在第三方机构申请到一个证书（registry.zhangsan.com）,这个证书的名字是registry.zhangsan.com.crt,私钥是registry.zhangsan.com.key，这两个文件，都存放在/data/zhangsan/这个文件夹下
1.修改harbor.yaml文件

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: registry.zhangsan.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/zhangsan/registry.zhangsan.com.crt
  private_key: /data/zhangsan/registry.zhangsan.com.key

2.转换registry.zhangsan.com.crt为registry.zhangsan.com.cert，供Docker使用。

openssl x509 -inform PEM -in registry.zhangsan.com.crt -out registry.zhangsan.com.cert

3.直接安装即可

./install.sh

3.配置开机自启动

使用systemd来启动关闭harbor，下面这个地址Environment=harbor_install_path=/root，需要修改成自己的harbor的安装路径位置

cat > /usr/lib/systemd/system/harbor.service << 'EOF'
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
Environment=harbor_install_path=/root
ExecStart=/usr/local/bin/docker-compose -f ${harbor_install_path}/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f ${harbor_install_path}/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF

配置开机自启动

systemctl enable --now harbor

四.访问harbor仓库（自签名的https仓库）

特别说明：该仓库是使用自签名的https的仓库，如果是使用第三方签名https的harbor仓库或者是http的方式，步骤会有些许的不一样

方式1：修改启动文件

1.修改启动文件之前登录，如下，会报错，因为系统无法认证自签名证书

(base) root@test:/etc/docker# docker login registry.harbor.com
Authenticating with existing credentials...
Login did not succeed, error: Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
Username (admin): admin
Password: 
Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
(base) root@test:/etc/docker# 

2.修改启动文件/lib/systemd/system/docker.service
最主要就是修改ExecStart后增加–insecure-registry=registry.harbor.com
就是启动的时候，把仓库地址给添加上去

  GNU nano 4.8                                                                        /lib/systemd/system/docker.service                                                                         Modified  
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
Wants=containerd.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry=registry.harbor.com
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

3.重新启动docker

(base) root@test:/etc/docker# systemctl daemon-reload 
(base) root@test:/etc/docker# systemctl restart docker

4.登录registry.harbor.com

(base) root@test:/etc/docker# docker login registry.harbor.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

5.向harbor服务器push镜像，push镜像前，现在本地随便给一个镜像tag一下，然后push上去

(base) root@test:/etc/docker# docker tag ubuntu:latest registry.harbor.com/library/ubuntu
(base) root@test:/etc/docker# docker push registry.harbor.com/library/ubuntu:v1 
The push refers to repository [registry.harbor.com/library/ubuntu]
e59fc9495612: Layer already exists 
v1: digest: sha256:aa6c2c047467afc828e77e306041b7fa4a65734fe3449a54aa9c280822b0d87d size: 529
(base) root@test:/etc/docker# 

方式2：分发ca.cert证书到其他docker引擎

1.将方式1中的步骤2的修改删除，并且重新启动docker（主要为了恢复成最开始的时候），然后再重新登录registry.harbor.com，会出现无法登录的情况

(base) root@test:/etc/docker# docker login registry.harbor.com
Authenticating with existing credentials...
Login did not succeed, error: Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
Username (admin): admin
Password: 
Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
(base) root@test:/etc/docker# 

2.将在k8s-master1节点中，使用自签名的根证书分发给需要访问harbor服务的docker机器上，并且重新启动docker

(base) root@test:/etc/docker# mkdir -p /etc/docker/certs.d/registry.harbor.com/
(base) root@test:/etc/docker# scp 192.168.100.240:/root/harbor/ssl/ca.crt /etc/docker/certs.d/registry.harbor.com/
root@192.168.100.240's password: 
ca.crt                                                                                                                                                                   100% 2065   392.9KB/s   00:00    
(base) root@test:/etc/docker# systemctl restart docker
(base) root@test:/etc/docker# 

3.登录registry.harbor.com

(base) root@test:/etc/docker# docker login registry.harbor.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
(base) root@test:/etc/docker# 

4.向harbor服务器push镜像，push镜像前，现在本地随便给一个镜像tag一下，然后push上去

(base) root@test:/etc/docker# docker tag ubuntu:latest registry.harbor.com/library/ubuntu
(base) root@test:/etc/docker# docker push registry.harbor.com/library/ubuntu:v1 
The push refers to repository [registry.harbor.com/library/ubuntu]
e59fc9495612: Layer already exists 
v1: digest: sha256:aa6c2c047467afc828e77e306041b7fa4a65734fe3449a54aa9c280822b0d87d size: 529
(base) root@test:/etc/docker# 

方式3：k8s的pod访问

1.在k8s-master1节点，k8s集群使用类型为docker-registry的Secret来提供身份认证

root@k8s-master1:~# kubectl create secret docker-registry registry-key --docker-server registry.harbor.com --docker-username admin --docker-password 123456 --dry-run -o yaml > secret-regisry.yaml
root@k8s-master1:~# kubectl apply -f secret-regisry.yaml 
secret/registry-key created
root@k8s-master1:~# kubectl get secrets 
NAME           TYPE                             DATA   AGE
registry-key   kubernetes.io/dockerconfigjson   1      5s

仓库的帐号密码是用base64进行加密，我们可以将其解密

root@k8s-master1:~# cat secret-regisry.yaml 
apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5oYXJib3IuY29tIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6IjEyMzQ1NiIsImF1dGgiOiJZV1J0YVc0Nk1USXpORFUyIn19fQ==
kind: Secret
metadata:
  creationTimestamp: null
  name: registry-key
type: kubernetes.io/dockerconfigjson
root@k8s-master1:~# echo eyJhdXRocyI6eyJyZWdpc3RyeS5oYXJib3IuY29tIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6IjEyMzQ1NiIsImF1dGgiOiJZV1J0YVc0Nk1USXpORFUyIn19fQ== |base64 -d
{"auths":{"registry.harbor.com":{"username":"admin","password":"123456","auth":"YWRtaW46MTIzNDU2"}}}root@k8s-master1:~# 

2.新建一个名称为test.yaml文件的pod，见下面配置已经在registry.harbor.com网站，存放了busybox镜像。

apiVersion: batch/v1
kind: Job
metadata:
  name: hello
spec:
  template:
    # 这里是 Pod 模板
    spec:
      imagePullSecrets:
      - name: registry-key
      containers:
      - name: hello
        image: registry.harbor.com/test/busybox:1.28
        command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
      restartPolicy: OnFailure

3.直接应用，会报错

root@k8s-master1:~# kubectl get pods 
NAME          READY   STATUS         RESTARTS   AGE
hello-whcs6   0/1     ErrImagePull   0          5s

root@k8s-master1:~# kubectl describe pods hello-whcs6
后会发现如下错误，无法拉取镜像

......
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  15s   default-scheduler  Successfully assigned default/hello-whcs6 to k8s-node1
  Normal   Pulling    14s   kubelet            Pulling image "registry.harbor.com/test/busybox:1.28"
  Warning  Failed     13s   kubelet            Failed to pull image "registry.harbor.com/test/busybox:1.28": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate has expired or is not yet valid: current time 2022-07-23T03:38:45Z is after 2021-04-28T12:00:00Z
  Warning  Failed     13s   kubelet            Error: ErrImagePull

4.在k8s-ndoe1和k8s-node2节点上，将该harbor网站的证书签发机构的证书放到，这个节点的docker上去
在k8s-node1上

root@k8s-node1:~# mkdir -p /etc/docker/certs.d/registry.harbor.com/
root@k8s-node1:~# scp 192.168.100.240:/root/harbor/ssl/ca.crt /etc/docker/certs.d/registry.harbor.com/
root@192.168.100.240's password: 
ca.crt                                                                                                                                                                   100% 2065     1.4MB/s   00:00    
root@k8s-node1:~# systemctl restart docker
root@k8s-node1:~# 

在k8s-node2上

root@k8s-node2:~# mkdir -p /etc/docker/certs.d/registry.harbor.com/
root@k8s-node2:~# scp 192.168.100.240:/root/harbor/ssl/ca.crt /etc/docker/certs.d/registry.harbor.com/
root@192.168.100.240's password: 
ca.crt                                                                                                                                                                   100% 2065     1.3MB/s   00:00    
root@k8s-node2:~# systemctl restart docker
root@k8s-node2:~# 

5.删除pod，重新部署

root@k8s-master1:~# kubectl delete -f test.yaml 
job.batch "hello" deleted
root@k8s-master1:~# kubectl apply -f test.yaml 
job.batch/hello created
root@k8s-master1:~# kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
hello-t4tdq   1/1     Running   0          3s
root@k8s-master1:~# kubectl get pods -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
hello-57ps9   1/1     Running   0          58s   10.244.1.12   k8s-node1   <none>           <none>

五.参考资料

1.harbor安装并配置https
https://blog.csdn.net/networken/article/details/107502461
2.Ubuntu16 安装搭建Harbor
https://blog.csdn.net/qq_35720307/article/details/8669175