podman加速器&harbor私有镜像仓库

Docker Registry(仓库)

网上有很多的Registry服务器都支持第三方用户注册，而后基于用户名去做自己的仓库，但是使用互联网上的Registry有一个缺陷，那就是我们去推送和下载镜像时都不会很快，而在生产环境中很可能并行启动的容器将达到几十、上百个，而且很有可能每个服务器本地是没有镜像的，此时如果通过互联网去下载镜像会有很多问题，比如下载速度会很慢、带宽会用很多等等，如果带宽不够的话，下载至启动这个过程可能要持续个几十分钟，这已然违背了使用容器会更加轻量、快速的初衷和目的。因此，很多时候我们很有可能需要去做自己的私有Registry。

Registry用于保存docker镜像，包括镜像的层次结构和元数据。用户可以自建Registry，也可以使用官方的Docker Hub。

Docker Registry分类：

• Sponsor Registry：第三方的Registry，供客户和Docker社区使用
• Mirror Registry：第三方的Registry，只让客户使用
• Vendor Registry：由发布docker镜像的供应商提供的registry
• Private Registry：通过设有防火墙和额外的安全层的私有实体提供的registry

事实上，如果运维的系统环境托管在云计算服务上，比如阿里云，那么用阿里云的Registry则是最好的选择。很多时候我们的生产环境不会在本地，而是托管在数据中心机房里，如果我们在数据中心机房里的某台主机上部署Registry，因为都在同一机房，所以属于同一局域网，此时数据传输走内网，效率会极大的提升。

所有的Registry默认情况下都是基于https工作的，这是Docker的基本要求，而我自建Registry时很可能是基于http工作的，但是Docker默认是拒绝使用http提供Registry服务的，除非明确的告诉它，我们就是要用http协议的Registry。

Docker Private(私人) Registry(仓库)

为了帮助我们快速创建私有Registry，Docker专门提供了一个名为Docker Distribution的软件包，我们可以通过安装这个软件包快速构建私有仓库。
问：既然Docker是为了运行程序的，Docker Distribution能否运行在容器中？
容器时代，任何程序都应该运行在容器中，除了Kernel和init。而为了能够做Docker Private Registry，Docker Hub官方直接把Registry做成了镜像，我们可以直接将其pull到本地并启动为容器即可快速实现私有Registry。

Registry的主要作用是托管镜像，Registry运行在容器中，而容器自己的文件系统是随着容器的生命周期终止和删除而被删除的，所以当我们把Registry运行在容器中时，客户端上传了很多镜像，随着Registry容器的终止并删除，所有镜像都将化为乌有，因此这些镜像应该放在存储卷上，而且这个存储卷最好不要放在Docker主机本地，而应该放在一个网络共享存储上，比如NFS。不过，镜像文件自己定义的存储卷，还是一个放在Docker本地、Docker管理的卷，我们可以手动的将其改成使用其它文件系统的存储卷。

这就是使用容器来运行Registry的一种简单方式。自建Registry的另一种方式，就是直接安装docker-distribution软件。

Harbor简介

Harbor是由VMWare在Docker Registry的基础之上进行了二次封装，加进去了很多额外程序，而且提供了一个非常漂亮的web界面。

Project Harbor is an open source trusted cloud native registry project that stores, signs, and scans context.
// 项目 harbor是一个开源受信任的云原生的仓库项目。用来存储、用户管理、查找镜像
Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management.
// Harbor通过添加用户通常需要的功能，如安全、身份和管理，扩展了开源Docker分发版。
Harbor supports advanced features such as user management, access control, activity monitoring, and replication between instances.
// Harbor支持高级特性，如用户管理、访问控制、活动监视和高可用

Harbor的功能

• Multi-tenant content signing and validation //用户的登陆和校验
• Security and vulnerability analysis //安装性和漏洞分析
• Audit logging //日志审计
• Identity integration and role-based access control //身份认证，基于角色的访问控制
• Image replication between instances //镜像的实例的高可用
• Extensible API and graphical UI //web 图形化的界面
• Internationalization（currently English and Chinese） //支持的语言 英语和中文

Docker compose

Harbor在物理机上部署是非常难的，而为了简化Harbor的应用，Harbor官方直接把Harbor做成了在容器中运行的应用，而且这个容器在Harbor中依赖类似redis、mysql、pgsql等很多存储系统，所以它需要编排很多容器协同起来工作，因此VMWare Harbor在部署和使用时，需要借助于Docker的单机编排工具(Docker compose)来实现。

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Docker Compose官网文档

harbor私有镜像仓库部署

---

实验环境：

主机名	IP	软件	系统
docker	192.168.164.141	docker、Docker Compose、harbor-offline-installer-v2.3.5	centos8
client	192.168.164.144	docker	centos8

harbor-offline-installer下载链接
两个主机的防火墙selinux都要关闭

1、两个主机上下载docker的源码包

[root@docker ~]# curl -o docker-ce.repo https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo 
[root@client ~]# curl -o docker-ce.repo https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo

2、下载当前Docker Compose 的稳定版本
*如果出现访问被拒绝或者下载失败多下几次即可，下载完成后在/usr/local/bin/目录下面会有一个docker-compose文件*

[root@docker ~]# curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

[root@docker ~]# cd /usr/local/bin/
[root@docker bin]# ls
docker-compose

3、给这个文件设置可执行的权限

[root@docker bin]# chmod +x docker-compose 
[root@docker bin]# ll
总用量 12440
-rwxr-xr-x 1 root root 12737304 12月 15 19:06 docker-compose

4、解压提前下载好的harbor仓库的压缩包，并修改配置文件

[root@docker ~]# tar xf harbor-offline-installer-v2.3.5.tgz  -C /usr/local/
[root@docker ~]# cd /usr/local/
[root@localhost local]# ls
bin  games   include  lib64    sbin   src
etc  harbor  lib      libexec  share

//复制原有的配置文件进行配置
[root@docker harbor]# cp harbor.yml.tmpl harbor.yml
[root@docker harbor]# ls
common.sh             harbor.yml       install.sh  prepare
harbor.v2.3.5.tar.gz  harbor.yml.tmpl  LICENSE

5、两个主机上添加仓库主机域名解析

//跟改主机的域名，也就是主机名
[root@docker ~]# hostname
docker
[root@docker ~]# hostname docker.example.com
[root@docker ~]# bash
[root@docker ~]# hostname
docker.example.com

//docker主机上添加域名解析
[root@docker ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.164.141  docker.example.com


//客户端上添加域名解析
[root@client ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.164.141  docker.example.com

//客户机上配置完成后ping一下 看是否能通
[root@client ~]# ping docker.example.com
PING docker.example.com (192.168.164.141) 56(84) bytes of data.
64 bytes from docker.example.com (192.168.164.141): icmp_seq=1 ttl=64 time=0.421 ms
64 bytes from docker.example.com (192.168.164.141): icmp_seq=2 ttl=64 time=0.393 ms
64 bytes from docker.example.com (192.168.164.141): icmp_seq=3 ttl=64 time=0.579 ms
^C
--- docker.example.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2069ms
rtt min/avg/max/mdev = 0.393/0.464/0.579/0.083 ms

6、docker主机上修改配置文件

[root@localhost harbor]# pwd
/usr/local/harbor
[root@localhost harbor]# vim harbor.yml
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: docker.example.com    //主机域名 就是刚刚设置的主机域名

# http related config
http:      //我们使用的是http，所以这里不用做任何修改
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
# https:   //因为我们是用的http不是https没有证书，所以注释这里即可
  # https port for harbor, default is 443
  # port: 443
  # The path of cert and key files for nginx
//这两个位置是证书的位置，如果购买了证书那么填写证书的位置即可
  # certificate: /your/certificate/path    
  # private_key: /your/private/key/path

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
//这里是登陆web管理界面的账户和密码 账户是admin 密码是Harbor12345
harbor_admin_password: Harbor12345

# Harbor DB configuration
database:   //数据库
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123  //数据库的密码
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
//最大空闲连接数
  max_idle_conns: 100
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
//最大打开连接数
  max_open_conns: 900

# The default data volume
//数据存放的位置，推荐把数据房子共享存储上面（NFS,NAS），这样主机出问题了数据也不会丢失
data_volume: /data   

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

# Trivy configuration
#
# Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases.
# It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached
# in the local file system. In addition, the database contains the update timestamp so Trivy can detect whether it
# should download a newer version from the Internet or use the cached one. Currently, the database is updated every
# 12 hours and published as a new release to GitHub.
trivy:
  # ignoreUnfixed The flag to display only fixed vulnerabilities
  ignore_unfixed: false
  # skipUpdate The flag to enable or disable Trivy DB downloads from GitHub
  #
  # You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues.
  # If the flag is enabled you have to download the `trivy-offline.tar.gz` archive manually, extract `trivy.db` and
  # `metadata.json` files and mount them in the `/home/scanner/.cache/trivy/db` path.
  skip_update: false
  #
  # insecure The flag to skip verifying registry certificate
  insecure: false
  # github_token The GitHub access token to download Trivy DB
  #
  # Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough
  # for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000
  # requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult
  # https://developer.github.com/v3/#rate-limiting
  #
  # You can create a GitHub token by following the instructions in
  # https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
  #
  # github_token: xxx

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
//日志滚动，一个文件超过200M的时候就滚动一次
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.3.0   //版本号

7、执行当前目录下的安装脚本
注意：执行这一步前需要确保防火墙、selinux是关闭的，并且docker服务时在运行的

[root@localhost harbor]# pwd
/usr/local/harbor
[root@localhost harbor]# ls
common              harbor.v2.3.5.tar.gz  install.sh
common.sh           harbor.yml            LICENSE
docker-compose.yml  harbor.yml.tmpl       prepare
[root@localhost harbor]# ./install.sh 

[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.11

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.29.2

[Step 2]: loading Harbor images ...
1e3f0dc884e2: Loading layer  39.45MB/39.45MB
8b0c54c9b24d: Loading layer   5.27MB/5.27MB
16fd3250cc89: Loading layer  6.219MB/6.219MB
8305a4ff482f: Loading layer  15.88MB/15.88MB
f56299486b92: Loading layer  29.29MB/29.29MB
e94faa0e0434: Loading layer  22.02kB/22.02kB
3795db0164b1: Loading layer  15.88MB/15.88MB
Loaded image: goharbor/notary-server-photon:v2.3.5
93bb3a91e25c: Loading layer  7.662MB/7.662MB
3691a58d5ac4: Loading layer  4.096kB/4.096kB
58d80447c5c1: Loading layer  3.072kB/3.072kB
a18f17bf310a: Loading layer  31.52MB/31.52MB
b8bde3306c8b: Loading layer  11.39MB/11.39MB
d771094af926: Loading layer   43.7MB/43.7MB
Loaded image: goharbor/trivy-adapter-photon:v2.3.5
9b3feb1ef69e: Loading layer    124MB/124MB
6338ca24efb3: Loading layer  3.584kB/3.584kB
9318b22b0102: Loading layer  3.072kB/3.072kB
0bbac171b9e5: Loading layer   2.56kB/2.56kB
.........
.........
# 这里它会自动拉取很多支撑web管理界面的容器并且运行
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating redis         ... done
Creating registry      ... done
Creating registryctl   ... done
Creating harbor-db     ... done
Creating harbor-portal ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----
# 安装完成


//查看刚刚自动拉取的镜像和容器运行情况
[root@localhost harbor]# docker images
REPOSITORY                      TAG       IMAGE ID       CREATED        SIZE
goharbor/harbor-exporter        v2.3.5    1730c6f650e2   5 days ago     81.9MB
goharbor/chartmuseum-photon     v2.3.5    47004f032938   5 days ago     179MB
goharbor/redis-photon           v2.3.5    3d0cedc89a0d   5 days ago     156MB
goharbor/trivy-adapter-photon   v2.3.5    5c0212e98070   5 days ago     133MB
goharbor/notary-server-photon   v2.3.5    f20a76c65359   5 days ago     111MB
goharbor/notary-signer-photon   v2.3.5    b9fa38eef4d7   5 days ago     108MB
goharbor/harbor-registryctl     v2.3.5    7a52567a76ca   5 days ago     133MB
goharbor/registry-photon        v2.3.5    cf22d3e386b8   5 days ago     82.6MB
goharbor/nginx-photon           v2.3.5    5e3b6d9ce11a   5 days ago     45.7MB
goharbor/harbor-log             v2.3.5    a03e4bc963d6   5 days ago     160MB
goharbor/harbor-jobservice      v2.3.5    2ac32df5a2e0   5 days ago     211MB
goharbor/harbor-core            v2.3.5    23baee01156f   5 days ago     193MB
goharbor/harbor-portal          v2.3.5    bb545cdedf5a   5 days ago     58.9MB
goharbor/harbor-db              v2.3.5    9826c57a5749   5 days ago     221MB
goharbor/prepare                v2.3.5    a1ceaabe47b2   5 days ago     255MB


[root@localhost harbor]# docker ps 
CONTAINER ID   IMAGE                                COMMAND                  CREATED         STATUS                   PORTS                                   NAMES
a4cff4361f8a   goharbor/harbor-jobservice:v2.3.5    "/harbor/entrypoint.…"   5 minutes ago   Up 5 minutes (healthy)                                           harbor-jobservice
713dadebde2c   goharbor/nginx-photon:v2.3.5         "nginx -g 'daemon of…"   5 minutes ago   Up 5 minutes (healthy)   0.0.0.0:80->8080/tcp, :::80->8080/tcp   nginx
77376cebd7c4   goharbor/harbor-core:v2.3.5          "/harbor/entrypoint.…"   5 minutes ago   Up 5 minutes (healthy)                                           harbor-core
67bd3c094539   goharbor/harbor-portal:v2.3.5        "nginx -g 'daemon of…"   5 minutes ago   Up 5 minutes (healthy)                                           harbor-portal
031b3ff26b57   goharbor/harbor-db:v2.3.5            "/docker-entrypoint.…"   5 minutes ago   Up 5 minutes (healthy)                                           harbor-db
09de1d92ef96   goharbor/harbor-registryctl:v2.3.5   "/home/harbor/start.…"   5 minutes ago   Up 5 minutes (healthy)                                           registryctl
964e47238ab9   goharbor/registry-photon:v2.3.5      "/home/harbor/entryp…"   5 minutes ago   Up 5 minutes (healthy)                                           registry
21d3b48fda27   goharbor/redis-photon:v2.3.5         "redis-server /etc/r…"   5 minutes ago   Up 5 minutes (healthy)                                           redis
d32dfefe5f4d   goharbor/harbor-log:v2.3.5           "/bin/sh -c /usr/loc…"   5 minutes ago   Up 5 minutes (healthy)   127.0.0.1:1514->10514/tcp               harbor-log


[root@localhost harbor]# ss -antl
State     Recv-Q    Send-Q       Local Address:Port       Peer Address:Port   Process    
LISTEN    0         128              127.0.0.1:1514            0.0.0.0:*                 
LISTEN    0         128                0.0.0.0:111             0.0.0.0:*                 
LISTEN    0         128                0.0.0.0:80              0.0.0.0:*                 
LISTEN    0         128                0.0.0.0:22              0.0.0.0:*                 
LISTEN    0         128                   [::]:111                [::]:*                 
LISTEN    0         128                   [::]:80                 [::]:*                 
LISTEN    0         128                   [::]:22                 [::]:*                 

IP访问即可


在客户端使用的时候如果不是用的https则必须要在客户端的/etc/docker/daemon.json配置文件中配置insecure-registries参数

8、修改客户端的配置文件
因为使用的http并不是https所有要修改配置文件insecure-registries参数

[root@client ~]# vim /etc/docker/daemon.json
{
"insecure-registries": ["docker.example.com"]
# 如果使用的是https不要在前面加insecure
}
[root@client ~]# systemctl restart docker

9、测试是否能上传成功，并且在web界面可以看见
登陆、上传镜像、web界面查看

//登陆到自己搭建的私有仓库中
[root@client ~]# docker login docker.example.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded


//随便拉取一个镜像测试，这里会自动到官方的仓库中去拉取，并不是自己搭建的仓库中拉取
[root@client ~]# docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
3cb635b06aa2: Pull complete 
Digest: sha256:b5cfd4befc119a590ca1a81d6bb0fa1fb19f1fbebd0397f25fae164abe1e8a6a
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest

[root@client ~]# docker images
REPOSITORY   TAG       IMAGE ID       CREATED      SIZE
busybox      latest    ffe9d497c324   7 days ago   1.24MB

[root@client ~]# docker tag busybox:latest  docker.example.com/library/busybox:v0.1
//tag  改名
[root@client ~]# docker images
REPOSITORY                           TAG       IMAGE ID       CREATED      SIZE
docker.example.com/library/busybox   v0.1      ffe9d497c324   7 days ago   1.24MB
busybox                              latest    ffe9d497c324   7 days ago   1.24MB


//把生成的镜像上传到自己的仓库中，然后去网页上查看是否上传成功
[root@client ~]# docker push docker.example.com/library/busybox:v0.1
The push refers to repository [docker.example.com/library/busybox]
64cac9eaf0da: Pushed 
v0.1: digest: sha256:50e44504ea4f19f141118a8a8868e6c5bb9856efa33f2183f5ccea7ac62aacc9 size: 527

10、从自己搭建的仓库中拉取镜像

[root@client ~]# docker rmi docker.example.com/library/busybox:v0.1 
Untagged: docker.example.com/library/busybox:v0.1
Untagged: docker.example.com/library/busybox@sha256:50e44504ea4f19f141118a8a8868e6c5bb9856efa33f2183f5ccea7ac62aacc9

[root@client ~]# docker images
REPOSITORY   TAG       IMAGE ID       CREATED      SIZE
busybox      latest    ffe9d497c324   7 days ago   1.24MB

//从自己搭建的仓库中拉取镜像
[root@client ~]# docker pull docker.example.com/library/busybox:v0.1
v0.1: Pulling from library/busybox
Digest: sha256:50e44504ea4f19f141118a8a8868e6c5bb9856efa33f2183f5ccea7ac62aacc9
Status: Downloaded newer image for docker.example.com/library/busybox:v0.1
docker.example.com/library/busybox:v0.1

[root@client ~]# docker images
REPOSITORY                           TAG       IMAGE ID       CREATED      SIZE
docker.example.com/library/busybox   v0.1      ffe9d497c324   7 days ago   1.24MB
busybox                              latest    ffe9d497c324   7 days ago   1.24MB

11、harbor的关闭和开启

//停止，停止过后所有支持web界面的容器就会关闭
[root@docker harbor]# docker-compose stop
Stopping harbor-jobservice ... done
Stopping nginx             ... done
Stopping harbor-core       ... done
Stopping harbor-portal     ... done
Stopping harbor-db         ... done
Stopping registryctl       ... done
Stopping registry          ... done
Stopping redis             ... done
Stopping harbor-log        ... done

//开启
[root@docker harbor]# docker-compose start
Starting log         ... done
Starting registry    ... done
Starting registryctl ... done
Starting postgresql  ... done
Starting portal      ... done
Starting redis       ... done
Starting core        ... done
Starting jobservice  ... done
Starting proxy       ... done