Haproxy 实现 harbor 高可用

1 实现 harbor 高可用

高可用实现方式1—基于镜像复制

高可用实现方式2—基于共享存储

Harbor 支持基于策略的 Docker 镜像复制功能，这类似于 MySQL 的主从同步， 其可以实现不同的数据中心、不同的运行环境之间同步镜像，并提供友好的管理界面，大大简化了实际运维中的镜像管理工作，已经有用很多互联网公司使用 harbor 搭建内网 docker 仓库的案例，并且还有实现了双向复制的案列

1.1 另外部署一台 harbor

两台配置基本一致

# 安装 harbor
apt -y install python3-pip
tar zxvf harbor-offline-installer-v2.2.3.tgz -C /usr/local/

# 安装 docker-compose
[root@harbor1 ~]# ls
docker-compose-Linux-x86_64  harbor-offline-installer-v2.2.3.tgz  snap
[root@harbor1 ~]# cp docker-compose-Linux-x86_64 /usr/bin/docker-compose
[root@harbor1 ~]# chmod +x /usr/bin/docker-compose

# 配置 harbor
[root@harbor1 ~]# grep -Ev "#|^$" /usr/local/harbor/harbor.yml
hostname: 172.18.8.214
http:
  port: 80
harbor_admin_password: 123456
database:
  password: root123
  max_idle_conns: 50
  max_open_conns: 1000
data_volume: /data
trivy:
  ignore_unfixed: false
  skip_update: false
  insecure: false
jobservice:
  max_job_workers: 12
notification:
  webhook_job_max_retry: 10
chart:
  absolute_url: disabled
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
_version: 2.2.0
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy
[root@harbor1 ~]#

# 启动 harbor
[root@harbor1 /usr/local/harbor]# pwd
/usr/local/harbor
[root@harbor1 /usr/local/harbor]# sudo ./install.sh

1.2 创建仓库管理

规则方式就是，写对方的 IP+用户名密码，然后点测试连接，确认可 以测试连接通过。

Harbor1：172.18.8.214 配置

Harbor2：172.18.8.215 配置

1.3 创建复制管理

Harbor1：172.18.8.214 配置

Harbor2：172.18.8.215 配置

1.4 测试

在 172.18.8.215 上给镜像打 tag ，然后上传到 Harbor（172.18.8.215）上

[root@harbor2 ~]# ifconfig enp0s9
enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.8.215  netmask 255.255.0.0  broadcast 172.18.255.255
        inet6 fe80::a00:27ff:fe94:60dc  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:94:60:dc  txqueuelen 1000  (Ethernet)
        RX packets 2025175  bytes 266910583 (266.9 MB)
        RX errors 0  dropped 33  overruns 0  frame 0
        TX packets 395778  bytes 536359586 (536.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@harbor2 ~]# docker tag harbor.tech.com/baseimages/jdk-base:v8.291 172.18.8.215/baseimages/jdk-base:v8.291
[root@harbor2 ~]# docker push 172.18.8.215/baseimages/jdk-base:v8.291
The push refers to repository [172.18.8.215/baseimages/jdk-base]
a7a8a6844438: Pushed
bbe2bf36de16: Pushed
279a97a7f0d5: Pushed
a28608ec1480: Pushed
a33878cdae1b: Mounted from baseimages/centos-base
9ccf2bd49c2d: Mounted from baseimages/centos-base
26288f6e6406: Mounted from baseimages/centos-base
174f56854903: Mounted from baseimages/centos-base
v8.291: digest: sha256:9ecb3e51153dd8d515397a71a8716a382e21adfcd24e3a1c9231f671b646dd43 size: 1991
[root@harbor2 ~]#

查看 172.18.8.215 已经上传成功

在另外一台 Harbor（172.18.8.214） 上查看，也已经成功复制过来了

1.5 测试2

在 172.18.8.214 上给镜像打 tag ，然后上传到 Harbor（172.18.8.214）上

[root@harbor1 ~]# ifconfig enp0s9
enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.8.214  netmask 255.255.0.0  broadcast 172.18.255.255
        inet6 fe80::a00:27ff:fe01:9095  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:01:90:95  txqueuelen 1000  (Ethernet)
        RX packets 2130325  bytes 280238232 (280.2 MB)
        RX errors 0  dropped 38  overruns 0  frame 0
        TX packets 42107  bytes 9885544 (9.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@harbor1 ~]# docker images |grep tomcat
harbor.tech.com/baseimages/tomcat-base   v8.5.45    58d6cee935fe   2 days ago     1.07GB
[root@harbor1 ~]# docker tag harbor.tech.com/baseimages/tomcat-base:v8.5.45 172.18.8.214/baseimages/tomcat-base:v8.5.45
[root@harbor1 ~]# docker push 172.18.8.214/baseimages/tomcat-base:v8.5.45
The push refers to repository [172.18.8.214/baseimages/tomcat-base]
f0897c7518e1: Pushed
6b7dd53c42e2: Pushed
fe91d5d84fc4: Pushed
a7a8a6844438: Mounted from baseimages/jdk-base
bbe2bf36de16: Mounted from baseimages/jdk-base
279a97a7f0d5: Mounted from baseimages/jdk-base
a28608ec1480: Mounted from baseimages/jdk-base
a33878cdae1b: Mounted from baseimages/jdk-base
9ccf2bd49c2d: Mounted from baseimages/jdk-base
26288f6e6406: Mounted from baseimages/jdk-base
174f56854903: Mounted from baseimages/jdk-base
v8.5.45: digest: sha256:3acb248fd84eb0913e6f64320505a5cd1525db27d578ed4725873c2b15eeba4b size: 2616
[root@harbor1 ~]

查看 172.18.8.214 已经上传成功

查看另外一台 Harbor（172.18.8.215）也已经复制成功了

2 Haproxy 负载均衡

haproxy 配置

[root@K8s-master1 ~]# docker run -it -d -p 80:80 -p 9999:9999 172.18.8.215/apps/haproxy-harbor:2.3.12
[root@K8s-master1 ~]# docker exec -it 26f77bb57636 /bin/bash
[root@26f77bb57636 /]# cat /etc/haproxy/haproxy.cfg
global
chroot /usr/local/haproxy
#stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
uid 99
gid 99
daemon
nbproc 1
pidfile /usr/local/haproxy/run/haproxy.pid
log 127.0.0.1 local3 info
defaults
option http-keep-alive
option forwardfor
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000ms

listen stats
  mode http
  bind 0.0.0.0:9999
  stats enable
  log global
  stats uri /haproxy-status
  stats auth haadmin:123456

listen harbor-80
  bind 0.0.0.0:80  # 宿主机可以监听IP，如果使用容器的话，必须这么写bind 0.0.0.0:80
  mode tcp
  balance source  # 相当于 Nginx 的源地址 hash
  server 172.18.8.214 172.18.8.214:80 check inter 3s fall 3 rise 5
  server 172.18.8.215 172.18.8.215:80 check inter 3s fall 3 rise 5
[root@26f77bb57636 /]#

注：Habor + Haproxy 只能实现在线的实时高可用，如果其中一台 harbor 宕机之后，客户端 push 的镜像，在其（宕机的机器）启动之后，镜像并不会同步过来