Harbor高可用部署使用

Harbor的高可用方案大致可以分为下面两种，一种依赖共享存储来保存镜像数据，另一种基于不同Harbor服务器间的镜像复制实现。如下图所示：

主机规划

本文基于第二种方案实现Harbor的高可用部署，使用主机如下：

192.168.122.10 harbor-server1.linux.io 2C4G Ubuntu20.04	harbor服务器1
192.168.122.11 harbor-server2.linux.io 2C4G Ubuntu20.04  harbor服务器2
192.168.122.12 haproxy-server1.linux.io 2C2G Ubuntu20.04  haproxy+keepalived服务器1
192.168.122.13 haproxy-server2.linux.io 2C2G Ubuntu20.04  haproxy+keepalived服务器2
192.168.122.188(VIP) harbor.linux.io

Harbor部署

先配置域名解析hosts文件，步骤略

在两个harbor-server节点部署Docker，步骤略

然后在两个harbor节点上部署harbor，步骤如下：

生成证书

两个harbor使用同一套证书

mkdir -p /apps/harbor/cert && cd /apps/harbor/cert
#生成根证书
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=linux.io" \
 -key ca.key \
 -out ca.crt

#生成harbor使用的证书
openssl genrsa -out harbor-server.key 4096
openssl req -sha512 -new \
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=linux.io" \
    -key harbor-server.key \
    -out harbor-server.csr
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=linux.io
DNS.2=harbor-server.linux.io
DNS.3=harbor-server1.linux.io
DNS.4=harbor-server2.linux.io
EOF
openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in harbor-server.csr \
    -out harbor-server.crt
rm -f harbor-server.csr	#删除证书请求文件 

然后将证书拷贝到两个harbor-server

cd  /apps/harbor/cert
scp ./harbor-server.*  root@harbor-server1:/apps/harbor/cert
scp ./harbor-server.*  root@harbor-server1:/apps/harbor/cert

下载安装包

wget https://github.com/goharbor/harbor/releases/download/v2.5.3/harbor-offline-installer-v2.5.3.tgz
tar xf installer-v2.5.3.tgz
cd harbor/
cp harbor.yml.tmpl harbor.yml

修改配置

修改两个节点上harbor/harbor.yml，两个节点的配置不同之处只有hostname配置项修改为和各自的hostname一致即可

hostname: harbor-server1.linux.io	#
http:
  port: 80
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /apps/harbor/cert/harbor-server.crt
  private_key: /apps/harbor/cert/harbor-server.key
harbor_admin_password: Passw0rd		#harbor界面的密码

data_volume: /data	#上传到harbor的镜像的保存目录

启动harbor

./install.sh --with-trivy --with-chartmuseum

配置镜像同步

先在两个harbor-server上创建相同的项目base-images

在harbor-server1上配置指向harbor-server2的复制规则

在harbor-server2上配置指向harbor-server1的复制规则，步骤和上面一样，只需要把目标地址改为harbor-server1的地址192.168.122.10即可

部署haproxy+keepalived

安装haproxy+keepalived

apt -y install haproxy keepalived

修改haproxy配置文件

修改/etc/haproxy/haproxy.cfg，添加下面内容，两个节点一样

listen harbor_80
       bind 192.168.122.188:80
       option tcplog
       mode tcp
       balance source
       server harbor1 192.168.122.10:80 check inter 2000 fall 3 rise 5
       server harbor2 192.168.122.11:80 check inter 2000 fall 3 rise 5
listen harbor_443
       bind 192.168.122.188:443
       option tcplog
       mode tcp
       balance source
       server harbor1 192.168.122.10:443 check inter 2000 fall 3 rise 5
       server harbor2 192.168.122.11:443 check inter 2000 fall 3 rise 5

修改keepalived配置文件

修改/etc/keepalived/keepalived.conf，haproxy-server1内容如下

global_defs {
   notification_email {
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id harbor-lvs1
}

vrrp_instance VI_1 {
    state MASTER
    interface enp1s0
    garp_master_delay 10
    smtp_alert
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.122.188

    }
}

haproxy-server2内容如下：

global_defs {
   notification_email {
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id harbor-lvs2
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp1s0
    garp_master_delay 10
    smtp_alert
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.122.188

    }
}

启动服务

systemctl start haproxy && systemctl enable haproxy
systemctl start keepalived && systemctl enable keepalived

测试

配置docker使用https访问harbor，将harbor的证书分发至docker

mkdir /etc/docker/certs.d/harbor-server.linux.io/
cp /apps/harbor/cert/harbor-server.crt /etc/docker/certs.d/harbor-server.linux.io/harbor-server.cert
cp /apps/harbor/cert/harbor-server.key /etc/docker/certs.d/harbor-server.linux.io/harbor-server.key
cp /apps/harbor/cert/ca.crt /etc/docker/certs.d/harbor-server.linux.io/ca.crt 

上传镜像

docker login harbor.server.linux.i
docker pull rockylinx:9.0
docker image tag rockylinux:9.0 harbor-server.linux.io/base-images/rockylinux:9.0
docker push harbor-server.linux.io/base-images/rockylinux:9.0

查看镜像是否同步

在两个harbor-server节点上都有这个镜像，说明镜像同步成功

关闭harbor-server1的harbor服务

再下载镜像测试

下载成功表示高可用没问题