Harbor高可用部署使用
一 简介
Harbor的高可用方案大致可以分为下面两种:
【1】依赖共享存储来保存镜像数据
【2】基于不同Harbor服务器间的镜像复制实现。
图示表示:
【1】基于共享存储的Harbor高可用
【2】基于镜像同步的Harbor高可用
二 主机规划
主机名(域名) | IP | 角色 | 安装软件 | OS |
---|---|---|---|---|
harbor-server1.linux.io | 11.0.1.141 | harbor服务器 | docker、docker-compose、harbor | Rocky 9.4 |
harbor-server2.linux.io | 11.0.1.142 | harbor服务器 | docker、docker-compose、harbor | Rocky 9.4 |
haproxy-server1.linux.io | 11.0.1.140 | 负载均衡服务器haproxy、keepalived | haproxy、keepalived | Rocky 9.4 |
haproxy-server2.linux.io | 11.0.1.143 | 负载均衡服务器 | haproxy、keepalived | Rocky 9.4 |
harbor-server.linux.io | 11.0.1.150 | VIP |
三 Harbor服务器部署
3.1 系统初始化
【1】设置主机名并配置hosts解析
~]# hostnamectl set-hostname harbor-server1.linux.io
~]# hostnamectl set-hostname harbor-server2.linux.io
~]# vim /etc/hosts
11.0.1.141 harbor-server1.linux.io
11.0.1.142 harbor-server2.linux.io
11.0.1.150 harbor-server.linux.io
11.0.1.140 haproxy-server1.linux.io
11.0.1.143 haproxy-server2.linux.io
【2】关闭防火墙
~]# systemctl disable --now firewalld
【3】 关闭selinux
~]# setenforce 0
~]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
【4】配置时钟同步
略
3.2 生成证书
注意:两个harbor使用同一套证书!!!
# 创建证书目录
~] mkdir -p /data/harbor/cert && cd /data/harbor/cert
# 生成ca证书私钥ca.key
~] openssl genrsa -out ca.key 4096
# 生成ca证书ca.crt
~] openssl req -x509 -new -nodes -sha512 -days 3650 \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=linux.io" \
-key ca.key \
-out ca.crt
# 生成服务器证书
## 生成服务器证书私钥
~] openssl genrsa -out harbor-server.key 4096
## 生成服务器证书
~] openssl req -sha512 -new \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=linux.io" \
-key harbor-server.key \
-out harbor-server.csr
## 生成 x509 v3 扩展名文件
~] cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=linux.io
DNS.2=harbor-server.linux.io
DNS.3=harbor-server1.linux.io
DNS.4=harbor-server2.linux.io
EOF
## 使用v3.ext生成服务器证书
~] openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in harbor-server.csr \
-out harbor-server.crt
#删除证书请求文件
~] rm -f harbor-server.csr
然后将证书拷贝到两个harbor-server
~]# cd /data/harbor/cert/
~]# scp ./harbor-server.* root@11.0.1.141:/data/harbor/cert
~]# scp ./harbor-server.* root@11.0.1.142:/data/harbor/cert
3.3 docker部署
略
可以参考 四步骤的 脚本
3.4 docker-compuse 部署
略
可以参考 六 的 脚本
3.5 harbor部署
【1】下载安装包
wget https://github.com/goharbor/harbor/releases/download/v2.5.3/harbor-offline-installer-v2.5.3.tgz
tar xf installer-v2.5.3.tgz -C /usr/local/
cd /usr/local/harbor/
cp harbor.yml.tmpl harbor.yml
【2】修改配置
修改两个节点上harbor/harbor.yml,两个节点的配置不同之处只有hostname配置项修改为和各自的hostname一致即可
如果harbor和haproxy部署在同一台服务器中,默认的80端口也需要改,否则会造成端口冲突
hostname: harbor-server1.linux.io #两个节点的差别只有这个配置项不同
http:
port: 80
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
certificate: /data/harbor/cert/harbor-server.crt
private_key: /data/harbor/cert/harbor-server.key
harbor_admin_password: Harbor12345 #harbor界面的密码,可以不改,默认的就是Harbor12345,部署好可以去界面改的
data_volume: /data #上传到harbor的镜像的保存目录
【3】启动harbor
./install.sh --with-trivy
3.6 配置镜像同步
【1】先在两个harbor-server上创建相同的项目 test_item(项目名按实际情况创建即可)
server1:
server2:
【2】在harbor-server1上配置指向harbor-server2的复制规则
添加完以后要先进行测试,测试成功后点击确定
这里会出现连接其他仓库报错(测试一直失败)
原因:
1 禁止ping导致
2 也可能就是解析不到域名(但是我直接用了ip还是不行,也确实是修改了容器的hosts搞定的)
解决办法:
1 查看barbor镜像中,/etc/hosts下的文件
~] docker ps |grep harbor-core
~] docker exec -it 98e654f768f3 /bin/bash
cat /etc/hosts #查看是否有hosts解析,连接其他仓库报错一般都是容器里面没有hosts解析
2 将本地的/etc/hosts挂载到容器中(两台都要)
# 修改docker-compose.yml挂载hosts文件
第一处:jobservice
~] vim /usr/local/harbor/docker-compose.yml
jobservice:
image: goharbor/harbor-jobservice:v2.10.0
container_name: harbor-jobservice
env_file:
- ./common/config/jobservice/env
restart: always
cap_drop:
- ALL
cap_add:
- CHOWN
- SETGID
- SETUID
volumes:
- /data/job_logs:/var/log/jobs:z
- /etc/hosts:/etc/hosts:z #添加这行
- type: bind
...........
第二处:core
core:
image: goharbor/harbor-core:v2.10.0
container_name: harbor-core
env_file:
- ./common/config/core/env
restart: always
cap_drop:
- ALL
cap_add:
- SETGID
- SETUID
volumes:
- /data/ca_download/:/etc/core/ca/:z
- /etc/hosts:/etc/hosts:z #添加这行
- /data/:/data/:z
- ./common/config/core/certificates/:/etc/core/certificates/:z
..........
3 重新加载harbor(两台都要)
cd /usr/local/harbor
docker-compose down
docker-compose up -d
docker-compose ps
# 如果执行命令报错,加上-f参数 指定一下配置文件路径或者进入harbor目录 cd /usr/local/harbor 执行
# 通常是找不到docker-compose.yml文件在哪
# ~]# docker-compose ps
no configuration file provided: not found
~ docker-compose -f /usr/local/harbor/docker-compose.yml up -d
4 再次在页面上进行测试,成功后点击确定
然后创建复制策略
创建成功后的图示
到此在harbor-server1上的配置就完成了
【3】在harbor-server2上配置指向harbor-server1的复制规则
步骤和上面一样,只需要把目标地址改为harbor-server1的地址192.168.122.10即可
简单来说,就是在harbor-server2上把 【2】步骤的操作再来一遍,目标地址换成harbor-server1的地址11.0.1.141即可
harbor-server2上创建好的图示
四 部署Haproxy+Keepalived
4.1 系统环境初始化
主机名修改
11.0.1.140 --- hostnamectl set-hostname haproxy-server1.linux.io
11.0.1.143 --- hostnamectl set-hostname haproxy-server2.linux.io
hosts 解析
~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
11.0.1.141 harbor-server1.linux.io
11.0.1.142 harbor-server2.linux.io
11.0.1.150 harbor-server.linux.io
11.0.1.140 haproxy-server1.linux.io
11.0.1.143 haproxy-server2.linux.io
关闭selinux、防火墙
# 首先临时关闭SELinux
setenforce 0
# 修改配置文件,永久关闭SELinux
vi /etc/sysconfig/selinux
# 设置
SELINUX=disabled
# 关闭防火墙
systemctl disable --now firewalld
添加IP转发
~] echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
~] echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
~] sysctl -p
# 1. net.ipv4.ip_forward = 1 开启允许数据包转发
# 2. net.ipv4.ip_nonlocal_bind = 1 开启允许绑定非本机的IP
5.2 安装keepalived和haproxy
这里直接yum安装,如果需要编译安装,请参考 五 的脚本
# keepalived
~] yum -y install keepalived
# haproxy
yum -y install haproxy
5.3 修改haproxy配置文件
修改/etc/haproxy/haproxy.cfg,或者直接备份原来的,直接将下面的内容重新添加,两个节点配置一样
~] mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
~] cat > /etc/haproxy/haproxy.cfg <<-EOF
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 20000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 20000
listen admin_stats # frontend and backend 监控
mode http
bind *:8118
stats enable
log global
stats uri /haproxy-status
stats auth admin:123456
stats hide-version
stats refresh 30s
listen harbor_80
bind 11.0.1.150:80
option tcplog
mode tcp
balance source
server harbor01 11.0.1.141:80 check inter 2000 fall 3 rise 5
server harboe02 11.0.1.142:80 check inter 2000 fall 3 rise 5
listen harbor_443
bind 11.0.1.150:443
option tcplog
mode tcp
balance source
server harbor01 11.0.1.141:443 check inter 2000 fall 3 rise 5
server harboe02 11.0.1.142:443 check inter 2000 fall 3 rise 5
EOF
5.4 修改keepalived配置文件
修改/etc/keepalived/keepalived.conf,或者直接备份原来的,直接将下面的内容重新添加(我这里设置的是非抢占模式的),两个节点配置文件不一样
haproxy-server1内容如下
~] mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
~] cat <<EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id harbor-lvs1
}
vrrp_script check_haproxy {
script "/etc/keepalived/scripts/check_haproxy.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface ens160
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
11.0.1.150/24
}
track_script {
check_haproxy
}
}
EOF
haproxy-server2内容如下:
~] mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
~] cat <<EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id harbor-lvs2
}
vrrp_script check_haproxy {
script "/etc/keepalived/scripts/check_haproxy.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface ens160
virtual_router_id 51
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
11.0.1.150/24
}
track_script {
check_haproxy
}
}
EOF
编写检测脚本,两个节点都一样
~]# cat /etc/keepalived/scripts/check_haproxy.sh
#!/bin/bash
#LOGFILE="/var/log/keepalived-haproxy-state.log" #这个日志建议不加
#date >> $LOGFILE
A=`ps -C haproxy --no-header | wc -l`
if [ $A -eq 0 ];then
#echo "fail: check_haproxy status" >> $LOGFILE
#echo "Try to start haproxy service" >> $LOGFILE
systemctl start haproxy #重启haproxy服务
sleep 2
if [ `ps -C haproxy --no-header | wc -l` -eq 0 ];then #如果启动失败,就进行VIP 转移
#echo "Can not start haproxy service, so will stop keepalived service" >> $LOGFILE
killall keepalived
fi
#else
#echo "success: check_haproxy status" >> $LOGFILE
fi
# 给执行权限
chmod +x /etc/keepalived/check_haproxy.sh
5.5 启动服务
systemctl start haproxy && systemctl enable haproxy
systemctl start keepalived && systemctl enable keepalived
5.6 登录验证
# 通过http://ip:port/haproxy-status访问,查看HAProxy统计状态
http://11.0.1.150:8118/haproxy-status
5.7 配置 haproxy日志输出
$ vim /etc/rsyslog.conf
# 在文件中,取消注释以下行:
# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
# 通过添加以下行来禁用私人身份验证消息:
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local2.none /var/log/messages
local2.* /var/log/haproxy.log
# 验证配置文件是否正常
$ rsyslogd -N1
# 重新启动Rsyslog,haproxy
$ systemctl restart haproxy
$ systemctl restart rsyslog
5.8 补充说明
# 如果需要修改harbor.yml文件
1) 需要先停止Harbor服务
docker-compose down
2)修改yml文件
3)进入harbor目录,执行 ./prepare
cd /usr/local/harbor
./prepare
4) 启动harbor
docker-compose up -d
# 未修改配置文件,怎么重启harbor
docker-compose start/stop/restart
五 测试Harbor高可用
5.1 docker客户端登录harbor仓库
配置docker使用https访问harbor,将harbor的证书分发至docker(如果配置了/etc/docker/daemon.json 这步就不用做了 )
mkdir /etc/docker/certs.d/harbor-server.linux.io/
scp /apps/harbor/cert/harbor-server.crt /etc/docker/certs.d/harbor-server.linux.io/harbor-server.cert
scp /apps/harbor/cert/harbor-server.key /etc/docker/certs.d/harbor-server.linux.io/harbor-server.key
scp /apps/harbor/cert/ca.crt /etc/docker/certs.d/harbor-server.linux.io/ca.crt
配置docker客户端 /etc/docker/daemon.json 文件
~] vim /etc/docker/daemon.json
{
"insecure-registries": ["harbor-server.linux.io"] # 注意 域名别写错了,要写vip的域名
}
如果两步都不做那么就会报错
报错详细:
Error response from daemon: Get "https://harbor-server.linux.io/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
登录验证
[root@haproxy-server2 weihu]# docker login harbor-server.linux.io
Username: admin #harbor账号
Password: #harbor密码
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded #登录成功
5.2 测试habror仓库镜像同步
上传一个镜像进行测试
# 这里用nginx镜像作为测试
~] docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx 1.24.2 a72860cb95fd 4 weeks ago 188MB
# 打标签,仓库名称要看好,别打错了,是vip的域名仓库
~] docker tag nginx:1.24.2 harbor-server.linux.io/test_item/nginx:1.24.2
# 推镜像到harbor仓库
~] docker push harbor-server.linux.io/test_item/nginx:1.24.2
查看镜像是否同步
11.0.1.141 – harbor-server1 :
11.0.1.142 – harbor-server2 :
在两个harbor-server节点上都有这个镜像,说明镜像同步成功啦
5.3 测试harbor仓库高可用
关闭harbor-server1的harbor服务
[root@harbor-server1 ~]# cd /usr/local/harbor/
[root@harbor-server1 harbor]# docker-compose stop
[root@harbor-server1 harbor]# ps -ef |grep harbor
root 34767 31822 0 20:45 pts/0 00:00:00 grep --color=auto harbor
再下载镜像测试
[root@haproxy-server1 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
[root@haproxy-server1 ~]# docker pull harbor-server.linux.io/test_item/nginx:1.24.2
1.24.2: Pulling from test_item/nginx
efc2b5ad9eec: Pull complete
8fe9a55eb80f: Pull complete
045037a63be8: Pull complete
7111b42b4bfa: Pull complete
3dfc528a4df9: Pull complete
9e891cdb453b: Pull complete
0f11e17345c5: Pull complete
Digest: sha256:4ac65f23061de2faef157760fa2125c954b5b064bc25e10655e90bd92bc3b354
Status: Downloaded newer image for harbor-server.linux.io/test_item/nginx:1.24.2
harbor-server.linux.io/test_item/nginx:1.24.2
[root@haproxy-server1 ~]#
下载成功表示高可用没问题
到此,Harbor仓库高可用部署已经完成并且测试成功
六 一键脚本实现部署
5.1 docker+docker compuse+Harbor
脚本说明:
脚本里面包含docker部署 和 docker-compose部署
需要提前将安装包放至 /home/weihu/src ,脚本中有会检测安装包,如果没有脚本会自动退出执行
如果需要放至其他目录,可以修改脚本里面的 “安装包存放位置” 里面的 SRC_DIR 参数
harbor配置文件说明:
如果harbor和haproxy部署在一台服务器中,那么需要修改harbor的80端口,会和haproxy端口冲突,造成harbor启动失败,如果分开部署,那么harbor可以使用默认的80端口即可
harbor.yaml文件只需要修改 hostname 配置项、certificate和private_key配置项即可(单节点)
密码可改可不改,默认密码为"Harbor12345"
两个节点的配置不同之处只有hostname配置项,只需要修改为和各自的hostname即可,certificate和private_key配置一样的路径即可
~]# cat /home/weihu/harbor_install.sh
############### 颜色 #################
COLOR_RED="echo -e \\033[01;31m" #
COLOR_GREEN="echo -e \\033[01;32m" #
COLOR_BLUE="echo -e \\033[01;36m" #
END='\033[0m' #
######################################
###### 安装包存放位置 ######
SRC_DIR=/home/weihu/src #
############################
################################################### 下载地址 ##########################################################################
#
# Docker 下载地址: "https://mirrors.aliyun.com/docker-ce/linux/static/stable/x86_64/" #
DOCKER_URL="https://mirrors.aliyun.com/docker-ce/linux/static/stable/x86_64/" #
DOCKER_FILE="docker-24.0.7.tgz" #
#
# Docker Compose下载地址:"https://github.com/docker/compose/releases/download/v2.23.3/docker-compose-linux-x86_64",请提前下载。 #
DOCKER_COMPOSE_URL="https://github.com/docker/compose/releases/download" #
DOCKER_COMPOSE_FILE="docker-compose-linux-x86_64" #
#
# Harbor下载地址:"https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz",请提前下载。 #
HARBOR_URL="https://github.com/goharbor/harbor/releases/download" #
HARBOR_FILE="harbor-offline-installer-v" #
HARBOR_VERSION="2.10.0" #
TAR=".tgz" #
HARBOR_INSTALL_DIR="/usr/local" #
HARBOR_DOMAIN="harbor.example.com" #
HARBOR_ADMIN_PASSWORD="Redhat123456" #
HARBOR_PORT="8888" #
#SSL/TLS证书文件的本机位置 #
CERTIFICATE="/data/harbor/cert/harbor-server.crt" #
#私钥文件的本机位置 #
PRIVATE_KEY="/data/harbor/cert/harbor-server.key" #
#######################################################################################################################################
######################################## 变量定义 ##########################################
#CPU个数 #
CPUS=`lscpu |awk '/^CPU: /{print $2}'` #
#网卡名称 #
NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'` #
#IP地址 #
IP=`ip addr show ${NET_NAME}| awk -F" +|/" '/global/{print $3}'|head -n1` #
#IP=`ip addr show ${NET_NAME} | sed -n 's/.* \([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)\/.*/\1/p` #
#主机名 #
HOSTNAME=`hostname` #
#系统类型 #
OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' /etc/os-release` #
#系统版本 #
OS_RELEASE_VERSION=`sed -rn '/^VERSION_ID=/s@.*="?([0-9]+)\.?.*"?@\1@p' /etc/os-release` #
############################################################################################
check_file(){
cd ${SRC_DIR}
if [ ! -e ${DOCKER_FILE} ];then
${COLOR_RED}"缺少${DOCKER_FILE}文件,如果是离线包,请把文件放到${SRC_DIR}目录下"${END}
${COLOR_RED}"下载地址为:${DOCKER_URL}"${END}
exit
#${COLOR_BLUE}'开始下载DOCKER二进制源码包'${END}
#wget ${DOCKER_URL}${DOCKER_FILE} || { ${COLOR_RED}"DOCKER二进制安装包下载失败"${END}; exit; }
elif [ ! -e ${DOCKER_COMPOSE_FILE} ];then
${COLOR_RED}"缺少${DOCKER_COMPOSE_FILE}文件,果是离线包,请把文件放到${SRC_DIR}目录下"${END}
${COLOR_RED}"下载地址为:${DOCKER_COMPOSE_URL}"${END}
exit
elif [ ! -e ${HARBOR_FILE}${HARBOR_VERSION}${TAR} ];then
${COLOR_RED}"缺少${HARBOR_FILE}${HARBOR_VERSION}${TAR}文件,请把文件放到${SRC_DIR}目录下"${END}
${COLOR_RED}"下载地址为:${HARBOR_URL}"${END}
exit
else
${COLOR_GREEN}"相关文件已准备好"${END}
fi
}
docker_install(){
${COLOR_BLUE}"开始安装Docker,请稍等..."${END}
sleep 3
read -p "请输入docker安装目录,默认/home/docker,如使用默认路径可直接回车,自定义请按默认路径格式输入: " dockerpath
if [ -z "$dockerpath" ];then
dockerpath=/home/docker
else
mkdir -p $dockerpath
fi
dockerstatus=$(systemctl status docker 2> /dev/null |grep Active |grep running |wc -l)
if [ $dockerstatus -eq 1 ];then
systemctl stop docker
fi
cd ${SRC_DIR} && \
tar -xf $DOCKER_FILE && \
chmod 755 docker/* && \
cp docker/* /usr/bin/ && \
groupadd docker
#增加docker服务
cat > /usr/lib/systemd/system/docker.service <<"EOF"
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H unix://var/run/docker.sock
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
if [ -d "/etc/docker/" ];then rm -rf /etc/docker/;fi && \
mkdir -p /etc/docker/ && \
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://docker.lpanel.live"],
"insecure-registries": ["${HARBOR_DOMAIN}"],
"exec-opts": ["native.cgroupdriver=systemd"],
"data-root": "$dockerpath",
"max-concurrent-downloads": 10,
"max-concurrent-uploads": 5,
"log-opts": {
"max-size": "300m",
"max-file": "2"
}
}
EOF
systemctl daemon-reload
systemctl enable --now docker > /dev/null 2>&1
systemctl is-active docker &> /dev/null && ${COLOR_GREEN}"Docker 服务启动成功"${END} || { ${COLOR_RED}"Docker 启动失败"${END};exit; }
docker version && ${COLOR_GREEN}"Docker 安装成功"${END} || ${COLOR_RED}"Docker 安装失败"${END}
}
docker_compose_install(){
${COLOR_BLUE}"开始安装Docker Compose,请稍等..."${END}
mv ${SRC_DIR}/${DOCKER_COMPOSE_FILE} /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose
docker-compose --version && ${COLOR_GREEN}"Docker Compose 安装完成"${END} || ${COLOR_RED}"Docker compose 安装失败"${END}
}
install_harbor(){
${COLOR_BLUE}"开始安装Harbor,请稍等..."${END}
[ -d ${HARBOR_INSTALL_DIR} ] || mkdir ${HARBOR_INSTALL_DIR}
tar xf ${SRC_DIR}/${HARBOR_FILE}${HARBOR_VERSION}${TAR} -C ${HARBOR_INSTALL_DIR}/
/usr/bin/cp -f ${HARBOR_INSTALL_DIR}/harbor/harbor.yml.tmpl ${HARBOR_INSTALL_DIR}/harbor/harbor.yml
sed -ri \
-e 's/^(hostname:) .*/\1 '${HOSTNAME}'/' \
#-e 's/^(https:)/#\1/' \
#-e 's/ (port: 443)/# \1/' \
#-e 's@ (certificate: .*)@# \1@' \
#-e 's@ (private_key: .*)@# \1@' \
-e 's@certificate:.*@certificate: '${CERTIFICATE}'@' \
-e 's@private_key:.*@private_key: '${PRIVATE_KEY}'@' \
#-e 's/^(harbor_admin_password:) .*/\1 '${HARBOR_ADMIN_PASSWORD}'/' \
#-e '/port: 80/s#80#'${HARBOR_PORT}'#' \
"${HARBOR_INSTALL_DIR}/harbor/harbor.yml"
if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> /dev/null;then
rpm -q python3 &> /dev/null || { ${COLOR_BLUE}"安装python3,请稍等..."${END};yum -y install python3 &> /dev/null; }
else
dpkg -s python3 &>/dev/null || { ${COLOR_BLUE}"安装python3,请稍等..."${END};apt -y install python3 &> /dev/null; }
fi
${HARBOR_INSTALL_DIR}/harbor/install.sh --with-trivy && ${COLOR_GREEN}"Harbor 安装完成"${END} || ${COLOR_RED}"Harbor 安装失败"${END}
cat > /lib/systemd/system/harbor.service <<-EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor
[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/docker-compose -f ${HARBOR_INSTALL_DIR}/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f ${HARBOR_INSTALL_DIR}/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable harbor &>/dev/null && ${COLOR_GREEN}"Harbor已配置为开机自动启动"${END}
}
main(){
check_file
[ -f /usr/bin/docker ] && ${COLOR_BLUE}"Docker已安装"${END} || docker_install
[ -f /usr/bin/docker-compose ] && ${COLOR_BLUE}"Docker Compose已安装"${END} || docker_compose_install
#systemctl is-active harbor &> /dev/null && ${COLOR_BLUE}"Harbor已安装"${END} || install_harbor
}
main
在harbor服务器中执行安装脚本
~]# sh harbor_install.sh
5.2 Haproxy
脚本说明:
需要提前将安装包上传至 /home/weihu/src 目录
~ cat haproxy_install.sh
#!/bin/bash
#源码包存放位置
SRC_DIR=/home/weihu/src
#颜色
COLOR_RED="echo -e \\033[01;31m"
COLOR_GREEN="echo -e \\033[01;33m"
COLOR_BLUE="echo -e \\033[01;36m"
END='\033[0m'
#lua下载地址:"https://www.lua.org/ftp/lua-5.4.6.tar.gz",请提前下载。
LUA_FILE=lua-5.4.6.tar.gz
#haproxy下载地址:"https://www.haproxy.org/download/2.8/src/haproxy-2.8.5.tar.gz",请提前下载。
HAPROXY_FILE=haproxy-2.8.5.tar.gz
#安装目录
HAPROXY_INSTALL_DIR=/usr/local/haproxy
STATS_AUTH_USER=admin
STATS_AUTH_PASSWORD=123456
#CPU个数
CPUS=`lscpu |awk '/^CPU: /{print $2}'`
#网卡名称
NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'`
#IP地址
IP=`ip addr show ${NET_NAME}| awk -F" +|/" '/global/{print $3}'`
#IP=`ip addr show ${NET_NAME} | sed -n 's/.* \([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)\/.*/\1/p`
os(){
OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' /etc/os-release`
}
check_file (){
cd ${SRC_DIR}
${COLOR_BLUE}'***** 检查Haproxy相关源码包 *****'${END}
if [ ! -e ${LUA_FILE} ];then
${COLOR_RED}"缺少${LUA_FILE}文件,请把文件放到${SRC_DIR}目录下"${END}
exit
elif [ ! -e ${HAPROXY_FILE} ];then
${COLOR_RED}"缺少${HAPROXY_FILE}文件,请把文件放到${SRC_DIR}目录下"${END}
exit
else
${COLOR_GREEN}"相关文件已准备好"${END}
fi
}
install_haproxy(){
[ -d ${HAPROXY_INSTALL_DIR} ] && { ${COLOR_RED}"Haproxy已存在,安装失败"${END};exit; }
${COLOR_BLUE}"开始安装Haproxy"${END}
${COLOR_BLUE}"开始安装Haproxy依赖包"${END}
if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> /dev/null;then
yum -y install gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel net-tools zlib-devel tcpdump &> /dev/null
else
apt update &> /dev/null;apt -y install gcc make libreadline-dev openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev libsystemd-dev &> /dev/null
fi
if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> /dev/null;then
rpm -q tar &> /dev/null || { ${COLOR_BLUE}"安装tar工具,请稍等..."${END};yum -y install tar &> /dev/null; }
fi
cd ${SRC_DIR}
tar xf ${LUA_FILE}
LUA_DIR=`echo ${LUA_FILE} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'`
cd ${LUA_DIR}
make linux test
cd ${SRC_DIR}
tar xf ${HAPROXY_FILE}
HAPROXY_DIR=`echo ${HAPROXY_FILE} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'`
cd ${HAPROXY_DIR}
make -j ${CPUS} ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_CPU_AFFINITY=1 USE_LUA=1 LUA_INC=${SRC_DIR}/${LUA_DIR}/src/ LUA_LIB=${SRC_DIR}/${LUA_DIR}/src/
make install PREFIX=${HAPROXY_INSTALL_DIR}
[ $? -eq 0 ] && $COLOR_GREEN"Haproxy编译安装成功"$END || { $COLOR_RED"Haproxy编译安装失败,退出!"$END;exit; }
[ -L /usr/sbin/haproxy ] || ln -s ${HAPROXY_INSTALL_DIR}/sbin/haproxy /usr/sbin/ &> /dev/null
cat > /lib/systemd/system/haproxy.service <<-EOF
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target
[Service]
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
[Install]
WantedBy=multi-user.target
EOF
[ -d /etc/haproxy ] || mkdir /etc/haproxy &> /dev/null
cat > /etc/haproxy/haproxy.cfg <<-EOF
global
maxconn 100000
chroot ${HAPROXY_INSTALL_DIR}
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
user haproxy
group haproxy
daemon
pidfile /var/lib/haproxy/haproxy.pid
log 127.0.0.1 local2 info
defaults
option http-keep-alive
option forwardfor
maxconn 100000
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000ms
listen stats
mode http
bind 0.0.0.0:9999
stats enable
log global
stats refresh 30s
stats uri /haproxy-status
stats auth ${STATS_AUTH_USER}:${STATS_AUTH_PASSWORD}
listen harbor_80
bind ${IP}:80
option tcplog
mode tcp
balance source
server harbor01 11.0.1.141:80 check inter 2000 fall 3 rise 5
server harboe02 11.0.1.142:80 check inter 2000 fall 3 rise 5
listen harbor_443
bind ${IP}:443
option tcplog
mode tcp
balance source
server harbor01 11.0.1.141:443 check inter 2000 fall 3 rise 5
server harboe02 11.0.1.142:443 check inter 2000 fall 3 rise 5
EOF
cat >> /etc/sysctl.conf <<-EOF
# 意思是启动haproxy的时候,允许忽视VIP的存在
net.ipv4.ip_nonlocal_bind = 1
# 确认是否开启内核的转发功能
net.ipv4.ip_forward = 1
EOF
sysctl -p &> /dev/null
[ -d /var/lib/haproxy/ ] || mkdir -p /var/lib/haproxy/ &> /dev/null
echo "PATH=${HAPROXY_INSTALL_DIR}/sbin:${PATH}" > /etc/profile.d/haproxy.sh
useradd -r -s /sbin/nologin -d /var/lib/haproxy haproxy
systemctl daemon-reload
systemctl enable --now haproxy &> /dev/null
systemctl is-active haproxy &> /dev/null && ${COLOR_GREEN}"Haproxy 服务启动成功!"${END} || { ${COLOR_RED}"Haproxy 启动失败,退出!"${END} ; exit; }
${COLOR_GREEN}"Haproxy安装完成"${END}
}
main(){
os
check_file
install_haproxy
}
main
执行脚本
# 分别在harbor01 和harbor02 上执行
[root@haproxy01 weihu]# sh haproxy_install.sh
[root@haproxy02 weihu]# sh haproxy_install.sh
配置文件详解
HAProxy 的配置文件haproxy.cfg由两大部分组成,分别是global和proxies部分
global:全局配置段
进程及安全配置相关的参数
性能调整相关参数
Debug参数
proxies:代理配置段
defaults:为frontend, backend, listen提供默认配置
frontend:前端,相当于nginx中的server {}
backend:后端,相当于nginx中的upstream {}
listen:同时拥有前端和后端配置,配置简单,生产推荐使用
[root@rocky9-2 ~]# cat /etc/haproxy/haproxy.cfg
global # global 配置参数
maxconn 100000 # 每个haproxy进程的最大并发连接数
chroot /apps/haproxy # 锁定运行目录
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin #socket文件,并可以通过此文件管理
user haproxy # user, group是运行haproxy的用户身份
group haproxy
daemon # 以守护进程运行
pidfile /var/lib/haproxy/haproxy.pid # 指定pid文件路径
log 127.0.0.1 local2 info # 定义全局的syslog服务器;日志服务器需要开启UDP协议,最多可以定义两个
# proxies:代理配置段
defaults # Proxies配置-defaults
option http-keep-alive # 开启与客户端的会话保持
option forwardfor # 透传客户端真实IP至后端web服务器
maxconn 100000 # 每个haproxy进程的最大并发连接数
mode http # 设置默认工作类型
timeout connect 300000ms # 客户端请求从haproxy到后端server最长连接等待时间(TCP连接之前),默认单位ms
timeout client 300000ms # 设置haproxy与客户端的最长非活动时间,默认单位ms,建议和timeout server相同
timeout server 300000ms # 客户端请求从haproxy到后端服务端的请求处理超时时长(TCP连接之后),默认单位ms,如果超时,会出现502错误,此值建议设置较大些,防止出现502错误
listen stats # 状态页配置项
mode http # 设置默认工作类型
bind 0.0.0.0:9999 # 指定HAProxy的监听地址,可以是IPV4或IPV6,可以同时监听多个IP或端口,可同时用于listen字段中
stats enable # 基于默认的参数启用stats page
log global # 开启当前状态页的日志功能,默认不记录日志
stats uri /haproxy-status # 自定义stats page uri
stats auth haadmin:123456 # 认证时的账号和密码,可定义多个用户,每行指定一个用户.默认:no authentication
listen web_port # Proxies配置-listen
bind 172.31.0.19:80 # 指定HAProxy的监听地址,可以是IPV4或IPV6,可以同时监听多个IP或端口,可同时用于listen字段中
mode http # 指定负载协议类型
log global # 开启当前web_port的日志功能,默认不记录日志
#balance roundrobin # balance是haproxy的算法,roundrobin为默认调度算法,基于权重的轮询动态调度算法,支持权重的运行时调整,如果不写就是默认的
#balance source # 源地址hash,基于用户源地址hash并将请求转发到后端服务器,后续同一个源地址请求将被转发至同一个后端web服务器。此方式当后端服务器数据量发生变化时,会导致很多用户的请求转发至新的后端服务器,默认为静态方式,但是可以通过hash-type选项进行更改,在harbor高可用的时候要配置balance source
server web1 127.0.0.1:8080 check inter 3000 fall 2 rise 5 # server配置,
# check 对指定real进行健康状态检查,如果不加此设置,默认不开启检查,只有check后面没有其它配置也可以启用检查功能;默认对相应的后端服务器IP和端口,利用TCP连接进行周期性健康性检查,注意必须指定端口才能实现健康性检查;
# inter 健康状态检查间隔时间,默认2000 ms;
# fall 后端服务器从线上转为线下的检查的连续失效次数,默认为3;
# rise 后端服务器从下线恢复上线的检查的连续有效次数,默认为2。
# 注意:如果需要绑定在非本机的IP,需要开启内核参数:net.ipv4.ip_nonlocal_bind=1
[root@rocky9-2 ~]# cat >> /etc/sysctl.conf <<-EOF
net.ipv4.ip_nonlocal_bind = 1
EOF
[root@rocky9-2 ~]# sysctl -p
5.3 Keepalived
脚本说明:
需要提前将安装包上传至 /home/weihu/src 目录
编写haproxy健康检查文件(两台都要写)
[root@ha01 ~]# cat /home/weihu/src/check_haproxy.sh
#!/bin/bash
err=0
for k in $(seq 1 3);do
check_code=$(pgrep haproxy)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
编写keepalived安装脚本
[root@rocky-03 weihu]# cat keepalived_install.sh
#!/bin/bash
SRC_DIR=/home/weihu/src
#颜色
COLOR_RED="echo -e \\033[01;31m"
COLOR_GREEN="echo -e \\033[01;33m"
COLOR_BLUE="echo -e \\033[01;36m"
END='\033[0m'
KEEPALIVED_URL=https://keepalived.org/software/
KEEPALIVED_FILE=keepalived-2.2.8.tar.gz
KEEPALIVED_INSTALL_DIR=/usr/local/keepalived
CPUS=`lscpu |awk '/^CPU\(s\)/{print $2}'`
NET_NAME=`ip a |awk -F"[: ]" '/^2/{print $3}'`
VIP=11.0.1.150
os(){
OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' /etc/os-release`
OS_RELEASE_VERSION=`sed -rn '/^VERSION_ID=/s@.*="?([0-9]+)\.?.*"?@\1@p' /etc/os-release`
}
check_file (){
cd ${SRC_DIR}
if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> /dev/null;then
rpm -q wget &> /dev/null || { ${COLOR_BLUE}"安装wget工具,请稍等..."${END};yum -y install wget &> /dev/null; }
fi
if [ ! -e ${KEEPALIVED_FILE} ];then
${COLOR_RED}"缺少${KEEPALIVED_FILE}文件,如果是离线包,请放到${SRC_DIR}目录下"${END}
${COLOR_BLUE}'开始下载Keepalived源码包'${END}
wget ${KEEPALIVED_URL}${KEEPALIVED_FILE} || { ${COLOR_RED}"Keepalived源码包下载失败"${END}; exit; }
elif [ ! -e check_haproxy.sh ];then
${COLOR_RED}"缺少check_haproxy.sh文件,请把文件放到${SRC_DIR}目录下"${END}
exit
else
${COLOR_GREEN}"相关文件已准备好"${END}
fi
}
install_keepalived(){
${COLOR_BLUE}"开始安装Keepalived,请稍等..."${END}
${COLOR_BLUE}"开始安装Keepalived依赖包,请稍等..."${END}
if [ "$OS_ID" == "CentOS" ] || [ "$OS_ID" == "Rocky" ]; then
if [ "$OS_RELEASE_VERSION" -eq 9 ]; then
yum -y install make gcc ipvsadm autoconf automake openssl-devel libnl3-devel iptables-devel ipset file net-snmp-devel glib2-devel pcre2-devel libnftnl libmnl systemd-devel &> /dev/null
elif [ "$OS_RELEASE_VERSION" -eq 8 ]; then
yum -y install make gcc ipvsadm autoconf automake openssl-devel libnl3-devel iptables-devel ipset-devel file-devel net-snmp-devel glib2-devel pcre2-devel libnftnl-devel libmnl-devel systemd-devel &> /dev/null
elif [ "$OS_RELEASE_VERSION" -eq 7 ]; then
yum -y install make gcc libnfnetlink-devel libnfnetlink ipvsadm libnl libnl-devel libnl3 libnl3-devel lm_sensors-libs net-snmp-agent-libs net-snmp-libs openssh-server openssh-clients openssl openssl-devel automake iproute &> /dev/null
fi
elif [ "$OS_ID" == "Debian" ] || [ "$OS_ID" == "Ubuntu" ]; then
if [ "$OS_RELEASE_VERSION" == "20" ] || [ "$OS_RELEASE_VERSION" == "22" ]; then
apt update &> /dev/null;apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libipset-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev
else
apt update &> /dev/null;apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf iptables-dev libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libipset-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev &> /dev/null
fi
fi
cd ${SRC_DIR}
tar xf ${KEEPALIVED_FILE}
KEEPALIVED_DIR=`echo ${KEEPALIVED_FILE} | sed -nr 's/^(.*[0-9]).*/\1/p'`
cd ${KEEPALIVED_DIR}
./configure --prefix=${KEEPALIVED_INSTALL_DIR} --disable-fwmark
make -j $CPUS && make install
[ $? -eq 0 ] && $COLOR_GREEN"Keepalived编译安装成功"$END;sleep 3 || { $COLOR_RED"Keepalived编译安装失败,退出!"$END;exit; }
[ -d /etc/keepalived ] || mkdir -p /etc/keepalived &> /dev/null
read -p "请输入是主服务断或备用服务端,例如(MASTER或BACKUP): " STATE
read -p "请输入优先级,例如(100或80): " PRIORITY
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check_haoroxy {
script "/etc/keepalived/check_haproxy.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state ${STATE}
nopreempt
interface ${NET_NAME}
virtual_router_id 51
priority ${PRIORITY}
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
${VIP}/24
}
track_script {
check_haproxy
}
}
EOF
cp ./keepalived/keepalived.service /lib/systemd/system/
mv check_haproxy.sh /etc/keepalived/check_haproxy.sh
chmod +x /etc/keepalived/check_haproxy.sh
echo "PATH=${KEEPALIVED_INSTALL_DIR}/sbin:${PATH}" > /etc/profile.d/keepalived.sh
systemctl daemon-reload
systemctl enable --now keepalived &> /dev/null
systemctl is-active keepalived &> /dev/null || { ${COLOR_RED}"Keepalived 启动失败,退出!"${END} ; exit; }
${COLOR_GREEN}"Keepalived安装完成"${END}
}
main(){
os
check_file
install_keepalived
}
main
执行脚本
# 分别在haproxy01和haproxy01执行安装
# 我这里设置的非抢占模式,所以都是 BACKUP
[root@harbor01 weihu]# sh keepalived_install.sh
...
请输入是主服务断或备用服务端,例如(MASTER或BACKUP): BACKUP
请输入优先级,例如(100或80): 100
[root@harbor02 weihu]# sh keepalived_install.sh
...
请输入是主服务断或备用服务端,例如(MASTER或BACKUP): BACKUP
请输入优先级,例如(100或80): 80