在Centos7搭建高可用Harbor集群
一、主机清单
IP地址 | 主机名 | 描述 |
---|---|---|
192.168.0.211 | / | 负载均衡VIP,5000端口 |
192.168.0.212 | harbor01 | Harbor实例1,5000端口 |
192.168.0.213 | harbor02 | Harbor实例2,5000端口 |
192.168.0.214 | Docker Server | 打包及上传docker镜像 |
-
harbor01与harbor02需要安装docker 、docker-compose、harbor、keepalived
-
Docker Server服务器需要安装docker 、docker-compose
-
二、系统升级
1、更新软件包列表
yum update -y
2、列出可用的 更新软件包
# yum list updates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.ustc.edu.cn
* updates: mirrors.aliyun.com
3、查看当前系统版本
# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
4、更新系统
# yum upgrade
# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
三、内核升级
1、查看当前内核版本
# uname -r
3.10.0-1160.71.1.el7.x86_64
2、查看可升级内核版本
# yum list kernel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.ustc.edu.cn
* updates: mirrors.aliyun.com
Installed Packages
kernel.x86_64 3.10.0-1160.71.1.el7 @anaconda
kernel.x86_64 3.10.0-1160.119.1.el7 @updates
可以看出只有一个内核版本3.10.0,而这次需要升级到内核6.9.7。不能直接使用yum update kernel-*打补丁升级了
3、查看是否安装ELRepo
[root@harbor01 ~]# yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
Loaded plugins: fastestmirror
Error getting repository data for elrepo-kernel, repository not found
ELRepo源是国外的一个只对Linux操作系统的第三方免费软件资源库,支持Linux和CentOS操作系统的软件安装和升级。
4、导入一个公钥
# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
5、安装一下ELRepo源
# yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
6、安装新内核
# yum --enablerepo=elrepo-kernel install kernel-ml -y
7、设置新安装的内核成为默认启动选项
将配置文件中GRUB_DEFAULT参数saved改为0
# sed -i s/saved/0/g /etc/default/grub
8、更新配置文件
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.9.7-1.el7.elrepo.x86_64
Found initrd image: /boot/initramfs-6.9.7-1.el7.elrepo.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1160.119.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.119.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1160.71.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.71.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-e227a3c248e94736968e30eee994bdb8
Found initrd image: /boot/initramfs-0-rescue-e227a3c248e94736968e30eee994bdb8.img
done
9、重启:reboot
# reboot
10、升级完成
# uname -r
6.9.7-1.el7.elrepo.x86_64
四、安装docker
1. 卸载旧版本(可选)
sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
2、安装需要的[软件包
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
3 、设置yum安装源
默认是国外的yum源,基本不可用,我们修改为阿里源。
//中央仓库
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
//国内建议安装阿里仓库
# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror
adding repo from: http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
grabbing file http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
[root@harbor01 ~]# yum makecache fast
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* elrepo: mirrors.tuna.tsinghua.edu.cn
* extras: mirrors.ustc.edu.cn
* updates: mirrors.aliyun.com
base | 3.6 kB 00:00:00
docker-ce-stable | 3.5 kB 00:00:00
elrepo | 3.0 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/2): docker-ce-stable/7/x86_64/updateinfo | 55 B 00:00:00
(2/2): docker-ce-stable/7/x86_64/primary_db | 152 kB 00:00:00
Metadata Cache Created
4、查看docker可用版本
# yum list docker-ce --showduplicates | sort -r
* updates: mirrors.aliyun.com
Loading mirror speeds from cached hostfile
Loaded plugins: fastestmirror
* extras: mirrors.ustc.edu.cn
* elrepo: mirrors.tuna.tsinghua.edu.cn
docker-ce.x86_64 3:26.1.4-1.el7 docker-ce-stable
docker-ce.x86_64 3:26.1.3-1.el7 docker-ce-stable
docker-ce.x86_64 3:26.1.2-1.el7 docker-ce-stable
docker-ce.x86_64 3:26.1.1-1.el7 docker-ce-stable
docker-ce.x86_64 3:26.1.0-1.el7 docker-ce-stable
docker-ce.x86_64 3:26.0.2-1.el7 docker-ce-stable
docker-ce.x86_64 3:26.0.1-1.el7 docker-ce-stable
docker-ce.x86_64 3:26.0.0-1.el7 docker-ce-stable
docker-ce.x86_64 3:25.0.5-1.el7 docker-ce-stable
docker-ce.x86_64 3:25.0.4-1.el7 docker-ce-stable
docker-ce.x86_64 3:25.0.3-1.el7 docker-ce-stable
docker-ce.x86_64 3:25.0.2-1.el7 docker-ce-stable
docker-ce.x86_64 3:25.0.1-1.el7 docker-ce-stable
docker-ce.x86_64 3:25.0.0-1.el7 docker-ce-stable
5、安装docker
# 安装最新版
yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
# 安装指定版本
yum install docker-ce-VERSION_STRING docker-ce-cli-VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin -y
6、 开启docker服务
# systemctl start docker
# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2024-06-29 00:23:30 CST; 33s ago
Docs: https://docs.docker.com
Main PID: 1692 (dockerd)
Tasks: 10
Memory: 32.2M
CGroup: /system.slice/docker.service
└─1692 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Jun 29 00:23:28 harbor01 systemd[1]: Starting Docker Application Container Engine...
Jun 29 00:23:28 harbor01 dockerd[1692]: time="2024-06-29T00:23:28.660489449+08:00" level=info msg="Starting up"
Jun 29 00:23:28 harbor01 dockerd[1692]: time="2024-06-29T00:23:28.900546935+08:00" level=info msg="Loading containers: start."
Jun 29 00:23:30 harbor01 dockerd[1692]: time="2024-06-29T00:23:30.264744262+08:00" level=info msg="Firewalld: interface dock...urning"
Jun 29 00:23:30 harbor01 dockerd[1692]: time="2024-06-29T00:23:30.532756602+08:00" level=info msg="Loading containers: done."
Jun 29 00:23:30 harbor01 dockerd[1692]: time="2024-06-29T00:23:30.606393989+08:00" level=info msg="Docker daemon" commit=de5...=26.1.4
Jun 29 00:23:30 harbor01 dockerd[1692]: time="2024-06-29T00:23:30.606732113+08:00" level=info msg="Daemon has completed init...zation"
Jun 29 00:23:30 harbor01 dockerd[1692]: time="2024-06-29T00:23:30.863205537+08:00" level=info msg="API listen on /run/docker.sock"
Jun 29 00:23:30 harbor01 systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
7、安装阿里云镜像加速器
tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://复制自己的加速器地址.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl restart docker
8、设置docker开机自启
# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
五、安装docker-compose(通过python的pip安装)
# 安装 epel-release
yum install -y epel-release
# 安装 python-pip
yum install -y python-pip
# 安装 python3-pip
yum install -y python3-pip
# 更新 pip
pip3 install --upgrade pip
# 安装 docker-compose
pip3 install docker-compose
# 查看 docker-compose 版本号
docker-compose -version
六、安装harbor
1、下载并解压harbor安装包
cd /usr/local
wget https://github.com/goharbor/harbor/releases/download/v2.11.0/harbor-offline-installer-v2.11.0.tgz
tar -zxvf harbor-offline-installer-v2.11.0.tgz
2、拷贝并编辑yml文件
cd harbor
cp harbor.yml.tmpl harbor.yml
vim harbor.yml
修改hostname、port,并注释https相关参数。
3、执行./install.sh
./install
4、执行./prepare.sh
./prepare
docker-compose down -v
docker-compose up -d
#开启防火墙端口
firewall-cmd --zone=public --add-port=5000/tcp --permanent
firewall-cmd --reload
5、登录
6、设置Harbor开机自启
vim /usr/lib/systemd/system/harbor.service
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor
[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /usr/local/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /usr/local/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target
7、服务启动
systemctl enable harbor
systemctl start harbor
8、修改daemon.json
分别在harbor01、harbor02、Docker Server三台主机添加 insecure-registries
# vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://复制自己的加速器地址.aliyuncs.com"],
"insecure-registries":["192.168.0.211:5000","192.168.0.212:5000","192.168.0.213:5000"]
}
# systemctl daemon-reload
# systemctl restart docker
在harbor01、harbor02主机执行以下命令
docker-compose down -v
docker-compose up -d
9、分别在3台主机验证登录harbor服务器
# docker login 192.168.0.212:5000
# docker login 192.168.0.213:5000
显示Login Succeeded代表登录成功
七、实现Harbor仓库双向同步
1、新建目标
2、创建复制规则
八、实现高可用
1、安装keepalived
#分别在harbor01、harbor02主机安装
yum -y install make gcc openssl-devel libnfnetlink-devel libnl3-devel net-snmp-devel
yum install -y keepalived
cd /usr/local/
#安装最新版本的keepalived
wget https://keepalived.org/software/keepalived-2.3.1.tar.gz
tar vxf keepalived-2.3.1.tar.gz -C /usr/local/src
cd src/keepalived-2.3.1/
./configure --prefix=/usr/local/keepalived
make -j 4 && make install
cd /usr/local/keepalived/
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
2、编写harbor健康检查脚本
# vim /usr/local/check_harbor.sh
脚本的具体内容
#!/bin/sh
# Harbor的健康检查URL
HARBOR_HEALTH_URL="http://localhost:5000/api/v2.0/health"
# 发送请求并检查状态码
if curl -s --output /dev/null --write-out "%{http_code}" "$HARBOR_HEALTH_URL" | grep -q '^200$'; then
exit 0 # Harbor健康
else
exit 1 # Harbor不健康
fi
修改sh文件为可以执行状态
chmod +x /usr/local/check_harbor.sh
3、修改keepalived配置
vim /etc/keepalived/keepalived.conf
主节点harbor01
! Configuration File for keepalived
global_defs {
notification_email {
111111111@qq.com
}
router_id harbor_master
}
vrrp_script check_harbor {
script "/usr/local/check_harbor.sh"
interval 10 # 每10秒检查一次
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface ens3
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass harbor1111
}
unicast_src_ip 192.168.0.212
unicast_peer {
192.168.0.213
}
virtual_ipaddress {
192.168.0.211
}
track_script {
check_harbor
}
}
备节点harbor02
! Configuration File for keepalived
global_defs {
notification_email {
111111111@qq.com
}
router_id harbor_backup
}
vrrp_script check_harbor {
script "/usr/local/check_harbor.sh"
interval 10 # 每10秒检查一次
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface ens3
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass harbor1111
}
unicast_src_ip 192.168.0.213
unicast_peer {
192.168.0.212
}
virtual_ipaddress {
192.168.0.211
}
track_script {
check_harbor
}
}
启动keepalived
systemctl start keepalived
systemctl enable keepalived
systemctl status keepalived
检测vip是否正常启用
检查在Docker Server使用vip登录是否正常
[root@dockerserver214 ~]# docker login 192.168.0.211:5000
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
4、向Harbor推送镜像
[root@dockerserver214 ~]# docker pull nginx
[root@dockerserver214 ~]# docker tag nginx:latest 192.168.0.211:5000/library/nginx:latest
[root@dockerserver214 ~]# docker push 192.168.0.211:5000/library/nginx
5、在harbor拉取镜像
[root@dockerserver214 ~]# docker pull 192.168.0.211:5000/library/nginx
Using default tag: latest
latest: Pulling from library/nginx
a2abf6c4d29d: Pull complete
a9edb18cadd1: Pull complete
589b7251471a: Pull complete
186b1aaa4aa6: Pull complete
b4df32aa5a72: Pull complete
a0bcbecc962e: Pull complete
Digest: sha256:ee89b00528ff4f02f2405e4ee221743ebc3f8e8dd0bfd5c4c20a2fa2aaa7ede3
Status: Downloaded newer image for 192.168.0.211:5000/library/nginx:latest
192.168.0.211:5000/library/nginx:latest
九、配置https
1、创建ssl证书
#创建目录
mkdir -p /usr/local/cret
cd /usr/local/cret/
#安装所需工具
yum -y install openssl
#创建ca密钥
openssl genrsa -out ca.key 4096
#创建ca证书,前提是已经创建了ca密钥
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=192.168.0.211" -key ca.key -out ca.crt
#在创建已宿主机IP地址为名称的私钥
openssl genrsa -out 192.168.0.211.key 4096
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=192.168.0.211" -key 192.168.0.211.key -out 192.168.0.211.csr
cat > v3.ext <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=192.168.1.11
EOF
openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in 192.168.0.211.csr -out 192.168.0.211.crt
所有ssl正式已经创建完成
2、 修改Harbor文件
3、启动harbor
docker-compose down -v
docker-compose up -d
4、修改harbor健康检查脚本
#!/bin/sh
# Harbor的健康检查URL
HARBOR_HEALTH_URL="https://localhost/api/v2.0/health"
# 发送请求并检查状态码
if curl -s --insecure --output /dev/null --write-out "%{http_code}" "$HARBOR_HEALTH_URL" | grep -q '^200$'; then
exit 0 # Harbor健康
else
exit 1 # Harbor不健康
fi
5、修改daemon.json
分别在harbor01、harbor02、Docker Server三台主机添加 insecure-registries
# vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://复制自己的加速器地址.aliyuncs.com"],
"insecure-registries":["192.168.0.211","192.168.0.212","192.168.0.213"]
}
# systemctl daemon-reload
# systemctl restart docker
6、重新新建目标
7、创建复制规则
8、向Harbor推送镜像
[root@dockerserver214 ~]# docker pull nginx
[root@dockerserver214 ~]# docker tag nginx:latest 192.168.0.211/library/nginx:latest
[root@dockerserver214 ~]# docker push 192.168.0.211/library/nginx:latest