搭建私有镜像站使用tunasync

tobias.b

已于 2024-02-20 10:14:41 修改

阅读量1.1k

点赞数 5

文章标签： linux 运维服务器

于 2024-01-24 19:43:50 首次发布

本文链接：https://blog.csdn.net/nasooo/article/details/135829887

版权

零摘要

本文主要介绍私有镜像站搭建，主要分两步一是下载同步公网镜像，而是提供web 服务，主要使用tunasync 实现下载，使用nginx 或者mirror-web 提供web 服务

一搭建同步工具tunasync

本文主要介绍以非root 用户搭建tunaasync

1.1 下载工具

https://github.com/tuna/tunasync

https://gitee.com/weiwei20180921/tunasync

1.1 基础配置

1.1.1 新建用户及工作目录 root 用户执行

[root@2023001 ~]# groupadd -g 2001 mirrorgroup
[root@2023001 ~]# useradd -u 2101 -g mirrorgroup mirrors
[root@2023001 ~]# passwd mirrors

1.1.2 建立应用目录（mirrors 执行）

主要存放应用配置文件及执行文件

[root@2023001 ~]# su - mirrors
[mirrors@2023001 ~]$ pwd
/home/mirrors
[mirrors@2023001 ~]$ mkdir -p /home/mirrors/tunasync/conf
[mirrors@2023001 ~]$ mkdir -p /home/mirrors/tunasync/db
[mirrors@2023001 ~]$ mkdir -p /home/mirrors/tunasync/bin

[mirrors@2023001 ~]$ tree
.
├── tunasync
│   ├── bin
│   ├── conf
│   └── db


3 directories, 0 files

1.1.3 建立数据目录（root 用户）

[root@2023001 data]# mkdir -p /data/mirrors
[root@2023001 data]# chown -R mirrors:mirrorgroup /data/mirrors/

1.2 部署及配置tunasync（mirrors 用户）

主要使用mirrors 用户部署

1.2.1 下载并上传到服务器

下载地址：
https://github.com/tuna/tunasync
或者
https://gitee.com/weiwei20180921/tunasync

1.2.2 部署

将tunasync-linux-amd64-bin.tar.gz 解压到/home/mirrors/tunasync/bin 目录

[mirrors@2023001 bin]$ pwd
/home/mirrors/tunasync/bin
[mirrors@2023001 bin]$ tar -xf /home/mirrors/tunasync-linux-amd64-bin.tar.gz
[mirrors@2023001 bin]$ ls
tunasync  tunasynctl
[mirrors@2023001 bin]$

1.2.3 配置环境变量

先备份，然后修改，最后source


[mirrors@2023001 ~]$ cp .bash_profile .bash_profile.bak.orig
[mirrors@2023001 ~]$ vim .bash_profile
[mirrors@2023001 ~]$ source .bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
TUNAPATH=/home/mirrors/tunasync

PATH=$PATH:$HOME/.local/bin:$HOME/bin:$TUNAPATH/bin

export PATH
~

验证

[mirrors@2023001 ~]$ tunasync -v
Version: 0.8.0
Git Hash: c07aaffe65da65217665d7c5df46ea2d857fdfa9
Build Date: 2021-09-02 22:24:51 +0800 CST
[mirrors@2023001 ~]$

1.2.4 配置tunasync

tunasync 的所有配置存放在/home/mirrors/tunasync/conf

1.2.4.1 manager 配置

创建/home/mirrors/tunasync/conf/manager.conf

debug = false

[server]
addr = "127.0.0.1"
port = 14242
ssl_cert = ""
ssl_key = ""

[files]
db_type = "bolt"
db_file = "/home/mirrors/tunasync/db/manager.db"
ca_cert = ""

配置说明

port：监听端口，默认是14242，
ssl_cert与ssl_key：ssl 配置，不需要 ssl 的同学不用配置
db_file：数据库文件，目前统一放在 /home/mirrors/tunasync/db/ 目录

端口14242 说明

[mirrors@2023001 conf]$ tunasynctl list --all
Failed to correctly get information of all jobs from manager server: Get http://localhost:14242/jobs: dial tcp [::1]:14242: connect: connection refused
[mirrors@2023001 conf]$ ll

若修改14242，则需修改tunasynctl 配置文件
一般在/etc/tunasync/ctl.conf 或者 ~/.config/tunasync/ctl.conf 两个位置

1.2.4.2 worker 配置

根据需要创建需同步的镜像配置文件。以 CentOS 镜像为例，同步 CentOS 需要创建 /home/mirrors/tunasync/conf/worker.conf，文件内容如下：

[global]
name = "myworker"
concurrent = 10
interval = 1440

[manager]
api_base = "http://localhost:14242"
token = "some_token"
ca_cert = ""

[cgroup]
enable = false
base_path = "/sys/fs/cgroup"
group = "tunasync"

[server]
hostname = "localhost"
listen_addr = "127.0.0.1"
listen_port = 16010
ssl_cert = ""
ssl_key = ""

[[mirrors]]
name = "7.5.1804"
log_dir = "/data/mirrors/log/tunasync/centos-vault/{{.Name}}"
mirror_dir = "/data/mirrors/centos-vault/7.5.1804"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.5.1804/"
use_ipv6 = false

[[mirrors]]
name = "7.6.1810"
log_dir = "/data/mirrors/log/tunasync/centos-vault/{{.Name}}"
mirror_dir = "/data/mirrors/centos-vault/7.6.1810"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810/"
use_ipv6 = false




[[mirrors]]
name = "7.9.2009"
log_dir = "/data/mirrors/log/tunasync/centos-vault/{{.Name}}"
mirror_dir = "/data/mirrors/centos-vault/7.9.2009"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.9.2009/"
use_ipv6 = false


[[mirrors]]
name = "epel"
log_dir = "/data/mirrors/log/tunasync/{{.Name}}"
mirror_dir = "/data/mirrors/epel"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/epel/"
use_ipv6 = false

mirror_dir = "/data/mirrors/centos-vault/7.9.2009
这个目录要手工创建mkdir

配置说明

global name：worker 进程名称，用于程序识别
log_dir：tunasync 的日志路径
mirror_dir：镜像下载地址
concurrent：并发线程数
interval：rsync 同步周期，以分钟为单位
api_base：manager 地址，注意，此处需与 manager 所在服务器配置保持一致
server listen_port：该 worker 自身的监听端口，如果同一台服务器上有多个 worker，各个 worker 的配置文件中务必配置不同端口
mirrors name：镜像名称，tunasync 会在镜像根目录下建立一个该名称的目录用于下载镜像
upstream：同步地址，注意，参数最后需要有“/”，否则启动 tunasync 时会报错

1.3 启动（mirrors 用户）

启动 tunasync 需开启 manager 进程与 worker 进程，先启动 manager，后启动 worker。为了便于监控系统进程情况，建立 /data/mirrors/log/plog 目录，所有进程的工作日志在该目录中（注意，此处日志为系统终端输出日志，与 tunasync 自身工作日志不同）。同样的，以下命令由 mirrors 用户操作。

1.3.1 建立日志目录

[mirrors@2023001 mirrors]$ mkdir -p /data/mirrors/log/plog

1.3.2 开启manager服务（后台进程）：

[mirrors@2023001 mirrors]$ tunasync manager --config /home/mirrors/tunasync/conf/manager.conf >> /data/mirrors/log/plog/manager.log &
[2] 2611
[mirrors@2023001 mirrors]$ netstat -anp | grep tunasy
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:28288         0.0.0.0:*               LISTEN      2611/tunasync
[mirrors@2023001 mirrors]$

1.3.2 开启worker服务（根据需要同步的镜像开启，这里以 CentOS7.9 为例）


[mirrors@2023001 mirrors]$ tunasync worker --config /home/mirrors/tunasync/conf/worker-centos.conf >> /data/mirrors/log/plog/worker-centos.log &
[3] 3233
[mirrors@2023001 mirrors]$ netstat -anp | grep tunasy
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:16010         0.0.0.0:*               LISTEN      3233/tunasync
tcp        0      0 127.0.0.1:28288         0.0.0.0:*               LISTEN      2611/tunasync
[mirrors@2023001 mirrors]$

1.4 启停脚本（mirrors 用户）

我这边是非root 用户部署和使用tunarsync ，所以启停脚本需要修改TUNASYNC_BIN，OPTIONS 这两个参数

1.4.1 下载并增加执行权限

脚本下载放到/home/mirrors/tunasync/bin
https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-worker
https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-manager

[mirrors@2023001 bin]$ pwd
/home/mirrors/tunasync/bin
[mirrors@2023001 bin]$ chmod u+x tunasync-manager.sh
[mirrors@2023001 bin]$ chmod u+x tunasync-worker.sh
[mirrors@2023001 bin]$ ls -al
total 42232
drwxr-xr-x 2 mirrors mirrorgroup       93 Jan 25 10:37 .
drwxr-xr-x 5 mirrors mirrorgroup       39 Jan 24 20:19 ..
-rwxr-xr-x 1 mirrors mirrorgroup 32456456 Sep  2  2021 tunasync
-rwxr--r-- 1 mirrors mirrorgroup     1487 Jan 25 10:32 tunasync-manager.sh
-rwxr-xr-x 1 mirrors mirrorgroup 10779559 Sep  2  2021 tunasynctl
-rwxr--r-- 1 mirrors mirrorgroup     1476 Jan 25 10:31 tunasync-worker.sh
[mirrors@2023001 bin]$

1.4.2 tunasync-manager.sh 修改

修改点


TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="manager --config /home/mirrors/tunasync/conf/manager.conf"
LOGFILE=/data/mirrors/log/plog/manager.log

 $TUNASYNC_BIN $OPTIONS >> $LOGFILE &

全量

#!/bin/sh
# Provides:          tunasync-manager
# Description:       Mirror job management tool.

# Author: whsir <wh@whsir.com>

NAME=tunasync
TUNASYNC_NAME=tunasync-manager
TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="manager --config /home/mirrors/tunasync/conf/manager.conf"
LOGFILE=/data/mirrors/log/plog/manager.log
StatBin=/bin/ps

case "$1" in
    start)
        if $StatBin -ef | grep -v grep | grep -q "tunasync manager";then
            TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync manager" | awk -F" " '{print $2}'`
            echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) already running."
        else
            $TUNASYNC_BIN $OPTIONS >> $LOGFILE &
            echo "Starting $TUNASYNC_NAME... "
        fi
    ;;

    status)
        if $StatBin -ef | grep -v grep | grep -q "tunasync manager";then
            TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync manager" | awk -F" " '{print $2}'`
            echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) is running."
        else
            echo "$TUNASYNC_NAME is not running."
        fi
    ;;

    stop)
        if $StatBin -ef | grep -v grep | grep -q "tunasync manager";then
            TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync manager" | awk -F" " '{print $2}'`
            kill $TUNASYNC_PID
            echo "Stop $TUNASYNC_NAME...."
        else
            echo "$TUNASYNC_NAME already stopping."
        fi
    ;;

    restart)
        $0 stop
        $0 start
    ;;

    *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
    ;;
esac

1.4.3 tunasync-worker.sh 修改

修改点：


TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="worker --config /home/mirrors/tunasync/conf/worker.conf"
LOGFILE=/data/mirrors/log/plog/worker.log
 $TUNASYNC_BIN $OPTIONS >> $LOGFILE &

全量

#!/bin/sh
# Provides:          tunasync-worker
# Description:       Mirror job management tool.

# Author: whsir <wh@whsir.com>

NAME=tunasync
TUNASYNC_NAME=tunasync-worker
TUNASYNC_BIN=/home/mirrors/tunasync/bin/$NAME
OPTIONS="worker --config /home/mirrors/tunasync/conf/worker.conf"
StatBin=/bin/ps
LOGFILE=/data/mirrors/log/plog/worker.log
case "$1" in
    start)
        if $StatBin -ef | grep -v grep | grep -q "tunasync worker";then
            TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync worker" | awk -F" " '{print $2}'`
            echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) already running."
        else
            $TUNASYNC_BIN $OPTIONS >> $LOGFILE &
            echo "Starting $TUNASYNC_NAME... "
        fi
    ;;

    status)
        if $StatBin -ef | grep -v grep | grep -q "tunasync worker";then
            TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync worker" | awk -F" " '{print $2}'`
            echo "$TUNASYNC_NAME (pid $TUNASYNC_PID) is running."
        else
            echo "$TUNASYNC_NAME is not running."
        fi
    ;;

    stop)
        if $StatBin -ef | grep -v grep | grep -q "tunasync worker";then
            TUNASYNC_PID=`ps -ef | grep -v grep | grep "tunasync worker" | awk -F" " '{print $2}'`
            kill $TUNASYNC_PID
            echo "Stop $TUNASYNC_NAME...."
        else
            echo "$TUNASYNC_NAME already stopping."
        fi
    ;;


    restart)
        $0 stop
        $0 start
    ;;

    *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
    ;;
esac

1.4.4 验证

[mirrors@2023001 ~]$ tunasync-manager.sh stop
Stop tunasync-manager....
[mirrors@2023001 ~]$ tunasync-manager.sh restart
tunasync-manager already stopping.
Starting tunasync-manager...
[mirrors@2023001 ~]$ tunasync-manager.sh status
tunasync-manager (pid 22809) is running.
[mirrors@2023001 ~]$

[mirrors@2023001 ~]$ tunasync-worker.sh status
tunasync-worker (pid 5877) is running.
[mirrors@2023001 ~]$ tunasync-worker.sh restart
Stop tunasync-worker....
tunasync-worker (pid 5877) already running.
[mirrors@2023001 ~]$

1.5 运维 tunasynctl 使用

主要介绍删除job,
使用参考
https://github.com/tuna/tunasync/blob/master/docs/zh_CN/tips.md
https://gitee.com/weiwei20180921/tunasync/blob/master/docs/zh_CN/tips.md

1.5.1 删除job

$ tunasynctl disable -w <worker_id> <mirror_name>
$ tunasynctl flush

[mirrors@2023001 log]$ tunasynctl disable -w centos_worker79 centos79
Successfully send the command
[mirrors@2023001 log]$ tunasynctl flush
Successfully flushed disabled jobs
[mirrors@2023001 log]$ tunasynctl list --all
[
  {
    "name": "7.9.2009",
    "is_master": true,
    "status": "syncing",
    "last_update": "0001-01-01 00:00:00 +0000",
    "last_update_ts": -62135596800,
    "last_started": "2024-01-25 11:06:56 +0800",
    "last_started_ts": 1706152016,
    "last_ended": "2024-01-25 10:56:53 +0800",
    "last_ended_ts": 1706151413,
    "next_schedule": "0001-01-01 00:00:00 +0000",
    "next_schedule_ts": -62135596800,
    "upstream": "rsync://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.9.2009/",
    "size": "unknown"
  },
  {
    "name": "epel",
    "is_master": true,
    "status": "syncing",
    "last_update": "0001-01-01 00:00:00 +0000",
    "last_update_ts": -62135596800,
    "last_started": "2024-01-25 14:35:53 +0800",
    "last_started_ts": 1706164553,
    "last_ended": "0001-01-01 00:00:00 +0000",
    "last_ended_ts": -62135596800,
    "next_schedule": "0001-01-01 00:00:00 +0000",
    "next_schedule_ts": -62135596800,
    "upstream": "rsync://mirrors.tuna.tsinghua.edu.cn/epel/",
    "size": "unknown"
  }
]
[mirrors@2023001 log]$

1.5.2 删除worker

$ tunasynctl rm-worker -w <worker_id>

[mirrors@2023001 bin]$ tunasynctl rm-worker -w centos_worker79
Successfully removed the worker
[mirrors@2023001 bin]$ tunasynctl workers
null
[mirrors@2023001 bin]$

1.5.3 热重载 worker.conf

当修改了worker.conf 可使用该命令，下面介绍增加worker.conf 配置，删除worker.conf 配置操作见官网示例

$ tunasynctl reload -w <worker_id>

示例：新增欧拉系统同步
首先修改worker.conf 配置
第二补充对应数据目录
第三执行热重载命令
第四验证

添加同步欧拉系统配置

[mirrors@2023001 conf]$ vi worker.conf
[[mirrors]]
name = "openEuler-22.03-LTS-SP3"
log_dir = "/mirrordata/mirrors/log/tunasync/{{.Name}}"
mirror_dir = "/mirrordata/mirrors/openeuler/openEuler-22.03-LTS-SP3"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS-SP3/"
use_ipv6 = false

第二补充对应数据目录



[mirrors@2023001 conf]$ cd /mirrordata/mirrors/
[mirrors@2023001 mirrors]$ mkdir -p ./openeuler/openEuler-22.03-LTS-SP3

第三执行热重载命令

tunasynctl reload -w myworker

验证

tunasynctl list --all

 {
    "name": "openEuler-22.03-LTS-SP3",
    "is_master": true,
    "status": "syncing",
    "last_update": "0001-01-01 00:00:00 +0000",
    "last_update_ts": -62135596800,
    "last_started": "2024-02-02 10:02:21 +0800",
    "last_started_ts": 1706839341,
    "last_ended": "0001-01-01 00:00:00 +0000",
    "last_ended_ts": -62135596800,
    "next_schedule": "0001-01-01 00:00:00 +0000",
    "next_schedule_ts": -62135596800,
    "upstream": "rsync://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS-SP3/",
    "size": "unknown"
  }

二搭建mirror-web

原相关代码在GitHub上，先导入gitee,然后再gitee 使用gitee

2.1 将github 代码迁移到gitee

https://github.com/tuna/mirror-web.git
https://github.com/tuna/mirrorz-help-ng.git

迁移略，
迁移后地址：
https://gitee.com/weiwei20180921/mirror-web.git
https://gitee.com/weiwei20180921/mirrorz-help-ng.git

2.2 制作mirror-web 镜像

操作参考：https://github.com/tuna/mirror-web

2.2.1 克隆代码

[root@2023001 code]# git clone https://gitee.com/weiwei20180921/mirror-web.git

2.2.2 嵌入帮助文档

修改mirror-web/.gitmodules
url 改成gitee 的，否则网不行
修改后如下

[submodule "helpz"]
        path = help/_posts/mirrorz-help-ng-transpiled
        url = https://gitee.com/weiwei20180921/mirrorz-help-ng.git
        branch = transpiled
~

执行嵌入命令

[root@2023001 mirror-web]# git submodule update --init
Submodule 'helpz' (https://gitee.com/weiwei20180921/mirrorz-help-ng.git) registered for path 'help/_posts/mirrorz-help-ng-transpiled'
Cloning into 'help/_posts/mirrorz-help-ng-transpiled'...
remote: Enumerating objects: 802, done.
remote: Counting objects: 100% (802/802), done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 802 (delta 642), reused 802 (delta 642), pack-reused 0
Receiving objects: 100% (802/802), 269.09 KiB | 98.00 KiB/s, done.
Resolving deltas: 100% (642/642), done.
Submodule path 'help/_posts/mirrorz-help-ng-transpiled': checked out '61639dd0bbc8a7e8482e755aff566f1b8714b0c8'
[root@2023001 mirror-web]#

2.2.3 制作镜像

[root@2023001 mirror-web]# docker build -t builden -f Dockerfile.build .

2.2.4 运行

卡在这一步了，运行不起来。

2.3 使用nginx 提供web 服务

安装略，此处最后给出配置文件

三常见问题

主要记录运维过程中出现的现象及问题以及解决办法

3.1 job 同步目录在增长，但看不到文件

比如如下两个job 的同步目录分别是是
/data/mirrors/openeuler/openEuler-22.03-LTS-SP3
/data/mirrors/openeuler/openEuler-22.03-LTS

这两个目录大小一直在增长，但是却看不到具体文件，
原因是文件都在隐藏临时目录._tmp 下，而且据观察是只有整个/data/mirrors/openeuler/openEuler-22.03-LTS 同步完，才会将隐藏目录里内容转移到对应的真实目录里。

[mirrors@junpanpro openeuler]$ du -h --max-depth=2 /data/mirrors/openeuler/
41G	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/EPOL
104G	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/ISO
6.7G	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/OS
211M	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/WSL
39G	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/debuginfo
170M	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/docker_img
2.8G	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/edge_img
6.5G	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/embedded_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/everything
0	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/raspi_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/source
0	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/stratovirt_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/update
0	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3/virtual_machine_img
199G	/data/mirrors/openeuler/openEuler-22.03-LTS-SP3
26G	/data/mirrors/openeuler/openEuler-22.03-LTS/EPOL
127G	/data/mirrors/openeuler/openEuler-22.03-LTS/ISO
10G	/data/mirrors/openeuler/openEuler-22.03-LTS/OS
5.4G	/data/mirrors/openeuler/openEuler-22.03-LTS/debuginfo
0	/data/mirrors/openeuler/openEuler-22.03-LTS/docker_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS/edge_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS/embedded_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS/everything
0	/data/mirrors/openeuler/openEuler-22.03-LTS/raspi_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS/source
0	/data/mirrors/openeuler/openEuler-22.03-LTS/stratovirt_img
0	/data/mirrors/openeuler/openEuler-22.03-LTS/update
0	/data/mirrors/openeuler/openEuler-22.03-LTS/virtual_machine_img
168G	/data/mirrors/openeuler/openEuler-22.03-LTS
366G	/data/mirrors/openeuler/
[mirrors@junpanpro openeuler]$