搭建镜像网站使用rsync

一 摘要

本文主要介绍基于rsync 以及shell 和crontab 完成,相对于使用tunarsync,我们可以更好的调整rsync 同步参数，更可控。维护也简单。

二 环境信息

[mirrors@localhost repodata]$ cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[mirrors@localhost repodata]$ rsync --version
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

三 实施

主要使用rsync + shell 实现增量同步，rsync 不使用临时目录，然后结合crontab 定时启动

3.1 同步docker-ce

以同步centos 7.9 里的docker-ce 安装包为例

rsync_docker_ce_centos_7.9.sh 同步脚本。

[mirrors@localhost script]$ cat rsync_docker_ce_centos_7.9.sh 
#!/bin/bash
#docer-ce
DIR_LOG="/home/mirrors/rsyncmirror/log"

FILE_LOG="${DIR_LOG}/docker-ce-centos7.9.`date +%F`.log"

mkdir -pv /data/mirrors/docker-ce/linux/centos/7.9 

nohup rsync -avzP --bwlimit=30720 --timeout=120 --contimeout=120   rsync://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/7.9/ /data/mirrors/docker-ce/linux/centos/7.9/ >> ${FILE_LOG}  2>&1 &

3.2 同步控制脚本

该脚本用于启动同步，停止同步，可配合crontab 执行

[mirrors@localhost script]$ cat mirrors_tool.sh 
#!/bin/bash
LOGFILE=/home/mirrors/rsyncmirror/log/rsyncl_all.log

case "$1" in
    start)
         /home/mirrors/rsyncmirror/script/rsync_docker_ce_centos_7.9.sh && echo "start rsync_docker_ce_centos_7.9.sh" >> $LOGFILE 
         echo "STARTING rsync miroor..."
    
    ;;  
    stop)
       #killall rsync && echo " kill all rsync " >> $LOGFILE
       PIDS=`ps -ef | grep -v grep | grep rsync | awk '{print $2}'`
       for pid in $PIDS
       do
          kill -9 $pid && echo "kill $pid" >> $LOGFILE
       done
       echo "STOP ALL rsync"

    ;;

    status)
       ps -ef | grep -v grep | grep rsync
    ;;
    
    restart)
        $0 stop
        $0 start
    ;; 

    *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
    ;;
esac

3.3 按需同步脚本

比如我只想同步openEuler 目下 2203SP3 版本操作系统

#!/bin/bash
#docer-ce
DIR_LOG="/home/mirrors/rsyncmirror/log"

FILE_LOG="${DIR_LOG}/openeuler.`date +%F`.log"

mkdir -pv /data/mirrors/openeuler 

nohup rsync -avzP  --bwlimit=30720 --timeout=120 --contimeout=120 --include="openEuler-22.03-LTS-SP3/" --include="openEuler-22.03-LTS-SP3/***"  --exclude="*" --exclude="*/" rsync://mirrors.tuna.tsinghua.edu.cn/openeuler/ /data/mirrors/openeuler/ >> ${FILE_LOG}  2>&1 &

3.4 使用include 参数同步脚本

rsync_centos_vault.sh 用于同步centos-vault 目录，但是不是该目录下所有的子目录都需要同步，需要同步的目录放到INCLUDE_FILE

[mirrors@junpanpro script]$ cat rsync_centos_vault.sh 
#!/bin/bash
#mariadb
APP_NAME=centos-vault
WORK_DIR="/home/mirrors/rsyncmirror"
DIR_LOG="${WORK_DIR}/log"
INCLUDE_FILE="${WORK_DIR}/includefile/${APP_NAME}.txt"
FILE_LOG="${DIR_LOG}/${APP_NAME}.`date +%F`.log"

mkdir -pv /data/mirrors/${APP_NAME} 

nohup rsync -avzP  --bwlimit=30720 --timeout=120 --contimeout=120 --include-from="${INCLUDE_FILE}"   --exclude="*" --exclude="*/" rsync://mirrors.tuna.tsinghua.edu.cn/${APP_NAME}/ /data/mirrors/${APP_NAME}/ >> ${FILE_LOG}  2>&1 &

INCLUDE_FILE 内容示例：

[mirrors@junpanpro script]$ cat ../includefile/centos-vault.txt 
7.3.1611/
7.3.1611/***
7.4.1708/
7.4.1708/***
7.5.1804/
7.5.1804/***
7.6.1810/
7.6.1810/***
7.7.1908/
7.7.1908/***
7.8.2003/
7.8.2003/***
7.9.2009/
7.9.2009/***
[mirrors@junpanpro script]$