inotify tools与 rsync

最新推荐文章于 2024-11-05 18:02:41 发布

Terry_Tsang

最新推荐文章于 2024-11-05 18:02:41 发布

阅读量180

点赞数

分类专栏： rsync 文章标签： rsync inotify linux

本文链接：https://blog.csdn.net/signmem/article/details/107839978

版权

rsync 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

目的

每个 client1,client2 … clientN 需要定时，不定时地同脚本，代码到某个固定目录
其目的只需要确保 client 中某个目录的代码保持最新即可

源架构

用户 client1, 2 … N 通过 git pull 方式获取数据
问题:

git server 并不是一个专门用于数据下载的服务器，因此具有并发量限制
在使用过程中，当超过 6 个用户并发同步数据，则打爆千兆网卡网络
git server 没有做好权限控制，总会有开发直接从 client push 代码到 server，而且历史原因没法祸首权限
偶尔有用户不按规定直接在 client 中直接修改代码，导致 git pull 无法完成代码同步
在 git pull 传输过程中， .git 目录空间占用了一半，浪费了网络流量带宽及磁盘空间

arhc2

新架构

为确保代码开发一致性，开发只可以按照规定提交，合并代码至 git server
开发者无权限访问 rsync1, 2 … N 服务器，而 rsync server 对 Client 提供代码同步功能
rsync1 为上传 git server 代码服务器， rsync 2, 3 … N 向 rsync1 进行代码同步
rsync 具有全量镜像同步优点，确保每个 rsync server 文件(内容，权限)一致性
client1, 2 … N 也使用全量镜像同步功能，避免了开发直接在 client 中修改代码导致 client 代码异步
使用 nginx 作为 rsync server 前端代理，增加了并发并可横向无缝扩展，不再担心流量并发量问题

arch

原理

1 git server 代码提交，合并方法（略）确保代码源安全性及一致性
2 git server 在代码合并后，通过 rsync 方式提交代码至 rsync1
3 通过 inotify 工具监控 rsync1 代码目录，当有文件改变，自动同步至 rsync2, rsync3
4 由于所有代码文件都是小文件，因此基本上可以理解为 rsync1, 2,3 文件都是实时同步
5 nginx 利用 tcp 代理方式，反向代理 rsync1,2…N 服务器
6 client1, 2 … N 通过 cron 不定期进行数据同步

软件安装

软件下载

rsync officail web
下载 rsync-3.2.2.tar.gz 稳定版
 github notify-tools
下载 notify-tools-3.13.tar.gz

编译 inotify-tools

检查内核是否具备 inotify 检测功能，当具备下面三个文件，则支持

 inotify-tools-3.13]# ls -l /proc/sys/fs/inotify/
total 0
-rw-r--r-- 1 root root 0 Aug  6 06:50 max_queued_events
-rw-r--r-- 1 root root 0 Aug  6 06:50 max_user_instances
-rw-r--r-- 1 root root 0 Aug  6 06:50 max_user_watches

安装编译需要的软件

yum install -y gcc gcc-c++ make

编译方法

tar xf inotify-tools-3.13.tar.gz
cd inotify-tools-3.13
./configure --prefix=/apps/svr/inotify
make
make install

编译 rsync

解决一些常见依赖关系

yum install -y lz4-devel openssl-devel

编译方法

./configure  --disable-zstd --disable-xxhash --prefix=/apps/svr/rsync  --with-rsync-path=/apps/svr/rsync/bin/ --with-rsyncd-conf=/apps/conf/rsync/rsyncd.conf

.....
configure.sh: creating ./config.status
config.status: creating Makefile
config.status: creating lib/dummy
config.status: creating zlib/dummy
config.status: creating popt/dummy
config.status: creating shconfig
config.status: creating config.h

    rsync 3.2.2 configuration successful


make 
make install

部署

git 相关（略）

rsync server

目的

对目录 /apps/dat/plugins 进行数据同步控制
通过 rsync server，对该目录区分读/写权限控制
rsync1 只能够对来源 git-server 允许写操作
rsync2, 3 …N 只对来源 rsync1 服务器允许写操作
对所有用户开放读操作

配置文件

rsync1 server: /apps/conf/rsync/rsyncd.conf 配置

cat /apps/conf/rsync/rsyncd.conf
uid = apps
gid = apps
use chroot = yes
max connections = 1000
timeout = 300
pid file = /apps/logs/rsync/rsyncd.pid
lock file = /apps/logs/rsync/rsync.lock
log file = /apps/logs/rsync/rsyncd.log
list = true
strict modes = false                <- 允许  /apps/conf/rsync/rsync.password 文件 owner 为非 root 用户

[plugins_read]
read only = true                                                 <- 改目录为只读控制
hosts allow = 0.0.0.0/0                                          <- 全局开放
comment = "read only sync plugins"
path = /apps/dat/plugins                                         <- 服务器目录位置  


[plugins_write]
read only = false                                                 <- 允许写操作
path = /apps/dat/plugins                                          <- 服务器目录位置  
comment = "read write sync plugins"
hosts allow = git-server.google.com                               <- 只允许该服务器执行写
hosts deny = 0.0.0.0/0
auth users =  terry                                               <- 用户执行写操作需要权限验证
secrets file = /apps/conf/rsync/rsync.password                    <- 用户校验密码存放位置， 注意必须  chmod o-r 属性

密码文件

echo  terry:123123 > /apps/conf/rsync/rsync.password     
chmod 600  /apps/conf/rsync/rsync.password  
chown  apps:apps  /apps/conf/rsync/rsync.password

rsync2, rsync3 , rsyncN 服务器配置

cat /apps/conf/rsync/rsyncd.conf
uid = apps
gid = apps
use chroot = yes
max connections = 200
timeout = 300
pid file = /apps/logs/rsync/rsyncd.pid
lock file = /apps/logs/rsync/rsync.lock
log file = /apps/logs/rsync/rsyncd.log
list = true
strict modes = false

[plugins_read]
read only = true
hosts allow = 0.0.0.0/0
comment = "read only sync plugins"
path = /apps/dat/plugins


[plugins_write]
read only = false
path = /apps/dat/plugins
comment = "read write sync plugins"
hosts allow = rsyn1-server.google.com      <- 只需要在这里进行源 IP 地址控制写入
hosts deny = 0.0.0.0/0

启动 rsync server

参考启动方法

/apps/svr/rsync/bin/rsync --daemon  --address=0.0.0.0 --config=/apps/conf/rsync/rsyncd.conf  --verbose --ipv4

ex: 参考命令帮助

/apps/svr/rsync/bin/rsync --daemon  --help

rsync server (el6)

#! /bin/sh

### BEGIN INIT INFO
# Provides:          rsyncd
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Should-Start:      $named
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: fast remote file copy program daemon
# Description:       rsync is a program that allows files to be copied to and
#                    from remote machines in much the same way as rcp.
#                    This provides rsyncd daemon functionality.
### END INIT INFO

set -e

# /etc/init.d/rsync: start and stop the rsync daemon

DAEMON=/apps/svr/rsync/bin/rsync
RSYNC_ENABLE=true
RSYNC_OPTS=''
RSYNC_DEFAULTS_FILE=/etc/default/rsync
RSYNC_CONFIG_FILE=/apps/conf/rsync/rsyncd.conf
RSYNC_PID_FILE=/apps/logs/rsync/rsyncd.pid
RSYNC_NICE_PARM=''
RSYNC_IONICE_PARM=''

test -x $DAEMON || exit 0

. /lib/lsb/init-functions

if [ -s $RSYNC_DEFAULTS_FILE ]; then
    . $RSYNC_DEFAULTS_FILE
    case "x$RSYNC_ENABLE" in
        xtrue|xfalse)   ;;
        xinetd)         exit 0
                        ;;
        *)              log_failure_msg "Value of RSYNC_ENABLE in $RSYNC_DEFAULTS_FILE must be either 'true' or 'false';"
                        log_failure_msg "not starting rsync daemon."
                        exit 1
                        ;;
    esac
    case "x$RSYNC_NICE" in
        x[0-9])         RSYNC_NICE_PARM="--nicelevel $RSYNC_NICE";;
        x1[0-9])        RSYNC_NICE_PARM="--nicelevel $RSYNC_NICE";;
        x)              ;;
        *)              log_warning_msg "Value of RSYNC_NICE in $RSYNC_DEFAULTS_FILE must be a value between 0 and 19 (inclusive);"
                        log_warning_msg "ignoring RSYNC_NICE now."
                        ;;
    esac
    case "x$RSYNC_IONICE" in
        x-c[123]*)      RSYNC_IONICE_PARM="$RSYNC_IONICE";;
        x)              ;;
        *)              log_warning_msg "Value of RSYNC_IONICE in $RSYNC_DEFAULTS_FILE must be -c1, -c2 or -c3;"
                        log_warning_msg "ignoring RSYNC_IONICE now."
                        ;;
    esac
fi

export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"

rsync_start() {
    if [ ! -s "$RSYNC_CONFIG_FILE" ]; then
        log_failure_msg "missing or empty config file $RSYNC_CONFIG_FILE"
        exit 0
    fi
    # See ionice(1)
    if [ -n "$RSYNC_IONICE_PARM" ] && [ -x /usr/bin/ionice ] &&
        /usr/bin/ionice "$RSYNC_IONICE_PARM" true 2>/dev/null; then
        /usr/bin/ionice "$RSYNC_IONICE_PARM" -p$$ > /dev/null 2>&1
    fi
    if start_daemon $DAEMON --daemon --config "$RSYNC_CONFIG_FILE" $RSYNC_OPTS
    then
        rc=0
        sleep 1
        if ! kill -0 $(cat $RSYNC_PID_FILE) >/dev/null 2>&1; then
            log_failure_msg "rsync daemon failed to start"
            rc=1
        else
                log_success_msg "rsync daemon started successfully"
        fi
    else
        rc=1
    fi
    if [ $rc -ne 0 ]; then
        rm -f $RSYNC_PID_FILE
    fi
} # rsync_start


case "$1" in
  start)
        if "$RSYNC_ENABLE"; then
            log_success_msg "Starting rsync daemon" "rsync"
            if [ -s $RSYNC_PID_FILE ] && kill -0 $(cat $RSYNC_PID_FILE) >/dev/null 2>&1; then
                        log_warning_msg "apparently already running"
                        exit 0
            fi
            rsync_start
    else
            if [ -s "$RSYNC_CONFIG_FILE" ]; then
                [ "$VERBOSE" != no ] && log_warning_msg "rsync daemon not enabled in $RSYNC_DEFAULTS_FILE, not starting..."
            fi
        fi
        ;;
  stop)
        log_success_msg "Stopping rsync daemon" "rsync"
        killproc -p $RSYNC_PID_FILE "$DAEMON" rsync
        rm -f $RSYNC_PID_FILE
        ;;

  reload|force-reload)
        log_warning_msg "Reloading rsync daemon: not needed, as the daemon"
        log_warning_msg "re-reads the config file whenever a client connects."
        ;;

  restart)
        set +e
        if $RSYNC_ENABLE; then
            log_success_msg "Restarting rsync daemon" "rsync"
            if [ -s $RSYNC_PID_FILE ] && kill -0 $(cat $RSYNC_PID_FILE) >/dev/null 2>&1; then
                        killproc -p $RSYNC_PID_FILE || true
                        sleep 1
            else
                        log_warning_msg "rsync daemon not running, attempting to start."
                rm -f $RSYNC_PID_FILE
            fi
            rsync_start
    else
        if [ -s "$RSYNC_CONFIG_FILE" ]; then
            [ "$VERBOSE" != no ] && log_warning_msg "rsync daemon not enabled in $RSYNC_DEFAULTS_FILE, not starting..."
        fi
        fi
        ;;

  status)
    set +e
    pidofproc -p $RSYNC_PID_FILE "$DAEMON" rsync 1>2 2>/dev/null
    case "$?" in
                0)
                        log_success_msg "$DAEMON is running..."
                        exit 0
                        ;;
                1)
                        log_warning_msg  "$DAEMON dead but pid file exists"
                        exit 1
                        ;;
                3)
                        log_warning_msg  "$DAEMON is not running"
                        exit 3
                        ;;
        esac
        ;;
  *)
        echo "Usage: /etc/init.d/rsync {start|stop|reload|force-reload|restart|status}"
        exit 1
esac

exit 0

rsync server (el6)

/usr/lib/systemd/system/rsyncd.service

[Unit]
Description=fast remote file copy program daemon
ConditionPathExists=/apps/conf/rsync/rsyncd.conf
After=network.target
Documentation=man:rsync(1) man:rsyncd.conf(5)

[Service]
ExecStart=/apps/svr/rsync/bin/rsync --daemon --no-detach
RestartSec=1

ProtectSystem=full
PrivateDevices=on
NoNewPrivileges=on

[Install]
WantedBy=multi-user.target

git server 上传数据到 rsync1 方法

检验当前 rsync1 上开放的目录

rsync -rtd rsync://rsync1-server.google.com:873/
plugins_read    "read only sync plugins"
plugins_write   "read write sync plugins"

上传

访问 write 目录方法

cd /需要同步代码的目录
rsync -avz --delete .  rsync://terry@rsync1-server.google.com:873/plugins_write/.
或
rsync -avz --delete /git/server/source/  rsync://terry@rsync1-server.google.com:873/plugins_write/.

下载

访问 read 目录方法

cd /需要同步代码的目录
rsync -avz --delete   rsync://terry@rsync1-server.google.com:873/plugins_write/  .
或
rsync -avz --delete /git/server/source/  rsync://terry@rsync1-server.google.com:873/plugins_write/   /git/server/source/

inotify-tools

目的

只需要在 rsync1 server 中执行
监控 rsync1 中目录文件变化
当文件发生变化，执行 rsync 命令同步变化文件至 rsync2, rsync3
至适用于小文件变化适用
对大文件，经常变化并不太合适

参考

监控事件

Events:
        access          file or directory contents were read
        modify          file or directory contents were written
        attrib          file or directory attributes changed
        close_write     file or directory closed, after being opened in
                        writeable mode
        close_nowrite   file or directory closed, after being opened in
                        read-only mode
        close           file or directory closed, regardless of read/write mode
        open            file or directory opened
        moved_to        file or directory moved to watched directory
        moved_from      file or directory moved from watched directory
        move            file or directory moved to or from watched directory
        create          file or directory created within watched directory
        delete          file or directory deleted within watched directory
        delete_self     file or directory was deleted
        unmount         file system containing file or directory unmounted

当发生上述文件变化事件，即可触发脚本

shell script example ( in rsync1)

#!/bin/bash

while /apps/svr/inotify/bin/inotifywait -r -e 'modify,create,delete' /apps/dat/plugins/
do
  rsync -avz --delete /apps/dat/plugins/ rsync://rsync2-server.google.com.com:873/plugins_write/
  rsync -avz --delete /apps/dat/plugins/ rsync://rsync3-server.google.com.com:873/plugins_write/
done

nginx

用于 tcp 反向代理
只提供最基础功能，执行进行优化适用

worker_processes  1;
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections  1024;
}
stream {
        upstream rsync_server {
            server rsync1-server.google.com:873 weight=1;
            server rsync2.server.google.com:873 weight=1;
            server rsync3-server.google.com:873 weight=1;
        }
        server {
            listen 873;
                proxy_pass rsync_server;
        }
}
include /etc/nginx/conf.d/*.conf;