使用inotifywait和rsync进行实时备份

安装inotifywait和rsync

$ sudo apt-get install inotify-tools rsync

inotifywait

inotifywait efficiently waits for changes to files using Linux’s inotify(7) interface. It is suitable for waiting for changes to files from shell scripts. It can either exit once an event occurs, or continually execute and output events as they occur.

inotifywait使用Linux的inotify接口对文件/文件夹的变化进行监控。当监控一个文件夹时，inotify会返回文件夹本身的事件，以及文件夹中文件的事件。

那么能够监控哪些事件呢：

• ACCESS : 被监控的文件或被监控的文件夹中的文件被读取了。
• MODIYF：被监控的文件或被监控的文件夹中的文件被修改了。
• ATTRIB：被监控的文件或被监控的文件夹中的文件的metadata（元数据）被修改了，元数据包括：时间戳、文件权限等。
• CLOSE_WRITE: 被监控的文件或被监控的文件夹中的文件以“写”方式打开并被关闭了。
• CLOSE_NOWRITE: 被监控的文件或被监控的文件夹中的文件以“只读”方式打开并被关闭了。
• CLOSE：被监控的文件或被监控的文件夹中的文件打开之后（写方式或者只读方式）被关闭了。这种事件其实是通过同时监控CLOSE_WRITE和CLOSE_NOWRITE事件来实现的，所有关闭事件发生时只会输出CLOSE_WRITE和CLOSE_NOWRITE中的一种，而不是输出CLOSE
• OPEN：被监控的文件或被监控的文件夹中的文件被打开了。
• MOVED_TO: 一个文件或文件夹被移动到了监控文件夹内。即使源文件夹和目的文件夹一样，该事件也会发生。
• MOVED_FROM: 一个文件或文件夹被移出了监控文件夹。即使源文件夹和目的文件夹一样，该事件也会发生。
• MOVE：moved_to 和 moved_from， 类似close
• MOVE_SELF: 被监控的文件或文件夹被move了，在这个事件之后，该文件和文件夹不再被监控。
• CREATE：被监控的文件夹中创建了新文件或文件夹。
• DELETE：被监控的文件夹中删除了一个文件或文件夹
• DELETE_SELF: 被监控的文件或文件夹被删除了。在此事件之后，该文件或文件夹不再被监控。注意：即使不显示的监听这个事件，这个事件也会发生。

一个例子：

#!/bin/bash

MONITOR_DIR=/path/to/monitored/folder

inotifywait -mrq --format '%e %w%f %T' --timefmt='%Y-%m-%d/%H:%M:%S' -e modify,create,delete,attrib,close_write,move $MONITOR_DIR  | \
while read INO_EVENT INO_FILE INO_TIME
do
    echo $INO_TIME" "$INO_FILE" "$INO_EVENT
done

修改MONITOR_DIR，然后在这个文件夹中进行新建、修改等操作。体会一下都有哪些事件发生了。

rsync

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a “quick check” algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.

rsync是一个快速、功能丰富的copy工具。它可以进行本地拷贝、从远端主机拷贝、拷贝到远端主机。rsync和remote host的通信可以使用两种方式：

• remote-shell transport： ssh
• rsync daemon： remote host上有rsync daemon

两种方式的命令格式：(主要区别在于HOST和SRC有几个’:’分隔， remote-shell为一个’:’，rsync-daemon为两个’:’或者以rsync://开头)

       Access via remote shell:
         Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
         Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

       Access via rsync daemon:
         Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
               rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
         Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
               rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

例子1

rsync -t *.c foo:src/

这个命令将所有当前目录下符合pattern “*.c”的文件传输到机器foo的src文件夹下。如果某些文件已经在远端主机foo上存在了，那么将使用rsync remote-update协议来处理，这样只会发送文件的差异。

-t 表示保留文件的修改时间戳。

例子2

rsync -avz foo:src/bar /data/tmp

这样会递归的将远端主机foo上的src/bar文件夹中的文件传输到本地机器的/data/tmp目录

-a 表示使用“archive”模式进行传输，这种模式保证符号连接、权限、所有者等信息保持一致。
-z 表示在传输前进行压缩。

例子3

rsync -avz foo:src/bar/ /data/tmp

与上面例子唯一的区别是源目录最后以’/’结尾。以‘/’结尾只会拷贝文件夹的内容，而不以‘/’结尾则会拷贝文件夹及内容。例如，下面两个命令的效果是一样的：

rsync -av /src/foo /dest
rsync -av /src/foo/ /dest/foo

例子4

rsync -av /foo/bar/baz.c remote:/tmp/

这将在/tmp/目录下创建名字为baz.c的文件，但是加上 -R选项就不一样了：

rsync -avR /foo/bar/baz.c remote:/tmp/

文件的路径将保持下来，即/tmp/foo/bar/baz.c将会被创建。

SORTED TRANSFER ORDER

Rsync always sorts the specified filenames into its internal transfer list. This handles the merging together of the contents of identically named directories, makes it easy to remove duplicate filenames, and may confuse someone when the files are transferred in a different order than what was given on the command-line.

rsync会排序、合并相同的文件差异，然后再进行传输。

使用 inotifywait & rsync进行实时备份

参考：http://www.ttlsa.com/web/let-infotify-rsync-fast/

backup.sh:

#!/bin/bash

MONITOR_DIR=/home/yourself/doc/
MONITOR_IGNORE_DIRS= # example: (data/cache/|data/locks/|data/index/|data/tmp/)'
BACKUP_LOG_LOCAL_PATH=/home/yourself/backup-log/
BACKUP_MACHINE_IP=10.0.0.1
BACKUP_MACHINE_USER=someone
BACKUP_MACHINE_PATH=/home/someone/backup

#
# function util
#
function LOG()
{
    LOG_FILE=$BACKUP_LOG_LOCAL_PATH`date +'%Y-%m-%d'`.log
    echo "$@">>$LOG_FILE
}

#
# start the real-time backup service 
#
function start() {

    LOG "==========================================="
    LOG "Backup Start at:" `date +%H:%M:%S`
    LOG "==========================================="

    cd $MONITOR_DIR || { LOG 'MONITOR_DIR ERROR' && exit 1; }
    inotifywait -mrq --excludei='$MONITOR_IGNORE_DIRS' --format '%e %w%f %T' --timefmt='%Y-%m-%d/%H:%M:%S' -e modify,create,delete,attrib,close_write,move .  | \
    while read INO_EVENT INO_FILE INO_TIME
    do
           #echo $INO_TIME" "$INO_FILE" "$INO_EVENT

           if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || 
              [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]] || 
              [[ $INO_EVENT =~ 'ATTRIB' ]]
           then
                #rsync -avzcR $(dirname ${INO_FILE}) ${BACKUP_MACHINE_USER}@${BACKUP_MACHINE_IP}:${BACKUP_MACHINE_PATH}          
                rsync -avzcR ${INO_FILE} ${BACKUP_MACHINE_USER}@${BACKUP_MACHINE_IP}:${BACKUP_MACHINE_PATH}          
           fi

           if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]]
           then
                    rsync -avzR --delete $(dirname ${INO_FILE}) ${BACKUP_MACHINE_USER}@${BACKUP_MACHINE_IP}:${BACKUP_MACHINE_PATH} 
           fi

           #if [[ $INO_EVENT =~ 'ATTRIB' ]]
           #then
           #            if [ ! -d "$INO_FILE" ] 
           #            then
           #                rsync -avzcR $(dirname ${INO_FILE}) ${BACKUP_MACHINE_USER}@${BACKUP_MACHINE_IP}:${BACKUP_MACHINE_PATH}
           #            fi
           #fi
    done

}

#
# stop the backup service
#
function stop() {
    LOG "==================================================="
    LOG "Backup stop at:" `date +%H:%M:%S`" >_< "
    LOG "==================================================="
    #sudo killall $(basename "$0")
}

#
# backup all data 
#
function oneshot() {
    LOG "==================================================="
    LOG "Backup Oneshot start at:" `date +%H:%M:%S`
    LOG "==================================================="

    cd $MONITOR_DIR || { LOG 'MONITOR_DIR ERROR' && exit 1; }
    rsync -avzcR ./ ${BACKUP_MACHINE_USER}@${BACKUP_MACHINE_IP}:${BACKUP_MACHINE_PATH}          

    LOG "======================================================"
    LOG "Backup Oneshot Finished at:" `date +%H:%M:%S`
    LOG "======================================================"
}

#
# main
#
if [ ! $# -eq 1 ]; then
    echo "Usage: $0 [start|stop|oneshot]"
    exit 1;
fi

case $1 in
  start|stop|oneshot) "$1" ;;
  *) echo "Unkonwn param: $1"
     echo "Usage: $0 [start|stop|oneshot]"
esac

使用之前需要将本机的公钥拷贝到远程机器上，保证rsync能正常工作

$ ssh-copy-id remoteuser@remotemachine

然后修改变量，然后启动：

$./backup.sh start

使用systemctl让脚本开机启动

参考: http://unix.stackexchange.com/questions/47695/how-to-write-startup-script-for-systemd

可以使用systemctl将备份服务做成开机启动。

首先创建/etc/systemd/system/backup.service

sudo vim /etc/systemd/system/backup.service 

内容如下：

[Unit]
Description=BackupService

[Service]
ExecStart=/home/yourself/backup.sh start
ExecStop=/home/yourself/backupw.sh stop

[Install]
WantedBy=multi-user.target

使用systemctl让backup.service开机启动。

$ sudo systemctl enable backup.service

最后启动服务：

$ sudo systemctl start backup.service

参考

https://segmentfault.com/a/1190000002427568
http://unix.stackexchange.com/questions/47695/how-to-write-startup-script-for-systemd