rsync all_in_one

#rsync configuration and usage in detail
#v 0.2
#11/13/2007
#v 0.1
#10/19/2007
#sense


Test platform:
RHEL5
Thinkpad R60
VMware Server 1.0.4 build-56528/RHEL5

Announcement:
The picture in Overview part and the shell script at the end of this document has maked reference to http://www.linuxfocus.org/English/March2004/article326.shtml.
Thanks to Brian Hone.
More information please read:
http://rsync.samba.org/ftp/rsync/rsync.html
http://rsync.samba.org/ftp/rsync/rsyncd.conf.html


Overview:
Rsync is a small and exquisite backup tool.Using rsync,you can easily create an up-to-date and powerful backup architecture over lots of transfer protocols,such as ssh,smb,nfs etc.

figure1/2 shows that we can use lots of transfer protocols as our basic backup media.all things in the rsync server can be backuped to the Backup Server's "Main Backup Dir",and those old files or directories that already being in the "Main Backup Dir" can be copied to an incremental directory called "Incremental Dirs" which name is make up of "YY-MM-DD".
the "Incremental Dirs" give us much more flexibility to manage backuped files,when we search for a langsyned files or directories.





rsync server end:
# vi /etc/rsyncd.conf
==============================================================
uid = nobody
gid = nobody
use chroot = no
max connections = 4
syslog facility = local5
pid file = /var/run/rsyncd.pid
motd file = /etc/rsyncd.motd

[vir]
        path = /home/sense/document/Linux/virtualization
        comment = virtualization doc

[trouble]
        path = /home/sense/document/Linux/troubleshooting
        comment = trouble doc
        auth users = sense, cluster1
        secrets file = /etc/rsyncd.secrets
===============================================================
# vi /etc/rsyncd.secrets
======================
sense:123456
cluster1:654321
======================
# vi /etc/rsyncd.motd
================================
****************************
welcome to ssn sync server
****************************
================================

# iptables -I INPUT -p tcp --dport 873 -j ACCEPT
# rsync --daemon



rsync client end:
1>rsync server file list:
# rsync --list-only 192.168.2.1::
****************************
welcome to ssn sync server
****************************

vir        virtualization doc
trouble        trouble doc
# rsync --list-only 192.168.2.1::vir
****************************
welcome to ssn sync server
****************************

drwxrwxr-x        4096 2007/10/07 18:36:33 .
-rwxr-xr-x       46592 2000/09/03 21:10:54 Qemu.doc
-rw-rw-r--     1439442 2007/09/26 15:20:32 Virtualization Guide5.pdf
-rw-rw-r--        7117 2007/10/01 19:08:11 qemu
# rsync --list-only sense@192.168.2.1::trouble
****************************
welcome to ssn sync server
****************************

Password:
drwxrwxr-x        4096 2007/09/30 10:51:28 .
-rwxr-xr-x      454013 2007/08/17 21:43:32 linux_rescue.pdf

[Note]
the permissions of rsync server's rsyncd.secrets file must not be readable by other users,or you will get this warning evenif you submit the right key:
server:
# ll /etc/rsyncd.secrets
-rw-r--r-- 1 root root 29 Oct 19 10:44 /etc/rsyncd.secrets
client:
#rsync --list-only sense@192.168.2.1::trouble
****************************
welcome to ssn sync server
****************************

Password:
@ERROR: auth failed on module trouble
rsync error: error starting client-server protocol (code 5) at main.c(1296) [receiver=2.6.8]

server:
# chmod 600 /etc/rsyncd.secrets
#ll /etc/rsyncd.secrets
-rw------- 1 root root 29 Oct 19 10:44 /etc/rsyncd.secrets
client:
#rsync --list-only sense@192.168.2.1::trouble
****************************
welcome to ssn sync server
****************************

Password:
drwxrwxr-x        4096 2007/09/30 10:51:28 .
-rwxr-xr-x      454013 2007/08/17 21:43:32 linux_rescue.pdf




2>remote rsync server file backup：
# rsync $parameters source destination
parameters explanation:
-a        leave everything originally.
-v        generate an verbose output information.
-z        compress file data during the transfer
-P        show progress during transfer,keep partially transferred files(so that we can transfer these partial files continuously next time.)

# rsync -avzP 192.168.2.1::vir vir
****************************
welcome to ssn sync server
****************************

receiving file list ...
4 files to consider
created directory vir
./
Qemu.doc
           0   0%    0.00kB/s    0:00:00       46592 100%   14.81MB/s    0:00:00 (xfer#1, to-check=2/4)
Virtualization Guide5.pdf
           0   0%    0.00kB/s    0:00:00     1439442 100%    5.40MB/s    0:00:00 (xfer#2, to-check=1/4)
qemu
           0   0%    0.00kB/s    0:00:00        7117 100%   27.26kB/s    0:00:00 (xfer#3, to-check=0/4)

sent 140 bytes  received 1072382 bytes  2145044.00 bytes/sec
total size is 1493151  speedup is 1.39

[second time backup]
# rsync -avzP 192.168.2.1::vir vir
****************************
welcome to ssn sync server
****************************

receiving file list ...
4 files to consider

sent 68 bytes  received 275 bytes  228.67 bytes/sec
total size is 1493151  speedup is 4353.21

# rsync -avzP sense@192.168.2.1::trouble trouble
****************************
welcome to ssn sync server
****************************

receiving file list ...
2 files to consider
created directory trouble
./
linux_rescue.pdf
           0   0%    0.00kB/s    0:00:00      454013 100%    2.64MB/s    0:00:00 (xfer#1, to-check=0/2)

sent 133 bytes  received 422839 bytes  76904.00 bytes/sec
total size is 454013  speedup is 1.07

[second time backup]
# rsync -avzP sense@192.168.2.1::trouble trouble
****************************
welcome to ssn sync server
****************************

receiving file list ...
2 files to consider

sent 105 bytes  received 253 bytes  55.08 bytes/sec
total size is 454013  speedup is 1268.19






Case Study:server backup implement
rsync backup basic syntax:
#rsync OPTIONS BACKUPDIR(from remote rsync server) BACKUPtoDIR(local backup directory)

backup command:
# rsync --force --ignore-errors --delete --delete-excluded --exclude-from=$EXCLUDE_file --backup --backup-dir=$BACKUPtoDIR/`date +%Y-%m-%d` --password-file=$PASS_file -av $BACKUPDIR $BACKUPtoDIR/mainbackup/
Explain:
--force            force deletion of dirs even if not empty
--ignore-errors        delete even if there are I/O errors
--delete        delete extraneous files from dest dirs
--delete-excluded    also delete excluded files from dest dirs
--exclude-from        read exclude patterns from FILE
--backup        make backups when needs to overlay the old one(incremental backup)
--backup-dir        make backups into hierarchy based in DIR
--password-file        read access password from $PASS_file
-a            archive mode; means:leave everything originally. equals -rlptgoD(recurse,copy symlinks,preserve permissions,preserve modification times,preserve group,preserve owner (implement this command as super-user only,or it will lost),same as --devices --specials{preserve device files (super-user only),preserve special files})
-v            verbose output

command explain:
1>backup all things in $BACKUPDIR on the remote rsync server to the local $BACKUPtoDIR/mainbackup/,and leave everything originally.
2>if the local $BACKUPtoDIR/mainbackup directory already have these files or directories,backup the local one to the $BACKUPtoDIR/`date +%Y-%m-%d`,and then implement incremental backup.
3>delete all things that not exist in the $BACKUPDIR from local $BACKUPtoDIR/mainbackup/,evenif these are directories and not emplty.
4>don't backup files or directories which has been specified in the $EXCLUDE_file.
5>generate an verbose output information.

Examples:
# echo "password" > /root/1
# rsync --force --ignore-errors --delete --delete-excluded --backup --backup-dir=/root/test/`date +%y-%m-%d` --password-file=/root/1 -av sense@192.168.2.1::trouble /root/test/main/
****************************
welcome to ssn sync server
****************************

receiving file list ... done
deleting 07-10-19/07-10-19/vir/qemu
deleting 07-10-19/07-10-19/vir/Virtualization Guide5.pdf
deleting 07-10-19/07-10-19/vir/Qemu.doc
deleting 07-10-19/07-10-19/vir/
deleting 07-10-19/07-10-19/trouble/ss/test
deleting 07-10-19/07-10-19/trouble/ss/
deleting 07-10-19/07-10-19/trouble/linux_rescue.pdf
deleting 07-10-19/07-10-19/trouble/
deleting 07-10-19/2

sent 154 bytes  received 294 bytes  896.00 bytes/sec
total size is 5696893  speedup is 12716.28



cron implemention:
make upstair command into a shell script-backup.sh
# crontab -e
0 23 * */1 * /root/backup.sh



referenced shell script from www.linuxfocus.org:
#!/bin/sh

#########################################################
# Script to do incremental rsync backups
# Adapted from script found on the rsync.samba.org
# Brian Hone 3/24/2002
# This script is freely distributed under the GPL
#########################################################

##################################
# Configure These Options
##################################

###################################
# mail address for status updates
#  - This is used to email you a status report
###################################
MAILADDR=your_mail_address_here

###################################
# HOSTNAME
#  - This is also used for reporting
###################################
HOSTNAME=your_hostname_here

###################################
# directory to backup
# - This is the path to the directory you want to archive
###################################
BACKUPDIR=directory_you_want_to_backup

###################################
# excludes file - contains one wildcard pattern per line of files to exclude
#  - This is a rsync exclude file.  See the rsync man page and/or the
#    example_exclude_file
###################################
EXCLUDES=example_exclude_file

###################################
# root directory to for backup stuff
###################################
ARCHIVEROOT=directory_to_backup_to

#########################################
# From here on out, you probably don't  #
#   want to change anything unless you  #
#   know what you're doing.             #
#########################################

# directory which holds our current datastore
CURRENT=main

# directory which we save incremental changes to
INCREMENTDIR=`date +%Y-%m-%d`

# options to pass to rsync
OPTIONS="--force --ignore-errors --delete --delete-excluded /
 --exclude-from=$EXCLUDES --backup --backup-dir=$ARCHIVEROOT/$INCREMENTDIR -av"

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# make sure our backup tree exists
install -d $ARCHIVEROOT/$CURRENT

# our actual rsyncing function
do_rsync()
{
   rsync $OPTIONS $BACKUPDIR $ARCHIVEROOT/$CURRENT
}

# our post rsync accounting function
do_accounting()
{
   echo "Backup Accounting for Day $INCREMENTDIR on $HOSTNAME:">/tmp/rsync_script_tmpfile
   echo >> /tmp/rsync_script_tmpfile
   echo "################################################">>/tmp/rsync_script_tmpfile
   du -s $ARCHIVEROOT/* >> /tmp/rsync_script_tmpfile
   echo "Mail $MAILADDR -s $HOSTNAME Backup Report < /tmp/rsync_script_tmpfile"
   Mail $MAILADDR -s $HOSTNAME Backup Report < /tmp/rsync_script_tmpfile
   echo "rm /tmp/rsync_script_tmpfile"
   rm /tmp/rsync_script_tmpfile
}

# some error handling and/or run our backup and accounting
if [ -f $EXCLUDES ]; then
    if [ -d $BACKUPDIR ]; then
        # now the actual transfer
        do_rsync && do_accounting
    else
        echo "cant find $BACKUPDIR"; exit
    fi
    else
        echo "cant find $EXCLUDES"; exit
fi