I want to copy (rsync to remote server) a directory tree whenever file uploaded or deleted in /var/www/html/upload/ directory under Linux operating systems for backup purpose and/or load balancing purpose without getting into complex file sharing setup such as NFS or GFS iscsi storage. How do I monitor /var/www/html/upload/ and its subdirectory for new files and executes rsync command to make copy back to www2.example.com:/var/www/html/upload/?
inotify is an inode-based filesystem notification technology. It provides possibility to simply monitor various events on files in filesystems. It is a very much powerful replacement of (obsolete) dnotify. inotify brings a comfortable way how to manage files used in your applications.
The incrond (inotify cron daemon) is a daemon which monitors filesystem events (such as add a new file, delete a file and so on) and executes commands or shell scripts. It’s use is generally similar to cron .
Install incron
Type the following command under RHEL / Fedora / CentOS Linux:$ sudo yum install incron
Type the following command under Debian / Ubuntu Linux:$ sudo apt-get install incron
Configuration Files
- /etc/incron.conf - Main incron configuration file
- /etc/incron.d/ - This directory is examined by incrond for system table files. You should put all your config file here as per directory or domain names.
- /etc/incron.allow - This file contains users allowed to use incron.
- /etc/incron.deny - This file contains users denied to use incron.
- /var/spool/incron - This directory is examined by incrond for user table files which is set by users running the incrontab command.
incron Syntax
The syntax is as follows:
<directory> <file change mask> <command or action> options /var/www/html IN_CREATE /root/scripts/backup.sh /sales IN_DELETE /root/scripts/sync.sh /var/named/chroot/var/master IN_CREATE,IN_ATTRIB,IN_MODIFY /sbin/rndc reload
Where,
- <directory> - It is nothing but path which is an absolute filesystem path such as /home/data. Any changes made to this path will result into command or action.
- <file change mask> - Mask is is nothing but various file system events such as deleting a file. Each event can result into command execution. Use the following masks:
- IN_ACCESS - File was accessed (read)
- IN_ATTRIB - Metadata changed (permissions, timestamps, extended attributes, etc.)
- IN_CLOSE_WRITE - File opened for writing was closed
- IN_CLOSE_NOWRITE - File not opened for writing was closed
- IN_CREATE - File/directory created in watched directory
- IN_DELETE - File/directory deleted from watched directory
- IN_DELETE_SELF - Watched file/directory was itself deleted
- IN_MODIFY - File was modified
- IN_MOVE_SELF - Watched file/directory was itself moved
- IN_MOVED_FROM - File moved out of watched directory
- IN_MOVED_TO - File moved into watched directory
- IN_OPEN - File was opened
- The IN_ALL_EVENTS symbol is defined as a bit mask of all of the above events.
- <command or action> - Run command or scripts when mask matched on given directory.
- options - It can be any one of the following with command (i.e. you can pass it as arg to your command):
- $$ - dollar sign
- $@ - watched filesystem path (see above)
- $# - event-related file name
- $% - event flags (textually)
- $& - event flags (numerically)
Turn Service On
Type the following command:# service incrond start
# chkconfig incrond on
Examples:
Type the following command to edit your incrontabincrontab -e
Run logger command when file created or deleted from /tmp directory:/tmp IN_ALL_EVENTS logger "/tmp action for $# file"
Save and close the file. Now cd to /tmp and create a file:$ cd /tmp
$ >foo
$ rm foo
To see message, enter:$ sudo tail -f /var/log/messages
Sample outputs:Jul 17 18:39:25 vivek-desktop logger: "/tmp action for foo file"
How Do I Run Rsync Command To Replicate Files For /var/www/html/upload Directory?
Type the following command:# incrontab -e
Append the following command:/var/www/html/upload/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/
Now, wherever files are uploaded in /var/www/html/upload/ directory, rsync will be executed to sync files to www2.example.com server. Make sure ssh keys are set for password less login.
How Do I Monitor /var/www/html/upload/ and Its Subdirectories Recursively?
You cannot monitor /var/www/html/upload/ directory recursively. However, you can use the find command to add all sub-directories as follows:find /var/www/html/upload -type d -print0 | xargs -0 -I{} echo "{} IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/" > /etc/incron.d/webroot.conf
This will create /etc/incron.d/webroot.conf config as follows:
/var/www/html/upload IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/css IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/1 IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/js IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/3 IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/2010 IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/2010/11 IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/2010/12 IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/2 IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/files IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/ /var/www/html/upload/images IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync --exclude '*.tmp' -a /var/www/html/upload/ user@www2.example.com:/var/www/html/upload/
How Do I Troubleshoot Problems?
You need to see /var/log/cron log file:# tail -f /var/log/cron
# grep something /var/log/cron