数据同步——rsync远程同步

rsync同步简介

关于rsync

■ 一款增量备份工具

Remote Sync,远程同步
支持本地复制,或者与其他SSH、rsync主机同步
官方网站: http://rsync.samba.org

配置rsync备份源

配置rsync源服务器

■ rsync同步源

指备份操作的远程服务器,也称备份源

配置rsync源

■ 基本思路

建立rsyncd.conf配置文件、独立的帐号文件
启用rsync的 --daemon模式

■ 应用示例

用户backuper,允许下行同步
操作目录为 /var/www/html(安装HTTP)

■ 配置文件rsyncd.conf

需手动建立,语法类似于Samba配置
认证配置auth users、secrets file,不加则为匿名

■ rsync帐号文件

采用 “用户名:密码” 的记录格式,每行一个用户记录
独立的账号数据,不依赖于系统账号

■ 启用rsync服务

通过 --daemin独自提供服务

使用rsync备份工具

■ rsync命令的用法

rsync [选项] 原始位置 目标位置

■ 常用选项

-a:归档模式,递归并保留对象属性
-v:显示同步过程的详细信息
-z:在传输文件时进行压缩
-H保留硬链接文件
-A:保留ACL属性信息
–delete:删除目标位置而原始位置没有的文件
–checksum:根据对象的校验和来决定是否跳过文件

■ 配置源的两种表示方法

格式1:用户名@主机地址::共享模块名
格式2:rsync://用户名@主机地址/共享模块名


格式一:
[root@localhost etc]# rsync backuper@20.0.0.3::wwwroot /opt/

格式二:
[root@localhost etc]#  rsync -avz rsync://backuper@20.0.0.3::wwwroot /opt/

rsync+inotify结合使用

rsync远程同步实验操作

要先关闭防火墙和核心防护
[root@localhost ~]# systemctl stop  firewalld  ###关闭防火墙 
[root@localhost ~]# vim /etc/selinux/config    ###关闭核心防护
SELINUX=disabled
[root@localhost ~]# iptables -F    ###清空防火墙规则
[root@localhost ~]# setenforce 0   ###临时关闭核心防护

####配置rsync源服务器####

uid = nobody
gid = nobody
use chroot = yes  ###禁锢家目录
address = 20.0.0.3   ###监听地址
port 873                      ###端口号
log file = /var/log/rsyncd.log   ###指定日志文件
pid file = /var/run/rsyncd.pid   ###开启pid
hosts allow = 20.0.0.0/24          ###运行哪些网段可以访问(白名单)


[wwwroot]                     ###共享模块名(自定义的)
path = /var/www/html    ###共享站点目录
comment = www.zk.cn  ###描述性信息,可以随便写
read only = no                 ###只读模式
dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2  ###里面这些格式不进行压缩(因为格式本来就是压缩格式)
auth users = backuper      ###创建一个身份验证的账户
secrets file = /etc/reyncd_users.db   ###配置了身份验证账户,那要有密码,去这里配置密码

[root@localhost ~]#vim  /etc/reyncd_users.db   ###添加密码
backuper:abc123

[root@localhost ~]#  chmod 600 /etc/reyncd_users.db   ###给读的权限

[root@localhost ~]# yum -y install httpd  ###安装一下apache,因为需要/var/www/html文件

[root@localhost ~]# reync --daemon   ###启动服务
[root@localhost ~]# rsync -antp |grep 873    ###过滤一下端口号看看起来了没

发起端配置

发起端:
格式一:
[root@localhost etc]# rsync backuper@20.0.0.3::wwwroot /opt/   ###进行同步
格式二:
[root@localhost etc]#  rsync -avz rsync://backuper@20.0.0.3::wwwroot /opt/    ###进行同步

免交互:
[root@localhost ]#cd /etc/  
[root@localhost etc]# touch server.pass  ###创建存密码文件
[root@localhost etc]#  vim server.pass     ###往里面写密码
abc123
[root@localhost etc]#  chmod 600 /etc/server.pass  ###给读的权限

[root@localhost etc]# rsync -avz --delete --password-file=/etc/server.pass   backuper@20.0.0.3::wwwroot /opt/  ###面交互式同步

发起端可以做计划任务定期同步

[root@localhost ]# crontab -e  ###可以用计划任务定期同步

rsync+inotify实验操作

发起端配置

[root@localhost ~]# vim /etc/sysctl.conf  ###调整inotify内核参数,在最后添加这三段
fs.inotify.max_queued_events = 16384     ###监控事件队列大小
fs.inotify.max_user_instances = 1024        ###最多监控实例数
fs.inotify.max_user_watches = 1048576    ###每个实例最多监控文件数,监控数应大于监控目标的总文件数

[root@localhost ~]# sysctl -p               ###刷新生效一下
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

将inotify-tools-3.14.tar工具上传上来
[root@localhost abc]# tar zxvf inotify-tools-3.14.tar.gz -C /opt/   ###将工具解压到/opt目录下

[root@localhost inotify-tools-3.14]# yum -y install gcc gcc-c++ make  ###安装一下gcc编译器

[root@localhost ~]# cd /opt/inotify-tools-3.14/                       ###到这个目录下
[root@localhost inotify-tools-3.14]# ./configure                      ###执行
[root@localhost inotify-tools-3.14]# make && make install   ###编译安装

[root@localhost ~]# yum -y install httpd   ###安装一下apache
[root@localhost ]# chmod 777 /var/www/html/    ###目录给个777权限

inotifywait  -mrq  -e  modify,create,move,delete  /var/www/html     ### -mrq:持续性监控,-e:操作,modify:修改,create:创建,move:移动,delete:删除  ,监控本地的/var/www/html

敲了上面的监控命令就会持续监控不能操作,需要在开一个发起端操作字符页面

进入到另开的发起端操作字符界面

[root@localhost ~]# cd /var/www/html/  ###进入到html目录
[root@localhost html]# touch abc    ###创建一个abc

------然后刚刚敲监控那边就会出现操作提示-----
/var/www/html/ CREATE abc

没问题的话,我们就回刚刚监控的发起端,ctrl+c或ctrl+z结束掉监控

[root@localhost ~]# cd /opt/   
[root@localhost opt]# vim inotify.sh  
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html backuper@20.0.0.3::wwwroot"
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if  [ $(pgrep rsync | wc -l) -le 0 ] ; then
         $RSYNC_CMD
    fi
done

[root@localhost opt]# chmod +x inotify.sh   ###给个执行权限


[root@localhost ]# ls -l /var/www/  ###用这条命令查看两边都查看一下html权限给足了没

在源端在调一个数据
前面配置的时候已经是no了,就不用操作这几步了,直接跳到下面的发起端启动脚本就好

[root@localhost ~]# vim /etc/rsyncd.conf  
read only = no   ###需要把wwwroot模块的只读模式改成no

[root@localhost ~]# netstat -ntap | grep rsync  ###虽然改了配置,但是还没重启,查看一下进程
tcp        0      0 20.0.0.3:873            0.0.0.0:*               LISTEN      20491/rsync  
 
[root@localhost ~]# kill -9 20491                        ###杀死进程
[root@localhost ~]# netstat -ntap | grep rsync   ###在查看一下应该就没有了

[root@localhost ~]# cd /var/run/                ###去这个目录下
[root@localhost run]# rm -rf rsyncd.pid     ###删掉这个pid
[root@localhost run]# rsync --daemon      ###启动一下就行

[root@localhost run]# netstat -ntap | grep rsync   ###在看一下就又有了
tcp        0      0 20.0.0.3:873            0.0.0.0:*               LISTEN      70903/rsync  

---------------------------发起端启动脚本----------------------

[root@localhost opt]# ./inotify.sh 

--------------需要在开一个发起端操作字符页面------------------

[root@localhost ~]# cd /var/www/html/
[root@localhost html]# echo "this is test" > test.txt

---------------------监控那边的发起端就会出现提示----------------------

[root@localhost opt]# ./inotify.sh  ###这个报错不影响同步
rsync: chgrp "/html" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/html/.test.txt.at3wRj" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]

--------------------源端查看一下是否同步过去了------------------------

[root@localhost run]# cd /var/www/html/
[root@localhost html]# cd html
[root@localhost html]# ls
test.txt

rsync的同步操作和rsync+inotify结合使用操作完成

Rsync服务常见问题汇总讲解

客户端的错误现象:No route to host

 rsync服务端开启的iptables防火墙

   [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

   rsync: failed to connect to 172.16.1.41: No route to host (113)

   rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6]

异常问题解决:

   关闭rsync服务端的防火墙服务(iptables)

   [root@backup mnt]# /etc/init.d/iptables stop

   iptables: Setting chains to policy ACCEPT: filter          [  OK  ]

   iptables: Flushing firewall rules:                         [  OK  ]

   iptables: Unloading modules:                               [  OK  ]

   [root@backup mnt]# /etc/init.d/iptables status

   iptables: Firewall is not running.

==============================================================================================

ERROR: The remote path must start with a module name not a /
rsync客户端执行rsync命令错误:

客户端的错误现象:

   [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::/backup

   ERROR: The remote path must start with a module name not a /

   rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6]

异常问题解决:

   rsync命令语法理解错误,::/backup是错误的语法,应该为::backup(rsync模块)

==============================================================================================

@ERROR: auth failed on module backup
@ERROR: auth failed on module oldboy

客户端的错误现象:

   [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

Password:

@ERROR: auth failed on module backup

rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6]

异常问题解决:

   1. 密码真的输入错误,用户名真的错误

   2. secrets file = /etc/rsync.password指定的密码文件和实际密码文件名称不一致

   3. /etc/rsync.password文件权限不是600

   4. rsync_backup:123456密码配置文件后面注意不要有空格

   5. rsync客户端密码文件中只输入密码信息即可,不要输入虚拟认证用户名称

==============================================================================================

@ERROR: Unknown module ‘backup’

4. Unknown module 'backup'  

[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

@ERROR: Unknown module 'backup'

rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6]

异常问题解决:

   1、 /etc/rsyncd.conf配置文件模块名称书写错误

   2、配置文件中网段限制不对

==============================================================================================

Permission denied

[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

Password:

sending incremental file list

hosts

rsync: mkstemp ".hosts.5z3AOA" (in backup) failed: Permission denied (13)

 

sent 196 bytes  received 27 bytes  63.71 bytes/sec

total size is 349  speedup is 1.57

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]  

异常问题解决:

   1. 共享目录的属主和属组不正确,不是rsync

   2. 共享目录的权限不正确,不是755

==============================================================================================

chdir failed

[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

Password:

@ERROR: chdir failed

rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6]

异常问题解决:

    1. 备份存储目录没有建立

    2. 建立的备份存储目录和配置文件定义不一致

    [root@backup backup]# /etc/init.d/xinetd restart

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

Stopping xinetd:                                           [  OK  ]

Starting xinetd: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
                                                                     [  OK  ]

   说明:如果没有备份存储目录,xinetd服务都不能正确启动

==============================================================================================

invalid uid rsync

[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

Password:

@ERROR: invalid uid rsync

rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6]
异常问题解决:

rsync服务对应rsync虚拟用户不存在了

==============================================================================================

客户端已经配置了密码文件,但免秘钥登录方式,依旧需要输入密码
password file must not be other-accessible

[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password

password file must not be other-accessible

continuing without password file

Password:

sending incremental file list

sent 26 bytes  received 8 bytes  5.23 bytes/sec

total size is 349  speedup is 10.26
异常问题解决:

rsync客户端的秘钥文件也必须是600权限

==============================================================================================

rsync客户端连接慢问题
错误日志输出

2017/03/08 20:14:43 [3422] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors

2017/03/08 20:14:43 [3422] name lookup failed for 172.16.1.31: Name or service not known

2017/03/08 20:14:43 [3422] connect from UNKNOWN (172.16.1.31)

2017/03/08 20:14:43 [3422] rsync to backup/ from rsync_backup@unknown (172.16.1.31)

2017/03/08 20:14:43 [3422] receiving file list

2017/03/08 20:14:43 [3422] sent 76 bytes  received 83 bytes  total size 349

正确日志输出

2017/03/08 20:16:45 [3443] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors

2017/03/08 20:16:45 [3443] connect from nfs02 (172.16.1.31)

2017/03/08 20:16:45 [3443] rsync to backup/ from rsync_backup@nfs02 (172.16.1.31)

2017/03/08 20:16:45 [3443] receiving file list

2017/03/08 20:16:45 [3443] sent 76 bytes  received 83 bytes  total size 349
异常问题解决:

查看日志进行分析

==============================================================================================

rsync服务没有正确启动Connection refused (111)

 [root@oldboy-muban ~]#  rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

rsync: failed to connect to 172.16.1.41: Connection refused (111)

rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6]

解决 rsync服务没开启

[root@oldboy-muban ~]# rsync --daemon

[root@oldboy-muban ~]# ss -lntup |grep rsync

tcp    LISTEN     0      5                     :::873                  :::*      users:(("rsync",1434,5))

tcp    LISTEN     0      5                      *:873                   *:*      users:(("rsync",1434,4))

[root@oldboy-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup

Password:

sending incremental file list

hosts

 sent 196 bytes  received 27 bytes  49.56 bytes/sec

total size is 349  speedup is 1.57

==============================================================================================

11 port 22: Connection refused
环境:本地服务器集群内部传输利用远程ssh 报错

利用(telnet 172.16.1.31 22) 排查服务监听状态后采取的解决方法

[root@oldboy-muban ~]# rsync /etc/hosts 172.16.1.31:/tmp

ssh: connect to host 172.16.1.31 port 22: Connection refused

rsync: connection unexpectedly closed (0 bytes received so far) [sender]

rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

排错思路:

[root@oldboy-muban ~]# ping 172.16.1.31

PING 172.16.1.31 (172.16.1.31) 56(84) bytes of data.

64 bytes from 172.16.1.31: icmp_seq=1 ttl=64 time=0.628 ms

64 bytes from 172.16.1.31: icmp_seq=2 ttl=64 time=0.393 ms

64 bytes from 172.16.1.31: icmp_seq=3 ttl=64 time=1.06 ms

64 bytes from 172.16.1.31: icmp_seq=4 ttl=64 time=0.745 ms

 

[root@oldboy-muban ~]# traceroute 172.16.1.31

traceroute to 172.16.1.31 (172.16.1.31), 30 hops max, 60 byte packets

 1  nfs01 (172.16.1.31)  0.597 ms  0.189 ms  0.965 ms

/etc/init.d/iptables status

iptables: Firewall is not running.

[root@backup ~]#

[root@backup ~]# netstat -lntup|grep 22

  p        0      0 10.0.0.31:22                0.0.0.0:*                   LISTEN      1187/sshd   

故障原因:无法连接

telnet 172.16.1.31 22

解决方法:

[root@oldboy-backup-41]# vim /etc/ssh/sshd_config

#Port 22

#AddressFamily any

#ListenAddress 10.0.0.31 改为 0.0.0.0

#ListenAddress ::

总结:内网传输通过SSH pro 22 表明22端口链接不上

==============================================================================================

12 --passwd-file=/etc/rsync.passwd: unknown option 没有正确输入password文件名
报错:–passwd-file=/etc/rsync.passwd: unknown option

错误案例

本地rsync.password 文件要保持一致缺少字母都会报错

echo "123456">>/etc/rsync.passwd

[root@nfs01 ~]# chmod 600 /etc/rsync.passwd

[root@nfs01 ~]# ll /etc/rsync.passwd

-rw------- 1 root root 7 Mar  9 13:47 /etc/rsync.passwd

[root@nfs01 ~]# rsync  -az -P /root/ rsync_backup@172.16.1.41::backup --passwd-file=/etc/rsync.passwd

rsync: --passwd-file=/etc/rsync.passwd: unknown option

rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]

正确做法:

[root@nfs01 ~]# echo "123456">>/etc/rsync.password

[root@nfs01 ~]# chmod 600 /etc/rsync.password

[root@nfs01 ~]# ll /etc/rsync.password

-rw------- 1 root root 7 Mar  9 13:49 /etc/rsync.password

rsync  -az -P /server/files/secure-20161219  rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password

sending incremental file list

secure-20161219

    51053780 100%   14.31MB/s    0:00:03 (xfer#1, to-check=0/1)

rsync: mkstemp ".secure-20161219.lcnuWA" (in backup) failed: Permission denied (13)

 

sent 2210982 bytes  received 27 bytes  491335.33 bytes/sec

total size is 51053780  speedup is 23.09

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

 

[root@backup ~]# ls /backup/

100.log          cc.txt       optimize-init_sys.sh

anaconda-ks.cfg

      1)可能是服务没有开启

      2)iptables SELinux

      3)本次遇见sshd传输受限 限制了传输的ip(安全)

==============================================================================================

Rsync服务端排错思路

查看rsync服务配置文件路径是否正确 /etc/rsyncd.conf

查看配置文件例的host allow,host deny,允许的ip网段是否是允许客户端访问的ip网段

查看配置文件中path参数里的路径是否存在,权限是否正确(正常应为配置文件中的UUID参数对应的属主和组)

查看rsync服务是否启动,端口是否存在 ps -ef  netstat -lntup

查看iptables防火墙和SELinux是否开启允许rsync服务通过,也可以关闭

查看服务端rsync配置文件里的密码权限是否为600 密码文件格式是否正确,正确格式(用户名:密码)文件路径和配置文件里的secrect files 参数对应

如果是推送数据,要查看,配置rsyncd.conf 文件中用户是否对模块下目录有可读的权限

==============================================================================================

客户端排错思路

查看客户端rsync配置的密码文件是否为600的权限,密码文件格式是否正确,注意:仅需要有密码,并且和服务端的密码一致

用telnet链接rsync服务器ip地址873端口,查看服务是否启动(可测试服务端防火墙是否阻挡telnet10.0.0.100 873)

客户端执行命令是 rsync -avzP rsync_backup@10.0.0.100::backup/test/test/ --password-file=/etc/rsync.password

此命令要记清楚尤其10.0.0.100::backup/test/处的双引号及随后的backup为模块名称
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值