NTPD同步失败问题

今天WPST出现了误差很高的问题，由于之前有过经验，我上来就检查了系统时间。果然发现WPST server的NTP时间跟simulator server的NTP时间有误差。WPST server从来没有同步过时间。跟同事确认，她在最开始安装的时候同步过，而这么长时间后，误差居然到了12秒，说明NTP时间误差在持续增大。那么我们应用的统计误差很高也可以解释了。

1. 首先检查ntp服务是否开启

   
   

    
    #ps -ef | grep ntp
   
   
   
   

    
    

   
   
   
   

    
    ntp      41939     1  0 14:46 ?        00:00:00 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -i /var/lib/ntp -I 10.170.65.54 -c /etc/ntp.conf
   
   

这里可以看到ntp服务其实是后台调/usr/sbin/ntpd命令。

2. 看ntp同步情况

#ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.170.65.54   .INIT.          16 -    - 1024    0    0.000    0.000   0.000

连续运行命令多次， 发现when的值一直是"-"，而不是递增数字。通常情况下，when值会一直增加到poll值，然后同步一次。

而这里一直不增加，说明同步有问题。而且reach为0，说明一次自动同步也没完成。

3. 首先我去检查了/etc/ntp.conf，检查同步机器是否对的。

发现有些restrict

#server 2001:1b70:200:105b:505::9
#server 2001:1b70:200:105b:1505::9
restrict default ignore
restrict -6 default ignore
restrict -6 2001:1b70:200:105b:505::9 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff nomodify notrap nopeer
restrict -6 2001:1b70:200:105b:1505::9 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff nomodify notrap nopeer
restrict -6 2001:1b70:200:1026::132:135 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff nomodify notrap nopeer
restrict -6 2001:1b70:200:1026::132:84 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff nomodify notrap nopeer
restrict -6 2001:1b70:200:1026::132:131 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff nomodify notrap nopeer
restrict -6 2001:1b70:200:1026::132:132 mask ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff nomodify notrap nopeer
restrict 127.0.0.1 nomodify notrap nopeer
restrict ::1 nomodify notrap nopeer

google了一把，得知这些restrict是使客户端连本机受限，只有目标网段客户端才能连上本机ntp server。那么显然对这个问题无关紧要，但是保险起见，都注掉了。而且去我们的ntp server那边也检查了ntp.conf，确保ntp server是绑定所有网卡的123UDP端口（NTP默认端口）。

4. 重启ntp

#cd /etc/init.d/
#./ntp stop
#./ntp status
#./ntp start

这里发现stop以后，进程一直在！说明stop有问题。那么我kill掉重新start，问题依旧。

5. 用debug模式看看

#cd /usr/sbin/
#ntpd -d

ntpd 4.2.4p8@1.1612-o Thu Nov 22 17:43:27 UTC 2012 (1)
addto_syslog: precision = 1.000 usec
addto_syslog: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
addto_syslog: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
addto_syslog: Listening on interface #1 wildcard, ::#123 Disabled
addto_syslog: Listening on interface #2 vlan6, 2001:1b70:206:1029::132:116#123 Enabled
addto_syslog: Listening on interface #3 vlan6, fe80::6e3b:e5ff:feba:ab#123 Enabled
addto_syslog: Listening on interface #4 vlan5, 2001:1b70:205:1029::132:116#123 Enabled
addto_syslog: Listening on interface #5 vlan4, 2001:1b70:204:1029::132:116#123 Enabled
addto_syslog: Listening on interface #6 vlan3, 2001:1b70:203:1029::132:116#123 Enabled
addto_syslog: Listening on interface #7 vlan2, 2001:1b70:202:1029::132:116#123 Enabled
addto_syslog: Listening on interface #8 vlan11, 2001:1b70:211:1029::132:116#123 Enabled
addto_syslog: Listening on interface #9 vlan1, 2001:1b70:201:1029::132:116#123 Enabled
addto_syslog: Listening on interface #10 vlan4, fe80::6e3b:e5ff:feba:a9#123 Enabled
addto_syslog: Listening on interface #11 eth4, fe80::6e3b:e5ff:feba:aa#123 Enabled
addto_syslog: Listening on interface #12 vlan2, fe80::6e3b:e5ff:feba:a8#123 Enabled
addto_syslog: Listening on interface #13 lo, ::1#123 Enabled
addto_syslog: Listening on interface #14 lo, 127.0.0.1#123 Enabled
addto_syslog: Listening on interface #15 lo, 127.0.0.2#123 Enabled
addto_syslog: Listening on interface #16 eth4, 10.175.132.116#123 Enabled
addto_syslog: Listening on interface #17 vlan3, 192.168.132.116#123 Enabled
local_clock: time 0 offset 0.000000 freq 0.000 state 0
addto_syslog: kernel time sync status 2040
peer_crypto_clear: at 0 next 0 assoc ID 7487
key_expire: at 0
peer_clear: at 0 next 1 assoc ID 7487 refid INIT
newpeer: 10.175.132.116->10.170.65.54 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000
addto_syslog: frequency initialized 0.000 PPM from /var/lib/ntp/drift/ntp.drift
local_clock: time 0 offset 0.000000 freq 0.000 state 1
report_event: system event 'event_restart' (0x01) status 'sync_alarm, sync_unspec, 1 event, event_unspec' (0xc010)
transmit: at 1 10.175.132.116->10.170.65.54 mode 3
auth_agekeys: at 1 keys 0 expired 0
timer: refresh ts 0
timer: interface update
receive: at 1 10.175.132.116<-10.170.65.54 mode 4 code 1 auth 0
peer 10.170.65.54 event 'event_reach' (0x84) status 'unreach, conf, 1 event, event_reach' (0x8014)
clock_filter: n 1 off -0.006533 del 0.000129 dsp 7.937501 jit 0.000001, age 0

看到连54的时候报 'unreach’ 事件。奇怪了……

6. 测试ntp server 是否真的连不通

#ping 10.170.65.54

结果是好的……囧

7. 检查/etc/init.d/ntp 文件，该文件是把ntp配成系统服务所读的文件

...
startproc $NTPD_BIN -p ${NTPD_PID_NOPREFIX} $NTPD_OPTIONS -I 10.170.65.54 -c $NTP_CONF 
...

里面有段" -I 10.170.65.54"，这个-I参数应该是本机地址，而不是目标机器地址。将这里改对，或者直接删除-I参数。

kill掉ntp进程，rcntp start或#/etc/init.d/./ntp start 启动ntp服务。

再次出现无语的事情。

./ntp status 看到状态是

#./ntp status
Checking for network time protocol daemon (NTPD):                                           unused

而用ntpq -p检查状态却是

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*10.170.65.54    10.175.250.66    6 u   43   64  37    0.117   -8.719   2.853

说明同步已经正常，并且ntp server已被认证为基准同步server。

不过，到这里，同步问题已经解决了，看到reach值知道已经同步了37次。

其实，在最开始ps的时候就应该可以看出问题了。看来看问题还是不够仔细啊。