实例重启 - ORA-27302: failure occurred at: sskgxpsnd1

Oracle 10.2.0.4 ,  Linux AS 5.4 , 64bit , 4 Nodes .  


程序员匆忙更新了一个function 中的SQL, 要求一个dba同事更新到production , 结果这两位老兄都没有测试就 直接上去了，
更新的东西也非常简单，就是下面的红色order by 部分 。 可是SQL中select 后面没有这个install_date 字段，结果出现
问题 。  奇怪的是， 节点3 在出现问题的时候居然 实例重新启动了， 不过server没有reboot,   查看了一下日志， 好像是
心跳网路问题， 但是找不到相关的资料 。  仅仅查看到了类似的文章，不过不是太吻合 ：

Troubleshooting ORA-27300 ORA-27301 ORA-27302 errors [ID 579365.1]  



修改加入order by 的SQL语句：  

SELECT  DISTINCT  bb.part_no,
                                 CASE
                                    WHEN bb.request_qty > 0
                                       THEN TO_CHAR (bb.unit_qty)
                                    ELSE '0'
                                 END AS part_count,
                                 'N/A' AS next_group,
                                 cc.component_sn AS ppid
                            FROM wip_d_sn_master aa,
                                 wip_d_wo_detail bb,
                                 wip_d_csn_master cc
                           WHERE aa.work_order = bb.work_order
                             AND aa.serial_number = v_sn
                             AND aa.serial_number = cc.ssn
                             -- AND bb.work_order = cc.work_order
                             AND bb.part_no = cc.part_no
                             AND cc.csn_status = '2'                           
                ORDER BY cc.install_date






节点3上的alert log :

Thu Feb 10 14:20:28 2011
Thread 3 advanced to log sequence 32222 (LGWR switch)
  Current log# 42 seq# 32222 mem# 0: /ocfs_ctrl_redo/mxdell/redo42_a.log
  Current log# 42 seq# 32222 mem# 1: /ocfs_data/mxdell/redo42_b.log
Thu Feb 10 14:35:02 2011
IPC Send timeout detected.Sender: ospid 31278
Receiver: inst 4 binc 429414865 ospid 12933
Thu Feb 10 14:35:03 2011
Errors in file /u01/product/admin/mxdell/udump/mxdell3_ora_31278.trc:
ORA-27508: IPC error sending a message
ORA-27300: OS system dependent operation:sendmsg failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: sskgxpsnd1
Thu Feb 10 14:35:03 2011
IPC Send timeout to 3.2 inc 32 for msg type 32 from opid 162
Thu Feb 10 14:35:03 2011
Communications reconfiguration: instance_number 4
Thu Feb 10 14:35:03 2011
Trace dumping is performing id=[cdmp_20110210143503]
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lmon_12757.trc:
ORA-29740: evicted by member 2, group incarnation 34
Thu Feb 10 14:36:48 2011
LMON: terminating instance due to error 29740
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms0_12762.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
LMON: terminating instance due to error 29740
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms0_12762.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lmd0_12760.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms2_12771.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms4_12783.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_dbw0_12793.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms3_12779.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms1_12766.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
System state dump is made for local instance
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lgwr_12799.trc:
ORA-29740: evicted by member , group incarnation
System State dumped to trace file /u01/product/admin/mxdell/bdump/mxdell3_diag_12753.trc
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_pmon_12751.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:48 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_dbw2_12797.trc:

Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lck0_12840.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:49 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_dbw1_12795.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:49 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_mman_12791.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:49 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_ckpt_12801.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:49 2011
Shutting down instance (abort)
License high water mark = 486
Thu Feb 10 14:36:51 2011
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_psp0_12755.trc:
ORA-29740: evicted by member , group incarnation
Thu Feb 10 14:36:57 2011
Instance terminated by LMON, pid = 12757
Thu Feb 10 14:36:59 2011
Instance terminated by USER, pid = 22331
Thu Feb 10 14:37:05 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.13.67.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 2000
  sessions                 = 2205
  resource_limit           = TRUE
  __shared_pool_size       = 3221225472
  __large_pool_size        = 100663296
  __java_pool_size         = 100663296
  __streams_pool_size      = 16777216

......



Starting background process QMNC
QMNC started with pid=30, OS id=22831
Thu Feb 10 14:37:51 2011
Completed: ALTER DATABASE OPEN
Thu Feb 10 14:47:23 2011
Thread 3 advanced to log sequence 32224 (LGWR switch)
  Current log# 38 seq# 32224 mem# 0: /ocfs_ctrl_redo/mxdell/redo38_a.log
  Current log# 38 seq# 32224 mem# 1: /ocfs_data/mxdell/redo38_b.log
Thu Feb 10 14:51:46 2011
Errors in file /u01/product/admin/mxdell/udump/mxdell3_ora_29327.trc:
ORA-00600: internal error code, arguments: [17087], [0x55AD25A18], [], [], [], [], [], []
Thu Feb 10 14:52:06 2011
Trace dumping is performing id=[cdmp_20110210145206]
Thu Feb 10 14:52:27 2011
Errors in file /u01/product/admin/mxdell/udump/mxdell3_ora_29327.trc:
ORA-07445: exception encountered: core dump [kksumc()+193] [SIGSEGV] [Address not mapped to object]

[0x00000007F] [] []
ORA-00600: internal error code, arguments: [17087], [0x55AD25A18], [], [], [], [], [], []
Thu Feb 10 14:52:27 2011
Trace dumping is performing id=[cdmp_20110210145227]
Thu Feb 10 14:52:30 2011
Trace dumping is performing id=[cdmp_20110210145229]
Thu Feb 10 14:56:56 2011
Trace dumping is performing id=[cdmp_20110210145655]
Thu Feb 10 14:58:17 2011
Thread 3 advanced to log sequence 32225 (LGWR switch)
  Current log# 39 seq# 32225 mem# 0: /ocfs_ctrl_redo/mxdell/redo39_a.log
  Current log# 39 seq# 32225 mem# 1: /ocfs_data/mxdell/redo39_b.log
Thu Feb 10 15:02:45 2011
Errors in file /u01/product/admin/mxdell/udump/mxdell3_ora_31699.trc:
ORA-00600: internal error code, arguments: [17087], [0x55AD25A18], [], [], [], [], [], []
Thu Feb 10 15:05:49 2011
Errors in file /u01/product/admin/mxdell/udump/mxdell3_ora_31699.trc:
ORA-07445: exception encountered: core dump [kksumc()+193] [SIGSEGV] [Address not mapped to object]

[0x00000007F] [] []
ORA-00600: internal error code, arguments: [17087], [0x55AD25A18], [], [], [], [], [], []
Thu Feb 10 15:05:49 2011
WARNING: inbound connection timed out (ORA-3136)
Thu Feb 10 15:05:49 2011
Trace dumping is performing id=[cdmp_20110210150549]
Thu Feb 10 15:05:52 2011
Trace dumping is performing id=[cdmp_20110210150552]
Thu Feb 10 15:11:11 2011
Thread 3 advanced to log sequence 32226 (LGWR switch)
  Current log# 40 seq# 32226 mem# 0: /ocfs_ctrl_redo/mxdell/redo40_a.log
  Current log# 40 seq# 32226 mem# 1: /ocfs_data/mxdell/redo40_b.log









节点3上节点实例自动重新启动后的trace文件达到398M 及 1.9G, 很难打开 .

mxrac03$
mxrac03$ls  mxdell3_ora_29327.trc   -alh
-rw-r----- 1 oracle dba 398M Feb 10 14:52 mxdell3_ora_29327.trc
mxrac03$ls  mxdell3_ora_31699.trc   -alh
-rw-r----- 1 oracle dba 1.9G Feb 10 15:05 mxdell3_ora_31699.trc
mxrac03$





节点3上的linux log :

Feb  6 04:03:01 mxrac03 syslogd 1.4.1: restart.
Feb  6 08:22:01 mxrac03 auditd[7519]: Audit daemon rotating log files
Feb  7 13:04:28 mxrac03 auditd[7519]: Audit daemon rotating log files
Feb  8 17:46:01 mxrac03 auditd[7519]: Audit daemon rotating log files
Feb  9 13:02:37 mxrac03 Server Administrator: Storage Service EventID: 2242  The Patrol Read has started.:  Controller 0 (PERC 6/i Integrated)
Feb  9 13:29:31 mxrac03 Server Administrator: Storage Service EventID: 2243  The Patrol Read has stopped.:  Controller 0 (PERC 6/i Integrated)
Feb  9 22:18:47 mxrac03 auditd[7519]: Audit daemon rotating log files
Feb 10 10:37:50 mxrac03 gconfd (root-9009): starting (version 2.14.0), pid 9009 user 'root'
Feb 10 10:37:50 mxrac03 gconfd (root-9009): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
Feb 10 10:37:50 mxrac03 gconfd (root-9009): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
Feb 10 10:37:50 mxrac03 gconfd (root-9009): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Feb 10 10:37:51 mxrac03 hcid[7929]: Default passkey agent (:1.300, /org/bluez/applet) registered
Feb 10 10:37:51 mxrac03 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Feb 10 10:37:51 mxrac03 last message repeated 4 times
Feb 10 10:37:51 mxrac03 nm-system-settings: Loaded plugin ifcfg-rh: (c) 2007 - 2008 Red Hat, Inc.  To report bugs please use the NetworkManager mailing list.
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth2 ...
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh:     read connection 'System eth2'
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-lo ...
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh:     error: Ignoring loopback device config.
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth5 ...
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh:     read connection 'System eth5'
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth4 ...
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh:     read connection 'System eth4'
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth1 ...
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh:     read connection 'System eth1'
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth0 ...
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh:     read connection 'System eth0'
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth3 ...
Feb 10 10:37:51 mxrac03 nm-system-settings:    ifcfg-rh:     read connection 'System eth3'
Feb 10 10:37:51 mxrac03 gconfd (root-9009): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
Feb 10 11:15:50 mxrac03 gconfd (root-9009): GConf server is not in use, shutting down.
Feb 10 11:15:50 mxrac03 gconfd (root-9009): Exiting
Feb 10 11:40:15 mxrac03 Server Administrator: Storage Service EventID: 2238  The controller debug log file has been exported.:  Controller 0 (PERC 6/i Integrated)
Feb 10 11:40:50 mxrac03 Server Administrator: Storage Service EventID: 2095  SCSI sense data Sense key:  5 Sense code: 24 Sense qualifier:  0:  Enclosure 0:0 Controller 0, Connector 0
Feb 10 14:29:48 mxrac03 kernel: printk: 935 messages suppressed.
Feb 10 14:29:48 mxrac03 kernel: ip_conntrack: table full, dropping packet.
Feb 10 14:29:51 mxrac03 last message repeated 9 times
Feb 10 14:29:54 mxrac03 kernel: printk: 10 messages suppressed.
Feb 10 14:29:54 mxrac03 kernel: ip_conntrack: table full, dropping packet.
Feb 10 14:29:59 mxrac03 kernel: printk: 7 messages suppressed.
Feb 10 14:29:59 mxrac03 kernel: ip_conntrack: table full, dropping packet.






节点3上的 ocssd.log


[    CSSD]2011-01-29 02:10:43.769 [1194670400] >TRACE:   clssgmInitialRecv: (0x98e3150) accepted a new connection from node 5 born at 10 active (4, 4), vers (10,3,1,2)
[    CSSD]2011-01-29 02:10:43.769 [1194670400] >TRACE:   clssgmInitialRecv: conns done (4/4)
[    CSSD]2011-01-29 02:10:43.769 [1236629824] >TRACE:   clssgmEstablishMasterNode: MASTER for 12 is node(4) birth(7)
[    CSSD]2011-01-29 02:10:43.769 [1236629824] >TRACE:   clssgmChangeMasterNode: requeued 0 RPCs
[    CSSD]2011-01-29 02:10:43.775 [1194670400] >TRACE:   clssgmHandleDBDone(): src/dest (4/65535) size(72) incarn 12
[    CSSD]CLSS-3000: reconfiguration successful, incarnation 12 with 4 nodes
[    CSSD]CLSS-3001: local node number 3, master node number 4
[    CSSD]2011-01-29 02:10:43.776 [1236629824] >TRACE:   clssgmReconfigThread:  completed for reconfig(12), with status(1)
[    CSSD]2011-01-29 02:10:43.856 [1194670400] >TRACE:   clssgmCommonAddMember: clsomon joined (3/0x1000000/#CSS_CLSSOMON)
[    CSSD]2011-02-10 14:36:52.747 [1194670400] >TRACE:   clscsendx: (0x9918a90) Connection not active
[    CSSD]2011-02-10 14:36:52.747 [1194670400] >TRACE:   clssgmSendClient: Send failed rc 6, con (0x9918a90), client (0x9918d90), proc ((nil))
[    CSSD]2011-02-10 14:36:57.517 [1194670400] >TRACE:   clscsendx: (0x9900850) Connection not active
[    CSSD]2011-02-10 14:36:57.517 [1194670400] >TRACE:   clssgmSendClient: Send failed rc 6, con (0x9900850), client (0x98fd480), proc ((nil))
[    CSSD]2011-02-10 14:36:57.517 [1194670400] >TRACE:   clscsendx: (0x990b880) Connection not active
[    CSSD]2011-02-10 14:36:57.517 [1194670400] >TRACE:   clssgmSendClient: Send failed rc 6, con (0x990b880), client (0x990bf10), proc ((nil))
[    CSSD]2011-02-10 14:36:57.550 [1194670400] >TRACE:   clscsendx: (0x9900e80) Connection not active
[    CSSD]2011-02-10 14:36:57.550 [1194670400] >TRACE:   clssgmSendClient: Send failed rc 6, con (0x9900e80), client (0x9909e60), proc ((nil))






节点3上的crsd.log  

2011-01-29 02:10:49.615: [  CRSRES][1493207360]0Attempting to start `ora.mxrac03.LISTENER_MXRAC03.lsnr` on member `mxrac03`
2011-01-29 02:10:50.151: [  CRSRES][1520490816]0CRS-1002: Resource 'ora.mxrac03.vip' is already running on member 'mxrac03'
2011-01-29 02:10:53.101: [  CRSRES][1493207360]0Start of `ora.mxrac03.LISTENER_MXRAC03.lsnr` on member `mxrac03` succeeded.
2011-01-29 02:10:53.766: [  CRSRES][1518389568]0CRS-1002: Resource 'ora.mxrac03.LISTENER_MXRAC03.lsnr' is already running on member 'mxrac03'
2011-01-29 02:11:15.450: [  CRSRES][1518389568]0startRunnable: setting CLI values
2011-01-29 02:11:15.476: [  CRSRES][1518389568]0Attempting to start `ora.mxrac03.ons` on member `mxrac03`
2011-01-29 02:11:16.939: [  CRSRES][1518389568]0Start of `ora.mxrac03.ons` on member `mxrac03` succeeded.
2011-01-29 02:11:17.229: [  CRSRES][1495308608]0Start of `ora.mxdell.mxdell3.inst` on member `mxrac03` succeeded.
2011-01-29 02:11:17.231: [  CRSRES][1493207360]0Skip online resource: ora.mxrac03.ons
2011-01-29 02:11:17.248: [  CRSRES][1520490816]0CRS-1002: Resource 'ora.mxrac03.LISTENER_MXRAC03.lsnr' is already running on member 'mxrac03'
2011-01-29 02:11:17.284: [  CRSRES][1495308608]0startRunnable: setting CLI values
2011-01-29 02:11:17.305: [  CRSRES][1495308608]0Attempting to start `ora.mxrac03.gsd` on member `mxrac03`
2011-01-29 02:11:17.638: [  CRSRES][1495308608]0Start of `ora.mxrac03.gsd` on member `mxrac03` succeeded.
2011-01-29 02:11:22.235: [  OCRUTL][1388292416]u_freem: mem passed is null
2011-02-10 14:36:49.335: [  CRSRES][1518389568]0In stateChanged, ora.mxdell.mxdell3.inst target is ONLINE
2011-02-10 14:36:49.336: [  CRSRES][1518389568]0ora.mxdell.mxdell3.inst on mxrac03 went OFFLINE unexpectedly
2011-02-10 14:36:49.336: [  CRSRES][1518389568]0StopResource: setting CLI values
2011-02-10 14:36:49.348: [  CRSRES][1518389568]0Attempting to stop `ora.mxdell.mxdell3.inst` on member `mxrac03`
2011-02-10 14:37:04.885: [  CRSRES][1518389568]0Stop of `ora.mxdell.mxdell3.inst` on member `mxrac03` succeeded.
2011-02-10 14:37:04.885: [  CRSRES][1518389568]0ora.mxdell.mxdell3.inst RESTART_COUNT=0 RESTART_ATTEMPTS=5
2011-02-10 14:37:04.886: [  CRSRES][1518389568]0Restarting ora.mxdell.mxdell3.inst on mxrac03
2011-02-10 14:37:04.896: [  CRSRES][1518389568]0startRunnable: setting CLI values
2011-02-10 14:37:04.896: [  CRSRES][1518389568]0Attempting to start `ora.mxdell.mxdell3.inst` on member `mxrac03`
2011-02-10 14:37:04.968: [  OCRUTL][1241434432]u_freem: mem passed is null
2011-02-10 14:37:54.511: [  CRSRES][1518389568]0Start of `ora.mxdell.mxdell3.inst` on member `mxrac03` succeeded.
2011-02-10 14:37:54.511: [  CRSRES][1518389568]0Successfully restarted ora.mxdell.mxdell3.inst on mxrac03, RESTART_COUNT=1
2011-02-10 14:37:54.919: [  CRSRES][1518389568]0ora.mxdell.mxdell3.inst Updated LAST_RESTART time in ocr
2011-02-10 14:37:59.534: [  OCRUTL][1241434432]u_freem: mem passed is null





节点3上的crs alert log  :

2011-01-16 18:48:22.394
[cssd(12242)]CRS-1607:CSSD evicting node mxrac01. Details in /u01/product/crs/log/mxrac03/cssd/ocssd.log.
2011-01-16 18:48:22.394
[cssd(12242)]CRS-1607:CSSD evicting node mxrac05. Details in /u01/product/crs/log/mxrac03/cssd/ocssd.log.
[cssd(12242)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mxrac03 mxrac04 .
2011-01-16 18:48:23.068
[crsd(11564)]CRS-1204:Recovering CRS resources for node mxrac01.
2011-01-16 18:48:23.116
[crsd(11564)]CRS-1204:Recovering CRS resources for node mxrac05.
[cssd(12242)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mxrac01 mxrac03 mxrac04 .
[cssd(12242)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mxrac01 mxrac03 mxrac04 mxrac05 .
2011-01-29 02:10:43.416
[cssd(12390)]CRS-1605:CSSD voting file is online: /ocfs_data/crs/votingdisk. Details in /u01/product/crs/log/mxrac03/cssd/ocssd.log.
[cssd(12390)]CRS-1601:CSSD Reconfiguration complete. Active nodes are mxrac01 mxrac03 mxrac04 mxrac05 .
2011-01-29 02:10:44.500
[crsd(11683)]CRS-1012:The OCR service started on node mxrac03.
2011-01-29 02:10:44.525
[evmd(11555)]CRS-1401:EVMD started on node mxrac03.
2011-01-29 02:10:45.747
[crsd(11683)]CRS-1201:CRSD started on node mxrac03.

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/35489/viewspace-687129/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/35489/viewspace-687129/