网络带宽不足引起arc进程争用导致两个standby失败

一.背景概述

2014/08/17，按照客户的要求对异地dataguard进行实施工作。实施完成后拓扑结构及相关配置信息如下:

实施异地前同城standby可以正常应用日志且稳定运行时间超过半年，完成异地灾备搭建工作后异地灾备中心的gap越来越大且alert日志经常出现kill hang process的动作。当addrprod业务量增加后addrprod_std出现不再应用日志且主库也不主动往addrprod_std归档，分析后判定为往异地网络带宽不够引起arc进程资源被ADDRCD耗尽。故考虑将异地灾备中心级联到ADDRPROD_STD之后防止出现ADDRCD耗尽进程资源引起ADDRPROD_STD无法正常工作，调整后同城灾备运行正常但是异地灾备中心仍不能正常工作。最终通过调整ADDRPROD_STD到ADDRCD的归档进程数量，减少抢占网络资源的进程数，调整后ADDRPROD再一定时间内成功自动处理了40GB的gap。

以下通过调整拓扑结构，增量备份处理gap及优化ARCn进程数几个步骤对本次配置优化和故障处理的过程进行描述。

二.调整拓扑结构

调整拓扑结构前，客户反应ADDRCD(异地)的GAP越来越大且不能自动修复。进行检查分析工作时发现同城ADDRPROD_STD灾备环境和主库的数据版本差距也越来越大且主库不再主动归档日志到ADDRPROD_STD,同时ADDRPROD_STD从ADDRPROD去FETCH ARCHIVE LOG一直不能成功。各自alert日志如下(已设置log_archive_trace=128):

源端

alert日志：

Reclaiming FAL entry from dead process [pid 29229110]
FAL[server, ARC2]: Begin FAL archive (dbid 0 branch 845289128 thread 1 sequence 11600 dest ADDRCD)
Thu Aug 28 09:29:31 2014
Killing 1 processes with pids 24182958 (Process by index) in order to remove hung processes. Requested by OS process 16056504 on instance 1
ARC5: Detected ARCH process failure
ARC5: STARTING ARCH PROCESSES
Thu Aug 28 09:29:33 2014
ARC0 started with pid=36, OS id=19988636
ARC0: Archival started
ARC5: STARTING ARCH PROCESSES COMPLETE
Reclaiming FAL entry from dead process [pid 24182958]
FAL[server, ARC0]: Begin FAL archive (dbid 0 branch 845289128 thread 1 sequence 11603 dest ADDRCD)
Thu Aug 28 09:34:04 2014
Thread 1 advanced to log sequence 11657 (LGWR switch)
  Current log# 5 seq# 11657 mem# 0: +DATADG/addrprod/onlinelog/group_5.325.845289135
Thu Aug 28 09:34:10 2014
Archived Log entry 162811 added for thread 1 sequence 11656 ID 0xffffffff8f4c47cc dest 1:
Thu Aug 28 09:55:34 2014
Killing 1 processes with pids 29556818 (Process by index) in order to remove hung processes. Requested by OS process 16056504 on instance 1
ARC5: Detected ARCH process failure
ARC5: STARTING ARCH PROCESSES
Thu Aug 28 09:55:36 2014
ARC4 started with pid=44, OS id=29556826
ARC4: Archival started
ARC5: STARTING ARCH PROCESSES COMPLETE
Reclaiming FAL entry from dead process [pid 29556818]
FAL[server, ARC4]: Begin FAL archive (dbid 0 branch 845289128 thread 1 sequence 11594 dest ADDRCD)
Thu Aug 28 09:57:04 2014
FAL[server, ARC2]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance addr11g1 - Archival Error. Archiver continuing.
FAL[server, ARC2]: Begin FAL archive (dbid 0 branch 845289128 thread 1 sequence 11605 dest ADDRPROD_STD)
ARC2: Archive log rejected (thread 1 sequence 11605) at host 'ADDRPROD_STD'
FAL[server, ARC2]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance addr11g1 - Archival Error. Archiver continuing.
FAL[server, ARC2]: Begin FAL archive (dbid 0 branch 845289128 thread 1 sequence 11498 dest addrcd)
FAL[server, ARC2]: FAL archive failed, see trace file.
FAL[server, ARC2]: Begin FAL archive (dbid 0 branch 845289128 thread 1 sequence 11605 dest ADDRCD)
Thu Aug