如何在 oracle 集群环境下修改私网信息 (文档 ID 2103317.1)

如何在 oracle 集群环境下修改私网信息 (文档 ID 2103317.1)

---

---

文档内容

目标

解决方案

例1： 更改私有主机名

例2：只更改私有 ip 地址不更改网卡、子网及子网掩码信息

例3：只改变私有网络的 MTU 值

例4：更改私有网卡名字，子网及掩码

A. 对于 11gR2 以前的集群管理软件

B. 对于 11gR2 和 12c 上没有使用 flex ASM 的版本

C. 对于 12C flex ASM 结构

关于 11gR2 的一些注意事项

关于 Windows 系统注意事项

使用 oifcfg 命令更改网卡名字的影响

Oifcfg 命令用法

例5 对于 11gR2 或以上版本的 HAIP 添加或删除集群私网

参考

---

适用于:

Oracle Database - Enterprise Edition - 版本 10.1.0.2 到 12.2.0.1 [发行版 10.1 到 12.2]
本文档所含信息适用于所有平台

目标

本文的目的是描述如何在 oracle 集群环境中更改或更新私有网络（cluster_interconnect）信息。

有时我们需要更改或更新网卡的名字，或者更改网卡的子网掩码，再或者更改当时原始安装时就输入了不正确的信息，也许还有其它的原因，oifcfg 工具当时没有成功的安装。

请参考文档：Note 276434.1 for modifying public network and VIP associated information
和文档： Note 1386709.1 for basics of IPv4 subnet and Oracle Clusterware。

提示：对于 oracle Engineered system（Exadata）和 oracle Applicance（ODA）不适用本文档。

 

解决方案

Oracle 集群中的网络信息（接口，子网及每个网卡的角色）都可以被’oifcfg’ 命令管理, 但是没有网卡的IP地址除外，oifcfg 命令不能修改IP地址信息. ‘oifcfg getif’ 命令可以用来显示OCR中当前网卡的配置信息：

% $CRS_HOME/bin/oifcfg getif 
eth0 10.2.156.0 global public 
eth1 192.168.0.0 global cluster_interconnect

在 Unix/Linux 系统中，网卡名字是被系统自动分配的，依据系统平台的不同而不同。对于 windows 系统，请参考下面的附带的文档. 上面的例子显示当前网卡 eth0 被用作公网并且子网为 10.2.156.0 eth1 被用作集群私网，子网为192.168.0.0。

‘公有’ 网络是服务器与客户端之间的通信（与 VIP 使用相同的网段并以不同的记录存储在 OCR 中）,与之对应的’cluster_interconnect’网络是用来在 RDBMS/ASM 节点间缓存融合。从 11gR2 开始，cluster_interconnect 同时被用作集群间的心跳，相对于 11gR2 以前的版本，当配置集群心跳信息时指定主机名而言，这是一个标志性的改变。

如果私有网卡的子网或接口名字配置不正确，我们需要使用 crs/grid 用户来更改。

例1： 更改私有主机名

在 11.2 oracle clusterware 之前的版本，私有主机名被记录在 OCR 中, 它不能被更改，一般情况下私有主机名是不需要改变的，它附属的 ip 可以被更改，只有使用删除/添加节点或重新安装 oracle clusterware 来更改私有主机名。

在 11.2 Grid 结构中，私有主机名不在记录在 OCR 中，并且不存在依赖关系，所以它可以在 /etc/hosts 文件中任意更改。

例2：只更改私有 ip 地址不更改网卡、子网及子网掩码信息

举例，私有 ip 地址从 192.168.1.10 更改至 192.168.1.21，网卡名字及子网保持不变。或者只改变MAC地址，保持private IP address/interface name/subnet/network不变

只要关闭需要更改主机上的 oracle clusterware 软件，在操作系统层，根据需求更改私有 ip 地址或者MAC地址（如：/etc/hosts，OS network config 等等）, 再重启启动 oracle clusterware 软件即可。

例3：只改变私有网络的 MTU 值

举例, 将私有网络 MTU 值从1500更改至9000（激活 jumbo frame），网卡名字保持不变。

1. 关闭集群中的所有节点。
2. 在操作系统层更改 MTU 需要设定的值，确保更改后 MTU 值的私有网卡可用并且可以 ping 通集群中的所有节点。
3. 重启所有节点的集群管理软件。

例4：更改私有网卡名字，子网及掩码

提示：当子网掩码被更改，但是子网标识没有改变时，如：
子网掩码从 255.255.0.0 更改至 255.255.255.0，私网 ip 为 192.168.0.x，子网标识保持不变 192.168.0.0，网卡名字没有改变.关闭所有需要更改的主机 oracle clusterware，在操作系统层修改私有网络IP地址（如：操作系统 网络配置等等）。并重启集群中所有节点，请注意，这种更改是不能采用轮转方式（rolling manaer）完成的。

当子网掩码被改变，附属的子网标识也经常会被改变，oracle 在 OCR 中只存储网卡名字及子网标识的信息，而不存储子网掩码。可以使用 oifcfg 命令完成这样的变更，oifcfg 命令只需在集群中的一个节点执行，而不是所有节点。

A. 对于 11gR2 以前的集群管理软件

1. 使用 oifcfg 命令添加新的私有网络信息，删除旧的私有网络信息：

% $ORA_CRS_HOME/bin/oifcfg/oifcfg setif -global <if_name>/<subnet>:cluster_interconnect
% $ORA_CRS_HOME/bin/oifcfg/oifcfg delif -global <if_name>[/<subnet>]]

举例：
% $ORA_CRS_HOME/bin/oifcfg setif -global  eth3 / 192.168.2.0 :cluster_interconnect
% $ORA_CRS_HOME/bin/oifcfg delif -global eth1/192.168.1.0

校验结果
% $ORA_CRS_HOME/bin/oifcfg getif   
eth0 10.2.166.0 global public 
eth3   192.168.2.0  global cluster_interconnect

2. 关闭 Oracle Clusterware

使用 root 用户执行： # crsctl stop crs

3. 在操作系统层面更改网络配置，修改集群内所有节点的 /etc/hosts 文件，确保集群内所有节点新的网络设置都已生效：

% ping <private hostname/IP>
% ifconfig -a  on Unix/Linux 
或
% ipconfig /all on windows

4. 重新启动 Oracle Clusterware

以 root 用户：# crsctl start crs

提示：如果在 linux 系统上正在运行 OCFS2，则可能还需要更改 OCFS2 运行在其它节点的私有 ip 地址. 更多详细的信息，请参考：  Note 604958.1 。

 

B. 对于 11gR2 和 12c 上没有使用 flex ASM 的版本

针对于 11.2 的结构，私有网络配置信息不但保存在 OCR 中，而且还保存在 gpnp 属性文件中。如果私有网络不可用或定义错误，则 CRSD 进程将无法启动，任何随后对于 OCR 的改变都是不可能完成的，因此需要注意当对私有网络的配置信息进行修改，正确的改变顺序是非常重要的。同时请注意，手动修改 gpnp 属性文件是不支持的。

在对集群中所有节点操作之前，请先备份 profile.xml 配置文件。作为 grid 用户执行：
$ cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
$ cp -p profile.xml profile.xml.bk

1. 确保集群中的所有节点都已启动并正常运行

2. 使用 grid 用户：

获取下面信息， 例如：

$ oifcfg getif
eth1 100.17.10.0 global public
eth0 192.168.0.0 global cluster_interconnect

加入新的集群私网通讯信息：

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect

例如：
a. 加入新的并有相同子网的接口卡 bond0
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect

b. add a new subnet with the same interface name but different subnet or new interface name
$ 添加一个新的子网具有相同网卡的名称但不同的子网或新的网卡名
或
$ oifcfg setif -global eth3/192.168.1.96:cluster_interconnect

 

1. 如果网卡不可用，需要使用 –global 选项来完成，而不能使用 –node 选项，它将导致节点被驱逐。

2. 如果网卡在服务器上可用，则可以使用下面命令识别子网地址：
$ oifcfg iflist

它列出了网卡及子网地址，即使 oracle 集群没有启动，此命令也可以被执行。 请注意，子网掩码有可能不是 x.y.z.0 的格式 ，它可以是 x.y.z.24，x.y.z.64 或 x.y.z.128 等格式。如：
$ oifcfg iflist 
lan1 18.1.2.0
lan2  10.2.3.64         << 这是一个私有网络子网地址附属的私有网络 ip 地址为 10.2.3.86

3. 如果需要添加第二个私有网络，而不是替换现有的私有网络， 则需要保证两个网卡的 MTU 值相同 ，否则实例将无法启动并报如下错误信息：
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if MTU failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcini2
ORA-27303: additional information: requested interface lan1:801 has a different MTU (1500) than lan3:801 (9000), which is not supported. Check output from ifconfig command

4. 对于 11gR2 或更高版本, 不建议在 ASM 或 database 的 spfile 或 pfile 中设置 cluster interconnects 参数。无论什么原因如果设置了该参数，则需要在集群关闭之前需将新的私网 ip 地址设置在 spfile 或 pfile 中，否则它会由于私网信息不匹配而导致重启失败。

校验更改后的值：

$ oifcfg getif

3. 使用 root 用户关闭集群中所有的节点并禁用集群：

# crsctl stop crs
# crsctl disable crs

4. 使网络配置信息都已在 OS 层更改完成，确保更改完成后新的接口在所有的节点都可用有效：

$ ifconfig -a
$ ping <private hostname>

5. 使用 root 用户激活 oracle 集群并重新启动集群中的所有节点：

# crsctl enable crs
# crsctl start crs

6. 如果需要去除旧接口卡信息：

$ oifcfg delif -global <if_name>[/<subnet>]
例如：
$ oifcfg delif -global eth0/192.168.0.0

 

C. 对于 12C flex ASM 结构

请检查上面部分B，并关注提示部分，按下面命令做备份：

在对集群中所有节点操作之前，请先备份 profile.xml 配置文件。 作为 grid 用户执行：
$ cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
$ cp -p profile.xml profile.xml.bk

1. 确保 oracle 集群中的所有节点都已正常运行。

2. 使用 grid 用户：

得到现有信息，如下：

$ oifcfg getif
eth1 100.17.10.0 global public
eth0 192.168.0.0 global cluster_interconnect,asm

上面例子显示网卡 ech0 被用作集群私网和 ASM 网络。

加入新的集群私网信息：

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect[,asm]

如：
a. 加入一个新的具有相同子网网卡 bond0
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect,asm

b. 加入一个新的并具有相同网卡名字的子网，或不同子网和具有新的接口名字
$ oifcfg setif -global eth0/192.68.10.0:cluster_interconnect,asm
或
$ oifcfg setif -global eth3/192.168.1.96:cluster_interconnect,asm

 如果有不同的网络用于私有网络和 ASM 网络，则可以对其进行相应的调整。

3. 当 ASMLISTENER 正被用作私有网络，如果对其修改则会影响 ASMLISTENER。需要添加一个新的 ASMLISTENER 及新的网络配置。如果 ASM 的子网网络没有改变则跳过这一步。

3.1. 加入一个新的 ASMLISTENE（例：ASMNEWLISNR_ASM）及新的子网，使用 grid 用户：

$ srvctl add  listener -asmlistener -l <new ASM LISTENER NAME> -subnet <new subnet>

如：
$ srvctl add listener -asmlistener -l ASMNEWLSNR_ASM -subnet 192.168.10.0

3.2. 删除现有的 ASMLISTENER（这个例子中 ASMLSNR_ASM）并去除依赖关系，使用 grid 用户：

$ srvctl update listener -listener ASMLSNR_ASM -asm -remove -force
$ lsnrctl stop ASMLSNR_ASM

 

注意. 需要使用 –force 选项，否则会出现下面错误：

$ srvctl update listener -listener ASMLSNR_ASM -asm -remove
PRCR-1025 : Resource ora.ASMLSNR_ASM.lsnr is still running
$ srvctl stop listener -l ASMLSNR_ASM
PRCR-1065 : Failed to stop resource ora.ASMLSNR_ASM.lsnr
CRS-2529: Unable to act on 'ora.ASMLSNR_ASM.lsnr' because that would require stopping or relocating 'ora.asm', but the force option was not specified

3.3 校验配置信息：

$ srvctl config listener -asmlistener
$ srvctl config asm

4. 使用 root 用户关闭集群中的所有节点并禁用集群：

# crsctl stop crs
# crsctl disable crs

5. 在操作系统层面更改网络配置，更改之后，确保所有节点上的新网卡生效：

$ ifconfig -a
$ ping <private hostname>

6. 使用 root 用户激活 oracle 集群并重新启动集群中的所有节点：

# crsctl enable crs
# crsctl start crs

7. 删除旧的网卡信息：

$ oifcfg delif -global <if_name>[/<subnet>]
如：
$ oifcfg delif -global eth0/192.168.0.0

 

 

关于 11gR2 的一些注意事项

1. 如果底层网络配置已经更改， 但是 oifcfg 尚未执行同样的变更，则重启 oracle 集群会导致 crsd 进程不能启动。

crsd.log 日志将会显示如下：

2010-01-30 09:22:47.234: [ default][2926461424] CRS Daemon Starting
..
2010-01-30 09:22:47.273: [ GPnP][2926461424]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=7153, tl=3, f=0
2010-01-30 09:22:47.282: [ OCRAPI][2926461424] clsu_get_private_ip_addresses: no ip addresses found.
2010-01-30 09:22:47.282: [GIPCXCPT][2926461424] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
2010-01-30 09:22:47.283: [GIPCXCPT][2926461424] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
[ OCRAPI][2926461424]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2010-01-30 09:22:47.285: [ OCRAPI][2926461424]a_init:13!: Clusterware init unsuccessful : [44]
2010-01-30 09:22:47.285: [ CRSOCR][2926461424] OCR context init failure. Error:  PROC-44: Error in network address and interface operations Network address and interface operations error [7]
2010-01-30 09:22:47.285: [ CRSD][2926461424][PANIC] CRSD exiting: Could not init OCR, code: 44
2010-01-30 09:22:47.285: [ CRSD][2926461424] Done.

以上错误显示操作系统层面的设置（oifcfg iflist）与 gpnp profile.xml 配置文件设置不匹配。

解决方法：恢复操作系统网络配置到最初的状态，启动 oracle 集群，然后再按照上面的步骤重新更改。

如果底层的网络并没有改变，但 oifcfg 已经被设置了一个错误的子网地址或接口名字，则会发生同样的问题。

2. 如果集群中的任何一个节点关闭，oifcfg 命令将会失败并显示错误：

$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster

解决方案：启动 oracle 集群中没有运行的节点，确保集群中所有的节点都已启动，如果由于操作系统原因不能启动的节点，请先将此节点从集群中删除在执行私网网络变更。

3. 如果执行上面命令的的用户非 GI 的拥有者，则会出现相同的错误：

$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster

解决方案：确保使用 GI 的拥有者登录并执行上面命令。

4. 从 11.2.0.2 开始，如果在没有加入一个新私有网卡，就试图删除最后一个私有网卡（集群私网）则会发生下面错误：

PRIF-31: Failed to delete the specified network interface because it is the last private interface

解决方案：在删除旧的私有网卡之前，先加入新的私有网卡。

5. 如果主机节点的 oracle 集群关闭在关闭状态，则会报下面错误：

$ oifcfg getif
PRIF-10: failed to initialize the cluster registry

解决方案：启动该主机节点上的 oracle 集群软件。

 

关于 Windows 系统注意事项

更改网卡的语法在 windows/RAC 和Unix/Linux 集群是一样的，但是网卡名称会略有不同，在 windows 系统上，默认分配给接口通常的名称为：

Local Area Connection
Local Area Connection 1 
Local Area Connection 2

如果使用一个网卡名称含有空格，则名称必须使用引号括起来，同时，请注意这是区分大小写的。例如，在 windows上，设置集群私网链接：

C:\oracle\product\10.2.0\crs\bin\oifcfg setif -global "Local Area Connection 1"/192.168.1.0:cluster_interconnect

然而，在 windows 上重新命名网卡按最佳实践更有意义，如重命名为”ocwpublic” 和”ocwprivate”。如果 oracle 集群安装完成后需要更改网卡名字，则需要运行”oifcfg”命令来添加新的网卡并删除旧的。综上所述。

您可以运行下面命令查看每个节点上可用的网卡名字。

oifcfg iflist -p -n

必须在每个节点上运行这个命令来验证网卡名称相同的定义。

使用 oifcfg 命令更改网卡名字的影响

对于私网网卡，数据库将使用存储在 OCR 中定义为集群互联的网卡作为节点间缓存融合通信。在告警日志开始的时候，就会显示集群互联有效的信息。在参数清单。例如：

For pre 11.2.0.2:
Cluster communication is configured to use the following interface(s) for this instance 
192.168.1.1


For 11.2.0.2+: (HAIP address will show in alert log instead of private IP)
Cluster communication is configured to use the following interface(s) for this instance
  169.254.86.97

如果上面信息不正确，则实例需要重启以便 OCR 条目修正，这同样适用于 ASM 实例和数据库实例。在 windows 系统上，实例被关闭后，在 OCR 将被重读之前，还需要停止/启动 OracleService < SID >（或 OracleASMService < ASMSID > 。

 

Oifcfg 命令用法

查看 oifcfg 命令的全部选项，只需输入：

$ <CRS_HOME>/bin/oifcfg

 

例5 对于 11gR2 或以上版本的 HAIP 添加或删除集群私网

1. 添加另外的私有网络到现有的使用 HAIP 的集群中，作为 grid 用户执行：

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect

例如：

$ oifcfg setif -global enp0s8/192.168.57.0:cluster_interconnect

 关闭 CRS 中的所有节点，通过重新启动 crs 中的所有节点，来使 HAIP 读入新的接口，不能使用滚动方式重启。

2. 在使用 HAIP 的集群中删除私有网络，作为 grid 用户执行：

$ oifcfg delif -global <if_name>

例如：
$ oifcfg delif -global enp0s8

HAIP 将切换至其它可用接口，在接口被删除后，集群/数据库会继续采用此方式运行。

删除多余的 HAIP 接口，应关闭 CRS 所有节点，然后重启 CRS 所有节点。不能采用以滚动的方式重新启动 CRS。

 

参考

NOTE:1386709.1  - The Basics of IPv4 Subnet and Oracle Clusterware
NOTE:276434.1  - How to Modify Public Network Information including VIP in Oracle Clusterware
NOTE:604958.1  - OCFS2 Node Fence Caused by Removing the External Network Cable
NOTE:1054902.1  - How to Validate Network and Name Resolution Setup for the Clusterware and RAC

---

---

How to Modify Private Network Information in Oracle Clusterware (文档 ID 283684.1)

In this Document

Goal

Solution

Case I. Changing private hostname

Case II. Changing private IP only without changing network interface, subnet and netmask               or changing private IP MAC address only without changing anything else

Case III. Changing private network MTU only

Case IV. Changing private network interface name, subnet or netmask

A. For pre-11gR2 Oracle Clusterware

B. For 11gR2 Oracle Clusterware and 12c Cluster without Flex ASM

C. For 12c Oracle Clusterware with Flex ASM

Something to note for 11gR2+

Notes for Windows Systems

Ramifications of Changing Interface Names Using oifcfg

Oifcfg Usage

Case V. Add or remove cluster_interconnect for 11gR2 and above with HAIP

References

---

APPLIES TO:

Oracle Database - Enterprise Edition - Version 10.1.0.2 to 12.2.0.1 [Release 10.1 to 12.2]
Information in this document applies to any platform.

GOAL

The purpose of this note is to describe how to change or update the private network (cluster_interconnect) information in Oracle Clusterware. 

It may be necessary to change or update interface names, or subnet associated with an interface if there is a network change affecting the servers, or if the original information that was input during the installation was incorrect.   It may also be the case that for some reason, the Oracle Interface Configuration Assistant  ('oifcfg')  did not succeed during the installation.

Please refer to Note 276434.1 for modifying public network and VIP associated information
and refer to Note 1386709.1 for basics of IPv4 subnet and Oracle Clusterware.

Note: for Oracle Engineered system (Exadata) and Oracle Database Appliance (ODA), please do not make such changes following this note.

 

SOLUTION

Network information(interface, subnet and role of each interface) for Oracle Clusterware is managed by 'oifcfg', but actual IP address for each interfaces are not, 'oifcfg' can not update IP address information. 'oifcfg getif' can be used to find out currently configured interfaces in OCR:

% $CRS_HOME/bin/oifcfg getif 
eth0 10.2.156.0 global public 
eth1 192.168.0.0 global cluster_interconnect

On Unix/Linux systems, the interface names are generally assigned by the OS, and standard names vary by platform. For Windows systems, see additional notes below. Above example shows currently interface eth0 is used for public with subnet 10.2.156.0, and eth1 for cluster_interconnect/private with subnet 192.168.0.0.

The 'public' network is for database client communication (VIP also uses the same network though it's stored in OCR as separate entry), whereas the 'cluster_interconnect' network is for RDBMS/ASM cache fusion. Starting with 11gR2, cluster_interconnect is also used for clusterware heartbeats - this is significant change compare to prior release as pre-11gR2 uses the private nodename that were specified at installation time for clusterware heartbeats.

If the subnet or interface name for 'cluster_interconnect' interface is incorrect, it needs to be changed as crs/grid user.

Case I. Changing private hostname

In pre-11.2 Oracle Clusterware, private hostname is recorded in OCR, it can not be updated. Generally private hostname is not required to change. Its associated IP can be changed. The only way to change private hostname is by deleting/adding nodes, or reinstall Oracle Clusterware.

In 11.2 Grid Infrastructure, private hostname is no longer recorded in OCR and there is no dependency on the private hostname. It can be changed freely in /etc/hosts.

Case II. Changing private IP only without changing network interface, subnet and netmask
              or changing private IP MAC address only without changing anything else

For example, private IP is changed from 192.168.1.10 to 192.168.1.21, network interface name and subnet remain the same, or MAC address is changed, private IP address/interface name/subnet/network all remain the same.

Simply shutdown Oracle Clusterware stack on the node where change required, make IP or MAC modification at OS layer as required (eg: /etc/hosts, OS network config etc) for private network, restart Oracle Clusterware stack will complete the task.

Case III. Changing private network MTU only

For example, private network MTU is changed from 1500 to 9000 (enable jumbo frame), network interface name and subnet remain the same.

1. Shutdown Oracle Clusterware stack on all nodes
2. Make the required network change of MTU size at OS network layer, ensure private network is available with the desired MTU size, ping with the desired MTU size works on all cluster nodes
3. Restart Oracle Clusterware stack on all nodes

Case IV. Changing private network interface name, subnet or netmask

Note: When the netmask is changed but the subnet ID doesn't change, for example:
The netmask is changed from 255.255.0.0 to 255.255.255.0 with private IP like 192.168.0.x, the subnet ID remains the same as 192.168.0.0, the network interface name is not changed.
Shutdown Oracle Clusterware stack on all cluster nodes where change required, make IP modification at OS layer (eg: OS network config etc) for private network, restart Oracle Clusterware stack on all nodes will complete the task. Please note, this change can not be done in rolling manner.

When the netmask is changed, the associated subnet ID is often changed. Oracle only store network interface name and subnet ID in OCR, not the netmask. Oifcfg command can be used for such change, oifcfg commands only require to run on 1 of the cluster node, not all.

A. For pre-11gR2 Oracle Clusterware

1. Use oifcfg to add the new private network information, delete the old private network information:

% $ORA_CRS_HOME/bin/oifcfg/oifcfg setif -global <if_name>/<subnet>:cluster_interconnect
% $ORA_CRS_HOME/bin/oifcfg/oifcfg delif -global <if_name>[/<subnet>]]

For example:
% $ORA_CRS_HOME/bin/oifcfg setif -global  eth3 / 192.168.2.0 :cluster_interconnect
% $ORA_CRS_HOME/bin/oifcfg delif -global eth1/192.168.1.0

To verify the change
% $ORA_CRS_HOME/bin/oifcfg getif   
eth0 10.2.166.0 global public 
eth3   192.168.2.0  global cluster_interconnect

2. Shutdown Oracle Clusterware stack

As root user: # crsctl stop crs

3. Make required network change at OS level, /etc/hosts file should be modified on all nodes to reflect the change.
Ensure the new network is available on all cluster nodes:

% ping <private hostname/IP>
% ifconfig -a  on Unix/Linux 
or 
% ipconfig /all on windows

4. restart the Oracle Clusterware stack

As root user: # crsctl start crs

Note:  If running OCFS2 on Linux, one  may also need to change the private IP address that OCFS2 is using to communicate with other nodes.   For more information, please refer to  Note 604958.1

 

B. For 11gR2 Oracle Clusterware and 12c Cluster without Flex ASM

As of 11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in the gpnp profile.  If the private network is not available or its definition is incorrect, the CRSD process will not start and any subsequent changes to the OCR will be impossible. Therefore care needs to be taken when making modifications to the configuration of the private network. It is important to perform the changes in the correct order. Please also note that manual modification of gpnp profile is not supported.

Please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:
$ cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
$ cp -p profile.xml profile.xml.bk

1. Ensure Oracle Clusterware is running on ALL cluster nodes in the cluster

2. As grid user:

Get the existing information. For example:

$ oifcfg getif
eth1 100.17.10.0 global public
eth0 192.168.0.0 global cluster_interconnect

Add the new cluster_interconnect information:

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect

For example:
a. add a new interface bond0 with the same subnet
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect

b. add a new subnet with the same interface name but different subnet or new interface name
$ oifcfg setif -global eth0/192.65.0.0:cluster_interconnect
or
$ oifcfg setif -global eth3/192.168.1.96:cluster_interconnect

 

1. This can be done with -global option even if the interface is not available yet, but this can not be done with -node option if the interface is not available, it will lead to node eviction.

2. If the interface is available on the server, subnet address can be identified by command:
$ oifcfg iflist

It lists the network interface and its subnet address. This command can be run even if Oracle Clusterware is not running.  Please note, subnet address might not be in the format of x.y.z.0,  it can be x.y.z.24, x.y.z.64 or x.y.z.128 etc. For example,
$ oifcfg iflist 
lan1 18.1.2.0
lan2  10.2.3.64         << this is the private network subnet address associated with private network IP: 10.2.3.86

3. If it is for adding a 2nd private network, not replacing the existing private network, please ensure  MTU size of both interfaces are the same , otherwise instance startup will report error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if MTU failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcini2
ORA-27303: additional information: requested interface lan1:801 has a different MTU (1500) than lan3:801 (9000), which is not supported. Check output from ifconfig command

4. For 11gR2 and higher, it is not recommended to set cluster_interconnects in ASM or Database spfile or pfile. If this parameter is set for any reason, it needs to be changed to the new private IP in spfile or pfile prior to clusterware shutdown, otherwise it will result a failure during restart due to the interconnect mismatch.

Verify the change:

$ oifcfg getif

3. Shutdown Oracle Clusterware on all nodes and disable the Oracle Clusterware as root user:

# crsctl stop crs
# crsctl disable crs

4. Make the network configuration change at OS level as required, ensure the new interface is available on all nodes after the change.

$ ifconfig -a
$ ping <private hostname>

5. Enable Oracle Clusterware and restart Oracle Clusterware on all nodes as root user:

# crsctl enable crs
# crsctl start crs

6. Remove the old interface if required:

$ oifcfg delif -global <if_name>[/<subnet>]
eg:
$ oifcfg delif -global eth0/192.168.0.0

 

C. For 12c Oracle Clusterware with Flex ASM

Please review above section B and pay attention to the Note section, take a backup as follows:

Please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:
$ cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
$ cp -p profile.xml profile.xml.bk

1. Ensure Oracle Clusterware is running on ALL cluster nodes in the cluster

2. As grid user:

Get the existing information. For example:

$ oifcfg getif
eth1 100.17.10.0 global public
eth0 192.168.0.0 global cluster_interconnect,asm

Above example shows network eth0 is used for both cluster_interconnect and ASM network.

Add the new cluster_interconnect information:

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect[,asm]

For example:
a. add a new interface bond0 with the same subnet
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect,asm

b. add a new subnet with the same interface name but different subnet or new interface name
$ oifcfg setif -global eth0/192.68.10.0:cluster_interconnect,asm
or
$ oifcfg setif -global eth3/192.168.1.96:cluster_interconnect,asm

 If different network is used for private network and ASM network, then modify them accordingly.

3. As ASMLISTENER is using the private network, modifying the private network will affect ASMLISTENER. It is required to add a new ASMLISTENER with the new network configuration. Skip this step if the subnet for the ASM network is not changed.

3.1. Add a new ASMLISTENER (for example: ASMNEWLSNR_ASM) with the new subnet, as grid user:

$ srvctl add  listener -asmlistener -l <new ASM LISTENER NAME> -subnet <new subnet>

eg:
$ srvctl add listener -asmlistener -l ASMNEWLSNR_ASM -subnet 192.168.10.0

3.2. Drop the existing ASMLISTENER (ASMLSNR_ASM in this example) and remove the dependency, as grid user:

$ srvctl update listener -listener ASMLSNR_ASM -asm -remove -force
$ lsnrctl stop ASMLSNR_ASM

 

Note. -force option is required, otherwise the following error will occur:

$ srvctl update listener -listener ASMLSNR_ASM -asm -remove
PRCR-1025 : Resource ora.ASMLSNR_ASM.lsnr is still running
$ srvctl stop listener -l ASMLSNR_ASM
PRCR-1065 : Failed to stop resource ora.ASMLSNR_ASM.lsnr
CRS-2529: Unable to act on 'ora.ASMLSNR_ASM.lsnr' because that would require stopping or relocating 'ora.asm', but the force option was not specified

3.3 Verify the configuration

$ srvctl config listener -asmlistener
$ srvctl config asm

4. Shutdown Oracle Clusterware on ALL nodes and disable the Oracle Clusterware as root user:

# crsctl stop crs
# crsctl disable crs

5. Make the network configuration change at OS level as required, ensure the new interface is available on all nodes after the change.

$ ifconfig -a
$ ping <private hostname>

6. Enable Oracle Clusterware and restart Oracle Clusterware on all nodes as root user:

# crsctl enable crs
# crsctl start crs

7. Remove the old interface if required:

$ oifcfg delif -global <if_name>[/<subnet>]
eg:
$ oifcfg delif -global eth0/192.168.0.0

 

 

Something to note for 11gR2+

1. If underlying network configuration has been changed, but oifcfg has not been run to make the same change,  then upon Oracle Clusterware restart, the CRSD will not be able to start.

The crsd.log will show:

2010-01-30 09:22:47.234: [ default][2926461424] CRS Daemon Starting
..
2010-01-30 09:22:47.273: [ GPnP][2926461424]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=7153, tl=3, f=0
2010-01-30 09:22:47.282: [ OCRAPI][2926461424] clsu_get_private_ip_addresses: no ip addresses found.
2010-01-30 09:22:47.282: [GIPCXCPT][2926461424] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
2010-01-30 09:22:47.283: [GIPCXCPT][2926461424] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
[ OCRAPI][2926461424]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2010-01-30 09:22:47.285: [ OCRAPI][2926461424]a_init:13!: Clusterware init unsuccessful : [44]
2010-01-30 09:22:47.285: [ CRSOCR][2926461424] OCR context init failure. Error:  PROC-44: Error in network address and interface operations Network address and interface operations error [7]
2010-01-30 09:22:47.285: [ CRSD][2926461424][PANIC] CRSD exiting: Could not init OCR, code: 44
2010-01-30 09:22:47.285: [ CRSD][2926461424] Done.

Above errors indicate a mismatch between OS setting (oifcfg iflist) and gpnp profile setting profile.xml.

Workaround: restore the OS network configuration back to the original status, start Oracle Clusterware. Then follow above steps to make the changes again. 

If the underlying network has not been changed, but oifcfg setif has been run with a wrong subnet address or interface name, same issue will happen.

2. If any one node is down in the cluster, oifcfg command will fail with error:

$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster

Workaround: start Oracle Clusterware on the node where it is not running. Ensure Oracle Clusterware is up on all cluster nodes. If the node is down for any OS reason, please remove the node from the cluster before performing private network change.

3. If a user other than Grid Infrastructure owner issues above command, it will fail with same error:

$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster

Workaround: ensure to login as Grid Infrastructure owner to perform such command.

4. From 11.2.0.2 onwards, if attempt to delete the last private interface (cluster_interconnect) without adding a new one first, following error will occur:

PRIF-31: Failed to delete the specified network interface because it is the last private interface

Workaround: Add new private interface first before deleting the old private interface.

5. If Oracle Clusterware is down on the node, the following error is expected:

$ oifcfg getif
PRIF-10: failed to initialize the cluster registry

Workaround: Start the Oracle Clusterware on the node

 

Notes for Windows Systems

The syntax for changing the interfaces on Windows/RAC clusters is the same as on Unix/Linux, but the interface names will be slightly different. On Windows systems, the default names assigned to the interfaces are generally named such as:

Local Area Connection
Local Area Connection 1 
Local Area Connection 2

If using an interface name that has space in it, the name must be enclosed in quotes. Also, be aware that it is case sensitive.  For example, on Windows,  to set cluster_interconnect:

C:\oracle\product\10.2.0\crs\bin\oifcfg setif -global "Local Area Connection 1"/192.168.1.0:cluster_interconnect

However, it is best practice on Windows to rename the interfaces to be more meaningful, such as renaming them to 'ocwpublic' and 'ocwprivate'.   If interface names are renamed after Oracle Clusterware is installed, then you will need to run 'oifcfg'  to add the new interface and delete the old one, as described above.

You can view the available interface names on each node by running the command:

oifcfg iflist -p -n

This command must be run on each node to verify the interface names are defined the same.

Ramifications of Changing Interface Names Using oifcfg

For the Private interface, the database will use the interface stored in the OCR and defined as a 'cluster_interconnect' for cache fusion traffic.  The cluster_interconnect information is available at startup in the alert log, after the parameter listing - for example:

For pre 11.2.0.2:
Cluster communication is configured to use the following interface(s) for this instance 
192.168.1.1


For 11.2.0.2+: (HAIP address will show in alert log instead of private IP)
Cluster communication is configured to use the following interface(s) for this instance
  169.254.86.97

If this is incorrect, then instance is required to restart once the OCR entry is corrected. This applies to ASM instances and Database instances alike. On Windows systems, after shutting down the instance, it is also required to stop/restart the OracleService<SID> (or OracleASMService<ASMSID> before the OCR will be re-read.

 

Oifcfg Usage

To see the full options of oifcfg, simply type:

$ <CRS_HOME>/bin/oifcfg

 

Case V. Add or remove cluster_interconnect for 11gR2 and above with HAIP

1. To add another private network into existing cluster using HAIP, as grid user:

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect

For example:

$ oifcfg setif -global enp0s8/192.168.57.0:cluster_interconnect

 Shutdown CRS on ALL nodes, then restart CRS on ALL nodes for HAIP to pick up the new interface. It is insufficient to restart CRS in rolling manner.

2. To remove a private network from a cluster with HAIP, as grid user:

$ oifcfg delif -global <if_name>

For example:
$ oifcfg delif -global enp0s8

HAIP will failover to the remaining interface and clusterware/database continue to function after the interface removal.

To remove the extra HAIP interface, it is required to shutdown CRS on ALL nodes, then restart CRS on ALL nodes. It is insufficient to restart CRS in rolling manner.

 

 

Database - RAC/Scalability Community
To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Database - RAC/Scalability Community

REFERENCES

NOTE:1054902.1  - How to Validate Network and Name Resolution Setup for the Clusterware and RAC
NOTE:1386709.1  - The Basics of IPv4 Subnet and Oracle Clusterware
NOTE:276434.1  - How to Modify Public Network Information including VIP in Oracle Clusterware
NOTE:604958.1  - OCFS2 Node Fence Caused by Removing the External Network Cable

小麦苗课程

小麦苗课堂开课啦，如下是现有的课程，欢迎咨询小麦苗：

课程名称	课时	上课时间	价格
OCP（从入门到专家）	每年1期，35课时左右/期	20：00-22：00	1588(可优惠)
OCM认证	每年N期，9课时/期	20：00-22：00	22888
高可用课程（rac+dg+ogg）	每年1期，20课时左右/期	20：00-22：00	1888(可优惠)
Oracle初级入门	每年1期，15课时左右/期	20：00-22：00	800
Oracle健康检查脚本	可微信或微店购买。		88
Oracle数据库技能直通车	包含如下3个课程： ①《11g OCP网络课程培训》（面向零基础） 价值1600元 ②《11g OCM网络班课程培训》（Oracle技能合集）价值10000+元 ③《RAC + DG + OGG 高可用网络班课程》 价值2000元 以上3个课程全部打包只要5888，只要5888所有课程带回家，终身指导！所有课程都是在线讲课，不是播放视频，课件全部赠送！ 注意：以上OCP和OCM课程只包括培训课程，不包括考试费用。OCM提供培训+视频，但是不提供练习环境和资料。报名一次，OCP和高可用的课程可以免费终身循环听课。		5888
OCP+高可用（rac+dg+ogg）	报名OCP+高可用课程，可以优惠300元，优惠后的价格为3188.		3188(可优惠)

注意：

1、每次上课前30分钟答疑。

2、OCM实时答疑，提供和考试一样的练习模拟环境，只要按照老师讲的方式来练习，可以保证100%通过。

3、授课方式：YY语音网络直播讲课（非视频） + QQ互动答疑 + 视频复习。

4、OCP课时可以根据大家学习情况进行增加或缩减。

5、以上所有课程均可循环听课。

6、12c OCM课程私聊。

7、Oracle初级入门课程，只教大家最实用+最常用的Oracle操作维护知识。

8、以上所有课程，可以加小麦苗微信（lhrbestxh）或QQ（646634621）详聊，优惠多多。
 

培训项目	连接地址
DB笔试面试历史连接	http://mp.weixin.qq.com/s/Vm5PqNcDcITkOr9cQg6T7w
OCP培训说明连接	https://mp.weixin.qq.com/s/2cymJ4xiBPtTaHu16HkiuA
OCM培训说明连接	https://mp.weixin.qq.com/s/7-R6Cz8RcJKduVv6YlAxJA
高可用（RAC+DG+OGG）培训说明连接	https://mp.weixin.qq.com/s/4vf042CnOdAD8zDyjUueiw
OCP最新题库解析历史连接（052）	http://mp.weixin.qq.com/s/bUgn4-uciSndji_pUbLZfA
微店地址	https://weidian.com/s/793741433?wfr=c&ifr=shopdetail

---

---

About Me

............................................................................................................................................. ● 本文作者：小麦苗，部分内容整理自网络，若有侵权请联系小麦苗删除 ● 本文在itpub（http://blog.itpub.net/26736162/abstract/1/）、博客园（http://www.cnblogs.com/lhrbest）和个人微信公众号（xiaomaimiaolhr）上有同步更新 ● 本文itpub地址：http://blog.itpub.net/26736162/abstract/1/ ● 本文博客园地址：http://www.cnblogs.com/lhrbest ● 本文pdf版、个人简介及小麦苗云盘地址：http://blog.itpub.net/26736162/viewspace-1624453/ ● 数据库笔试面试题库及解答：http://blog.itpub.net/26736162/viewspace-2134706/ ● DBA宝典今日头条号地址：http://www.toutiao.com/c/user/6401772890/#mid=1564638659405826 ............................................................................................................................................. ● QQ群号：230161599（满）、618766405 ● 微信群：可加我微信，我拉大家进群，非诚勿扰 ● 联系我请加QQ好友（646634621），注明添加缘由 ● 于 2018-07-01 06:00 ~ 2018-07-31 24:00 在魔都完成 ● 最新修改时间：2018-07-01 06:00 ~ 2018-07-31 24:00 ● 文章内容来源于小麦苗的学习笔记，部分整理自网络，若有侵权或不当之处还请谅解 ● 版权所有，欢迎分享本文，转载请保留出处 ............................................................................................................................................. ● 小麦苗的微店：https://weidian.com/s/793741433?wfr=c&ifr=shopdetail ● 小麦苗出版的数据库类丛书：http://blog.itpub.net/26736162/viewspace-2142121/ ● 小麦苗OCP、OCM、高可用网络班：http://blog.itpub.net/26736162/viewspace-2148098/ ............................................................................................................................................. 使用微信客户端扫描下面的二维码来关注小麦苗的微信公众号（xiaomaimiaolhr）及QQ群（DBA宝典）、添加小麦苗微信，学习最实用的数据库技术。    小麦苗的微信公众号      小麦苗的DBA宝典QQ群2       小麦苗的微信二维码          小麦苗的微店 .............................................................................................................................................

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/26736162/viewspace-2157364/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/26736162/viewspace-2157364/