oracle gi 安装时需要设置多播( multicast )

云环境 ecs 默认是禁止组播的,需要安装aliyun自己开发的工具,否则会报错(multicast or broadcast check),提示节点无法访问.

规划如下

#node1
10.10.7.101       nodea1
10.10.7.201       nodea1-vip #
192.168.56.101    nodea1-priv

#node2
10.10.7.102   	  nodea2
10.10.7.202       nodea2-vip #
192.168.56.102    nodea2-priv

#scan ip
10.10.7.221   	  clustera-scan #
10.10.7.222   	  clustera-scan #
10.10.7.223   	  clustera-scan #

网卡要首先支持 broadcast,multicast

# ifconfig -a 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

安装

安装组播多播代理程序

# uname -a
Linux nodea1 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# yum list installed | grep -i kernel-devel
kernel-devel.x86_64                     3.10.0-957.5.1.el7             @updates 

# yum list installed | grep -i rpm-build
rpm-build-libs.x86_64                   4.11.3-35.el7                  @anaconda

# yum install kernel-devel rpm-build

从 https://github.com/aliyun/multicast_proxy 下载程序
文件夹说明

multicast_kernel: 是linux下多播代理程序内核版本。
multicast_win: 是windows下多播代理程序。
multicast_linux: 是linux下多播代理程序。

Suggest:
1. 如果出现用户态组播工具占用CPU过高的情况，建议使用内核版本。
2. 如果有组播性能要求，建议使用内核版本。

The more infomation in the detail README in the directory of different version.

# cd /opt
# unzip multicast_proxy-master.zip
# ls -l 
drwxr-xr-x 7 root root    4096 May  8 18:29 multicast_proxy-master
-rw-r--r-- 1 root root 4460945 May 24 09:02 multicast_proxy-master.zip

# cd multicast_proxy-master/
# patch -p1 < multicast_kernel/patch/support_broadcast.patch
# ls -l
total 60
-rw-r--r-- 1 root root 35141 May  8 18:29 LICENSE
drwxr-xr-x 6 root root  4096 May  8 18:29 multicast_kernel
drwxr-xr-x 8 root root  4096 May  8 18:29 multicast_linux
drwxr-xr-x 5 root root  4096 May  8 18:29 multicast_win
-rw-r--r-- 1 root root   431 May  8 18:29 README
drwxr-xr-x 2 root root  4096 May  8 18:29 rpm_version
drwxr-xr-x 2 root root  4096 May  8 18:29 script

# cd multicast_kernel/

# ls -l
total 44
drwxr-xr-x 2 root root 4096 May  8 18:29 conf_auto_make_script
-rw-r--r-- 1 root root  203 May  8 18:29 Makefile
-rw-r--r-- 1 root root 1517 May  8 18:29 multicast_auto_build.sh
drwxr-xr-x 7 root root 4096 May  8 18:29 multi_client
-rw-r--r-- 1 root root 1154 May  8 18:29 multi_client.spec
drwxr-xr-x 7 root root 4096 May  8 18:29 multi_server
-rw-r--r-- 1 root root 1153 May  8 18:29 multi_server.spec
drwxr-xr-x 2 root root 4096 May  8 18:29 patch
-rw-r--r-- 1 root root  527 May  8 18:29 README
-rwxr-xr-x 1 root root 1169 May  8 18:29 tmcc_client_auto_rpm.sh
-rwxr-xr-x 1 root root 1154 May  8 18:29 tmcc_server_auto_rpm.sh

# ./tmcc_client_auto_rpm.sh;
./tmcc_server_auto_rpm.sh;

# rpm -Uvh multi_server-1.1-1.x86_64.rpm;
rpm -Uvh multi_client-1.1-1.x86_64.rpm;

配置启动

# chkconfig --list |grep -i multi

# chkconfig multis on --level 2345;
chkconfig multis off --level 016;

chkconfig multic on --level 2345;
chkconfig multic off --level 016;


# /etc/init.d/multis start;
/etc/init.d/multic start;

组播配置

注意 224.1.2.3 是固定XX地址,这里需要注意的是,下面的设置是 precheck 可以通过的设置,后面 crs,rac 正常运行时需要重新设置 230.0.1.0 和 224.0.0.251 的 42424 端口.
node1 配置组播服务端

# 
/usr/local/sbin/multis_admin -A -m 230.0.1.0 -j 192.168.56.102 ;
/usr/local/sbin/multis_admin -A -m 224.0.0.251 -j 192.168.56.102 ;
/usr/local/sbin/multis_admin -A -m 224.1.2.3 -j 192.168.56.102 ;

node1 配置组播客户端

# 
/usr/local/sbin/multic_admin -A -i 192.168.56.102 -p 42424 -m 230.0.1.0;
/usr/local/sbin/multic_admin -A -i 192.168.56.102 -p 8524 -m 224.0.0.251;
/usr/local/sbin/multic_admin -A -i 192.168.56.102 -p 65535 -m 224.1.2.3;

node2 配置组播服务端

# 
/usr/local/sbin/multis_admin -A -m 230.0.1.0 -j 192.168.56.101 ;
/usr/local/sbin/multis_admin -A -m 224.0.0.251 -j 192.168.56.101 ;
/usr/local/sbin/multis_admin -A -m 224.1.2.3 -j 192.168.56.101 ;

node2 配置组播客户端

#
/usr/local/sbin/multic_admin -A -i 192.168.56.101 -p 42424 -m 230.0.1.0;
/usr/local/sbin/multic_admin -A -i 192.168.56.101 -p 8524 -m 224.0.0.251;
/usr/local/sbin/multic_admin -A -i 192.168.56.101 -p 65535 -m 224.1.2.3;

查看配置情况

# /usr/local/sbin/multis_admin -S ;
/usr/local/sbin/multic_admin -L ;

/usr/local/sbin/multis_admin -P ;
/usr/local/sbin/multic_admin -P ;

清除配置

# /usr/local/sbin/multis_admin -C ;
/usr/local/sbin/multic_admin -C ;

/etc/init.d/multis stop;
/etc/init.d/multic stop;
/etc/init.d/multis start;
/etc/init.d/multic start;

# service multis stop;
service multic stop;
service multis start;
service multic start;


# lsmod |grep -i multi
multi_client           30089  0 
multi_server           59384  0 

也可以查看配置文件是否配置正确：

# cat /usr/local/etc/multi_server_startup_config;
cat /usr/local/etc/multi_server_running_config;
cat /usr/local/etc/multi_client_startup_config;
cat /usr/local/etc/multi_client_running_config;

tcpdump 抓取

node1上

# tcpdump -i eth1 -nnvv host 192.168.56.102 and udp

node2上

# tcpdump -i eth1 -nnvv host 192.168.56.101 and udp 

使用 oracle 提供的组播测试工具 mcasttest.本次测试无效

# su - grid
$ cd mcasttest/
$ ./mcasttest.pl -n nodea1,nodea2 -i eth1,eth1

# su - grid
$ cd $ORACLE_HOME
$ ./runcluvfy.sh stage -pre crsinst -n nodea1,nodea2 -verbose

参考:
https://help.aliyun.com/document_detail/58463.html?spm=a2c4g.11186623.2.10.26e44c078NlcyP

<<Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement (文档 ID 1212703.1)>>
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=519604170813151&id=1212703.1&_adf.ctrl-state=u3rk4310y_245

<<How to Validate Network and Name Resolution Setup for the Clusterware and RAC (文档 ID 1054902.1)>>
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=201099149491597&id=1054902.1&_adf.ctrl-state=13f3zs5een_77