dsc动态扩展节点

dsc动态扩展节点

1.配置环境

dsc0 192.168.6.71 centos7 64位 外部服务ip 和内部通信ip

dsc1 192.168.6.75 centos7 64位 外部服务ip 和内部通信ip

dsc2 192.168.6.69 centos7 64位 外部服务ip 和内部通信ip

确认第三节点已安装数据库软件

2.操作步骤

2.1在192.168.6.71(DSC0)机器上dmasmcmd导出备份export dmdcr_cfg_bak.ini

[dmdba@root bin]$ ./dmasmcmd
DMASMCMD V8
ASM>export dcrdisk  '/dev/raw/raw1' to '/home/dmdba/dmdbms/config/dmdcr_cfg_bak.ini'
ASMCMD export DCRDISK success.
Used time: 49.005(ms).

2.2为新增节点准备日志文件(DSC0)

2.2.1至少增加两个日志文件，路径必须asm文件格式，大小可以参考两个活动节点

[dmdba@root bin]$ ./disql
disql V8
用户名:
密码:

服务器[LOCALHOST:5236]:处于普通打开状态
登录使用时间: 8.705(毫秒)
SQL> alter database add node logfile '+DMLOG/log/DSC2_log01.log' size 256, '+DMLOG/log/DSC2_log02.log' size 256;
操作已执行
已用时间: 404.147(毫秒). 执行号:3.

2.2.2使用dmctlcvt工具将dm.ctl转换为文本文件dmctl.txt,查看dmctl.txt.

[dmdba@root bin]$ ./dmctlcvt TYPE=1 SRC=+DMDATA/data/dsc/dm.ctl DEST=/home/dmdba/dmctl.txt DCR_INI=/home/dmdba/dmdbms/config/dmdcr.ini
DMCTLCVT V8
convert ctl to txt success!

2.2.3使用dmasmtool工具登录asm文件系统，也可以看到新增日志节点

[dmdba@root bin]$ ./dmasmtool dcr_ini=/home/dmdba/dmdbms/config/dmdcr.ini
DMASMTOOL V8
ASM>ls +DMLOG/log
	file : dsc0_log01.log
	file : dsc0_log02.log
	file : dsc1_log01.log
	file : dsc1_log02.log
	file : DSC2_log01.log
	file : DSC2_log02.log
total count 6.
Used time: 3.977(ms).
ASM>

3.为新增节点准备config_path

将 192.168.6.71 机器/home/dmdba/dmdbms/config/dsc0_config 目录拷贝到 192.168.6.69 机器相

同目录下，修改名字为/home/dmdba/dmdbms/config/dsc2_config。

[dmdba@root config]$ scp -r dsc0_config/ dmdba@192.168.6.69:/home/dmdba/dmdbms/config/
dmdba@192.168.6.69's password: dmdba

[dmdba@dm3 config]$ ls
dsc0_config  trace
[dmdba@dm3 config]$ mv dsc0_config/ dsc2_config
[dmdba@dm3 config]$ ls
dsc2_config  trace
[dmdba@dm3 config]$ 

3.1修改dm.ini

[dmdba@dm3 dsc2_config]$ cat dm.ini |grep INSTANCE
		INSTANCE_NAME                   = DSC2              #Instance name
[dmdba@dm3 dsc2_config]$ cat dm.ini |grep CONFIG_PATH
		CONFIG_PATH                     = /home/dmdba/dmdbms/config/dsc2_config            #config path
		

3.2修改dmarch.ini

[dmdba@root dsc0_config]$ cat dmarch.ini 
[LOCAL_LOCAL1]
ARCH_TYPE=LOCAL
ARCH_DEST=/home/dmdba/dmdbms/dmarch
ARCH_FILE_SIZE=1024
ARCH_SPACE_LIMIT=10240
[dmdba@root dsc0_config]$ 


[dmdba@root dsc0_config]$ scp /home/dmdba/dmdbms/config/dsc0_config/dmarch.ini  dmdba@192.168.6.71:/home/dmdba/dmdbms/config/dsc1_config/dmarch.ini

dmdba@192.168.6.71's password: 
dmarch.ini                                                                                                                               100%  110   224.7KB/s   00:00    

[dmdba@root dsc0_config]$ scp /home/dmdba/dmdbms/config/dsc0_config/dmarch.ini  dmdba@192.168.6.69:/home/dmdba/dmdbms/config/dsc2_config/dmarch.ini
dmdba@192.168.6.69's password: 
dmarch.ini                                                                                                                               100%  110   124.6KB/s   00:00    
[dmdba@root dsc0_config]$ 

4.新建dmdcr.ini

新建 dmdcr.ini 配置文件，保存到节点3 192.168.6.69 的/home/dmdba/dmdbms/config目录下面,注意设置 dmdcr_seqo 为 2，修改 dm.ini 路径

DMDCR_PATH = /dev/raw/raw1
DMDCR_MAL_PATH = /home/dmdba/dmdbms/config/dmasvrmal.ini
DMDCR_SEQNO = 2

#ASM重启参数，命令行方式启动
#DMDCR_ASM_RESTART_INTERVAL = 30 #DMCSS 认定 DMASM 节点故障重启的时间间隔，默认60S
#DMDCR_ASM_STARTUP_CMD = /home/dmdba/dmdbms/bin/DmASMSvrServiceDSC2 start

#DB重启参数，命令行方式启动
#DMDCR_DB_RESTART_INTERVAL = 60 #DMCSS 认定 DMDSC 节点故障重启的时间间隔
#DMDCR_DB_STARTUP_CMD = /home/dmdba/dmdbms/bin/DmServiceDSC2 start

#指定参数后，如果超过指定时间节点实例还未启动，DMCSS 自动将未启动节点踢出集群环境，变为 ERROR 节点，之后其他活动 OK 节点可以正常启动
#DMDCR_AUTO_OPEN_CHECK=60

5. 修改当前环境的 MAL 配置文件

直接修改当前环境的 dmasvrmal.ini 文件，添加新增节点信息，使用 DMASM 的所有

节点都要配置，内容完全一样，并且将新增信息后的 dmasvrmal.ini 文件拷贝到节点3

192.168.6.69的/home/dmdba/dmdbms/ 目录下。

[dmdba@root config]$ cat  dmasvrmal.ini 
[MAL_INST1]
MAL_INST_NAME = ASM0
MAL_HOST = 192.168.6.71
MAL_PORT = 7236
[MAL_INST2]
MAL_INST_NAME = ASM1
MAL_HOST = 192.168.6.75
MAL_PORT = 7237
[MAL_INST3]
MAL_INST_NAME = ASM2
MAL_HOST = 192.168.6.69
MAL_PORT = 7238

[dmdba@root config]$ scp dmasvrmal.ini  dmdba@192.168.6.75:/home/dmdba/dmdbms/config/
dmdba@192.168.6.75's password: 
 
[dmdba@root config]$ scp dmasvrmal.ini dmdba@192.168.6.69:/home/dmdba/dmdbms/config/
dmdba@192.168.6.69's password: 
  

直接修改 dmserver 三个实例的 dmmal.ini，添加新增节点信息，所有节点都要配置

相同内容，保存到各自的 dsc_config 目录下。

[mal_inst0]
    mal_inst_name  = DSC0 
    mal_host       = 192.168.6.71 
    mal_port       = 9340 

[mal_inst1]
    mal_inst_name  = DSC1 
    mal_host       = 192.168.6.75 
    mal_port       = 9341 

[mal_inst2]
    mal_inst_name  = DSC2
    mal_host       = 192.168.6.69
    mal_port       = 9342
[dmdba@root bin]$ scp /home/dmdba/dmdbms/config/dsc0_config/dmmal.ini dmdba@192.168.6.75:/home/dmdba/dmdbms/config/dsc1_config/dmmal.ini

[dmdba@root bin]$ scp /home/dmdba/dmdbms/config/dsc0_config/dmmal.ini dmdba@192.168.6.69:/home/dmdba/dmdbms/config/dsc2_config/dmmal.ini

6.修改 dmdcr_cfg_bak.ini，添加新增节点信息，CSS/ASMSVR/DB 都要配置

DCR_GRP_N_EP = 3

DCR_GRP_EP_ARR = {0,1,2}

注意端口号不要冲突

[dmdba@root config]$ cat dmdcr_cfg_bak.ini 
# the file is auto-created by system, self edit is invalid!
#DCR HDR
DCR_N_GRP              = 3
DCR_VTD_PATH           = /dev/raw/raw2
DCR_OGUID              = 63635

[GRP]
DCR_GRP_TYPE           = CSS
DCR_GRP_NAME           = GRP_CSS
DCR_GRP_N_EP           = 3
DCR_GRP_EP_ARR         = {0,1,2}
DCR_GRP_N_ERR_EP       = 0
DCR_GRP_ERR_EP_ARR     = {}
DCR_GRP_DSKCHK_CNT     = 60

[GRP]
DCR_GRP_TYPE           = ASM
DCR_GRP_NAME           = GRP_ASM
DCR_GRP_N_EP           = 3
DCR_GRP_EP_ARR         = {0,1,2}
DCR_GRP_N_ERR_EP       = 0
DCR_GRP_ERR_EP_ARR     = {}
DCR_GRP_DSKCHK_CNT     = 60

[GRP]
DCR_GRP_TYPE           = DB
DCR_GRP_NAME           = GRP_DSC
DCR_GRP_N_EP           = 3
DCR_GRP_EP_ARR         = {0,1,2}
DCR_GRP_N_ERR_EP       = 0
DCR_GRP_ERR_EP_ARR     = {}
DCR_GRP_DSKCHK_CNT     = 60

[GRP_CSS]
DCR_EP_NAME        = CSS0
DCR_EP_HOST        = 192.168.6.71
DCR_EP_PORT        = 9341

[GRP_CSS]
DCR_EP_NAME        = CSS1
DCR_EP_HOST        = 192.168.6.75
DCR_EP_PORT        = 9343

[GRP_CSS]
DCR_EP_NAME        = CSS2
DCR_EP_HOST        = 192.168.6.69
DCR_EP_PORT        = 9345

[GRP_ASM]
DCR_EP_NAME        = ASM0
DCR_EP_SHM_KEY     = 93360
DCR_EP_SHM_SIZE    = 10
DCR_EP_HOST        = 192.168.6.71
DCR_EP_PORT        = 9349
DCR_EP_ASM_LOAD_PATH  = /dev/raw

[GRP_ASM]
DCR_EP_NAME        = ASM1
DCR_EP_SHM_KEY     = 93361
DCR_EP_SHM_SIZE    = 10
DCR_EP_HOST        = 192.168.6.75
DCR_EP_PORT        = 9351
DCR_EP_ASM_LOAD_PATH  = /dev/raw

[GRP_ASM]
DCR_EP_NAME        = ASM2
DCR_EP_SHM_KEY     = 93362
DCR_EP_SHM_SIZE    = 10
DCR_EP_HOST        = 192.168.6.69
DCR_EP_PORT        = 9352
DCR_EP_ASM_LOAD_PATH  = /dev/raw

[GRP_DSC]
DCR_EP_NAME        = DSC0
DCR_EP_SEQNO       = 0
DCR_EP_PORT        = 5236
DCR_CHECK_PORT     = 9741

[GRP_DSC]
DCR_EP_NAME        = DSC1
DCR_EP_SEQNO       = 1
DCR_EP_PORT        = 5236
DCR_CHECK_PORT     = 9742

[GRP_DSC]
DCR_EP_NAME        = DSC2
DCR_EP_SEQNO       = 2
DCR_EP_PORT        = 5236
DCR_CHECK_PORT     = 9743

7.使用dmasmcmd将新增节点信息写回磁盘，新增节点作为 error 节点

[dmdba@root bin]$ ./dmasmcmd
DMASMCMD V8
ASM>extend dcrdisk '/dev/raw/raw1' from '/home/dmdba/dmdbms/config/dmdcr_cfg_bak.ini'
ASMCMD extend node for dcr disk success.
ASMCMD extend node for vote disk success.
Used time: 113.999(ms).

8.dmcssm监视器

[dmdba@root bin]$ ./dmcssm ini_path=/home/dmdba/dmdbms/config/dmcssm.ini 
[monitor]         2020-09-03 03:01:13: CSS MONITOR V8
[monitor]         2020-09-03 03:01:13: CSS MONITOR SYSTEM IS READY.

[monitor]         2020-09-03 03:01:13: Wait CSS Control Node choosed...
[monitor]         2020-09-03 03:01:16: Wait CSS Control Node choosed succeed.

extend node
[monitor]         2020-09-03 03:01:25: 执行扩展节点动作

[monitor]         2020-09-03 03:01:32: 接收CSS(seqno: 0)消息超时
[monitor]         2020-09-03 03:01:32: 通知当前活动的CSS执行清理操作
[monitor]         2020-09-03 03:01:32: 命令EXTENT NODE 执行失败

show

monitor current time:2020-09-03 03:02:02, n_group:3
=================== group[name = GRP_CSS, seq = 0, type = CSS, Control Node = 0] ========================================

[CSS0] global info:
[ASM0] auto restart = TRUE
[DSC0] auto restart = TRUE

[CSS1] global info:
[ASM1] auto restart = TRUE
[DSC1] auto restart = TRUE

[CSS2] global info:
Connect to [CSS2] failed, please check the network or the CSSM_CSS_IP config in [/home/dmdba/dmdbms/config/dmcssm.ini] .

ep:	css_time               inst_name     seqno     port    mode         inst_status        vtd_status   is_ok        active       guid              ts              
	2020-09-03 03:02:02    CSS0          0         9341    Control Node OPEN               WORKING      OK           TRUE         322342780         322356227       
	2020-09-03 03:02:02    CSS1          1         9343    Normal Node  OPEN               WORKING      OK           TRUE         322343353         322356957       
	2020-09-03 03:02:02    CSS2          2         9345    Normal Node  SHUTDOWN           UNKNOWN      OK           FALSE        0                 0               

=================== group[name = GRP_ASM, seq = 1, type = ASM, Control Node = 0] ========================================

n_ok_ep = 2
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)

sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL

crash process over flag is TRUE
ep:	css_time               inst_name     seqno     port    mode         inst_status        vtd_status   is_ok        active       guid              ts              
	2020-09-03 03:02:02    ASM0          0         9349    Control Node OPEN               WORKING      OK           TRUE         322353464         322367075       
	2020-09-03 03:02:02    ASM1          1         9351    Normal Node  OPEN               WORKING      OK           TRUE         322353716         322367335       
	2020-09-03 03:02:02    ASM2          2         9352    Normal Node  SHUTDOWN           UNKNOWN      ERROR        FALSE        0                 0               

=================== group[name = GRP_DSC, seq = 2, type = DB, Control Node = 0] ========================================

n_ok_ep = 2
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)

sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL

crash process over flag is TRUE
ep:	css_time               inst_name     seqno     port    mode         inst_status        vtd_status   is_ok        active       guid              ts              
	2020-09-03 03:02:02    DSC0          0         5236    Control Node OPEN               WORKING      OK           TRUE         1359998485        1360012046      
	2020-09-03 03:02:02    DSC1          1         5236    Normal Node  OPEN               WORKING      OK           TRUE         1359999413        1360012977      
	2020-09-03 03:02:02    DSC2          2         5236    Normal Node  SHUTDOWN           UNKNOWN      ERROR        FALSE        0                 0               

==================================================================================================================

ASMSVR/dmserver 是 error 节点， 程 序 会 通 知ASMSVR/dmserver 更新 MAL 信息

9.启动DMCSS，DMASM服务程序

在第三个节点启动dmcss，dmasmsvr程序

[dmdba@dm3 bin]$ ./dmcss dcr_ini=/home/dmdba/dmdbms/config/dmdcr.ini 
DMCSS V8
DMCSS IS READY

[dmdba@dm3 bin]$ ./dmasmsvr dcr_ini=/home/dmdba/dmdbms/config/dmdcr.ini 
ASM SELF EPNO:2
DMASMSVR V8
dmasmsvr task worker thread startup
the ASM server is Ready.


[dmdba@dm3 bin]$./dmserver  /home/dmdba/dmdbms/config/dsc2_config/dm.ini dcr_ini=/home/dmdba/dmdbms/config/dmdcr.ini

9.1注册服务

[root@dm3 root]# ./dm_service_installer.sh -t dmcss -p dm3 -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini

[root@dm3 root]# ./dm_service_installer.sh -t dmasmsvr -p dm3 -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini -y DmCSSServicedm3

[root@dm3 root]# ./dm_service_installer.sh -t dmserver -p dm3 -dm_ini /home/dmdba/dmdbms/config/dsc2_config/dm.ini -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini -y DmASMSvrServicedm3

注册dmcss
[root@dm3 root]# ./dm_service_installer.sh  -t dmcss -p dm3 -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini  
Created symlink from /etc/systemd/system/multi-user.target.wants/DmCSSServicedm3.service to /usr/lib/systemd/system/DmCSSServicedm3.service.
创建服务(DmCSSServicedm3)完成


注册dmasm
[root@dm3 root]# ./dm_service_installer.sh  -t dmasmsvr -p dm3 -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini 
dmasmsvr(RAC)服务需设置依赖服务(dmcss)!

[root@dm3 root]# ./dm_service_installer.sh  -t dmasmsvr -p dm3 -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini  -y DmCSSServicedm3
Created symlink from /etc/systemd/system/multi-user.target.wants/DmASMSvrServicedm3.service to /usr/lib/systemd/system/DmASMSvrServicedm3.service.
创建服务(DmASMSvrServicedm3)完成


注册dmservice
[root@dm3 root]# ./dm_service_installer.sh  -t dmserver  -p dm3 -dm_ini /home/dmdba/dmdbms/config/dsc2_config/dm.ini -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini
dmserver(RAC)服务需设置依赖服务(dmasmsvr)!
[root@dm3 root]# ./dm_service_installer.sh  -t dmserver  -p dm3 -dm_ini /home/dmdba/dmdbms/config/dsc2_config/dm.ini -dcr_ini /home/dmdba/dmdbms/config/dmdcr.ini -y DmASMSvrServicedm3
Created symlink from /etc/systemd/system/multi-user.target.wants/DmServicedm3.service to /usr/lib/systemd/system/DmServicedm3.service.


##删除服务
[root@dm3 root]# ./dm_service_uninstaller.sh  -n DmServicedm3
是否删除服务(DmServicedm3)?(Y/y:是 N/n:否): y
Removed symlink /etc/systemd/system/multi-user.target.wants/DmServicedm3.service.
删除服务文件(/usr/lib/systemd/system/DmServicedm3.service)完成
删除服务(DmServicedm3)完成




提示可以输入-h

9.2报错

[dmdba@dm3 bin]$ ./dmcss dcr_ini=/home/dmdba/dmdbms/config/dmdcr.ini 
dmcss startup failed:[CODE:-4546],error info:Invalid file path[/dev/raw/raw1]

解决

[root@dm3 ~]# cat  /etc/udev/rules.d/60-raw.rules
ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdb2", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdb3", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sdb4", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="raw[1-4]", OWNER="dmdba", GROUP="dinstall", MODE="660"

[root@dm3 ~]# vi   /etc/udev/rules.d/60-raw.rules
[root@dm3 ~]# partprobe /dev/sdb
[root@dm3 ~]# ll /dev/raw/
total 0
crw-rw---- 1 dmdba dinstall 162, 1 Sep  3 03:21 raw1
crw-rw---- 1 dmdba dinstall 162, 2 Sep  3 03:21 raw2
crw-rw---- 1 dmdba dinstall 162, 3 Sep  3 03:21 raw3
crw-rw---- 1 dmdba dinstall 162, 4 Sep  3 03:21 raw4
crw-rw---- 1 root  disk     162, 0 Sep  3 03:11 rawctl

又报错

Waitpid error!
sh: /home/dmdba/dmdbms/bin/DmASMSvrServiceDSC2: No such file or directory
 [CSS]: 重启本地ASM实例，命令：[/home/dmdba/dmdbms/bin/DmASMSvrServiceDSC2 start]
 asm文件配置有问题，检查一遍