Oracle案例：深入解析ASM rebalance无法启动

点击上方蓝字关注我们

某银行ODS系统的一体机(数据库版本为19.8)上，由于某个存储节点掉了4块盘，磁盘处于offline状态，在超过了“_asm_disk_repair_time”时间内没有online，被磁盘组自动drop force，之后在drop disk rebalance未完成的情况下，将4块盘重新加入了磁盘组，由于担心rebalance影响ODS跑批业务，所以在跑批阶段中断rebalance操作，在空闲时重新发起rebalance，反复启停rebalance很多次，但是在某一次中断rebalance之后，发现rebalance就再也无法启动了。

2021-06-30T17:17:21.183103+08:00
SQL> alter diskgroup datac1 rebalance power 0 
2021-06-30T17:17:21.186100+08:00
NOTE: cache closing disk 77 of grp 1: (not open) _DROPPED_0077_DATAC1
2021-06-30T17:17:21.186162+08:00
NOTE: cache closing disk 107 of grp 1: (not open) _DROPPED_0107_DATAC1
2021-06-30T17:17:21.186216+08:00
NOTE: cache closing disk 113 of grp 1: (not open) _DROPPED_0113_DATAC1
2021-06-30T17:17:21.186268+08:00
NOTE: cache closing disk 119 of grp 1: (not open) _DROPPED_0119_DATAC1
2021-06-30T17:17:21.187265+08:00
NOTE: GroupBlock outside rolling migration privileged region
2021-06-30T17:17:21.227738+08:00
NOTE: stopping process ARBA
2021-06-30T17:17:25.269379+08:00
NOTE: rebalance interrupted for group 1/0xa1b08317 (DATAC1)
...
2021-06-30T17:18:57.863092+08:00
SQL>  alter diskgroup datac1 rebalance power 5 
2021-06-30T17:18:57.866314+08:00
NOTE: cache closing disk 77 of grp 1: (not open) _DROPPED_0077_DATAC1
2021-06-30T17:18:57.866376+08:00
NOTE: cache closing disk 107 of grp 1: (not open) _DROPPED_0107_DATAC1
2021-06-30T17:18:57.866430+08:00
NOTE: cache closing disk 113 of grp 1: (not open) _DROPPED_0113_DATAC1
2021-06-30T17:18:57.866482+08:00
NOTE: cache closing disk 119 of grp 1: (not open) _DROPPED_0119_DATAC1
2021-06-30T17:18:57.867615+08:00
NOTE: GroupBlock outside rolling migration privileged region

从日志中可以发现，从17点17分中断rebalance之后，在18分重新发起rebalance，从alert日志以及rbal trace文件可以看到rbal进程再也没有ARBn进程去做rebalance。

仔细的查阅了rebalance相关的后台进程trace以及ASM alert日志都没有任何有用的信息。从发起rebalance命令的进程trace中，发现了非常重要的信息。

ksedsts()+426<-kfnmGroupBlockGlobal()+659<-kfnmGroupBlockPriv()+318<-kfgFinalize()+334<-kfxdrvAlter()+3415<-kfxdrvEntry()+1417<-opiexe()+28735<-opiosq0()+4494<-kpooprx()+387<-kpoal8()+830<-opiodr()+1202<-ttcpip()+1222<-opitsk()+1903<-opiino()+936<-opiodr()+1202
<-opidrv()+1094<-sou2o()+165<-opimai_real()+422<-ssthrdmain()+417<-main()+256<-__libc_start_main()+245 
----- End of Abridged Call Stack Trace -----
Partial short call stack signature: 0xb0ac14de6c5e2e9c
SQL> alter diskgroup datac1 rebalance modify power 2
kfgbSendRebalUpdate: xic 2600134044 gn 1 power 2 phase 0 flags 0x1 op=0
NOTE: detected a paused rebalance of group DATAC1
kfgpCreate: max_fg_rel 4, max_disk_part 8
kfgpPartners: NOT appliance.
kfgpPartners: max_fg_rel, max_disk_part(4, 8) has been adjusted to (3, 8) due to actual FG, disk configuration (3, 144, num_singledisk_fg 0)
kfgpPartner: necessary rebalancing detected. Avail slot for disk120 7 target 8
WARNING: Too many uncompleted reconfigurations. Rebalance needs completion.

从trace的报错来看，熟悉ASM元数据和ASM相关函数的应该知道kfgp开头的都是PST相关的函数，虽然报错相关信息比较陌生，但是至少可以给问题分析确定了一个方向