HP-UX oracle RAC增加磁阵遇到故障

该磁阵分为4套，已经挂上2套，然后今天又有2套，我也不清楚磁盘是哪些，反正看到磁盘权限不对的，就挂。

chmod 666  

chown grid:asmadmin   

racdb2#[/home/grid]crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

结果重启数据库，起不来了。日志是

[ohasd(3316)]CRS-2807:Resource 'ora.cssd' failed to start automatically.
2018-01-31 19:29:33.760
[ohasd(3316)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.
2018-01-31 19:29:33.760
[ohasd(3316)]CRS-2807:Resource 'ora.evmd' failed to start automatically.
2018-01-31 19:29:33.770
[mdnsd(3458)]CRS-5602:mDNS service stopping by request.
2018-01-31 19:29:35.469
[gpnpd(3463)]CRS-2329:GPNPD on node racdb1 shutdown. 
racdb1#[/u01/app/grid/product/11.2.0/grid/log/racdb1]tail -50f alert*.log
2018-01-31 18:32:57.880
[gpnpd(2660)]CRS-2329:GPNPD on node racdb1 shutdown. 
2018-01-31 19:02:53.523
[ohasd(3316)]CRS-2112:The OLR service started on node racdb1.
2018-01-31 19:02:54.070
[ohasd(3316)]CRS-1301:Oracle High Availability Service started on node racdb1.
2018-01-31 19:02:54.083
[ohasd(3316)]CRS-8017:location: /var/opt/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2018-01-31 19:03:16.703
[ohasd(3316)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). 
2018-01-31 19:03:16.920
[gpnpd(3463)]CRS-2328:GPNPD started on node racdb1. 
2018-01-31 19:03:19.932
[cssd(3482)]CRS-1713:CSSD daemon is started in clustered mode
2018-01-31 19:03:21.729
[ohasd(3316)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2018-01-31 19:13:18.637
[/u01/app/grid/product/11.2.0/grid/bin/cssdagent(3475)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/grid/product/11.2.0/grid/log/racdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2018-01-31 19:13:18.638
[cssd(3482)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/product/11.2.0/grid/log/racdb1/cssd/ocssd.log
2018-01-31 19:13:18.639
[cssd(3482)]CRS-1603:CSSD on node racdb1 shutdown by user.
2018-01-31 19:16:26.542
[ohasd(3316)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'racdb1'.
2018-01-31 19:16:30.103
[ohasd(3316)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2018-01-31 19:16:38.314
[cssd(8712)]CRS-1713:CSSD daemon is started in clustered mod

结果不对，create diskgroup 一直卡住

结果发现，有些磁盘不能动，日志也报错了

racdb1#[/u01/app/grid/product/11.2.0/grid/log/racdb1]tail -10f alertracdb1.log 
2018-01-31 19:29:33.760
[ohasd(3316)]CRS-2807:Resource 'ora.cssd' failed to start automatically.
2018-01-31 19:29:33.760
[ohasd(3316)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.
2018-01-31 19:29:33.760
[ohasd(3316)]CRS-2807:Resource 'ora.evmd' failed to start automatically.
2018-01-31 19:29:33.770
[mdnsd(3458)]CRS-5602:mDNS service stopping by request.

结果吧权限修改回去,数据库正常了。   

chmod 640 

chown bin:sys

[ohasd(9806)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2018-01-31 20:23:41.526
[cssd(12258)]CRS-1713:CSSD daemon is started in clustered mode
2018-01-31 20:24:21.288
[cssd(12258)]CRS-1707:Lease acquisition for node racdb1 number 1 completed
2018-01-31 20:24:22.801
[cssd(12258)]CRS-1605:CSSD voting file is online: /dev/rdisk/disk3375; details in /u01/app/grid/product/11.2.0/grid/log/racdb1/cssd/ocssd.log.
2018-01-31 20:24:22.899
[cssd(12258)]CRS-1605:CSSD voting file is online: /dev/rdisk/disk3376; details in /u01/app/grid/product/11.2.0/grid/log/racdb1/cssd/ocssd.log.
2018-01-31 20:24:22.998
[cssd(12258)]CRS-1605:CSSD voting file is online: /dev/rdisk/disk3377; details in /u01/app/grid/product/11.2.0/grid/log/racdb1/cssd/ocssd.log.
2018-01-31 20:24:32.277
[cssd(12258)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racdb1 .
2018-01-31 20:24:34.383
[ctssd(12340)]CRS-2403:The Cluster Time Synchronization Service on host racdb1 is in observer mode.
2018-01-31 20:24:34.811
[ctssd(12340)]CRS-2407:The new Cluster Time Synchronization Service reference node is host racdb1.
2018-01-31 20:24:34.812
[ctssd(12340)]CRS-2401:The Cluster Time Synchronization Service started on host racdb1.
2018-01-31 20:25:50.377
[crsd(12684)]CRS-1012:The OCR service started on node racdb1.
2018-01-31 20:25:51.318
[evmd(12344)]CRS-1401:EVMD started on node racdb1.
2018-01-31 20:25:53.850
[crsd(12684)]CRS-1201:CRSD started on node racdb1.
2018-01-31 20:25:57.063
[/u01/app/grid/product/11.2.0/grid/bin/oraagent.bin(12753)]CRS-5016:Process "/u01/app/grid/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/grid/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/grid/product/11.2.0/grid/log/racdb1/agent/crsd/oraagent_grid/oraagent_grid.log"
2018-01-31 20:25:57.063
[/u01/app/grid/product/11.2.0/grid/bin/oraagent.bin(12753)]CRS-5016:Process "/u01/app/grid/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/grid/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/grid/product/11.2.0/grid/log/racdb1/agent/crsd/oraagent_grid/oraagent_grid.log"
2018-01-31 20:25:57.142
[/u01/app/grid/product/11.2.0/grid/bin/oraagent.bin(12753)]CRS-5016:Process "/u01/app/grid/product/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/app/grid/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/grid/product/11.2.0/grid/log/racdb1/agent/crsd/oraagent_grid/oraagent_grid.log"
2018-01-31 20:25:57.282
[crsd(12684)]CRS-2772:Server 'racdb1' has been assigned to pool 'Generic'.

racdb2#[/home/grid]crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

红色框部分，是恢复回来的。