12201跑rootupgrade.sh出现rootcrs.pl -upgrade‘ execution failed

Rootupgrade.sh遇到CLSRSC-400: A system reboot is required to continue installing.

运行rootupgrad.sh遇到以下告警

[root@ora12c1 rpm]# /u01/app/12.0.0/grid_1/rootupgrade.sh

Performing root user operation.

The following environment variables are set as:

    ORACLE_OWNER= grid

    ORACLE_HOME=  /u01/app/12.0.0/grid_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Using configuration parameter file: /u01/app/12.0.0/grid_1/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/ora12c1/crsconfig/rootcrs_ora12c1_2018-11-14_09-34-27PM.log

2018/11/14 21:34:31 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.

2018/11/14 21:34:31 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2018/11/14 21:35:13 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2018/11/14 21:35:13 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.

2018/11/14 21:35:18 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.

2018/11/14 21:35:19 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.

2018/11/14 21:35:19 CLSRSC-464: Starting retrieval of the cluster configuration data

2018/11/14 21:35:24 CLSRSC-515: Starting OCR manual backup.

2018/11/14 21:35:27 CLSRSC-516: OCR manual backup successful.

2018/11/14 21:35:35 CLSRSC-486:

 At this stage of upgrade, the OCR has changed.

 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2018/11/14 21:35:35 CLSRSC-541:

 To downgrade the cluster:

 1. All nodes that have been upgraded must be downgraded.

2018/11/14 21:35:35 CLSRSC-542:

 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2018/11/14 21:35:35 CLSRSC-615:

 3. The last node to downgrade cannot be a Leaf node.

2018/11/14 21:35:38 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2018/11/14 21:35:39 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.

2018/11/14 21:35:40 CLSRSC-363: User ignored prerequisites during installation

2018/11/14 21:35:48 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.

2018/11/14 21:35:51 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.

2018/11/14 21:35:59 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.

2018/11/14 21:36:02 CLSRSC-470: Starting non-rolling migration of Oracle ASM

2018/11/14 21:36:02 CLSRSC-482: Running command: '/u01/app/12.0.0/grid_1/bin/asmca -silent -upgradeNodeASM -nonRolling true -oldCRSHome /u01/app/12.0.0/grid -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2018/11/14 21:36:05 CLSRSC-471: Successfully initiated non-rolling migration of Oracle ASM

2018/11/14 21:36:14 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2018/11/14 21:36:34 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

2018/11/14 21:36:35 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.

2018/11/14 21:36:36 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.

2018/11/14 21:36:39 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.

2018/11/14 21:36:39 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.

2018/11/14 21:36:41 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.

2018/11/14 21:36:44 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.

2018/11/14 21:36:59 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

2018/11/14 21:37:40 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora12c1'

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora12c1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

2018/11/14 21:38:23 CLSRSC-400: A system reboot is required to continue installing.

The command '/u01/app/12.0.0/grid_1/perl/bin/perl -I/u01/app/12.0.0/grid_1/perl/lib -I/u01/app/12.0.0/grid_1/crs/install /u01/app/12.0.0/grid_1/crs/install/rootcrs.pl  -upgrade' execution failed

该状态集群属于升级到一半，12102软件已无法启动，12201的软件可以启动，但是在检查版本时，认为是12102，db升级无法继续

查看日志

路径在/u01/app/grid/crsdata/ora12c1/crsconfig/

关键部分如下

>  CRS-4621: Oracle High Availability Services autostart is disabled.

>End Command output

2018-11-16 10:08:11: /u01/app/12.0.0/grid_1/bin/crsctl disable crs ... CRS disabled, ready for reboot.

2018-11-16 10:08:11: USM driver install status is 3

2018-11-16 10:08:11: ACFS drivers unable to be installed.

2018-11-16 10:08:11: Executing cmd: /u01/app/12.0.0/grid_1/bin/clsecho -p has -f clsrsc -m 400

2018-11-16 10:08:11: Command output:

>  CLSRSC-400: A system reboot is required to continue installing.

>End Command output

2018-11-16 10:08:11: CLSRSC-400: A system reboot is required to continue installing.

发现是ACFS组件的问题

根据mos上的文档27769638.8，发现应该是Bug 27769638

根据要求打补丁后，报错依旧。。。。

根据一篇类似安装情况的文档，按文档说法该组件非必须，然后尝试重启服务器，crsctl enable crs的形式，再次执行

报错还是一样。。。

转换思路，先检查该组件

核心日志有该组件安装失败的日志

>  ACFS-9308: Loading installed ADVM/ACFS drivers.

>  ACFS-9321: Creating udev for ADVM/ACFS.

>  ACFS-9323: Creating module dependencies - this may take some time.

>  ACFS-9176: Entering 'ld usm drvs'

>  ACFS-9154: Loading 'oracleoks.ko' driver.

>  modprobe: FATAL: Module oracleoks not found.

>  ACFS-9109: oracleoks.ko driver failed to load.

>  ACFS-9178: Return code = USM_FAIL

>  ACFS-9177: Return from 'ld usm drvs'

>  ACFS-9428: Failed to load ADVM/ACFS drivers. A system reboot is recommended.

>  ACFS-9310: ADVM/ACFS installation failed.

>  ACFS-9178: Return code = USM_REBOOT_RECOMMENDED

>  ACFS-9177: Return from 'install'

>  ACFS-9176: Entering 'acroot ex'

>  ACFS-9178: Return code = 3

>  ACFS-9177: Return from 'acroot ex'

用以下命令检查ACFS组件的状态

/u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ supported

/u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ installed

/u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ loaded

检查状态是supported，installed，not loaded，说明该组件并没有载入

使用/u01/app/12.0.0/grid_1/bin/acfsload start

反馈不明

使用/u01/app/12.0.0/grid_1/bin/acfsroot install –v

发现这个输出与核心日志一致

>  modprobe: FATAL: Module oracleoks not found.

判断这个是关键原因

/sbin/modprobe -v oracleoks

发现无法找到lib

使用

find /lib/modules | grep oracle

发现找到了lib

[root@ora12c1 3.10.0-862.el7.x86_64]# ls -l /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracle*

-rw-r--r--. 1 root root 55953519 Nov 16 13:02 /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracleacfs.ko

-rw-r--r--. 1 root root 15504635 Nov 16 13:02 /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracleadvm.ko

-rw-r--r--. 1 root root 10299710 Nov 16 13:02 /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracleoks.ko

根据文档说明，这个modprobe工具找的是/lib/modules/`uname -r`的路径

拷贝这些lib到uname –r的路径下

再执行并检查

[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/bin/acfsroot install –v

ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.10.0-862.el7.x86_64'

[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ supported

ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.10.0-862.el7.x86_64'

ACFS-9201: Not Supported

[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ installed

ACFS-9203: true

发现已不支持acfs组件

重新跑rootupgrade.sh

[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/rootupgrade.sh

Performing root user operation.

The following environment variables are set as:

    ORACLE_OWNER= grid

    ORACLE_HOME=  /u01/app/12.0.0/grid_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Using configuration parameter file: /u01/app/12.0.0/grid_1/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/ora12c1/crsconfig/rootcrs_ora12c1_2018-11-16_01-14-58PM.log

2018/11/16 13:15:01 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.

2018/11/16 13:15:01 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2018/11/16 13:15:01 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.

2018/11/16 13:16:40 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.

2018/11/16 13:16:51 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2018/11/16 13:16:51 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.

2018/11/16 13:16:54 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.

2018/11/16 13:16:59 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.

2018/11/16 13:16:59 CLSRSC-464: Starting retrieval of the cluster configuration data

2018/11/16 13:17:05 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2018/11/16 13:17:05 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.

2018/11/16 13:17:06 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.

2018/11/16 13:17:07 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.

2018/11/16 13:17:07 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.

2018/11/16 13:17:07 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.

2018/11/16 13:17:08 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.

2018/11/16 13:17:09 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.

2018/11/16 13:17:54 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.

2018/11/16 13:17:56 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.

2018/11/16 13:17:57 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.

2018/11/16 13:18:12 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

2018/11/16 13:18:33 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora12c1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'ora12c1'

CRS-2673: Attempting to stop 'ora.gipcd' on 'ora12c1'

CRS-2673: Attempting to stop 'ora.evmd' on 'ora12c1'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'ora12c1'

CRS-2677: Stop of 'ora.gipcd' on 'ora12c1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'ora12c1' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'ora12c1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'ora12c1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora12c1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

2018/11/16 13:19:00 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.

2018/11/16 13:19:03 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora12c1'

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora12c1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.evmd' on 'ora12c1'

CRS-2672: Attempting to start 'ora.mdnsd' on 'ora12c1'

CRS-2676: Start of 'ora.mdnsd' on 'ora12c1' succeeded

CRS-2676: Start of 'ora.evmd' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'ora12c1'

CRS-2676: Start of 'ora.gpnpd' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'ora12c1'

CRS-2676: Start of 'ora.gipcd' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ora12c1'

CRS-2676: Start of 'ora.cssdmonitor' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'ora12c1'

CRS-2672: Attempting to start 'ora.diskmon' on 'ora12c1'

CRS-2676: Start of 'ora.diskmon' on 'ora12c1' succeeded

CRS-2676: Start of 'ora.cssd' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ora12c1'

CRS-2672: Attempting to start 'ora.ctssd' on 'ora12c1'

CRS-2676: Start of 'ora.ctssd' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.drivers.acfs' on 'ora12c1'

CRS-2674: Start of 'ora.drivers.acfs' on 'ora12c1' failed

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'ora12c1'

CRS-2676: Start of 'ora.asm' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'ora12c1'

CRS-2676: Start of 'ora.storage' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'ora12c1'

CRS-2676: Start of 'ora.crf' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'ora12c1'

CRS-2676: Start of 'ora.crsd' on 'ora12c1' succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

CRS-6017: Processing resource auto-start for servers: ora12c1

CRS-2672: Attempting to start 'ora.ora12c1.vip' on 'ora12c1'

CRS-2672: Attempting to start 'ora.scan1.vip' on 'ora12c1'

CRS-2672: Attempting to start 'ora.ons' on 'ora12c1'

CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'ora12c1'

CRS-2676: Start of 'ora.ora12c1.vip' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'ora12c1'

CRS-2676: Start of 'ora.scan1.vip' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'ora12c1'

CRS-2676: Start of 'ora.MGMTLSNR' on 'ora12c1' succeeded

CRS-2676: Start of 'ora.ons' on 'ora12c1' succeeded

CRS-2676: Start of 'ora.LISTENER.lsnr' on 'ora12c1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'ora12c1' succeeded

CRS-2672: Attempting to start 'ora.orcl.db' on 'ora12c1'

CRS-2676: Start of 'ora.orcl.db' on 'ora12c1' succeeded

CRS-6016: Resource auto-start has completed for server ora12c1

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2018/11/16 13:21:28 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 12c Release 1.

Successfully taken the backup of node specific configuration in OCR.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

2018/11/16 13:21:36 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.

2018/11/16 13:21:39 CLSRSC-474: Initiating upgrade of resource types

2018/11/16 13:22:15 CLSRSC-482: Running command: 'srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p first'

2018/11/16 13:22:15 CLSRSC-475: Upgrade of resource types successfully initiated.

Start upgrade invoked..

2018/11/16 13:22:20 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded

2018/11/16 13:22:20 CLSRSC-482: Running command: '/u01/app/12.0.0/grid_1/bin/crsctl set crs activeversion'

Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.

Started to upgrade CSS.

CSS was successfully upgraded.

Started to upgrade CRS.

CRS was successfully upgraded.

Successfully upgraded the active version of Oracle Clusterware.

Oracle Clusterware active version was successfully set to 12.2.0.1.0.

2018/11/16 13:23:25 CLSRSC-479: Successfully set Oracle Clusterware active version

2018/11/16 13:23:25 CLSRSC-476: Finishing upgrade of resource types

2018/11/16 13:23:42 CLSRSC-482: Running command: 'srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p last'

2018/11/16 13:23:42 CLSRSC-477: Successfully completed upgrade of resource types

2018/11/16 13:24:01 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.

2018/11/16 13:24:06 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

成功

总结：这个问题打了相关的补丁也没有解决，与modeprobe找lib相关性大。我倾向于是系统版本的问题（redhat7.5），可能属于个例情况。