故障现象

ASM创建磁盘组，报错如下

ORA-15018: diskgroup cannot be created
ORA-15137: The ASM cluster is in rolling patch state.

故障分析

• 检测当前集群状态，为ROLLING PATCH，而正常状态应该为NORMAL

$ crsctl query crs activeversion -f 
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [81469846].

$ asmcmd showclusterstate
In Rolling Patch

• 3个节点补丁状态一致，都为如下内容

[grid@node01 trace]$ /u01/app/12.2/grid/OPatch/opatch lspatches 
27060167;
24396050;OCW Interim patch for 24396050
29770090;ACFS JUL 2019 RELEASE UPDATE 12.2.0.1.190716 (29770090)

OPatch succeeded.
[grid@node01 trace]$  

[oracle@node01 ~]$ /u01/app/oracle/12.2/OPatch/opatch lspatches 
24396050;OCW Interim patch for 24396050

OPatch succeeded.
[oracle@node01 ~]$

• 根据文档ORA-15137 - The ASM Cluster Is In Rolling Patch State (Doc ID 2949951.1)
  进行检查

 $ <GRID_HOME>/bin/crsctl query crs releasepatch
 $ <GRID_HOME>/bin/crsctl query crs softwarepatch
 $ <GRID_HOME>/bin/crsctl query crs releaseversion
 $ <GRID_HOME>/bin/crsctl query crs softwareversion
 $ <GRID_HOME>/bin/kfod op=patches
 $ <GRID_HOME>/bin/kfod op=patchlvl
 $ <GRID_HOME>/OPatch/opatch lspatches

 $ <GRID_HOME>/bin/cluutil -ckpt -oraclebase <ORACLE_BASE> -chkckpt -name ROOTCRS_PREPATCH -status

3个节点的结果如下：
releasepatch和softwarepatch不一致

[grid@node02 ~]$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [2674111865] and the complete list of patches [24396050 27060167 ] have been applied on the local node.
[grid@node02 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node node02 is [81469846].
[grid@node02 ~]$ 
[grid@node02 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]
[grid@node02 ~]$  crsctl query crs softwareversion
Oracle Clusterware version on node [node02] is [12.2.0.1.0]
[grid@node02 ~]$  kfod op=patches
---------------
List of Patches
===============
24396050
27060167
[grid@node02 ~]$ kfod op=patchlvl
-------------------
Current Patch level
===================
2674111865
[grid@node02 ~]$ cluutil -ckpt -oraclebase $ORACLE_BASE -chkckpt -name ROOTCRS_PREPATCH -status
SUCCESS
[grid@node02 ~]$ cluutil -ckpt -oraclebase $ORACLE_BASE -chkckpt -name ROOTCRS_POSTPATCH -status
SUCCESS
[grid@node02 ~]$ 

解决方案

根据文档ORA-15137 After Rolling Back Failed Patch at GI_HOME (Doc ID 1968727.1)
Case 1. OCR holds patch details which was not updated.
从上面检查来看，排除第二种原因
Case 2. Patch was not applied at all nodes like 2nd node in a 2 node RAC.

解决方案如下：

• 所有节点执行
  需要集群启动时候执行，并且执行过程未影响集群和数据库运行

# /u01/app/12.2/grid/bin/clscfg -patch
clscfg: -patch mode specified
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.

--releasepatch和softwarepatch变为一致

# /u01/app/12.2/grid/bin/crsctl  query crs softwarepatch
Oracle Clusterware patch level on node node01 is [2674111865].

• 其中一个节点执行

# /u01/app/12.2/grid/bin/crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level [2674111865].

• 所有节点查看状态

# /u01/app/12.2/grid/bin/crsctl query crs activeversion -f 
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [2674111865].

• 正常创建磁盘组

参考文档

ORA-15137 - The ASM Cluster Is In Rolling Patch State (Doc ID 2949951.1)
ORA-15137 After Rolling Back Failed Patch at GI_HOME (Doc ID 1968727.1)

订阅号：DongDB手记
墨天轮：https://www.modb.pro/u/231198