10.2.0.4 RAC CRS-0184错误 CRS 无法启动案例

今天一个客户系统遇到一个问题，RAC的一个CRS节点无法启动，当然检查状态时遇到CRS-0184错误：

[root@rac2 tmp]# /opt/oracle/product/10.2.0/crs10g/bin/crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

在系统的message文件里记录了一系列的错误提示：

[root@rac2 tmp]# tail -20 /var/log/messages
Dec  1 13:34:46 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5234.
Dec  1 13:34:46 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5129.
Dec  1 13:35:46 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4974.
Dec  1 13:35:46 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5234.
Dec  1 13:35:46 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5129.
Dec  1 13:36:46 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4974.
Dec  1 13:36:47 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5234.
Dec  1 13:36:47 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5129.
Dec  1 13:37:47 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.4974.
Dec  1 13:37:47 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5234.
Dec  1 13:37:47 rac2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5129.

检查crsctl的跟踪文件发现如下错误，原来是OCR设备不能访问：

[root@rac2 tmp]# cat /tmp/crsctl.4974
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

再进一步的检查，发现主机端这个节点无法发现存储，fdisk -l输出仅有本地存储：

[root@rac2 ~]# fdisk -l

Disk /dev/cciss/c0d0: 440.3 GB, 440346238976 bytes
255 heads, 63 sectors/track, 53535 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1   *           1          13      104391   83  Linux
/dev/cciss/c0d0p2              14       20410   163838902+  83  Linux
/dev/cciss/c0d0p3           20411       22960    20482875   83  Linux
/dev/cciss/c0d0p4           22961       53535   245593687+   5  Extended
/dev/cciss/c0d0p5           22961       24235    10241406   83  Linux
/dev/cciss/c0d0p6           24236       25510    10241406   83  Linux
/dev/cciss/c0d0p7           25511       26785    10241406   83  Linux
/dev/cciss/c0d0p8           26786       27829     8385898+  82  Linux swap
/dev/cciss/c0d0p9           27830       53535   206483413+  83  Linux

看来需要派人去机房查看为什么主机无法发现存储了。