检查一下集群运行状态

检查集群状态
查看数据库安装log
如果是oracle 数据库bug，为规避该故障，可以在安装命令中增加参数，手动设置ORACLE_NAME。

[oracle@dbn01~]#./runInstaller ORACLE_NAME=dbn01

使用命令
 /app/grid/11.2.0/bin/crsctl start crs
在Oracle11g RAC的不同结点上分别启动，结果结点1启动报错，结点2启动成功。
节点一上
错误如下：
# /app/grid/11.2.0/bin/crsctl start crs
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
检查crs服务
# /app/grid/11.2.0/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services demon
CRS-4534: Cannot communicate with Event Manager
查看警告日志
 2013-08-27 14:24:03.374
[ohasd(2058)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac01'.
2013-08-27 14:54:09.465
[ohasd(2058)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac01'.
2013-08-27 15:24:16.595
[ohasd(2058)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac01'.
只是提示crsd启动失败，查看其他日志信息，也没有明显的错误提示

执行如下命令也报错
# ./crsctl query crs activeversion
Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, pn=kgfoAl06, dep=29701, loc=kgfokge
ORA-29701: unable to connect to Cluster Synchronization Service

# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, pn=kgfoAl06, dep=29701, loc=kgfokge
ORA-29701: unable to connect to Cluster Synchronization Service
] [7]

好像是物理存储ASM错误。
如是切换到grid用户，sqlplus连接
$ sqlplus / as sysdba
...

ERROR:
ORA-09817: Write to audit file failed.
Linux-x86_64 Error: 28: No space left on device
Additional information: 12
ORA-09945: Unable to initialize the audit trail file
Linux-x86_64 Error: 28: No space left on device

提示No space left on device，检查磁盘空间，发现该磁盘可用为零，如是，删除不需要的相关文件释放空间。

如是，停止crs服务

# /app/grid/11.2.0/bin/crsctl stop crs
...

然后重新执行
# /app/grid/11.2.0/bin/crsctl start crs

或者

# /app/grid/11.2.0/bin/crsctl start cluster

启动集群服务。
最后，检查验证
$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

$ crsctl status res -t
...

恢复正常。