超时可能与网络或者共享磁盘的连接有关；

从11.2.0.2开始，由于新特性rebootless restart的原因，节点不会被重启，而是集群管理软件进行重启。

具体文案找不到了，等待其他大神补充下了～

Rebootless Node Fencing

In versions before 11.2.0.2 Oracle Clusterware tried to prevent a split-brain with a fast reboot (better: reset) of the server(s) without waiting for ongoing I/O operations or synchronization of the file systems. This mechanism has been changed in version 11.2.0.2 (first 11g Release 2 patch set). After deciding which node to evict, the clusterware:

. attempts to shut down all Oracle resources/processes on the server (especially processes generating I/Os)

. will stop itself on the node

. afterwards Oracle High Availability Service Daemon (OHASD) will try to start the Cluster Ready Services (CRS) stack again. Once the cluster interconnect is back online,all relevant cluster resources on that node will automatically start

. kill the node if stop of resources or processes generating I/O is not possible (hanging in kernel mode, I/O path, etc.)

This behavior change is particularly useful for non-cluster aware applications.


Prior to 11g R2, during voting disk failures the node will be rebooted to protect the integrity of the cluster. But rebooting cannot be necessarily just the communication issue. The node can be hanging or the IO operation can be hanging so potentially the reboot decision can be the incorrect one. So Oracle Clusterware will fence the node without rebooting. This is a big (and big) achievement and changes in the way the cluster is designed.

The reason why we will have to avoid the reboot is that during reboots resources need to re-mastered and the nodes remaining on the cluster should be re-formed. In a big cluster with many numbers of nodes, this can be potentially a very expensive operation so Oracle fences the node by killing the offending process so the cluster will shutdown but the node will not be shutdown. Once the IO path is available or the network heartbeat is available, the cluster will be started again. Be assured the data will be protected but it will be done without any pain rebooting the nodes. But in the cases where the reboot is needed to protect the integrity, the cluster will decide to reboot the node.

可否提供下两个节点的asm日志、gpnp日志、gipc日志、数据库alert日志、两个节点的主机日志、OSW或者主机层面的监控信息。

从提供的附件里面看，初步判断是节点2的主机先宕机重启，集群是受害者。也就是初步判断主机因为某种原因重启，集群在正常运行时遭遇主机宕机，导致后台没有任何日志输出，节点1探测到节点2的网络不通，才有的节点1日志里面那些输出。

需要题主提供更多的信息来判断分析。

首先是心跳检测失败，要将节点2踢出，并重启节点2，但是一直无法终止节点2，重启失败，最终重启了机器

检查

1.cluvfy comp clocksync -n all各个节点服务是否正常

2.crsctl start res ora.ctssd -init 看一下ctss 

运行模式如果active 要看一下两个节点时间是否一直如果

observer模式看一下ntp服务是否正常

10:11:59 后节点2就已经重启了，节点1的心跳不通是10:12，这时机器已经down了