5M凌晨5:00多收到服务器重启告警短信，一早上班检查应用正常。放下心慢慢查找重启原因，以下是通过日志看到的情况，麻烦各位大神进一步帮忙分机具体什么原因导致的，网络心跳故障（心跳交换机、网线啥的问题）还是磁盘心跳（FC问题）
节点1重启，在节点一系统日志，数据库日志，集群日志里面都没发现异常。
–通过系统日志确定服务器在5:03重启了
Nov 19 05:03:36 wscard04 syslogd 1.4.1: restart.
节点2 系统日志里面发现5:01有个报错
Nov 19 05:01:11 wscard05 kernel: rport-2:0-2: blocked FC remote port time out: removing rport
集群日志里面 5:00分有心跳超时的记录
2020-11-19 05:00:40.505
[cssd(12101)]CRS-1612:node wscard04 (1) at 50% heartbeat fatal, eviction in 29.068 seconds
crs日志发现05:01分节点1被踢出集群
2020-11-19 05:01:31.152: [ CRSEVT][570329408]0CAAMonitorHandler :: 0:Could not join /oracle/app/oracle/product/10.2.0/crs_1/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
附件无法下载
从[cssd(12101)]CRS-1612:node wscard04 (1) at 50% heartbeat fatal, eviction in 29.068 seconds
可以看出，一般情况下，是心跳网络超时，导致节点重启
评论
有用 0分析：
aler_ecard1.log中：
Thu Nov 19 00:52:54 CST 2020
SUCCESS: diskgroup ARCH was mounted
Thu Nov 19 00:53:03 CST 2020 <<<<<此刻数据库有输出日志
SUCCESS: diskgroup ARCH was dismounted
Thu Nov 19 05:04:56 CST 2020 <<<<再无任何日志输出，知道05:04分，主机重启后输出数据库重启信息。
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.1.0 configured from OCR for use as a cluster interconnect
message日志中:
Nov 19 04:59:40 wscard04 yum-updatesd: error getting update info: Cannot retrieve repository metadata (repomd.xml) for repository: public_el5_latest. Please verify its path and try again
Nov 19 05:00:01 wscard04 xinetd[9410]: START: time-stream pid=0 from=188.0.1.56
Nov 19 05:00:01 wscard04 xinetd[9410]: START: time-stream pid=0 from=188.0.1.55 <<<<此刻时间点之后主机应该突然宕机了
Nov 19 05:03:36 wscard04 syslogd 1.4.1: restart. <<<<<<此时间点主机重启，能输出日志
Nov 19 05:03:36 wscard04 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Nov 19 05:03:36 wscard04 kernel: Initializing cgroup subsys cpuset
Nov 19 05:03:36 wscard04 kernel: Initializing cgroup subsys cpu
wscard05的集群日志检测出节点wscard04离线
2020-11-19 05:00:40.505
[cssd(12101)]CRS-1612:node wscard04 (1) at 50% heartbeat fatal, eviction in 29.068 seconds
2020-11-19 05:00:41.505
[cssd(12101)]CRS-1612:node wscard04 (1) at 50% heartbeat fatal, eviction in 28.068 seconds
2020-11-19 05:00:55.507
[cssd(12101)]CRS-1611:node wscard04 (1) at 75% heartbeat fatal, eviction in 14.068 seconds
总结：主机异常宕机，导致集群重启。请关注wscard04硬件情况及当时的主机资源使用情况。
建议：
如果主机部署了oswatch或者nmon等主机资源监控软件，可以看下主机当时的资源使用情况。
如果缺乏有效的监控手段，可以通过以下方式粗略的查看资源情况，找到对应的时间点。
cd /var/log/sa
houzhiqing@dqg6ddspt1:/var/log/sa> ls -lFrt
sar18
-rw-r–r-- 1 root root 17326668 Nov 18 23:59 sa18
-rw-r–r-- 1 root root 10468428 Nov 19 14:29 sa19
houzhiqing@dqg6ddspt1:/var/log/sa> sar -f sa19
Linux 3.0.76-0.11-default (dqg6ddspt1) 11/19/2020 x86_64 (64 CPU)
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:01:01 AM all 12.90 0.00 2.29 5.68 0.00 79.14
12:02:01 AM all 12.77 0.00 2.31 3.53 0.00 81.39
评论 有用 0
墨值悬赏