暂无图片
分享
吕治波
2019-03-15
CRS-2878: 无法重新启动资源 'ora.storage',CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)

oracle 12C 的rac集群,节点2在启动crs时日志如下:

2019-03-15 08:36:55.332 [OHASD(5084)]CRS-8500: Oracle Clusterware OHASD 进程以操作系统进程 ID 5084 开头

2019-03-15 08:36:55.338 [OHASD(5084)]CRS-0714: Oracle Clusterware 发行版 12.2.0.1.0。

2019-03-15 08:36:55.376 [OHASD(5084)]CRS-2112: 已在节点 wfmdb2 上启动 OLR 服务。

2019-03-15 08:36:55.404 [OHASD(5084)]CRS-1301: 已在节点 wfmdb2 上启动 Oracle 高可用性服务。

2019-03-15 08:36:55.418 [OHASD(5084)]CRS-8017: 位置: /etc/oracle/lastgasp 具有 2 个重新启动指导日志文件, 0 个已发布, 0 个出现错误

2019-03-15 08:36:55.783 [ORAROOTAGENT(5166)]CRS-8500: Oracle Clusterware ORAROOTAGENT 进程以操作系统进程 ID 5166 开头

2019-03-15 08:36:55.900 [CSSDAGENT(5191)]CRS-8500: Oracle Clusterware CSSDAGENT 进程以操作系统进程 ID 5191 开头

2019-03-15 08:36:55.907 [CSSDMONITOR(5195)]CRS-8500: Oracle Clusterware CSSDMONITOR 进程以操作系统进程 ID 5195 开头

2019-03-15 08:36:55.943 [ORAAGENT(5184)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 5184 开头

2019-03-15 08:36:56.816 [ORAAGENT(5378)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 5378 开头

2019-03-15 08:36:56.902 [EVMD(5399)]CRS-8500: Oracle Clusterware EVMD 进程以操作系统进程 ID 5399 开头

2019-03-15 08:36:56.913 [MDNSD(5397)]CRS-8500: Oracle Clusterware MDNSD 进程以操作系统进程 ID 5397 开头

2019-03-15 08:36:57.963 [GPNPD(5431)]CRS-8500: Oracle Clusterware GPNPD 进程以操作系统进程 ID 5431 开头

2019-03-15 08:36:58.990 [GIPCD(5495)]CRS-8500: Oracle Clusterware GIPCD 进程以操作系统进程 ID 5495 开头

2019-03-15 08:36:58.996 [GPNPD(5431)]CRS-2328: 已在节点 wfmdb2 上启动 GPNPD。

2019-03-15 08:37:01.017 [CSSDMONITOR(5519)]CRS-8500: Oracle Clusterware CSSDMONITOR 进程以操作系统进程 ID 5519 开头

2019-03-15 08:46:30.769 [CSSDAGENT(7522)]CRS-8500: Oracle Clusterware CSSDAGENT 进程以操作系统进程 ID 7522 开头

2019-03-15 08:46:30.971 [OCSSD(7540)]CRS-8500: Oracle Clusterware OCSSD 进程以操作系统进程 ID 7540 开头

2019-03-15 08:46:32.013 [OCSSD(7540)]CRS-1713: CSSD 守护程序已在 hub 模式下启动

2019-03-15 08:47:36.685 [OCSSD(7540)]CRS-1707: 节点 wfmdb2 (编号为 2) 的租约获取已完成

2019-03-15 08:47:37.818 [OCSSD(7540)]CRS-1605: CSSD 表决文件联机: AFD:OCR2; 详细资料见 /u01/app/grid/diag/crs/wfmdb2/crs/trace/ocssd.trc。

2019-03-15 08:47:37.860 [OCSSD(7540)]CRS-1605: CSSD 表决文件联机: AFD:OCR1; 详细资料见 /u01/app/grid/diag/crs/wfmdb2/crs/trace/ocssd.trc。

2019-03-15 08:47:37.909 [OCSSD(7540)]CRS-1605: CSSD 表决文件联机: AFD:OCR3; 详细资料见 /u01/app/grid/diag/crs/wfmdb2/crs/trace/ocssd.trc。

2019-03-15 08:47:39.403 [OCSSD(7540)]CRS-1601: CSSD 重新配置完毕。活动节点为 wfmdb1 wfmdb2 。

2019-03-15 08:47:39.451 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 08:47:42.012 [OCTSSD(8891)]CRS-8500: Oracle Clusterware OCTSSD 进程以操作系统进程 ID 8891 开头

2019-03-15 08:47:42.018 [OCSSD(7540)]CRS-1720: 集群同步服务守护程序 (CSSD) 已准备好进行操作。

2019-03-15 08:47:42.983 [OCTSSD(8891)]CRS-2403: 主机 wfmdb2 上的集群时间同步服务处于观察程序模式。

2019-03-15 08:47:43.525 [ORAROOTAGENT(5166)]CRS-5019: 所有 OCR 位置均位于 ASM 磁盘组 [OCR] 上, 但未装载这些磁盘组中的任何一个。有关详细信息, 请访问 "(:CLSN00140:)" (在 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc" 中)。

2019-03-15 08:47:44.123 [OCTSSD(8891)]CRS-2407: 新的集群时间同步服务引用节点为主机 wfmdb1。

2019-03-15 08:47:44.124 [OCTSSD(8891)]CRS-2401: 已在主机 wfmdb2 上启动了集群时间同步服务。

2019-03-15 08:47:54.426 [ORAROOTAGENT(5166)]CRS-5019: 所有 OCR 位置均位于 ASM 磁盘组 [OCR] 上, 但未装载这些磁盘组中的任何一个。有关详细信息, 请访问 "(:CLSN00140:)" (在 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc" 中)。

2019-03-15 08:57:50.535 [ORAROOTAGENT(5166)]CRS-5818: 已中止命令 'start' (对于资源 'ora.storage')。详细资料见 (:CRSAGF00113:) {0:9:4} (位于 /u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc)。

2019-03-15 08:57:51.106 [ORAROOTAGENT(5166)]CRS-5017: 资源操作 "ora.storage start" 遇到以下错误: 

2019-03-15 08:57:51.106+Storage agent start action aborted。有关详细信息, 请参阅 "(:CLSN00107:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc" 中)。

2019-03-15 08:57:51.110 [OHASD(5084)]CRS-2757: 命令 'Start' 在等待来自资源 'ora.storage' 的响应时超时。详细资料见 (:CRSPE00221:) {0:9:4} (位于 /u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd.trc)。

2019-03-15 08:58:57.754 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 08:59:02.138 [ORAROOTAGENT(5166)]CRS-5019: 所有 OCR 位置均位于 ASM 磁盘组 [OCR] 上, 但未装载这些磁盘组中的任何一个。有关详细信息, 请访问 "(:CLSN00140:)" (在 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc" 中)。

2019-03-15 09:08:57.916 [ORAROOTAGENT(5166)]CRS-5818: 已中止命令 'start' (对于资源 'ora.storage')。详细资料见 (:CRSAGF00113:) {0:1:17} (位于 /u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc)。

2019-03-15 09:09:01.891 [ORAROOTAGENT(5166)]CRS-5017: 资源操作 "ora.storage start" 遇到以下错误: 

2019-03-15 09:09:01.891+Storage agent start action aborted。有关详细信息, 请参阅 "(:CLSN00107:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc" 中)。

2019-03-15 09:09:01.894 [OHASD(5084)]CRS-2757: 命令 'Start' 在等待来自资源 'ora.storage' 的响应时超时。详细资料见 (:CRSPE00221:) {0:1:17} (位于 /u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd.trc)。

2019-03-15 09:09:08.158 [OHASD(5084)]CRS-2878: 无法重新启动资源 'ora.storage'

2019-03-15 09:10:08.229 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:10:08.232 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:11:08.388 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:11:08.393 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:12:08.540 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:12:08.544 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:13:08.688 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:13:08.693 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:14:08.884 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:14:08.889 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:15:09.060 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:15:09.065 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:16:09.217 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:16:09.221 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:17:09.354 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:17:09.358 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:18:09.502 [ORAROOTAGENT(5166)]CRS-5021: 检查存储失败: 详细信息见 "(:CLSN00117:)" (位于 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc")

2019-03-15 09:18:09.505 [OHASD(5084)]CRS-2771: 已达到资源 'ora.storage' 的最大重新启动尝试次数; 将不会重新启动。

2019-03-15 09:19:14.250 [ORAROOTAGENT(5166)]CRS-5019: 所有 OCR 位置均位于 ASM 磁盘组 [OCR] 上, 但未装载这些磁盘组中的任何一个。有关详细信息, 请访问 "(:CLSN00140:)" (在 "/u01/app/grid/diag/crs/wfmdb2/crs/trace/ohasd_orarootagent_root.trc" 中)。


ohasd_orarootagent_root.trc中有如下:

2019-03-15 08:59:22.018 : USRTHRD:1474291456: {0:1:17} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS


2019-03-15 08:59:22.029 : USRTHRD:1474291456: {0:1:17} 8033 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS


2019-03-15 08:59:22.104 : CLSCRED:1474291456: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.1eed137e4fab5fe8bf1a148274ffbf94.root not found

2019-03-15 08:59:22.104 : USRTHRD:1474291456: {0:1:17} 7755 Error 4 opening dom root in 0x7f85441387e0


2019-03-15 08:59:22.128 : CSSCLNT:1474291456: clsssinit: initialized context: (0x7f8544138ce0) flags 0x104

2019-03-15 08:59:22.130 : CSSCLNT:1474291456: clsssterm: terminating context (0x7f8544138ce0)

2019-03-15 08:59:22.130 : default:1474291456: clsCredDomClose: Credctx deleted 0x7f854417b7e0

2019-03-15 08:59:23.132 :    GPNP:1474291456: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'

2019-03-15 08:59:23.151 : CSSCLNT:1474291456: clsssinit: initialized context: (0x7f8544138ce0) flags 0x115

2019-03-15 08:59:23.152 : CSSCLNT:1474291456: clsssterm: terminating context (0x7f8544138ce0)

2019-03-15 08:59:23.152 :   CLSNS:1474291456: clsns_SetTraceLevel:trace level set to 1.

2019-03-15 08:59:23.153 :    GPNP:1474291456: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'

2019-03-15 08:59:23.157 : default:1474291456: Inited LSF context: 0x7f85440e7400 

2019-03-15 08:59:23.161 : CLSCRED:1474291456: clsCredCommonInit: Inited singleton credctx.

2019-03-15 08:59:23.162 : CLSCRED:1474291456: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.

2019-03-15 08:59:23.211 : USRTHRD:1474291456: {0:1:17} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS


2019-03-15 08:59:23.220 : USRTHRD:1474291456: {0:1:17} 8033 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS


2019-03-15 08:59:23.302 : CLSCRED:1474291456: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.1eed137e4fab5fe8bf1a148274ffbf94.root not found

2019-03-15 08:59:23.302 : USRTHRD:1474291456: {0:1:17} 7755 Error 4 opening dom root in 0x7f8544308460


2019-03-15 08:59:23.329 : CSSCLNT:1474291456: clsssinit: initialized context: (0x7f8544096f50) flags 0x104

2019-03-15 08:59:23.330 : CSSCLNT:1474291456: clsssterm: terminating context (0x7f8544096f50)

2019-03-15 08:59:23.331 : default:1474291456: clsCredDomClose: Credctx deleted 0x7f8544150d90

2019-03-15 08:59:23.332 : USRTHRD:1474291456: {0:1:17} -- trace dump on error exit --


2019-03-15 08:59:23.332 : USRTHRD:1474291456: {0:1:17} Error [kgfoAl06] in [kgfokge] at kgfo.c:3083


2019-03-15 08:59:23.332 : USRTHRD:1474291456: {0:1:17} ORA-27300: 操作系统系统相关操作: sslssunreghdlr 失败, 状态为: 0

ORA-27301: 操作系统故障消息: Error 0

ORA-27302: 错误发生在: sskgpres


2019-03-15 08:59:23.332 : USRTHRD:1474291456: {0:1:17} Category: 7


2019-03-15 08:59:23.332 : USRTHRD:1474291456: {0:1:17} DepInfo: 27300


2019-03-15 08:59:23.332 : USRTHRD:1474291456: {0:1:17} -- trace dump end --


2019-03-15 08:59:23.332 :CLSDYNAM:1474291456: [ora.storage]{0:1:17} [start] StorageAgent::parsekgforetcodes retcode = 7, kgfoCheckMount(OCR), flag 2

2019-03-15 08:59:23.332 :CLSDYNAM:1474291456: [ora.storage]{0:1:17} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 27300, other: ORA-27300: 操作系统系统相关操作: sslssunreghdlr 失败, 状态为: 0

ORA-27301: 操作系统故障消息: Error 0

ORA-27302: 错误发生在: sskgpres

2019-03-15 08:59:23.332 :CLSDYNAM:1474291456: [ora.storage]{0:1:17} [start] StorageAgent::check kgfo returncode 1

2019-03-15 08:59:23.332 :CLSDYNAM:1474291456: [ora.storage]{0:1:17} [start] (:CLSN00140:)StorageAgent::parsekgforretcodes OCR dgName OCR state 1

2019-03-15 08:59:23.333 :CLSDYNAM:1474291456: [ora.storage]{0:1:17} [start] waiting for check to not return PARTIALor UNPLANNED_OFFLINE 1

2019-03-15 08:59:24.334 :CLSDYNAM:1474291456: [ora.storage]{0:1:17} [start] StorageAgent::check NODEROLE_HUB getOCRdetails

2019-03-15 08:59:24.339 :    GPNP:1474291456: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'

2019-03-15 08:59:24.340 :    GPNP:1474291456: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'

2019-03-15 08:59:24.359 : CSSCLNT:1474291456: clsssinit: initialized context: (0x7f854417c0c0) flags 0x115

2019-03-15 08:59:24.360 : CSSCLNT:1474291456: clsssterm: terminating context (0x7f854417c0c0)

2019-03-15 08:59:24.361 :   CLSNS:1474291456: clsns_SetTraceLevel:trace level set to 1.

2019-03-15 08:59:24.362 :    GPNP:1474291456: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'

2019-03-15 08:59:24.365 : default:1474291456: Inited LSF context: 0x7f85441926b0 

2019-03-15 08:59:24.369 : CLSCRED:1474291456: clsCredCommonInit: Inited singleton credctx.

2019-03-15 08:59:24.369 : CLSCRED:1474291456: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.

2019-03-15 08:59:24.412 : USRTHRD:1474291456: {0:1:17} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS


2019-03-15 08:59:24.421 : USRTHRD:1474291456: {0:1:17} 8033 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS


2019-03-15 08:59:24.493 : CLSCRED:1474291456: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.1eed137e4fab5fe8bf1a148274ffbf94.root not found

2019-03-15 08:59:24.494 : USRTHRD:1474291456: {0:1:17} 7755 Error 4 opening dom root in 0x7f8544229570


2019-03-15 08:59:24.519 : CSSCLNT:1474291456: clsssinit: initialized context: (0x7f85440ef710) flags 0x104

2019-03-15 08:59:24.522 : CSSCLNT:1474291456: clsssterm: terminating context (0x7f85440ef710)

2019-03-15 08:59:24.522 : default:1474291456: clsCredDomClose: Credctx deleted 0x7f8544108490




查看主机资源信息

[root@wfmdb2 ~]# crsctl stat res -t -init

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details       

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.cluster_interconnect.haip

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.crf

      1        ONLINE  OFFLINE                               STABLE

ora.crsd

      1        ONLINE  OFFLINE                               STABLE

ora.cssd

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.cssdmonitor

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.ctssd

      1        ONLINE  ONLINE       wfmdb2                   OBSERVER,STABLE

ora.diskmon

      1        OFFLINE OFFLINE                               STABLE

ora.driver.afd

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.drivers.acfs

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.evmd

      1        ONLINE  INTERMEDIATE wfmdb2                   STABLE

ora.gipcd

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.gpnpd

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.mdnsd

      1        ONLINE  ONLINE       wfmdb2                   STABLE

ora.storage

      1        ONLINE  OFFLINE                               STABLE

--------------------------------------------------------------------------------



查看主机映射盘


[root@wfmdb2 ~]# ll /dev/asm*

brw-rw---- 1 grid asmadmin 8, 17 3月  15 08:34 /dev/asmdata1

brw-rw---- 1 grid asmadmin 8, 33 3月  15 08:34 /dev/asmdata2

brw-rw---- 1 grid asmadmin 8, 18 3月  15 08:34 /dev/asmdata3

brw-rw---- 1 grid asmadmin 8, 34 3月  15 08:34 /dev/asmdata4

brw-rw---- 1 grid asmadmin 8, 19 3月  15 08:34 /dev/asmdisk1

brw-rw---- 1 grid asmadmin 8, 35 3月  15 08:34 /dev/asmdisk2

brw-rw---- 1 grid asmadmin 8, 21 3月  15 08:34 /dev/asmfocr1

brw-rw---- 1 grid asmadmin 8, 37 3月  15 08:34 /dev/asmfocr2

brw-rw---- 1 grid asmadmin 8, 22 3月  15 08:34 /dev/asmfocr3

brw-rw---- 1 grid asmadmin 8, 38 3月  15 08:34 /dev/asmfocr4

brw-rw---- 1 grid asmadmin 8, 23 3月  15 08:34 /dev/asmfocr5

brw-rw---- 1 grid asmadmin 8, 39 3月  15 08:34 /dev/asmfocr6

brw-rw---- 1 grid asmadmin 8, 24 3月  15 08:34 /dev/asmfragd1

brw-rw---- 1 grid asmadmin 8, 40 3月  15 08:34 /dev/asmfragd2

brw-rw---- 1 grid asmadmin 8, 25 3月  15 08:34 /dev/asmfragd3

brw-rw---- 1 grid asmadmin 8, 41 3月  15 08:34 /dev/asmfragd4

brw-rw---- 1 grid asmadmin 8, 26 3月  15 08:34 /dev/asmgm1

brw-rw---- 1 grid asmadmin 8, 42 3月  15 08:34 /dev/asmgm2



在主机操作系统的message中有如下:

Mar 15 01:02:13 wfmdb2 sshd[46114]: error: no more sessions

Mar 15 03:08:10 wfmdb2 rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.

Mar 15 08:34:43 wfmdb2 sshd[3285]: Accepted password for root from 10.212.62.4 port 49533 ssh2

Mar 15 08:34:46 wfmdb2 udevd[3360]: GOTO 'pulseaudio_check_usb' has no matching label in: '/lib/udev/rules.d/90-pulseaudio.rules'

Mar 15 08:34:46 wfmdb2 kernel: udev: starting version 147

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> (eth2): device state change: activated -> disconnected (reason 'none') [8 3 0]

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> (eth2): deactivating device (reason 'none') [0]

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) starting connection 'Auto eth2'

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> (eth2): device state change: disconnected -> prepare (reason 'none') [3 4 0]

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 1 of 5 (Device Prepare) scheduled...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 1 of 5 (Device Prepare) started...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 2 of 5 (Device Configure) scheduled...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 1 of 5 (Device Prepare) complete.

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 2 of 5 (Device Configure) starting...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> (eth2): device state change: prepare -> config (reason 'none') [4 5 0]

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 2 of 5 (Device Configure) successful.

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 3 of 5 (IP Configure Start) scheduled.

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 2 of 5 (Device Configure) complete.

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 3 of 5 (IP Configure Start) started...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> (eth2): device state change: config -> ip-config (reason 'none') [5 7 0]

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 4 of 5 (IP4 Configure Get) scheduled...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 3 of 5 (IP Configure Start) complete.

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 4 of 5 (IP4 Configure Get) started...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 5 of 5 (IP Configure Commit) scheduled...

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 4 of 5 (IP4 Configure Get) complete.

Mar 15 08:34:47 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 5 of 5 (IP Configure Commit) started...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> (eth2): device state change: ip-config -> activated (reason 'none') [7 8 0]

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Policy set 'Auto eth2' (eth2) as default for IPv4 routing and DNS.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) successful, device activated.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth2) Stage 5 of 5 (IP Configure Commit) complete.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> (eth3): device state change: activated -> disconnected (reason 'none') [8 3 0]

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> (eth3): deactivating device (reason 'none') [0]

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Policy set 'Auto eth2' (eth2) as default for IPv4 routing and DNS.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Policy set 'Auto eth2' (eth2) as default for IPv4 routing and DNS.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) starting connection 'System eth3'

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> (eth3): device state change: disconnected -> prepare (reason 'none') [3 4 0]

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 1 of 5 (Device Prepare) scheduled...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 1 of 5 (Device Prepare) started...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 2 of 5 (Device Configure) scheduled...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 1 of 5 (Device Prepare) complete.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 2 of 5 (Device Configure) starting...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> (eth3): device state change: prepare -> config (reason 'none') [4 5 0]

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 2 of 5 (Device Configure) successful.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 3 of 5 (IP Configure Start) scheduled.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 2 of 5 (Device Configure) complete.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 3 of 5 (IP Configure Start) started...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> (eth3): device state change: config -> ip-config (reason 'none') [5 7 0]

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 4 of 5 (IP4 Configure Get) scheduled...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 3 of 5 (IP Configure Start) complete.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 4 of 5 (IP4 Configure Get) started...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 5 of 5 (IP Configure Commit) scheduled...

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 4 of 5 (IP4 Configure Get) complete.

Mar 15 08:34:48 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 5 of 5 (IP Configure Commit) started...

Mar 15 08:34:49 wfmdb2 NetworkManager[2852]: <info> Policy set 'Auto eth2' (eth2) as default for IPv4 routing and DNS.

Mar 15 08:34:49 wfmdb2 NetworkManager[2852]: <info> (eth3): device state change: ip-config -> activated (reason 'none') [7 8 0]

Mar 15 08:34:49 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) successful, device activated.

Mar 15 08:34:49 wfmdb2 NetworkManager[2852]: <info> Activation (eth3) Stage 5 of 5 (IP Configure Commit) complete.

Mar 15 08:35:08 wfmdb2 udevd[3361]: GOTO 'pulseaudio_check_usb' has no matching label in: '/lib/udev/rules.d/90-pulseaudio.rules'

Mar 15 08:35:47 wfmdb2 sshd[4641]: Accepted password for root from 10.212.62.4 port 49587 ssh2

Mar 15 08:36:45 wfmdb2 root: exec /u01/app/12.2.0/grid/perl/bin/perl -I/u01/app/12.2.0/grid/perl/lib /u01/app/12.2.0/grid/bin/crswrapexece.pl /u01/app/12.2.0/grid/crs/install/s_crsconfig_wfmdb2_env.txt /u01/app/12.2.0/grid/bin/ohasd.bin "reboot"

Mar 15 08:36:55 wfmdb2 Oracle Clusterware: 2019-03-15 08:36:55.333#012[(5084)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 5084

Mar 15 08:38:21 wfmdb2 sshd[5736]: Accepted password for root from 10.212.62.4 port 49677 ssh2

Mar 15 08:44:11 wfmdb2 sshd[6899]: Accepted password for root from 10.212.62.4 port 49932 ssh2

Mar 15 08:47:42 wfmdb2 Oracle Clusterware: 2019-03-15 08:47:42.012#012[(8891)]CRS-8500:Oracle Clusterware OCTSSD process is starting with operating system process ID 8891

Mar 15 08:48:49 wfmdb2 sshd[9098]: Accepted password for root from 10.212.62.4 port 50084 ssh2

Mar 15 08:52:25 wfmdb2 sshd[10602]: Accepted password for root from 10.212.62.4 port 50254 ssh2

并未看到错误信息

收藏
分享
5条回答
默认
最新
盖国强

这个问题,需要背景信息描述,如果是升级后出现的问题,这匹配到一个Bug。


如果是正常重启,那需要进一步分析。


当然,如果这类问题已经影响到服务提供,应该呼唤现场服务了。

暂无图片 评论
暂无图片 有用 0
吕治波

 之前网络不稳定,经常造成其中一个节点重启,于是就将一个节点crs stop了,过了几天后再启动这个节点的时候出现了这个错误

暂无图片 评论
暂无图片 有用 0
盖国强

你可以在出问题的这个节点,用kfed读一下磁盘头,read,如果访问没有问题,就不是存储问题。


暂无图片 评论
暂无图片 有用 0
吕治波

kfed read /dev/asmfocr1 text=/home/grid/asmfocr1.txt 

kfed read /dev/asmfocr3 text=/home/grid/asmfocr3.txt 

kfed read /dev/asmfocr5 text=/home/grid/asmfocr5.txt 

cat /home/grid/asmfocr1.txt 

kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL

kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER

kfdhdb.dskname:                    OCR3 ; 0x028: length=4

kfdhdb.grpname:                     OCR ; 0x048: length=3

kfdhdb.fgname:                     OCR3 ; 0x068: length=4


cat /home/grid/asmfocr3.txt 

kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL

kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER

kfdhdb.dskname:                    OCR1 ; 0x028: length=4

kfdhdb.grpname:                     OCR ; 0x048: length=3

kfdhdb.fgname:                     OCR1 ; 0x068: length=4


cat /home/grid/asmfocr5.txt 

kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL

kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER

kfdhdb.dskname:                    OCR2 ; 0x028: length=4

kfdhdb.grpname:                     OCR ; 0x048: length=3

kfdhdb.fgname:                     OCR2 ; 0x068: length=4



OCR的三个盘通过kfed read可以看到头信息




暂无图片 评论
暂无图片 有用 0
盖国强
问题已关闭: 问题已经得到解决
暂无图片 评论
暂无图片 有用 0
回答交流
提交
问题信息
请登录之后查看
邀请回答
暂无人订阅该标签,敬请期待~~
暂无图片墨值悬赏