5M集群的一个节点的其中仲裁盘的磁盘组掉线，导致节点集群服务宕掉，asm日志如下： 想知道这个具体能分析出什么问题吗？想知道为什么io就无法请求了，还需要什么日志吗?
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Fri Oct 25 21:38:12 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 17591] opening OCR file
Fri Oct 25 21:39:12 2019
NOTE: [ocrcheck.bin@rac3 (TNS V1-V3) 27038] opening OCR file
Mon Oct 28 09:06:16 2019
NOTE: [ocrcheck.bin@rac3 (TNS V1-V3) 79860] opening OCR file
Wed Oct 30 09:02:20 2019
NOTE: [ocrcheck.bin@rac3 (TNS V1-V3) 31239] opening OCR file
Sat Nov 02 17:02:59 2019
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 4.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 4.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 4.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 4.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 4.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 4.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 5.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 5.
Sat Nov 02 17:02:59 2019
NOTE: process b000+asm1 (59876) initiating offline of disk 0.3915948152 (OCRVOTE_0000) with mask 0x7e in group 4
NOTE: process b000+asm1 (59876) initiating offline of disk 1.3915948154 (OCRVOTE_0001) with mask 0x7e in group 4
NOTE: process b000+asm1 (59876) initiating offline of disk 2.3915948153 (OCRVOTE_0002) with mask 0x7e in group 4
NOTE: checking PST: grp = 4
GMON checking disk modes for group 4 at 90 for pid 37, osid 59876
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 4 done.
NOTE: initiating PST update: grp = 4, dsk = 0/0xe968a078, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 4, dsk = 1/0xe968a07a, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 4, dsk = 2/0xe968a079, mask = 0x6a, op = clear
GMON updating disk modes for group 4 at 91 for pid 37, osid 59876
ERROR: no read quorum in group: required 2, found 0 disks
Sat Nov 02 17:02:59 2019
NOTE: cache dismounting (not clean) group 4/0x35985399 (OCRVOTE)
NOTE: messaging CKPT to quiesce pins Unix process pid: 59878, image: oracle@rac3 (B001)
Sat Nov 02 17:02:59 2019
NOTE: halting all I/Os to diskgroup 4 (OCRVOTE)
Sat Nov 02 17:02:59 2019
NOTE: LGWR doing non-clean dismount of group 4 (OCRVOTE)
NOTE: LGWR sync ABA=2.50 last written ABA 2.50
WARNING: Offline for disk OCRVOTE_0000 in mode 0x7f failed.
WARNING: Offline for disk OCRVOTE_0001 in mode 0x7f failed.
WARNING: Offline for disk OCRVOTE_0002 in mode 0x7f failed.
Sat Nov 02 17:02:59 2019
kjbdomdet send to inst 2
detach from dom 4, sending detach message to inst 2
Sat Nov 02 17:03:00 2019
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 24)
Global Resource Directory partially frozen for dirty detach
- dirty detach - domain 4 invalid = TRUE
2 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Sat Nov 02 17:03:00 2019
WARNING: dirty detached from domain 4
NOTE: cache dismounted group 4/0x35985399 (OCRVOTE)
SQL> alter diskgroup OCRVOTE dismount force /* ASM SERVER:899175321 /
Sat Nov 02 17:03:00 2019
NOTE: cache deleting context for group OCRVOTE 4/0x35985399
GMON dismounting group 4 at 92 for pid 44, osid 59878
NOTE: Disk OCRVOTE_0000 in mode 0x7f marked for de-assignment
NOTE: Disk OCRVOTE_0001 in mode 0x7f marked for de-assignment
NOTE: Disk OCRVOTE_0002 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 4
Sat Nov 02 17:03:01 2019
ASM Health Checker found 1 new failures
Sat Nov 02 17:03:30 2019
SUCCESS: diskgroup OCRVOTE was dismounted
SUCCESS: alter diskgroup OCRVOTE dismount force / ASM SERVER:899175321 /
Sat Nov 02 17:03:30 2019
NOTE: diskgroup resource ora.OCRVOTE.dg is offline
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group OCRVOTE
Sat Nov 02 17:03:30 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Sat Nov 02 19:20:18 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
WARNING: requested mirror side 1 of virtual extent 6 logical extent 0 offset 577536 is not allocated; I/O request failed
WARNING: requested mirror side 2 of virtual extent 6 logical extent 1 offset 577536 is not allocated; I/O request failed
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26689.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Sat Nov 02 19:20:18 2019
SQL> alter diskgroup OCRVOTE check / proxy /
ORA-15032: not all alterations performed
ORA-15001: diskgroup “OCRVOTE” does not exist or is not mounted
ERROR: alter diskgroup OCRVOTE check / proxy */
Sat Nov 02 19:20:43 2019
NOTE: client exited [17591]
Sat Nov 02 19:20:45 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34286] opening OCR file
Sat Nov 02 19:20:46 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34407] opening OCR file
Sat Nov 02 19:20:48 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34430] opening OCR file
Sat Nov 02 19:20:51 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34443] opening OCR file
Sat Nov 02 19:20:53 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34456] opening OCR file
Sat Nov 02 19:20:55 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34475] opening OCR file
Sat Nov 02 19:20:57 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34488] opening OCR file
Sat Nov 02 19:20:59 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34508] opening OCR file
Sat Nov 02 19:21:01 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34521] opening OCR file
Sat Nov 02 19:21:03 2019
NOTE: [crsd.bin@rac3 (TNS V1-V3) 34617] opening OCR file
Sun Nov 03 02:33:57 2019
一下是同一时间的系统日志，不太会分析：
Nov 2 12:13:43 rac3 kernel: sbd_fun_linux INFO 2016412896,thread 5ca4 LinuxAgentServi,[SetErrCodeEx]:[disk:45]Runtime Err: [WorkRegion: 0x0] [UserS: 0x0] [SysS: 0x0]
Nov 2 12:13:43 rac3 kernel: sbd_fun_linux INFO 2016412896,thread 5ca4 LinuxAgentServi,[SetErrCodeEx]:[disk:46]Runtime Err: [WorkRegion: 0x0] [UserS: 0x0] [SysS: 0x0]
Nov 2 12:13:43 rac3 kernel: sbd_fun_linux INFO 2016412896,thread 5ca4 LinuxAgentServi,[SetErrCodeEx]:[disk:47]Runtime Err: [WorkRegion: 0x0] [UserS: 0x0] [SysS: 0x0]
Nov 2 14:48:19 rac3 kernel: sbd_fun_linux INFO 2025688962,thread 60cc LinuxAgentServi,[GetNextBakBlock]:NextBitPos == LastBlock on GetNextBakBlock, lastbit: 0x8baec4, nextbit: 0x8baec4, secsOnLast: 0x5c
Nov 2 15:40:16 rac3 kernel: disksbd_linux INFO 2028806037,thread 12264 PsThread,[filter_bio_rw]😗**check bdev, mutex_trylock is fail, bd_openers: 0x91
Nov 2 16:05:43 rac3 kernel: disksbd_linux INFO 2030332706,thread 12264 PsThread,[filter_bio_rw]😗**check bdev, mutex_trylock is fail, bd_openers: 0x94
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033723618,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 11, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033723719,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 12, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033723820,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 13, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033723921,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 14, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033724022,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 15, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033724123,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 16, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033724224,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 17, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:14 rac3 kernel: sbd_fun_linux INFO 2033724325,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 18, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:15 rac3 kernel: sbd_fun_linux INFO 2033724426,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 19, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:15 rac3 kernel: sbd_fun_linux INFO 2033724527,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 20, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:15 rac3 kernel: sbd_fun_linux INFO 2033724628,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 21, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:15 rac3 kernel: sbd_fun_linux INFO 2033724729,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 22, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:15 rac3 kernel: sbd_fun_linux INFO 2033724830,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 23, remain_len: 0xf70, DataLength: 0x1040
Nov 2 17:02:48 rac3 kernel: sbd_fun_linux INFO 2033758234,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 11, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:48 rac3 kernel: sbd_fun_linux INFO 2033758335,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 12, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033758436,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 13, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033758537,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 14, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033758638,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 15, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033758739,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 16, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033758840,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 17, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033758941,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 18, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033759042,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 19, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033759144,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 20, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033759245,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 21, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:49 rac3 kernel: sbd_fun_linux INFO 2033759346,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 22, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033759447,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 23, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033759548,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 24, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033759649,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 25, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033759750,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 26, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033759851,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 27, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033759952,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 28, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033760053,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 29, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033760154,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 30, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033760255,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 31, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:50 rac3 kernel: sbd_fun_linux INFO 2033760356,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 32, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033760457,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 33, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033760558,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 34, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033760659,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 35, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033760760,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 36, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033760861,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 37, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033760962,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 38, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033761063,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 39, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033761164,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 40, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033761265,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 41, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:51 rac3 kernel: sbd_fun_linux INFO 2033761366,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 42, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033761467,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 43, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033761568,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 44, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033761669,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 45, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033761770,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 46, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033761871,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 47, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033761972,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 48, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033762073,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 49, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033762174,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 50, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033762275,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 51, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:52 rac3 kernel: sbd_fun_linux INFO 2033762376,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 52, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033762477,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 53, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033762578,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 54, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033762679,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 55, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033762780,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 56, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033762881,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 57, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033762982,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 58, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763083,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 59, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763184,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 60, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[LuxSocketSendForNotWait]:ret == -EAGAIN, trySend, tryCount: 61, remain_len: 0xcd0, DataLength: 0x1040
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[TcpKnlSendCDPData]:Failed to Sock Send cdp data, Err: 0xfffffff5, ullStartSec: 0x10d2f868, secs: 0x8!
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[SendCDPPack]:Failed to send cdp data, will DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ConvertCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before ConvertCDP
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:6)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:13)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:14)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:15)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:32)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:33)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:14)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:34)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:35)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[ProcessDevStatusKernal]:–(disk34)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[ProcessDevStatusKernal]:–(disk34)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:47)DSCC_DRIVER_CDP_TO_INCREMENT…
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:34)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[ProcessDevStatusKernal]:–(disk13)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[CDPMemListToBakQueue]:(disk:14)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ProcessDevStatusKernal]:–(disk6)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[ProcessDevStatusKernal]:–(disk13)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:14)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ProcessDevStatusKernal]:–(disk6)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:13)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:6)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[ProcessDevStatusKernal]:–(disk33)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[ProcessDevStatusKernal]:–(disk32)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[ProcessDevStatusKernal]:–(disk15)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[CDPMemListToBakQueue]:(disk:13)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[ProcessDevStatusKernal]:–(disk33)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[ProcessDevStatusKernal]:–(disk32)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[ProcessDevStatusKernal]:–(disk15)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:13)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[CDPMemListToBakQueue]:(disk:6)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:33)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:32)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:15)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[ProcessDevStatusKernal]:–(disk13)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:6)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[ProcessDevStatusKernal]:–(disk35)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[ProcessDevStatusKernal]:–(disk47)WORK_SET_INCREMENT–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[CDPMemListToBakQueue]:(disk:33)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ProcessDevStatusKernal]:–(disk6)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[ProcessDevStatusKernal]:–(disk35)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[ProcessDevStatusKernal]:–(disk47)WORK_CLEAR_CDP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:33)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[CDPMemListToBakQueue]:(disk:32)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:35)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before CDPMemListToBakQueue (id:47)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[ProcessDevStatusKernal]:–(disk33)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:32)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[CDPMemListToBakQueue]:(disk:15)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[ProcessDevStatusKernal]:–(disk32)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:15)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[CDPMemListToBakQueue]:(disk:35)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[ProcessDevStatusKernal]:–(disk15)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:35)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[CDPMemListToBakQueue]:(disk:47)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[ProcessDevStatusKernal]:–(disk35)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:47)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[ProcessDevStatusKernal]:–(disk47)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[CDPMemListToBakQueue]:(disk:34)resend 0x0 on cdp memory list bak queue
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[WorkClearCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 after CDPMemListToBakQueue (id:34)
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[ProcessDevStatusKernal]:–(disk34)WORK_CDP_CONVERT_LOG–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[WorkMergerCdpHelperBitmap]:–(disk14)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 2
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[WorkMergerCdpHelperBitmap]:–(disk14)end merger cdp helper bitmap, Status: 0x0, old used bit count: 2
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatus]:–(disk14) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ProcessDevStatusKernal]:–(disk6)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[WorkMergerCdpHelperBitmap]:–(disk6)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 8096
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[ProcessDevStatusKernal]:–(disk32)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12264 PsThread,[WorkMergerCdpHelperBitmap]:–(disk32)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 18772
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[ProcessDevStatusKernal]:–(disk47)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[ProcessDevStatusKernal]:–(disk13)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[WorkMergerCdpHelperBitmap]:–(disk47)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 3
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[ProcessDevStatusKernal]:–(disk33)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[WorkMergerCdpHelperBitmap]:–(disk13)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 2
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[WorkMergerCdpHelperBitmap]:–(disk47)end merger cdp helper bitmap, Status: 0x0, old used bit count: 3
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 12268 PsThread,[WorkMergerCdpHelperBitmap]:–(disk33)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 19311
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[WorkMergerCdpHelperBitmap]:–(disk13)end merger cdp helper bitmap, Status: 0x0, old used bit count: 2
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[ProcessDevStatusKernal]:–(disk47)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1fb0 PsThread,[ProcessDevStatus]:–(disk47) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[ProcessDevStatusKernal]:–(disk13)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[ProcessDevStatusKernal]:–(disk15)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[ProcessDevStatusKernal]:–(disk34)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[WorkMergerCdpHelperBitmap]:–(disk15)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 2
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dab PsThread,[ProcessDevStatus]:–(disk13) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[WorkMergerCdpHelperBitmap]:–(disk34)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 7
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[WorkMergerCdpHelperBitmap]:–(disk15)end merger cdp helper bitmap, Status: 0x0, old used bit count: 2
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[WorkMergerCdpHelperBitmap]:–(disk34)end merger cdp helper bitmap, Status: 0x0, old used bit count: 7
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[ProcessDevStatusKernal]:–(disk15)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[ProcessDevStatusKernal]:–(disk34)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1daf PsThread,[ProcessDevStatus]:–(disk15) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226c PsThread,[ProcessDevStatus]:–(disk34) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[ProcessDevStatusKernal]:–(disk35)WORK_MERGER_CDP_HELPER_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[WorkMergerCdpHelperBitmap]:–(disk35)start merger cdp helper bitmap, helper used bit count: 0, old used bit count: 7
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[WorkMergerCdpHelperBitmap]:–(disk35)end merger cdp helper bitmap, Status: 0x0, old used bit count: 7
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[ProcessDevStatusKernal]:–(disk35)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1226f PsThread,[ProcessDevStatus]:–(disk35) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[WorkMergerCdpHelperBitmap]:–(disk6)end merger cdp helper bitmap, Status: 0x0, old used bit count: 8096
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ProcessDevStatusKernal]:–(disk6)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ProcessDevStatus]:–(disk6) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763286,thread 12268 PsThread,[WorkMergerCdpHelperBitmap]:–(disk33)end merger cdp helper bitmap, Status: 0x0, old used bit count: 19311
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763286,thread 12268 PsThread,[ProcessDevStatusKernal]:–(disk33)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763286,thread 12268 PsThread,[ProcessDevStatus]:–(disk33) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763286,thread 12264 PsThread,[WorkMergerCdpHelperBitmap]:–(disk32)end merger cdp helper bitmap, Status: 0x0, old used bit count: 18772
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763286,thread 12264 PsThread,[ProcessDevStatusKernal]:–(disk32)WORK_CLEAR_CLOSE_CDP_FILE_BITMAP–
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763286,thread 12264 PsThread,[ProcessDevStatus]:–(disk32) waiting for DSCC is false, DevCmd: 0x86248, DevCmdParam: 0x1
Nov 2 18:03:41 rac3 kernel: MxAgent[17927]: segfault at 7 ip 00007fd5e7a2cf36 sp 00007fd5e9109220 error 4 in libpassivemonitorsrv.so[7fd5e799e000+15e000]
Nov 2 18:03:42 rac3 abrt[2998]: Saved core dump of pid 17916 (/usr/local/mxagent/bmc/bin/MxAgent) to /var/spool/abrt/ccpp-2019-11-02-18:03:42-17916 (139194368 bytes)
Nov 2 18:03:42 rac3 abrtd: Directory ‘ccpp-2019-11-02-18:03:42-17916’ creation detected
Nov 2 18:03:42 rac3 abrtd: Executable ‘/usr/local/mxagent/bmc/bin/MxAgent’ doesn’t belong to any package and ProcessUnpackaged is set to ‘no’
Nov 2 18:03:42 rac3 abrtd: ‘post-create’ on ‘/var/spool/abrt/ccpp-2019-11-02-18:03:42-17916’ exited with 1
Nov 2 18:03:42 rac3 abrtd: Deleting problem directory ‘/var/spool/abrt/ccpp-2019-11-02-18:03:42-17916’
Nov 2 18:05:17 rac3 kernel: disksbd_linux INFO 2037506523,thread 12268 PsThread,[filter_bio_rw]😗**check bdev, mutex_trylock is fail, bd_openers: 0x90
Nov 2 19:09:57 rac3 kernel: sbd_fun_linux INFO 2041387387,thread 60e5 LinuxAgentServi,[CompleteBak]:start CompleteBak
Nov 2 19:09:57 rac3 kernel: sbd_fun_linux INFO 2041387387,thread 60e5 LinuxAgentServi,[CompleteBak]:bytes: 0x0, g_mem_total_km: 0x16f5bb3, g_mem_total_vm: 0x10101000, g_mem_total_pm: 0x4a000, g_mem_total_lookaside: 0x0, g_mem_total_lookcount: 0x0
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:13)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:14)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 1dab PsThread,[ProcessDevStatusKernal]:–(disk13)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:15)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:32)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:33)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 12264 PsThread,[ProcessDevStatusKernal]:–(disk32)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:34)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:35)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:47)DSCC_COMPLETE_BAK…
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 1226c PsThread,[ProcessDevStatusKernal]:–(disk34)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 1daf PsThread,[ProcessDevStatusKernal]:–(disk15)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 1226f PsThread,[ProcessDevStatusKernal]:–(disk35)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 1fb0 PsThread,[ProcessDevStatusKernal]:–(disk47)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:58 rac3 kernel: sbd_fun_linux INFO 2041388388,thread 12268 PsThread,[ProcessDevStatusKernal]:–(disk33)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:59 rac3 kernel: sbd_fun_linux INFO 2041388453,thread 60e5 LinuxAgentServi,[GetConvertWork]:(disk:6)DSCC_COMPLETE_BAK…
Nov 2 19:09:59 rac3 kernel: sbd_fun_linux INFO 2041388453,thread 1d9d PsThread,[ProcessDevStatusKernal]:–(disk6)WORK_COMPLETE_CREATE_BAK–
Nov 2 19:09:59 rac3 kernel: sbd_fun_linux INFO 2041388476,thread 60e5 LinuxAgentServi,[ExDeleteNPagedLookasideList]:kfree mcache name,ffffffffa0b70800,ffff8802ab5f4080,filter-cache-228,65536
Nov 2 19:09:59 rac3 kernel: sbd_fun_linux INFO 2041388478,thread 60e5 LinuxAgentServi,[FreeCmpressBufDesc]:FreeCmpressBufDesc: SaveNodeCount: 0x0, LockNodeCount: 0x0, NodeTotalCount: 0xfff, TotalAddNodeCount: 0x695f
Nov 2 19:09:59 rac3 kernel: sbd_fun_linux INFO 2041388514,thread 60e5 LinuxAgentServi,[CompleteBak]:end CompleteBak
Nov 2 19:09:59 rac3 kernel: sbd_fun_linux INFO 2041388514,thread 60e5 LinuxAgentServi,[CompleteBak]:bytes: 0x0, g_mem_total_km: 0x140926b, g_mem_total_vm: 0x0, g_mem_total_pm: 0x0, g_mem_total_lookaside: 0x0, g_mem_total_lookcount: 0x0
Nov 2 19:38:36 rac3 kernel: sbd_fun_linux INFO 2043105463,thread 93c2 LinuxAgentServi,[CDPToIncrement]:CDPToIncrement…
Nov 2 19:42:57 rac3 kernel: sbd_fun_linux INFO 2043367198,thread a891 LinuxAgentServi,[GetConvertWork]:(disk:6)DSCC_INCREMENT_TO_NO_WORK…
Nov 2 19:42:57 rac3 kernel: sbd_fun_linux INFO 2043367198,thread 1d9d PsThread,[ProcessDevStatusKernal]:–(disk6)WORK_CLEAR_INCREMENT–
Nov 2 19:42:57 rac3 kernel: sbd_fun_linux INFO 2043367228,thread a891 LinuxAgentServi,[GetConvertWork]:(disk:13)DSCC_INCREMENT_TO_NO_WORK…
Nov 2 19:42:57 rac3 kernel: sbd_fun_linux INFO 2043367228,thread 1dab PsThread,[ProcessDevStatusKernal]:–(disk13)WORK_CLEAR_INCREMENT–
Nov 2 19:42:57 rac3 kernel: sbd_fun_linux INFO 2043367271,thread a891 LinuxAgentServi,[GetConvertWork]:(disk:14)DSCC_INCREMENT_TO_NO_WORK…
Nov 2 19:42:57 rac3 kernel: sbd_fun_linux INFO 2043367271,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_CLEAR_INCREMENT–
Nov 2 19:42:57 rac3 kernel: sbd_fun_linux INFO 2043367310,thread a891 LinuxAgentServi,[GetConvertWork]:(disk:15)DSCC_INCREMENT_TO_NO_WORK…
评论
有用 0存储连接超时
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 3.
操作系统也有相关报错
Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[TcpKnlSendCDPData]:Failed to Sock Send cdp data, Err: 0xfffffff5, ullStartSec: 0x10d2f868, secs: 0x8! Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[SendCDPPack]:Failed to send cdp data, will DSCC_DRIVER_CDP_TO_INCREMENT… Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[ConvertCDP]:CDPCacheRemain: 0x5a0afc00, CDPCacheCurrentSize: 0x0 before ConvertCDP Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:6)DSCC_DRIVER_CDP_TO_INCREMENT… Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:13)DSCC_DRIVER_CDP_TO_INCREMENT… Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:14)DSCC_DRIVER_CDP_TO_INCREMENT… Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:15)DSCC_DRIVER_CDP_TO_INCREMENT… Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1d9d PsThread,[GetConvertWork]:(disk:32)DSCC_DRIVER_CDP_TO_INCREMENT… Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_SET_INCREMENT– Nov 2 17:02:53 rac3 kernel: sbd_fun_linux INFO 2033763285,thread 1dad PsThread,[ProcessDevStatusKernal]:–(disk14)WORK_CLEAR_CDP–
让存储的人帮忙检查下存储吧。I/O超时15秒，ASM就是自动断开。
评论 有用 0这个磁盘组掉线了 然后手都挂载就挂载上了，而且观察几天都没在复发，那这样的情况还会再次出现吗？目前还是没有搞清楚存储哪里出了问题，检查存储 没有任何告警信息，而且磁盘组掉线还是集群的一个节点，另一个节点没问题啊?这个后续还要怎么分析呢？
评论 有用 0墨值悬赏