2019-11-17T05:42:54.481877+08:00
Error 3150 trapped in 2PC on transaction 3.15.163075. Cleaning up.
Error stack returned to user:
ORA-03150: 数据库链接的通信通道的文件结尾
ORA-02063: 紧接着 line (起自 LINK_170_IDEALOAGT)

2019-11-17T21:21:57.409496+08:00
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x10D29747, kxsxsi()+439] [flags: 0x0, count: 1]
Errors in file /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_61456.trc (incident=822016):
ORA-07445: 出现异常错误: 核心转储 [kxsxsi()+439] [SIGSEGV] [ADDR:0x0] [PC:0x10D29747] [SI_KERNEL(general_protection)] []
Incident details in: /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/incident/incdir_822016/racdb1_ora_61456_i822016.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

2019-11-18T09:03:21.023121+08:00
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x10A305E3, ksmem_malloc()+227] [flags: 0x0, count: 1]
Errors in file /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_99348.trc (incident=821984):
ORA-07445: 出现异常错误: 核心转储 [ksmem_malloc()+227] [SIGSEGV] [ADDR:0x0] [PC:0x10A305E3] [SI_KERNEL(general_protection)] []
ORA-12612: TNS: 连接正忙
ORA-24920: 列大小对于客户机过大
Incident details in: /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/incident/incdir_821984/racdb1_ora_99348_i821984.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x1277340, ksmem_dump()+80] [flags: 0x0, count: 2]
Errors in file /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_99348.trc (incident=821985):
ORA-07445: 出现异常错误: 核心转储 [ksmem_dump()+80] [SIGSEGV] [ADDR:0x0] [PC:0x1277340] [SI_KERNEL(general_protection)] []
ORA-07445: 出现异常错误: 核心转储 [ksmem_malloc()+227] [SIGSEGV] [ADDR:0x0] [PC:0x10A305E3] [SI_KERNEL(general_protection)] []
ORA-12612: TNS: 连接正忙
ORA-24920: 列大小对于客户机过大
Incident details in: /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/incident/incdir_821985/racdb1_ora_99348_i821985.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2019-11-18T09:03:23.233042+08:00
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0xF85D81B, dbgexDumpErrDesc()+107] [flags: 0x0, count: 3]
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x238] [PC:0x3F4ECD1, kopdarnew()+417] [flags: 0x0, count: 3]
Use ADRCI or Support Workbench to package the incident.

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x238] [PC:0x3F4ECD1, kopdarnew()+417] [flags: 0x0, count: 2]
Errors in file /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_99348.trc (incident=821986):
ORA-07445: 出现异常错误: 核心转储 [kopdarnew()+417] [SIGSEGV] [ADDR:0x238] [PC:0x3F4ECD1] [Address not mapped to object] []
Incident details in: /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/incident/incdir_821986/racdb1_ora_99348_i821986.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2019-11-18T09:03:24.612565+08:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x238] [PC:0x3F4ECD1, kopdarnew()+417] [flags: 0x0, count: 3]
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2019-11-18T09:03:44.313607+08:00
Dumping diagnostic data in directory=[cdmp_20191118090344], requested by (instance=2, osid=35138), summary=[incident=491607].
2019-11-18T09:09:06.099007+08:00
WARNING: Heavy swapping observed on system in last 5 mins.
Heavy swapping can lead to timeouts, poor performance, and instance eviction.
Errors in file /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbrm_93729.trc (incident=820928):
ORA-00700: 软内部错误, 参数: [kskvmstatact: excessive swapping observed], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/incident/incdir_820928/racdb1_dbrm_93729_i820928.trc
2019-11-18T09:09:09.376603+08:00
Dumping diagnostic data in directory=[cdmp_20191118090909], requested by (instance=1, osid=93729 (DBRM)), summary=[incident=820928].
2019-11-18T09:10:16.553509+08:00
Warning: VKTM detected a time stall.
Time drifts can result in unexpected behavior such as time-outs.
Please see the VKTM trace file for more details:
/opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_vktm_93709.trc
2019-11-18T09:22:49.828899+08:00
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x608082B, __intel_memset()+1339] [flags: 0x0, count: 1]
Errors in file /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_126447.trc (incident=822064):
ORA-07445: 出现异常错误: 核心转储 [__intel_memset()+1339] [SIGSEGV] [ADDR:0x0] [PC:0x608082B] [SI_KERNEL(general_protection)] []
Incident details in: /opt/oracle/app/oracle/diag/rdbms/racdb/racdb1/incident/incdir_822064/racdb1_ora_126447_i822064.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

节点2

2019-11-18T09:03:40.598946+08:00
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x10D29747, kxsxsi()+439] [flags: 0x0, count: 1]
Errors in file /opt/oracle/app/oracle/diag/rdbms/racdb/racdb2/trace/racdb2_ora_35138.trc (incident=491607):
ORA-07445: 出现异常错误: 核心转储 [kxsxsi()+439] [SIGSEGV] [ADDR:0x0] [PC:0x10D29747] [SI_KERNEL(general_protection)] []
Incident details in: /opt/oracle/app/oracle/diag/rdbms/racdb/racdb2/incident/incdir_491607/racdb2_ora_35138_i491607.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2019-11-18T09:03:44.314011+08:00
Dumping diagnostic data in directory=[cdmp_20191118090344], requested by (instance=2, osid=35138), summary=[incident=491607].
2019-11-18T09:09:09.376692+08:00
Dumping diagnostic data in directory=[cdmp_20191118090909], requested by (instance=1, osid=93729 (DBRM)), summary=[incident=820928].
2019-11-18T09:22:54.520602+08:00
Dumping diagnostic data in directory=[cdmp_20191118092254], requested by (instance=1, osid=126447), summary=[incident=822064].