2025-09-30T15:11:36.467669+08:00
Errors in file /app/u01/app/oracle/diag/rdbms/xxxx/xxxx/trace/xxxx_ora_181296.trc:
ORA-00600: internal error code, arguments: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []
2025-09-30T15:11:36.467934+08:00
Errors in file /app/u01/app/oracle/diag/rdbms/xxxx/xxxx/trace/xxxx_ora_181296.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []
2025-09-30T15:11:36.468158+08:00
Errors in file /app/u01/app/oracle/diag/rdbms/xxxx/xxxx/trace/xxxx_ora_181296.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
Errors in file /app/u01/app/oracle/diag/rdbms/xxxx/xxxx/trace/xxxx_ora_181296.trc (incident=1997124):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /app/u01/app/oracle/diag/rdbms/xxxx/xxxx/incident/incdir_1997124/xxxx_ora_181296_i1997124.trc
2025-09-30T15:11:37.263549+08:00
到达现场后，首先我在进行open resetlogs操作之前，打开10046 trace:
alter database mount;
recover database using backup controlfile until cance;
oradebug setmypid
oradebug tracefile_name
alter session set events '10046 trace name context forever,level 12';
alter session set db_file_multiblock_read_count=1;
alter database open;
oradebug EVENT 10046 trace name context off;
alter database open resetlogs;
startup mount;
oradebug setmypid
oradebug tracefile_name
alter session set events '10046 trace name context forever,level 12';
alter session set db_file_multiblock_read_count=1;
alter database open;
oradebug EVENT 10046 trace name context off;
--接下来我们来看看此时的trace 文件内容:
====================
PARSING IN CURSOR #140014359655632 len=165 dep=3 uid=0 oct=3 lid=0 tim=101032237863 hv=1641257687 ad='1bffd8740' sqlid='2sxqgx5hx76qr'
select /*+ rule */ bucket, endpoint, col#, epvalue, epvalue_raw, ep_repeat_count, endpoint_enc from histgrm$ where obj#=:1 and intcol#=:2 and row#=:3 order by bucket
END OF STMT
PARSE #140014359655632:c=407,e=407,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=0,tim=101032237862
BINDS #140014359655632:
Bind#0
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=00 fl2=1000001 frm=00 csi=00 siz=72 off=0
kxsbbbfp=7f57a22b9de8 bln=22 avl=02 flg=05
value=4
Bind#1
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
kxsbbbfp=7f57a22b9e00 bln=22 avl=02 flg=01
value=3
Bind#2
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=48
kxsbbbfp=7f57a22b9e18 bln=22 avl=01 flg=01
value=0
EXEC #140014359655632:c=754,e=833,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=3312420081,tim=101032238782
WAIT #140014359655632: nam='db file sequential read' ela= 10 file#=1 block#=601 blocks=1 obj#=65 tim=101032238831
WAIT #140014359655632: nam='db file sequential read' ela= 6 file#=1 block#=602 blocks=1 obj#=65 tim=101032238876 --这里是报起点，先查询obj，发现有事务再去查询的undo，然后最终报错
WAIT #140014359640384: nam='PGA memory operation' ela= 9 p1=65536 p2=1 p3=0 obj#=65 tim=101032239079
=====================
PARSING IN CURSOR #140014359640384 len=142 dep=4 uid=0 oct=3 lid=0 tim=101032239436 hv=361892850 ad='1affc8188' sqlid='7bd391hat42zk'
select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1
END OF STMT
PARSE #140014359640384:c=494,e=493,p=0,cr=0,cu=0,mis=1,r=0,dep=4,og=3,plh=0,tim=101032239435
WAIT #140014359640384: nam='PGA memory operation' ela= 9 p1=65536 p2=1 p3=0 obj#=65 tim=101032239934
BINDS #140014359640384:
Bind#0
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7f57a22b74f8 bln=22 avl=02 flg=05
value=14
........
=====================
PARSING IN CURSOR #140014359640384 len=142 dep=4 uid=0 oct=3 lid=0 tim=101032240723 hv=361892850 ad='1affc8188' sqlid='7bd391hat42zk'
select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1
END OF STMT
PARSE #140014359640384:c=17,e=17,p=0,cr=0,cu=0,mis=0,r=0,dep=4,og=3,plh=906473769,tim=101032240722
BINDS #140014359640384:
Bind#0
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7f57a22b74f8 bln=22 avl=02 flg=05
value=16
........
ktuisc detected lost changes to transaction table:
xid of interest: xid: 0x0010.009.00bdf194 xid from block : xid: 0x0010.009.00bdf16e
*** 2025-09-30T15:54:56.264929+08:00
2025-09-30T15:54:56.264914+08:00
Incident 2125123 created, dump file: /app/u01/app/oracle/diag/rdbms/xxxx/xxxx/incident/incdir_2125123/xxxx_ora_184420_i2125123.trc
ORA-00600: internal error code, arguments: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []
=====================
PARSE ERROR #140014359557496:len=66 dep=1 uid=0 oct=3 lid=0 tim=101033738243 err=600
看上面信息来看，file 1 block 602 可能有些问题。进一步查看incident详细信息，能看到该block 确实存在活跃事务，正好是跟前面报错提到的回滚段14,16 匹配。
Object id on Block? Y
seg/obj: 0x41 csc: 0x000018e68035b860 itc: 3 flg: - typ: 2 - INDEX
fsl: 0 fnx: 0x0 ver: 0x01
Itl Xid Uba Flag Lck Scn/Fsc
0x01 0x000a.001.000004d5 0x0100028e.0116.01 CBU- 0 scn 0x000000000016413c
0x02 0x000e.00c.014de471 0x011a5e21.00be.08 ---- 1 fsc 0x0000.00000000 --Flag等于三个-，代表活动事务，影响行数1行
0x03 0x0010.009.00bdf194 0x011ed923.2edf.09 ---- 1 fsc 0x0000.00000000
0x000e.00c.014de471=回滚段编号.回滚序列号.回滚记录号 , e 和10 分部对于十进制14和16
0x011a5e21.00be.08=回滚段地址(文件号+块号) .回滚序列. 回滚记录号
--blcok trc:
row#0[8020] flag: ------, lock: 0, len=12
col 0; len 2; (2): c1 04
col 1; len 6; (6): 01 00 02 0b 00 05
row#1[8008] flag: ------, lock: 0, len=12
col 0; len 2; (2): c1 10
col 1; len 6; (6): 01 00 02 0b 00 03
row#2[7996] flag: ---D--, lock: 0, len=12
col 0; len 2; (2): c1 15
col 1; len 6; (6): 01 00 02 0b 00 00
row#3[7984] flag: ------, lock: 1, len=12
col 0; len 2; (2): c1 1a
col 1; len 6; (6): 01 00 02 0b 00 06
row#4[7972] flag: ------, lock: 1, len=12
col 0; len 2; (2): c1 1d
col 1; len 6; (6): 01 00 02 0b 00 02
row#5[7960] flag: ------, lock: 0, len=12
col 0; len 2; (2): c1 1e
col 1; len 6; (6): 01 00 02 0b 00 04
row#6[7948] flag: ------, lock: 0, len=12
col 0; len 2; (2): c1 2a
col 1; len 6; (6): 01 00 02 0b 00 07
row#7[7936] flag: ------, lock: 0, len=12
col 0; len 2; (2): c1 2f
col 1; len 6; (6): 01 00 02 0b 00 01
还应该看到类似的信息(这个信息其他地方copy的不一致)，记录LOCK =1的offset ，后面需要修改
--下面用bbed修改事务状态和索引条目状态
set file 1 block 602
BBED> p ktbbh
struct ktbbh, 96 bytes @20
...........
struct ktbbhitl[0], 24 bytes @44
............
struct ktbbhitl[1], 24 bytes @68
struct ktbitxid, 8 bytes @68
ub2 kxidusn @68 0x000e
ub2 kxidslt @70 0x000c
ub4 kxidsqn @72 0x014de471
struct ktbituba, 8 bytes @76
ub4 kubadba @76 0x011a5e21
ub2 kubaseq @80 0x00be
ub1 kubarec @82 0x08
ub2 ktbitflg @84 0x0001 (NONE) --修这里修改成0080
union _ktbitun, 2 bytes @86
sb2 _ktbitfsc @86 0 --如果这里不为0需要修改成0
ub2 _ktbitwrp @86 0x0000
ub4 ktbitbas @88 0x00000000
struct ktbbhitl[2], 24 bytes @92
struct ktbitxid, 8 bytes @92
ub2 kxidusn @92 0x0010
ub2 kxidslt @94 0x0009
ub4 kxidsqn @96 0x00bdf194
struct ktbituba, 8 bytes @100
ub4 kubadba @100 0x011ed923
ub2 kubaseq @104 0x2edf
ub1 kubarec @106 0x09
ub2 ktbitflg @108 0x0001 (NONE) --修这里修改成0080
union _ktbitun, 2 bytes @110
sb2 _ktbitfsc @110 0 --如果这里不为0需要修改成0
ub2 _ktbitwrp @110 0x0000
ub4 ktbitbas @112 0x00000000
modify /x 0080 offset 84
modify /x 0080 offset 108
sum apply
--修改索引条目
BBED> p kd_off
sb2 kd_off[0] @148 8008
sb2 kd_off[1] @150 3456
......
sb2 kd_off[321] @790 2521
BBED> p *kd_off[83]
BBED> x /rcccccnnnnnnn
rowdata[5062] @6748
-------------
flag@6748: 0x00 (NONE)
lock@6749: 0x02
keydata[8]: 0x00 0x42 0x42 0xad 0x00 0x00 0x02 0x00
data key:
col 0[2] @6759: .!
col 1[2] @6762: ..
BBED> d /v offset 6749 count 2
File: /app/u01/app/oracle/oradata/xxxx/system01.dbf (1)
Block: 602 Offsets: 6749 to 6750 Dba:0x0040025a
-------------------------------------------------------
0200 l ..
<16 bytes per line>
BBED> modify /x 00 offset 6749
File: /app/u01/app/oracle/oradata/xxxx/system01.dbf (1)
Block: 602 Offsets: 6749 to 6750 Dba:0x0040025a
------------------------------------------------------------------------
0000
<32 bytes per line>
BBED> p *kd_off[244]
rowdata[2260]
-------------
ub1 rowdata[2260] @3946 0x00
BBED> x /rcccccnnnnn
rowdata[2260] @3946
-------------
flag@3946: 0x00 (NONE)
lock@3947: 0x03
keydata[8]: 0x00 0x42 0x1f 0x91 0x00 0x00 0x02 0x00
data key:
col 0[3] @3957: ..+
col 1[2] @3961: ..
BBED> modify /x 00 offset 3947
File: /app/u01/app/oracle/oradata/xxxx/system01.dbf (1)
Block: 602 Offsets: 3947 to 3948 Dba:0x0040025a
------------------------------------------------------------------------
0000
<32 bytes per line>
BBED> sum apply
Check value for File 1, Block 602:
current = 0x1db4, required = 0x1db4
修改完成之后，我们来尝试启动数据库试试【注意：已经设置 undo_management=manual】。
「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文章的来源（墨天轮），文章链接，文章作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
