某客户的一个意外掉电case

很久没写文章了；实在不知道写什么。姑且将一个网友的问题写出来吧。对方问到如何处理一个ora-00600错误；通过日志简单看了一下，是一个非常常见且熟悉的错误；很明显对方没有看过我的blog。

Errors in file /u1/app/diag/rdbms/fast/fast/trace/fast_ora_82483.trc  (incident=67409):
ORA-00600: internal error code, arguments: [2662], [2369], [89010686], [2369], [89168901], [54526080], [], [], [], [], [], []
Incident details in: /u1/app/diag/rdbms/fast/fast/incident/incdir_67409/fast_ora_82483_i67409.trc
Fri Aug 30 11:11:14 2019
Dumping diagnostic data in directory=[cdmp_20190830111114], requested by (instance=1, osid=82483), summary=[incident=67409].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u1/app/diag/rdbms/fast/fast/trace/fast_ora_82483.trc:
ORA-00600: internal error code, arguments: [2662], [2369], [89010686], [2369], [89168901], [54526080], [], [], [], [], [], []
Errors in file /u1/app/diag/rdbms/fast/fast/trace/fast_ora_82483.trc:
ORA-00600: internal error code, arguments: [2662], [2369], [89010686], [2369], [89168901], [54526080], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 82483): terminating the instance due to error 600
Instance terminated by USER, pid = 82483
ORA-1092 signalled during: alter database open resetlogs...
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u1/app/diag/rdbms/fast/fast/trace/fast_ora_82483.trc:
ORA-00600: internal error code, arguments: [2662], [2369], [89010689], [2369], [89168901], [54526080], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [2369], [89010688], [2369], [89168901], [54526080], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [2369], [89010686], [2369], [89168901], [54526080], [], [], [], [], [], []
Errors in file /u1/app/diag/rdbms/fast/fast/trace/fast_ora_82483.trc:
ORA-27300: OS system dependent operation:semctl failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgpwpost1
ORA-27303: additional information: semid = 622594
ORA-00600: internal error code, arguments: [2662], [2369], [89010689], [2369], [89168901], [54526080], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [2369], [89010688], [2369], [89168901], [54526080], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [2369], [89010686], [2369], [89168901], [54526080], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.

从错误来看；数据库在open的时候报错。针对该错误写过太多的文章和案例了，这里就不多说了。简单说下处理思路：

1、通过bbed 修改该数据块，提交问题事务

2、通过Oradebug poke去修改scn；推进整个db的scn。

需要注意的是，由于网友这里是11204版本，因此按照网上的处理方法就失败了。

SQL> oradebug setmypid
Statement processed.
SQL> oradebug DUMPvar SGA kcsgscn_
kcslf kcsgscn_ [06001AE70, 06001AEA0) = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6001AB50 00000000
SQL> oradebug poke 0x06001AE74  4 0x00000943
BEFORE: [06001AE74, 06001AE78) = 00000000
AFTER:  [06001AE74, 06001AE78) = 00000943
SQL> oradebug DUMPvar SGA kcsgscn_
kcslf kcsgscn_ [06001AE70, 06001AEA0) = 00000000 00000943 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6001AB50 00000000
SQL> 
SQL> alter database open resetlogs;

Database altered.

SQL>

可以看到，推进scn后很容易就打开了数据库。后面又提人到说有个文件之前被人重建控制文件丢掉了；现在状态是missing。

SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/u01/oradata/fast/system01.dbf
/u01/oradata/fast/sysaux01.dbf
/u1/app/11.2.0/db/dbs/MISSING00003
/u01/oradata/fast/users01.dbf
/u01/oradata/fast/WMS6D01.dbf
/u01/oradata/fast/WMS6D02.dbf
/u01/oradata/fast/WMS6D03.dbf
/u01/oradata/fast/WMS6D04.dbf
/u01/oradata/fast/WMS6D05.dbf
/u01/oradata/fast/WMSZZ01.dbf
/u01/oradata/fast/WMSZZ02.dbf
/u01/oradata/fast/WMSZZ03.dbf
/u01/oradata/fast/undotbs02.dbf

对于这个问题，之前blog写过类似问题处理方式，基本上没有别的方法，只能bbed去修改；否则直接恢复会报错：

SQL> alter database datafile 3 online;
alter database datafile 3 online
*
ERROR at line 1:
ORA-01190: control file or data file 3 is from before the last RESETLOGS
ORA-01110: data file 3: '/u01/oradata/fast/ogg_01.dbf'

处理方式就是直接bbed修改文件头就好了；这里不再描述了。

SQL> recover datafile 3;
Media recovery complete.
SQL> alter database datafile 3 online;

Database altered.

SQL> select file#,name,status,RESETLOGS_CHANGE#,CHECKPOINT_CHANGE# from v$datafile_header order by 1;

     FILE# NAME                                                         STATUS      RESETLOGS_CHANGE#   CHECKPOINT_CHANGE#
---------- ------------------------------------------------------------ ------- --------------------- --------------------
         1 /u01/oradata/fast/system01.dbf                               ONLINE         10174866534908       10183367458817
         2 /u01/oradata/fast/sysaux01.dbf                               ONLINE         10174866534908       10183367458817
         3 /u01/oradata/fast/ogg_01.dbf                                 ONLINE         10174866534908       10183367557538
         4 /u01/oradata/fast/users01.dbf                                ONLINE         10174866534908       10183367458817
         5 /u01/oradata/fast/WMS6D01.dbf                                ONLINE         10174866534908       10183367458817
         6 /u01/oradata/fast/WMS6D02.dbf                                ONLINE         10174866534908       10183367458817
         7 /u01/oradata/fast/WMS6D03.dbf                                ONLINE         10174866534908       10183367458817
         8 /u01/oradata/fast/WMS6D04.dbf                                ONLINE         10174866534908       10183367458817
         9 /u01/oradata/fast/WMS6D05.dbf                                ONLINE         10174866534908       10183367458817
        10 /u01/oradata/fast/WMSZZ01.dbf                                ONLINE         10174866534908       10183367458817
        11 /u01/oradata/fast/WMSZZ02.dbf                                ONLINE         10174866534908       10183367458817
        12 /u01/oradata/fast/WMSZZ03.dbf                                ONLINE         10174866534908       10183367458817
        13 /u01/oradata/fast/undotbs02.dbf                              ONLINE         10174866534908       10183367458817

13 rows selected.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

SQL>

对于Oracle实在没有太多可以写的东西了；下次写点MySQl、PostgreSQL的文章吧。