某天晚上22点16分左右，收到了一个RAC数据库两节点的所有实例宕库告警，登录服务器，启动数据库时，数据库open失败。
1、排查alert日志，日志中发现了ora-600报错，信息如下：
▼▼▼Mem# 1: +DG_DATA_SSD_1/NSFDB/ONLINELOG/group_4.272.10080280192021-01-04T22:26:37.030965+08:00OS process OFSD (ospid 5144) idle for 30 seconds, exiting2021-01-04T22:26:38.543347+08:00Errors in file u01/app/oracle/diag/rdbms/nsfdb/nsfdb1/trace/nsfdb1_ora_20131.trc (incident=993166):ORA-00600: internal error code, arguments: [krccfl_chunk], [0x9FFFFFFFBDE54DF0], [2], [], [], [], [], [], [], [], [], []Incident details in: u01/app/oracle/diag/rdbms/nsfdb/nsfdb1/incident/incdir_993166/nsfdb1_ora_20131_i993166.trc2021-01-04T22:26:40.532334+08:00Dumping diagnostic data in directory=[cdmp_20210104222640], requested by (instance=1, osid=2298720), summary=[incident=993166].
匹配到了一个文章“Ora-600 Krccfl_chunk When Block Change Tracking enabled (Doc ID 2046745.1)”
文章中提到了“Backup change tracking is enabled，Datafile in Rac added to local destination rather then shared location and later taken offline”，出现了Ora-600 Krccfl_chunk报错，需要先禁用块改变跟踪，才能open数据库。
节点1
▼▼▼2021-01-04T20:11:39.709767+08:00alter database datafile '+DG_DATA_SSD_2/NSFDB/DATAFILE/tbs_data.282.1060977401' onlineCompleted: alter database datafile '+DG_DATA_SSD_2/NSFDB/DATAFILE/tbs_data.282.1060977401' online2021-01-04T20:13:15.647462+08:00Thread 1 advanced to log sequence 68674 (LGWR switch) Current log# 4 seq# 68674 mem# 0: +DG_DATA_SSD_1/NSFDB/ONLINELOG/group_4.271.1008028017 Current log# 4 seq# 68674 mem# 1: +DG_DATA_SSD_1/NSFDB/ONLINELOG/group_4.272.10080280192021-01-04T20:13:16.273721+08:00
节点2
▼▼▼2021-01-04T19:46:14.210687+08:00alter tablespace TBS_DATA add datafile'DG_DATA_SSD_2' size 30g autoextend offORA-1537 signalled during: alter tablespace TBS_DATA add datafile'DG_DATA_SSD_2' size 30g autoextend off...2021-01-04T19:46:21.320437+08:00alter tablespace TBS_DATA add datafile'DG_DATA_SSD_2' size 30g autoextend offORA-1537 signalled during: alter tablespace TBS_DATA add datafile'DG_DATA_SSD_2' size 30g autoextend off...2021-01-04T19:51:10.043864+08:002021-01-04T19:56:41.886951+08:00ALTER DATABASE MOVE DATAFILE '/u01/app/oracle/product/12.2.0/db_1/dbs/DG_DATA_SSD_2' TO '+DG_DATA_SSD_2'2021-01-04T19:56:41.908716+08:00Moving datafile /u01/app/oracle/product/12.2.0/db_1/dbs/DG_DATA_SSD_2 (686) to +DG_DATA_SSD_22021-01-04T20:00:11.018835+08:00Thread 2 advanced to log sequence 54383 (LGWR switch) Current log# 5 seq# 54383 mem# 0: +DG_DATA_SSD_1/NSFDB/ONLINELOG/group_5.273.1008028023 Current log# 5 seq# 54383 mem# 1: +DG_DATA_SSD_1/NSFDB/ONLINELOG/group_5.274.10080280252021-01-04T20:00:11.726628+08:00TT02: Standby redo logfile selected for thread 2 sequence 54383 for destination LOG_ARCHIVE_DEST_22021-01-04T20:00:16.340789+08:00Archived Log entry 178702 added for T-2.S-54382 ID 0x5a823d6f LAD:12021-01-04T20:01:29.321397+08:00Move operation committed for file +DG_DATA_SSD_2/NSFDB/DATAFILE/tbs_data.282.10609774012021-01-04T20:01:31.385729+08:00Completed: ALTER DATABASE MOVE DATAFILE '/u01/app/oracle/product/12.2.0/db_1/dbs/DG_DATA_SSD_2' TO '+DG_DATA_SSD_2'
咨询了项目组同事，确认了操作时间和操作步骤都符合
22点定时备份任务开始启动，数据库备份时读取块改变跟踪的记录文件时，因为读不到了在本地添加的数据文件，触发了数据库数据保护机制，从而导致数据库实例cash掉。
2、那就根据文章中的提供的workaround，先启动实例到mount，再执行如下命令：
alter database disable block change tracking;
alter database open;
alter database enable block change tracking using file ‘+磁盘组名称/文件名’;
注意：以前的块改变跟踪信息因为disable/enable操作丢失了，需要重新进行一次0级备份之后，块改变跟踪信息才能生效。
增加操作审核，避免误操作 在使用Move table online操作时，要注意块改变跟踪的问题。
更多精彩干货分享
点击下方名片关注
IT那活儿
