事故场景
场景：级联dg，A>B>C，C库failover后，A库交换机流量被占满，影响A库正常业务。
下面是故障复现及恢复的步骤：
复现步骤
1、不规范配置
级联C库比正常配置多了如下归档传输至A库和B库配置：
ALTER SYSTEM SET log_archive_dest_2='SERVICE=gbkdbdg LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=gbkdbdg' SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_dest_3='SERVICE=gbkdb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=gbkdb' SCOPE=BOTH SID='*';
2、B库defer传输至C库
在时间点T1，B库执行defer传输至C库
--B库执行
SQL> alter system set log_archive_dest_state_3=defer;
System altered.
3、A库正常切换归档
A库模拟业务，切换10次归档
SQL> alter system switch logfile;
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
4、B库应用正常，但是C库因为没有接收到归档而延迟
--B库
SQL> col name for a24
SQL> col value for a30
SQL> col UNIT for a35
SQL> col TIME_COMPUTED for a40
SQL> col DATUM_TIME for a40
SQL> select * from v$dataguard_stats;
NAME VALUE UNIT TIME_COMPUTED DATUM_TIME
------------------------ ------------------------------ ----------------------------------- ---------------------------------------- ----------------------------------------
transport lag +00 00:00:00 day(2) to second(0) interval 03/19/2023 12:42:02 03/19/2023 12:42:01
apply lag +00 00:00:00 day(2) to second(0) interval 03/19/2023 12:42:02 03/19/2023 12:42:01
apply finish time +00 00:00:00.000 day(2) to second(3) interval 03/19/2023 12:42:02
estimated startup time 6 second 03/19/2023 12:42:02
SQL>
SQL> select thread#,sequence#,process,status from v$managed_standby;
THREAD# SEQUENCE# PROCESS STATUS
---------- ---------- --------------------------- ------------------------------------
1 34 ARCH CLOSING
1 33 ARCH CLOSING
1 22 ARCH CLOSING
1 32 ARCH CLOSING
0 0 RFS IDLE
0 0 RFS IDLE
1 35 RFS IDLE
1 35 MRP0 APPLYING_LOG
8 rows selected.
SQL> set lines 200
SQL> col checkpoint_change# for 999999999999999999999
SQL> SELECT CURRENT_SCN FROM V$DATABASE;
CURRENT_SCN
-----------
4569055246
RMAN> list incarnation;
using target database control file instead of recovery catalog
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 GBKDB 845494326 PARENT 1 24-AUG-13
2 2 GBKDB 845494326 PARENT 925702 29-DEC-21
3 3 GBKDB 845494326 ORPHAN 4568599459 13-FEB-23
4 4 GBKDB 845494326 PARENT 4568599516 13-FEB-23
5 5 GBKDB 845494326 ORPHAN 4568606764 13-FEB-23
6 6 GBKDB 845494326 PARENT 4568606764 13-FEB-23
7 7 GBKDB 845494326 PARENT 4568627631 13-FEB-23
8 8 GBKDB 845494326 PARENT 4568628222 13-FEB-23
9 9 GBKDB 845494326 PARENT 4568952372 18-MAR-23
10 10 GBKDB 845494326 CURRENT 4569006964 18-MAR-23
--C库，延迟了10个归档
SQL> set lines 200
SQL> col name for a24
SQL> col value for a30
SQL> col UNIT for a35
SQL> col TIME_COMPUTED for a40
SQL> col DATUM_TIME for a40
SQL> select * from v$dataguard_stats;
NAME VALUE UNIT TIME_COMPUTED DATUM_TIME
------------------------ ------------------------------ ----------------------------------- ---------------------------------------- ----------------------------------------
transport lag +00 00:06:44 day(2) to second(0) interval 03/19/2023 12:42:31 03/19/2023 12:35:34
apply lag +00 00:06:44 day(2) to second(0) interval 03/19/2023 12:42:31 03/19/2023 12:35:34
apply finish time day(2) to second(3) interval 03/19/2023 12:42:31
estimated startup time 6 second 03/19/2023 12:42:31
SQL>
SQL> select thread#,sequence#,process,status from v$managed_standby;
THREAD# SEQUENCE# PROCESS STATUS
---------- ---------- --------------------------- ------------------------------------
1 20 ARCH CLOSING
1 23 ARCH CLOSING
0 0 ARCH CONNECTED
1 24 ARCH CLOSING
0 0 RFS IDLE
0 0 RFS IDLE
0 0 RFS IDLE
1 25 MRP0 WAIT_FOR_LOG
0 0 RFS IDLE
0 0 RFS IDLE
0 0 RFS IDLE
11 rows selected.
SQL> SELECT CURRENT_SCN FROM V$DATABASE;
CURRENT_SCN
-----------
4569051565
5、C库failover
故障发生：
B库因为收到resetlogs_id，B库的数据库化身发生变化，MRP异常；因此二次请求A库传输归档，从defer到C库之后，主库新增的10个归档。对于生产环境，如果相差的归档较多，归档日志一起重新传输，占用了交换机流量，影响A库的业务。
SQL> alter database recover managed standby database finish;
Database altered.
SQL> alter database commit to switchover to primary with session shutdown;
Database altered.
SQL> shutdown abort
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 626327552 bytes
Fixed Size 2255832 bytes
Variable Size 234882088 bytes
Database Buffers 381681664 bytes
Redo Buffers 7507968 bytes
Database mounted.
Database opened.
SQL>
--C库日志，因为配置问题，C库resetlog后的日志传输到备库B库
Sun Mar 19 12:28:36 2023
RFS[6]: Assigned to RFS process 3080
RFS[6]: Opened log for thread 1 sequence 21 dbid 845494326 branch 1131820798
Archived Log entry 77 added for thread 1 sequence 21 rlc 1131820798 ID 0x34bcb66a dest 3:
Media Recovery Log /u01/app/oracle/gbkdrarch/1_21_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_22_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_23_1131820798.dbf
Media Recovery Waiting for thread 1 sequence 24
Sun Mar 19 12:28:51 2023
RFS[6]: Selected log 11 for thread 1 sequence 24 dbid 845494326 branch 1131820798
Sun Mar 19 12:28:51 2023
Archived Log entry 78 added for thread 1 sequence 24 ID 0x34bcb66a dest 1:
Sun Mar 19 12:28:51 2023
Media Recovery Log /u01/app/oracle/gbkdrarch/1_24_1131820798.dbf
Media Recovery Waiting for thread 1 sequence 25
Sun Mar 19 12:47:33 2023
alter database recover managed standby database finish
Terminal Recovery: Stopping real time apply
Sun Mar 19 12:47:33 2023
MRP0: Background Media Recovery cancelled with status 16037
Errors in file /u01/app/oracle/diag/rdbms/gbkdbdr/gbkdbdr/trace/gbkdbdr_pr00_2898.trc:
ORA-16037: user requested cancel of managed recovery operation
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Sun Mar 19 12:47:33 2023
MRP0: Background Media Recovery process shutdown (gbkdbdr)
Terminal Recovery: Stopped real time apply
All dispatchers and shared servers shutdown
CLOSE: killing server sessions.
CLOSE: all sessions shutdown successfully.
Sun Mar 19 12:47:34 2023
SMON: disabling cache recovery
Attempt to do a Terminal Recovery (gbkdbdr)
Media Recovery Start: Managed Standby Recovery (gbkdbdr)
started logmerger process
Sun Mar 19 12:47:34 2023
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 2 slaves
Media Recovery Waiting for thread 1 sequence 25
Killing 6 processes with pids 2890,2894,2892,3066,3074,3080 (all RFS, wait for I/O) in order to disallow current and future RFS connections. Requested by OS process 3269
Sun Mar 19 12:47:37 2023
idle dispatcher 'D000' terminated, pid = (17, 1)
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
Terminal Recovery timestamp is '03/19/2023 12:47:38'
Terminal Recovery: applying standby redo logs.
Terminal Recovery: thread 1 seq# 25 redo required
Media Recovery Waiting for thread 1 sequence 25
Terminal Recovery: End-Of-Redo log allocation
MRP: Validating standby redo logfile 11
Media Recovery Log /u01/app/oracle/oradata/gbkdbdr/sredo11.log
Terminal Recovery: log 11 reserved for thread 1 sequence 25
Recovery of Online Redo Log: Thread 1 Group 11 Seq 25 Reading mem 0
Mem# 0: /u01/app/oracle/oradata/gbkdbdr/sredo11.log
Identified End-Of-Redo (failover) for thread 1 sequence 25 at SCN 0xffff.ffffffff
Incomplete Recovery applied until change 4569051567 time 03/19/2023 12:28:49
Media Recovery Complete (gbkdbdr)
Terminal Recovery: successful completion
Sun Mar 19 12:51:00 2023
VKRM started with pid=27, OS id=3415
FAL[server, ARC3]: Error 16009 creating remote archivelog file 'gbkdb'
FAL[server, ARC3]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance gbkdbdr - Archival Error. Archiver continuing.
Sun Mar 19 12:51:04 2023
Thread 1 advanced to log sequence 3 (LGWR switch)
Current log# 3 seq# 3 mem# 0: /u01/app/oracle/oradata/gbkdbdr/redo03.log
Archived Log entry 83 added for thread 1 sequence 2 ID 0x34bc8f2c dest 1:
ARC2: Standby redo logfile selected for thread 1 sequence 2 for destination LOG_ARCHIVE_DEST_2
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
LNS: Standby redo logfile selected for thread 1 sequence 3 for destination LOG_ARCHIVE_DEST_2
--B库日志，因为收到resetlogs_id，B库的数据库化身发生变化，MRP异常；并且二次请求A库传输归档，从defer到C库之后，主库新增的10个归档
Sun Mar 19 12:50:57 2023
RFS[4]: Assigned to RFS process 3401
RFS[4]: New Archival REDO Branch: 1131886078 Current: 1131820798
RFS[4]: Selected log 13 for thread 1 sequence 1 dbid 845494326 branch 1131886078
A new recovery destination branch has been registered
RFS[4]: Standby in the future of new recovery destinationBranch(resetlogs_id) 1131886078
Incomplete Recovery SCN: 4569052162
Resetlogs SCN: 4569051568
Standby Became Primary SCN: 4569051565
Flashback database to SCN 4569051565 to follow new branch
Flashback database to SCN 4569051565 to follow new branch
RFS[4]: New Archival REDO Branch(resetlogs_id): 1131886078 Prior: 1131820798
RFS[4]: Archival Activation ID: 0x34bc8f2c Current: 0x34bcb66a
RFS[4]: Effect of primary database OPEN RESETLOGS
RFS[4]: Managed Standby Recovery process is active
RFS[4]: Incarnation entry added for Branch(resetlogs_id): 1131886078 (gbkdb)
Sun Mar 19 12:50:57 2023
Setting recovery target incarnation to 11
Sun Mar 19 12:50:57 2023
MRP0: Incarnation has changed! Retry recovery...
Errors in file /u01/app/oracle/diag/rdbms/gbkdbdg/gbkdb/trace/gbkdb_pr00_3082.trc:
ORA-19906: recovery target incarnation changed during recovery
Managed Standby Recovery not using Real Time Apply
Sun Mar 19 12:50:57 2023
Archived Log entry 229 added for thread 1 sequence 1 ID 0x34bc8f2c dest 1:
Recovery interrupted!
Recovered data files to a consistent state at change 4569055775
Sun Mar 19 12:50:58 2023
started logmerger process
Sun Mar 19 12:50:58 2023
Managed Standby Recovery starting Real Time Apply
Warning: Recovery target destination is in a sibling branch
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 4569055775) is orphaned on incarnation#=10
MRP0: Detected orphaned datafiles!
Recovery will possibly be retried after flashback...
Errors in file /u01/app/oracle/diag/rdbms/gbkdbdg/gbkdb/trace/gbkdb_pr00_3407.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '/u01/app/oracle/oradata/gbkdbdg/system01.dbf'
Managed Standby Recovery not using Real Time Apply
Recovery Slave PR00 previously exited with exception 19909
RFS[4]: Opened log for thread 1 sequence 25 dbid 845494326 branch 1131820798
Archived Log entry 230 added for thread 1 sequence 25 rlc 1131820798 ID 0x34bcb66a dest 2:
RFS[4]: Selected log 13 for thread 1 sequence 2 dbid 845494326 branch 1131886078
Sun Mar 19 12:51:04 2023
Archived Log entry 231 added for thread 1 sequence 2 ID 0x34bc8f2c dest 1:
Sun Mar 19 12:51:04 2023
Primary database is in MAXIMUM PERFORMANCE mode
RFS[5]: Assigned to RFS process 3425
RFS[5]: Selected log 13 for thread 1 sequence 3 dbid 845494326 branch 1131886078
Sun Mar 19 12:51:19 2023
MRP0: Background Media Recovery process shutdown (gbkdb)
Sun Mar 19 12:51:53 2023
RFS[6]: Assigned to RFS process 3100
RFS[6]: Opened log for thread 1 sequence 27 dbid 845494326 branch 1131820798
Archived Log entry 232 added for thread 1 sequence 27 rlc 1131820798 ID 0x34bcb66a dest 2:
RFS[6]: Opened log for thread 1 sequence 28 dbid 845494326 branch 1131820798
Archived Log entry 233 added for thread 1 sequence 28 rlc 1131820798 ID 0x34bcb66a dest 2:
RFS[6]: Opened log for thread 1 sequence 29 dbid 845494326 branch 1131820798
Archived Log entry 234 added for thread 1 sequence 29 rlc 1131820798 ID 0x34bcb66a dest 2:
RFS[6]: Opened log for thread 1 sequence 30 dbid 845494326 branch 1131820798
Archived Log entry 235 added for thread 1 sequence 30 rlc 1131820798 ID 0x34bcb66a dest 2:
Sun Mar 19 12:51:53 2023
RFS[7]: Assigned to RFS process 3439
RFS[7]: Opened log for thread 1 sequence 26 dbid 845494326 branch 1131820798
RFS[6]: Opened log for thread 1 sequence 31 dbid 845494326 branch 1131820798
Archived Log entry 236 added for thread 1 sequence 26 rlc 1131820798 ID 0x34bcb66a dest 2:
Archived Log entry 237 added for thread 1 sequence 31 rlc 1131820798 ID 0x34bcb66a dest 2:
RFS[6]: Opened log for thread 1 sequence 33 dbid 845494326 branch 1131820798
RFS[7]: Opened log for thread 1 sequence 32 dbid 845494326 branch 1131820798
Archived Log entry 238 added for thread 1 sequence 33 rlc 1131820798 ID 0x34bcb66a dest 2:
Archived Log entry 239 added for thread 1 sequence 32 rlc 1131820798 ID 0x34bcb66a dest 2:
RFS[6]: Opened log for thread 1 sequence 34 dbid 845494326 branch 1131820798
Archived Log entry 240 added for thread 1 sequence 34 rlc 1131820798 ID 0x34bcb66a dest 2:
6、恢复B库
--A库化身情况
RMAN> list incarnation;
using target database control file instead of recovery catalog
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 GBKDB 845494326 PARENT 1 2013-08-24 11:37:30
2 2 GBKDB 845494326 PARENT 925702 2021-12-29 22:29:12
3 3 GBKDB 845494326 ORPHAN 4568599459 2023-02-13 14:53:01
4 4 GBKDB 845494326 PARENT 4568599516 2023-02-13 15:05:13
5 5 GBKDB 845494326 ORPHAN 4568606764 2023-02-13 16:54:05
6 6 GBKDB 845494326 PARENT 4568606764 2023-02-13 16:59:56
7 7 GBKDB 845494326 PARENT 4568627631 2023-02-13 17:03:11
8 8 GBKDB 845494326 PARENT 4568628222 2023-02-13 17:06:18
9 9 GBKDB 845494326 PARENT 4568952372 2023-03-18 12:24:33
10 10 GBKDB 845494326 CURRENT 4569006964 2023-03-18 18:39:58
--B库修改化身为A库的当前化身
RMAN> list incarnation;
using target database control file instead of recovery catalog
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 GBKDB 845494326 PARENT 1 24-AUG-13
2 2 GBKDB 845494326 PARENT 925702 29-DEC-21
3 3 GBKDB 845494326 ORPHAN 4568599459 13-FEB-23
4 4 GBKDB 845494326 PARENT 4568599516 13-FEB-23
5 5 GBKDB 845494326 ORPHAN 4568606764 13-FEB-23
6 6 GBKDB 845494326 PARENT 4568606764 13-FEB-23
7 7 GBKDB 845494326 PARENT 4568627631 13-FEB-23
8 8 GBKDB 845494326 PARENT 4568628222 13-FEB-23
9 9 GBKDB 845494326 PARENT 4568952372 18-MAR-23
10 10 GBKDB 845494326 PARENT 4569006964 18-MAR-23
11 11 GBKDB 845494326 CURRENT 4569051568 19-MAR-23
RMAN> reset database to incarnation 10;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of reset database command on default channel at 03/19/2023 13:10:01
ORA-19910: can not change recovery target incarnation in control file
RMAN> shutdown abort
Oracle instance shut down
RMAN> startup mount
connected to target database (not started)
Oracle instance started
database mounted
Total System Global Area 626327552 bytes
Fixed Size 2255832 bytes
Variable Size 234882088 bytes
Database Buffers 381681664 bytes
Redo Buffers 7507968 bytes
RMAN> reset database to incarnation 10;
database reset to incarnation 10
RMAN> list incarnation;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 GBKDB 845494326 PARENT 1 24-AUG-13
2 2 GBKDB 845494326 PARENT 925702 29-DEC-21
3 3 GBKDB 845494326 ORPHAN 4568599459 13-FEB-23
4 4 GBKDB 845494326 PARENT 4568599516 13-FEB-23
5 5 GBKDB 845494326 ORPHAN 4568606764 13-FEB-23
6 6 GBKDB 845494326 PARENT 4568606764 13-FEB-23
7 7 GBKDB 845494326 PARENT 4568627631 13-FEB-23
8 8 GBKDB 845494326 PARENT 4568628222 13-FEB-23
9 9 GBKDB 845494326 PARENT 4568952372 18-MAR-23
10 10 GBKDB 845494326 CURRENT 4569006964 18-MAR-23
11 11 GBKDB 845494326 ORPHAN 4569051568 19-MAR-23
RMAN>
SQL> alter database recover managed standby database using current logfile disconnect from session;
Database altered.
--B库日志，恢复正常，并且即使C库再次传输日志，也不会影响到B库
Sun Mar 19 13:11:01 2023
alter database recover managed standby database using current logfile disconnect from session
Attempt to start background Managed Standby Recovery process (gbkdb)
Sun Mar 19 13:11:01 2023
MRP0 started with pid=26, OS id=3779
MRP0: Background Managed Standby Recovery process started (gbkdb)
started logmerger process
Sun Mar 19 13:11:06 2023
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/app/oracle/gbkarch/1_35_1131820798.dbf
Media Recovery Waiting for thread 1 sequence 36 (in transit)
Recovery of Online Redo Log: Thread 1 Group 12 Seq 36 Reading mem 0
Mem# 0: /u01/app/oracle/oradata/gbkdbdg/sredo12.log
Completed: alter database recover managed standby database using current logfile disconnect from session
Sun Mar 19 13:12:30 2023
RFS[2]: Selected log 11 for thread 1 sequence 37 dbid 845494326 branch 1131820798
Sun Mar 19 13:12:30 2023
Archived Log entry 242 added for thread 1 sequence 36 ID 0x34bcb66a dest 1:
Sun Mar 19 13:12:30 2023
Media Recovery Waiting for thread 1 sequence 37 (in transit)
Recovery of Online Redo Log: Thread 1 Group 11 Seq 37 Reading mem 0
Mem# 0: /u01/app/oracle/oradata/gbkdbdg/sredo11.log
Sun Mar 19 13:14:22 2023
RFS[3]: Assigned to RFS process 3871
RFS[3]: Opened log for thread 1 sequence 4 dbid 845494326 branch 1131886078
Archived Log entry 243 added for thread 1 sequence 4 rlc 1131886078 ID 0x34bc8f2c dest 2:
Sun Mar 19 13:14:22 2023
RFS[4]: Assigned to RFS process 3873
RFS[4]: Selected log 13 for thread 1 sequence 3 dbid 845494326 branch 1131886078
Sun Mar 19 13:14:22 2023
Archived Log entry 244 added for thread 1 sequence 3 ID 0x34bc8f2c dest 1:
Sun Mar 19 13:14:26 2023
RFS[5]: Assigned to RFS process 3875
RFS[5]: Selected log 13 for thread 1 sequence 5 dbid 845494326 branch 1131886078
Sun Mar 19 13:14:26 2023
Archived Log entry 245 added for thread 1 sequence 5 ID 0x34bc8f2c dest 1:
Sun Mar 19 13:14:26 2023
Primary database is in MAXIMUM PERFORMANCE mode
RFS[6]: Assigned to RFS process 3877
RFS[6]: Selected log 13 for thread 1 sequence 6 dbid 845494326 branch 1131886078
Sun Mar 19 13:14:31 2023
RFS[2]: Selected log 12 for thread 1 sequence 38 dbid 845494326 branch 1131820798
Archived Log entry 246 added for thread 1 sequence 37 ID 0x34bcb66a dest 1:
Sun Mar 19 13:14:31 2023
Media Recovery Waiting for thread 1 sequence 38 (in transit)
Recovery of Online Redo Log: Thread 1 Group 12 Seq 38 Reading mem 0
Mem# 0: /u01/app/oracle/oradata/gbkdbdg/sredo12.log
7、为避免再次出现问题，去掉C库异常配置
SQL> ALTER SYSTEM SET log_archive_dest_2='';
System altered.
SQL> ALTER SYSTEM SET log_archive_dest_3='';
System altered.
8、闪回恢复C库为级联DG备库
--C库执行
SQL> SELECT CURRENT_SCN,STANDBY_BECAME_PRIMARY_SCN FROM V$DATABASE;
CURRENT_SCN STANDBY_BECAME_PRIMARY_SCN
----------- --------------------------
4569055074 4569051565
SQL> select oldest_flashback_scn, to_char(oldest_flashback_time,'yyyy-mm-dd HH24:mi:ss') oldest_flashback_time from v$flashback_database_log;
OLDEST_FLASHBACK_SCN OLDEST_FLASHBACK_TIME
-------------------- ---------------------------------------------------------
4568952074 2023-03-18 12:18:42
SQL> startup mount force
ORACLE instance started.
Total System Global Area 626327552 bytes
Fixed Size 2255832 bytes
Variable Size 234882088 bytes
Database Buffers 381681664 bytes
Redo Buffers 7507968 bytes
Database mounted.
SQL> flashback database to scn 4569051565;
Flashback complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
SQL> alter database convert to physical standby;
Database altered.
SQL> shutdown abort
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 626327552 bytes
Fixed Size 2255832 bytes
Variable Size 234882088 bytes
Database Buffers 381681664 bytes
Redo Buffers 7507968 bytes
Database mounted.
Database opened.
SQL>
SQL> alter database recover managed standby database using current logfile disconnect from session;
Database altered.
--B库enable归档传输到C库
SQL> alter system set log_archive_dest_state_3=enable;
System altered.
--C库应用日志报错
Sun Mar 19 13:33:22 2023
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/app/oracle/gbkdrarch/1_24_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_25_1131820798.dbf
Identified End-Of-Redo (failover) for thread 1 sequence 25 at SCN 0x1.105631af
Resetting standby activation ID 884782698 (0x34bcb66a)
Incomplete Recovery applied until change 4569051567 time 03/19/2023 12:47:38
MRP0: Background Media Recovery applied all available redo. Recovery will be restarted once new redo branch is registered
Errors in file /u01/app/oracle/diag/rdbms/gbkdbdr/gbkdbdr/trace/gbkdbdr_pr00_4507.trc:
ORA-19906: recovery target incarnation changed during recovery
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
RMAN> list incarnation ;
using target database control file instead of recovery catalog
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 GBKDB 845494326 PARENT 1 24-AUG-13
2 2 GBKDB 845494326 PARENT 925702 29-DEC-21
3 3 GBKDB 845494326 ORPHAN 4568599459 13-FEB-23
4 4 GBKDB 845494326 PARENT 4568599516 13-FEB-23
5 5 GBKDB 845494326 ORPHAN 4568606764 13-FEB-23
6 6 GBKDB 845494326 PARENT 4568606764 13-FEB-23
7 7 GBKDB 845494326 PARENT 4568627631 13-FEB-23
8 8 GBKDB 845494326 PARENT 4568628222 13-FEB-23
9 9 GBKDB 845494326 PARENT 4568952372 18-MAR-23
10 10 GBKDB 845494326 CURRENT 4569006964 18-MAR-23
11 11 GBKDB 845494326 ORPHAN 4569051568 19-MAR-23
RMAN>
--此时的问题在于25号归档被C库重新构建后传输到了B库，并非A库的25号归档。尝试从A库拷贝25号归档到C库注册后，C库MRP应用日志正常。
25号归档报错选择了重新搭建，下面测试为卡住66号归档后恢复的日志显示：
--B库，failover之前的66号归档
[oracle@linux8 gbkarch]$ ls -l 1_66_1131820798.dbf
-rw-r----- 1 oracle oinstall 2077184 Mar 19 16:11 1_66_1131820798.dbf
[oracle@linux8 gbkarch]$ ls -l 1_67_1131820798.dbf
-rw-r----- 1 oracle oinstall 1536 Mar 19 16:11 1_67_1131820798.dbf
[oracle@linux8 gbkarch]$ md5sum 1_66_1131820798.dbf
6b3dd2a27e7f227a1626c4758a35d1b7 1_66_1131820798.dbf
[oracle@linux8 gbkarch]$ md5sum 1_67_1131820798.dbf
6a167be35ce82f18e3784d302d9d0174 1_67_1131820798.dbf
--C库failover后，B库的66号归档是C库构建出来的Identified End-Of-Redo (failover) 归档
--C库传输自己构建的66号归档给B库
Sun Mar 19 16:15:14 2023
CJQ0 started with pid=29, OS id=11136
PING[ARC2]: Heartbeat failed to connect to standby 'gbkdb'. Error is 16009.
ARC1: Error 16009 Creating archive log file to 'gbkdb'
ARC1: Standby redo logfile selected for thread 1 sequence 1 for destination LOG_ARCHIVE_DEST_2
Archived Log entry 42 added for thread 1 sequence 1 ID 0x34bced0d dest 1:
Sun Mar 19 16:15:16 2023
NSA3 started with pid=27, OS id=11148
Setting Resource Manager plan SCHEDULER[0x32DF]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Starting background process VKRM
Sun Mar 19 16:15:17 2023
VKRM started with pid=28, OS id=11150
Sun Mar 19 16:15:21 2023
Thread 1 advanced to log sequence 3 (LGWR switch)
Current log# 3 seq# 3 mem# 0: /u01/app/oracle/oradata/gbkdbdr/redo03.log
ARC0: STARTING ARCH PROCESSES
Sun Mar 19 16:15:21 2023
ARC4 started with pid=35, OS id=11162
ARC4: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
krsk_srl_archive_int: Enabling archival of deferred physical standby SRLs
Archived Log entry 44 added for thread 1 sequence 2 ID 0x34bced0d dest 1:
Archived Log entry 45 added for thread 1 sequence 66 ID 0x34bcb66a dest 1:
--66号归档已经发生变化
[oracle@linux8 gbkarch]$ ls -l 1_66_1131820798.dbf
-rw-r----- 1 oracle oinstall 1024 Mar 19 16:15 1_66_1131820798.dbf
[oracle@linux8 gbkarch]$ ls -l 1_67_1131820798.dbf
-rw-r----- 1 oracle oinstall 1536 Mar 19 16:15 1_67_1131820798.dbf
[oracle@linux8 gbkarch]$
--C库日志应用报错
Sun Mar 19 16:25:55 2023
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/app/oracle/gbkdrarch/1_66_1131820798.dbf
Identified End-Of-Redo (failover) for thread 1 sequence 66 at SCN 0x1.10567df0
Resetting standby activation ID 0 (0x0)
Incomplete Recovery applied until change 4569071088 time 03/19/2023 16:12:28
MRP0: Background Media Recovery applied all available redo. Recovery will be restarted once new redo branch is registered
Errors in file /u01/app/oracle/diag/rdbms/gbkdbdr/gbkdbdr/trace/gbkdbdr_pr00_11620.trc:
ORA-19906: recovery target incarnation changed during recovery
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
--C库注册新拷贝的A库的归档
RMAN> catalog start with '/u01/app/oracle/gbkdrarch/1_66_1131820798.dbf';
--拷贝注册后，C库应用日志正常
Sun Mar 19 16:30:45 2023
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/app/oracle/gbkdrarch/1_66_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_67_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_68_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_69_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_70_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_71_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_72_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_73_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_74_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_75_1131820798.dbf
Media Recovery Log /u01/app/oracle/gbkdrarch/1_76_1131820798.dbf
Media Recovery Waiting for thread 1 sequence 77
最后修改时间：2023-03-23 10:26:51
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文章的来源（墨天轮），文章链接，文章作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
