OGG源端进程Time since Chkpt状态Unknow(OGG-01438)故障处理

故障现象：
1.源端抽取、投递进程Time Since Chkpt状态Unknow，数据不同步，进程状态running（EXX抽取、DXX投递、RXX复制）
GGSCI (test1db01) 1> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING DXX 00:00:00 unknown
EXTRACT RUNNING EXX 00:00:00 unknown

GGSCI (test1db01) 11> info EXX
EXTRACT EXX Last Started 2021-12-06 09:54 Status RUNNING
Checkpoint Lag 00:00:00 (updated unknown ago)
Process ID 35266
Log Read Checkpoint Oracle Redo Logs
2021-12-07 08:22:57 Seqno 41205, RBA 207084144
SCN 0.2400510283 (2400510283)

GGSCI (test1db01) 12> info DXX
EXTRACT DXX Last Started 2021-12-06 09:56 Status RUNNING
Checkpoint Lag 00:00:00 (updated unknown ago)
Process ID 35314
Log Read Checkpoint File /home/db/oracle/ogg/dirdat/wy000000004
2021-12-06 20:46:56.000000 RBA 9956

2.report无明显ERROR，有以下Warning提示
2021-12-06 09:54:54 WARNING OGG-01438 Checkpoint marked as from graceful shutdown, but records found after checkpoint in trail /home/db/oracle/ogg/dirdat/xx. Expected EOF Seqno 0, RBA 0. Found Seqno 3, RBA 14613.

3.抽取、投递进程Hang死，无法stop（可kill）
GGSCI (test1db01) 7> stats EXX
Sending STATS request to EXTRACT EXX ...
2021-12-06 09:12:44 ERROR OGG-15163 There was a problem sending a message to EXTRACT EXX (Timeout waiting for message).

GGSCI (test1db01) 6> stop DXX
Sending STOP request to EXTRACT DXX ...
2021-12-06 09:50:37 ERROR OGG-15163 There was a problem sending a message to EXTRACT DXX (Timeout waiting for message).

故障原因：
参考 EXTRACT Pump Is Hung State(Doc ID 1638187.1)

解决方法：
1.查看seqno
GGSCI (test1db01) 5> info DXX detail
...
Trail Name Seqno RBA Max MB Trail Type
/home/db/oracle/ogg/dirdat/xx 4 10233 500 RMTTRAIL
...

GGSCI (test1db01) 12> info rmttrail *
...
Seqno: 4
RBA: 10233
File Size: 500M
...

2.kill抽取、投递进程
GGSCI (test1db01) 7> kill EXX
Sending KILL request to MANAGER ...
Killed process (35266) for EXTRACT EXX

GGSCI (test1db01) 9> kill DXX
Sending KILL request to MANAGER ...
Killed process (35314) for EXTRACT DXX

GGSCI (test1db01) 10> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT ABENDED DXX 00:00:00 unknown
EXTRACT ABENDED EXX 00:00:00 unknown

3.delete rmttrail /home/db/oracle/ogg/dirdat/xx
GGSCI (test1db01) 46> delete rmttrail /home/db/oracle/ogg/dirdat/xx
Deleting extract trail /home/db/oracle/ogg/dirdat/xx for extract EXX

4.add rmttrail /home/db/oracle/ogg/dirdat/xx, seqno 4,rba 0,extract DXX
GGSCI (test1db01) 49> add rmttrail /home/db/oracle/ogg/dirdat/xx, seqno 4,rba 0,extract DXX
RMTTRAIL added.

5.alter extract DXX, ETROLLOVER
GGSCI (test1db01) 50> alter extract DXX, ETROLLOVER
2021-12-06 11:53:42 INFO OGG-01520 Rollover performed. For each affected output trail of Version 10 or higher format, after starting the source extract, issue ALTER EXTSEQNO for that trail's reader (either pump EXTRACT or REPLICAT) to move the reader's scan to the new trail file; it will not happen automatically.
EXTRACT altered.

6.start抽取投递进程
GGSCI> start EXX
报错：
2021-12-06 11:54:58 ERROR OGG-01044 The trail '/home/db/oracle/ogg/dirdat/xx' is not assigned to extract 'EXX'. Assign the trail to the extract with the command "ADD EXTTRAIL/RMTTRAIL /home/db/oracle/ogg/dirdat/wy, EXTRACT EXX".
2021-12-06 11:54:58 ERROR OGG-01668 PROCESS ABENDING.
执行：
GGSCI (test1db01) 57>ADD EXTTRAIL /home/db/oracle/ogg/dirdat/xx, EXTRACT EXX
EXTTRAIL added.
重启：
GGSCI (test1db01) 59> start EXX
GGSCI (test1db01) 65> start DXX
源端恢复正常：
MANAGER RUNNING
EXTRACT RUNNING DXX 00:00:00 00:00:09
EXTRACT RUNNING EXX 00:00:00 00:00:09

7.目标端复制进程：
停复制进程：
GGSCI (bdpuat1db01) 3> stop RXX
Sending STOP request to REPLICAT RXX ...
Request processed.
alter replicat <replicat_name>, extseqno <new rmttrail written by the pump extract>,extrba 0：
GGSCI (bdpuat1db01) 5> alter replicat RXX, extseqno 5 , extrba 0
2021-XX-XX 12:05:20 INFO OGG-06594 Replicat RXX has been altered. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start RXX with NOFILTERDUPTRANSACTIONS option.
REPLICAT altered.
启复制进程：
GGSCI (bdpuat1db01) 6> start RXX
Sending START request to MANAGER ...
REPLICAT RXX starting
目标端恢复正常。