Oracle 19C 使用impdp+network_link迁移带有分区表数据的BUG 30321076

问题现象

使用impdp + network_link参数迁移数据到PDB

$ impdp system/Oracle_321@testpdb network_link=to_test directory=PUMP_DIR schemas='USER1','USER2' parallel=4 logfile=impdp_test_20200427.log EXCLUDE=STATISTICS cluster=N

Import: Release 19.0.0.0.0 - Production on Mon Apr 27 14:39:59 2020
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Starting "SYSTEM"."SYS_IMPORT_SCHEMA_01":  system/********@testpdb network_link=to_test directory=PUMP_DIR schemas=USER1,USER2 parallel=4 logfile=impdp_test_20200427.log EXCLUDE=STATISTICS cluster=N 
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 355.5 GB
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TABLE/TABLE
ORA-39014: One or more workers have prematurely exited.
ORA-39029: worker 4 with process name "DW03" prematurely terminated
ORA-31671: Worker process DW03 had an unhandled exception.

ORA-39029: worker 2 with process name "DW01" prematurely terminated
ORA-31671: Worker process DW01 had an unhandled exception.

ORA-39014: One or more workers have prematurely exited.
ORA-39029: worker 5 with process name "DW03" prematurely terminated
ORA-31671: Worker process DW03 had an unhandled exception.

ORA-39029: worker 6 with process name "DW01" prematurely terminated
ORA-31671: Worker process DW01 had an unhandled exception.

Job "SYSTEM"."SYS_IMPORT_SCHEMA_01" stopped due to fatal error at Mon Apr 27 14:41:33 2020 elapsed 0 00:01:30

此时出现ORA-39029和ORA-31671错误信息，查看DW03进程的trace日志文件

[oracle@test:/u02/app/oracle/diag/rdbms/db4/db41/trace]$ cat db41_dw03_137773.trc
Trace file /u02/app/oracle/diag/rdbms/db4/db41/trace/db41_dw03_137773.trc
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.6.0.0.0
Build label:    RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417
ORACLE_HOME:    /u02/app/oracle/product/19.0.0/db_1
System name:    Linux
Node name:      test
Release:        3.10.0-862.el7.x86_64
Version:        #1 SMP Wed Mar 21 18:14:51 EDT 2018
Machine:        x86_64
Instance name: db41
Redo thread mounted by this instance: 1
Oracle process number: 559
Unix process pid: 137773, image: oracle@test (DW03)


*** 2020-04-27T14:41:06.258595+08:00
*** SESSION ID:(3797.47194) 2020-04-27T14:41:06.258612+08:00
*** CLIENT ID:() 2020-04-27T14:41:06.258617+08:00
*** SERVICE NAME:(test) 2020-04-27T14:41:06.258621+08:00
*** MODULE NAME:(Data Pump Worker) 2020-04-27T14:41:06.258626+08:00
*** ACTION NAME:(SYS_IMPORT_SCHEMA_01) 2020-04-27T14:41:06.258630+08:00
*** CLIENT DRIVER:() 2020-04-27T14:41:06.258634+08:00
*** CONTAINER ID:(5) 2020-04-27T14:41:06.258639+08:00
 
2020-04-27T14:41:06.258398+08:00
Incident 404475 created, dump file: /u02/app/oracle/diag/rdbms/db4/db41/incident/incdir_404475/db41_dw03_137773_i404475.trc
ORA-00600: internal error code, arguments: [qesmaGetPamR-NullCtx], [], [], [], [], [], [], [], [], [], [], []


*** 2020-04-27T14:41:10.190015+08:00
2020-04-27T14:41:10.190003+08:00
Incident 404476 created, dump file: /u02/app/oracle/diag/rdbms/db4/db41/incident/incdir_404476/db41_dw03_137773_i404476.trc
ORA-00600: internal error code, arguments: [qesmaGetPamR-NullCtx], [], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.DBMS_SQL", line 1721
ORA-06512: at "SYS.KUPD$DATA", line 1308
ORA-06512: at "SYS.KUPD$DATA", line 1400
ORA-06512: at "SYS.KUPD$DATA", line 3815
ORA-06512: at "SYS.KUPD$DATA", line 4149
ORA-06512: at "SYS.KUPD$DATA", line 6024
ORA-06512: at "SYS.KUPD$DATA", line 7193
ORA-06512: at "SYS.KUPW$WORKER", line 23049
ORA-06512: at "SYS.KUPW$WORKER", line 4427
ORA-06512: at "SYS.KUPW$WORKER", line 13767
ORA-06512: at "SYS.KUPW$WORKER", line 2429
ORA-06512: at line 2


*** 2020-04-27T14:41:33.289161+08:00 (test(5))
KUPP: Error 600 detected in worker process DW03, worker id=5
[oracle@test:/u02/app/oracle/diag/rdbms/db4/db41/trace]$ 

DW03进程的trace日志文件报出了错误信息ORA-00600: internal error code, arguments: [qesmaGetPamR-NullCtx]
参考MOS文档
Bug 30321076 - ORA-00600 [qesmaGetPamR-NullCtx] when importing a partitioned table over network link (Doc ID 30321076.8)

If datapump import over a network link, where the target version is 12.2 or
greater and the source version is less than 12.2, results in the internal
error ora-600 [qesmaGetPamR-NullCtx] when importing a partitioned table, then
you may have encountered this bug.

修复问题

下载相应补丁并安装

[oracle@test:/home/oracle]$ unzip /opt/p30321076_196000DBRU_Linux-x86-64.zip -d /orapath/
Archive:  /opt/p30321076_196000DBRU_Linux-x86-64.zip
   creating: /orapath/30321076/
   creating: /orapath/30321076/files/
   creating: /orapath/30321076/files/lib/
   creating: /orapath/30321076/files/lib/libgeneric19.a/
  inflating: /orapath/30321076/files/lib/libgeneric19.a/qct.o  
   creating: /orapath/30321076/etc/
   creating: /orapath/30321076/etc/config/
  inflating: /orapath/30321076/etc/config/inventory.xml  
  inflating: /orapath/30321076/etc/config/actions.xml  
  inflating: /orapath/30321076/README.txt  
  inflating: /orapath/PatchSearch.xml  

[oracle@test:/home/oracle]$ sqlplus / as sysdba
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            MOUNTED
         5 test                           READ WRITE NO

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

[oracle@test:/orapath/30321076]$ /u02/app/oracle/product/19.0.0/db_1/OPatch/opatch apply
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u02/app/oracle/product/19.0.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u02/app/oracle/product/19.0.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.7.0
Log file location : /u02/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-04-27_15-49-46PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30321076  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u02/app/oracle/product/19.0.0/db_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30321076' to OH '/u02/app/oracle/product/19.0.0/db_1'

Patching component oracle.rdbms.rsf, 19.0.0.0.0...

Patching component oracle.rdbms, 19.0.0.0.0...
Patch 30321076 successfully applied.
Log file location: /u02/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-04-27_15-49-46PM_1.log

OPatch succeeded.

[oracle@test:/orapath/30321076]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Apr 27 15:51:03 2020
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 4.2950E+11 bytes
Fixed Size                 30412272 bytes
Variable Size            6.7646E+10 bytes
Database Buffers         3.6131E+11 bytes
Redo Buffers              506458112 bytes
Database mounted.
Database opened.
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            MOUNTED
         5 test                           MOUNTED

SQL> alter pluggable database test open instances=all;

Pluggable database altered.

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            MOUNTED
         5 test                           READ WRITE NO
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.6.0.0.0
[oracle@test:/orapath/30321076]$ 

[oracle@test:/home/oracle]$ $ORACLE_HOME/OPatch/opatch lspatches  
30321076;RTI 22224358   ORA-600 [QESMAGETPAMR-NULLCTX],                            <<<<<<<<<<<<<<<<<<<<<<<<<
29901419;INCREMENTAL STATS DEGRADE THE PERFORMANCE FOR ADD COLUMN DDL
29041775;ORA-41401 IN ALERT.LOG EXACM
30484981;OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981)
30557433;Database Release Update : 19.6.0.0.200114 (30557433)
30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)

OPatch succeeded.

修复后测试

$ impdp system/Oracle_321@testpdb network_link=to_test directory=PUMP_DIR schemas='USER1','USER2' parallel=4 logfile=impdp_test_20200427.log EXCLUDE=STATISTICS cluster=N

Import: Release 19.0.0.0.0 - Production on Tue Apr 28 08:40:03 2020
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Starting "SYSTEM"."SYS_IMPORT_SCHEMA_02":  system/********@testpdb network_link=to_test directory=PUMP_DIR schemas='USER1','USER2' parallel=4 logfile=impdp_test_20200427.log EXCLUDE=STATISTICS cluster=N
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 347.9 GB
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TABLE/TABLE
. . imported "USER1"."USER1":"USER1_201905" 1871329 rows
. . imported "USER1"."USER1":"USER1_201907" 1783783 rows