Troubleshooting ORA-600 [H_MK_WRITING_CR_BG] 11.2.0.3 standby instance crash

env: 11.2.0.3 RAC Dataguard ON Exadata x2, a standby instance crash, alert log show ora-600 internal error.

# db alert log
2017-06-12 08:39:10.148000 +08:00
Archived Log entry 70956 added for thread 1 sequence 247664 ID 0x94a6c6e9 dest 1:
2017-06-12 10:04:32.837000 +08:00
Media Recovery Waiting for thread 2 sequence 236820 (in transit)
Recovery of Online Redo Log: Thread 2 Group 32 Seq 236820 Reading mem 0
  Mem# 0: +DATA/rptstby/onlinelog/group_32.556.906801599
2017-06-12 10:11:11.510000 +08:00
Errors in file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc  (incident=224233):
ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2017-06-12 10:11:12.599000 +08:00
Dumping diagnostic data in directory=[cdmp_20170612101112], requested by (instance=1, osid=14567 (DBWA)), summary=[incident=224233].
Errors in file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc:
ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
DBWa (ospid: 14567): terminating the instance due to error 471
System state dump requested by (instance=1, osid=14567 (DBWA)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_diag_14516.trc
2017-06-12 10:11:13.759000 +08:00
ORA-1092 : opitsk aborting process
License high water mark = 66
2017-06-12 10:11:17.917000 +08:00
Instance terminated by DBWa, pid = 14567
USER (ospid: 10154): terminating the instance
Instance terminated by USER, pid = 10154
2017-06-12 10:11:19.223000 +08:00
Starting ORACLE instance (normal)

adrci> show trace /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
Output the results to file: /tmp/utsout_22055_13995_5.ado
/oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
----------------------------------------------------------
LEVEL PAYLOAD
----- ------------------------------------------------------------------------------------------------------------------------------------------------
      Trace file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
      Data Mining and Real Application Testing options
      ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
      System name:      Linux
      Node name:        qdexa1db01.anbob.com
      Release:  2.6.32-400.35.2.el5uek
      Version:  #1 SMP Thu Feb 13 15:00:30 PST 2014
      Machine:  x86_64
      Instance name: oradb1
      Redo thread mounted by this instance: 1
      Oracle process number: 29
      Unix process pid: 14567, image: oracle@qdexa1db01.anbob.com (DBWA)

      *** 2017-06-12 10:11:11.522
      *** SESSION ID:(632.1) 2017-06-12 10:11:11.522
      *** CLIENT ID:() 2017-06-12 10:11:11.522
      *** SERVICE NAME:(SYS$BACKGROUND) 2017-06-12 10:11:11.522
      *** MODULE NAME:() 2017-06-12 10:11:11.522
      *** ACTION NAME:() 2017-06-12 10:11:11.522
1>     ***** Incident 224233 created, dump file:  *****
       /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
1<     ***** incident_file *****
1>     ***** Error Stack *****
       ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
1<     ***** Error Stack *****
      error 471 detected in background process
      ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
      kjzduptcctx: Notifying DIAG for crash event
      ----- Abridged Call Stack Trace -----
      ksedsts()+461<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+53<-ksuitm()+1332<-ksbrdp()+3344<-opirip()+623<-opidrv()+603
      <-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36
      ----- End of Abridged Call Stack Trace -----
      *** 2017-06-12 10:11:12.765
      DBWa (ospid: 14567): terminating the instance due to error 471
      ksuitm: waiting up to [5] seconds before killing DIAG(14516)
      
adrci> show trace  /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
Output the results to file: /tmp/utsout_22055_13995_7.ado
/oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
----------------------------------------------------------
LEVEL PAYLOAD
----- ------------------------------------------------------------------------------------------------------------------------------------------------
      Dump file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
      Data Mining and Real Application Testing options
      ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
      System name:      Linux
      Node name:        qdexa1db01.anbob.com
      Release:  2.6.32-400.35.2.el5uek
      Version:  #1 SMP Thu Feb 13 15:00:30 PST 2014
      Machine:  x86_64
      Instance name: oradb1
      Redo thread mounted by this instance: 1
      Oracle process number: 29
      Unix process pid: 14567, image: oracle@qdexa1db01.anbob.com (DBWA)

      *** 2017-06-12 10:11:11.522
      *** SESSION ID:(632.1) 2017-06-12 10:11:11.522
      *** CLIENT ID:() 2017-06-12 10:11:11.522
      *** SERVICE NAME:(SYS$BACKGROUND) 2017-06-12 10:11:11.522
      *** MODULE NAME:() 2017-06-12 10:11:11.522
      *** ACTION NAME:() 2017-06-12 10:11:11.522
      Dump continued from file: /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
1>     ***** Error Stack *****
       ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
1<     ***** Error Stack *****
1>     ***** Dump for incident 224233 (ORA 600 [H_MK_WRITING_CR_BG]) *****
2>      ***** Beginning of Customized Incident Dump(s) *****
        BH (0x867cbf798) file#: 1093 rdba: 0x130caad3 (76/830163) class: 1 ba: 0x865298000
          set: 59 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0
          dbwrid: 10 obj: 2541940 objn: -1 tsn: 11 afn: 1093 hint: f
          hash: [0x227d2d950,0x707b5dcd0] lru: [0x60fcf9188,0x507bd2408]
          lru-flags: hot_buffer
          ckptq: [NULL] fileq: [NULL] objq: [0x2dfb2e1b0,0x9e07cfa20] objaq: [0x60fcf91c0,0x507bd2440]
          use: [0xa45bfc838,0xa45bfc838] wait: [NULL]
          st: MEDIA_WRITING md: SHR rsop: 0x9fd89d2e0 tch: 77 atm: 3732566919,3577170450 rlscn: 0x0dec.039476d9
          flags: only_sequential_access block_written_once affinity_lock
        Dump of buffer cache at level 10 for tsn=11 rdba=319597267
        BH (0x707b5dc18) file#: 1093 rdba: 0x130caad3 (76/830163) class: 1 ba: 0x7017a4000
          set: 59 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0
          dbwrid: 10 obj: 2541940 objn: -1 tsn: 11 afn: 1093 hint: f
          hash: [0x867cbf850,0xa59dea248] lru: [0x13fbfad88,0x567c1d288]
          obj-flags: object_ckpt_list
          ckptq: [0x13fbfac98,0x567c1d198] fileq: [0x567c1d1a8,0x13fbfaca8] objq: [0x13fbfadb0,0x567c1d2b0] objaq: [0x13fbfadc0,0x567c1d2c0]
          st: MEDIA_RCV md: NULL rsop: 0x9fd89d2e0 tch: 1 atm: 3300241033,3577178658 rlscn: 0x0dec.039477d6
          flags: buffer_dirty only_sequential_access block_written_once affinity_lock
          buffer tsn: 11 rdba: 0x130caad3 (76/830163)
          scn: 0x0dec.039477d7 seq: 0x01 flg: 0x04 tail: 0x77d70601
          frmt: 0x02 chkval: 0xa4a8 type: 0x06=trans data
        Hex dump of block: st=0, typ_found=1
      
       ----- Abridged Call Stack Trace -----
       ksedsts()+461<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdpcrshnfy()+43<-kstdmp()+282<-dbkedDefDump()+10477<-ksedmp()+41<-ksfdmp()+69<-dbgexPhaseI
       I()+1764<-dbgexExplicitEndInc()+755<-dbgeEndDDEInvocationImpl()+772<-dbgeEndDDEInvocation()+52<-kcbbic2()+9165<-kcbbic1()+1216
       <-kcbbiop()+1042<-kcbbdrv()+733<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603
       ----- End of Abridged Call Stack Trace -----
       

Search in MOS, that's easy to found a node hit a bug.
Cause
This issue was further investigated and fixed in unpublished BUG 16915139.

Solution
Apply Patch 16915139 if it is available for your platform and version.The final fix is included in RDBMS Versions: 11.2.0.4, 12.1.0.2, 12.2.