ASM归档无法删除分析[存档故障]

本套环境的Oracle archivelog，定时每天crontab任务删除7天之前的归档，并部署有监控系统获取ASM归档磁盘组的使用率。某天归档脚本并未正常完成，磁盘组使用率触发告警。

1.环境介绍

操作系统:Centos 6.9
数据库版本: Oracle RAC 11.2.0.4

2.无法删除归档日志

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of delete command on ORA_DISK_1 channel at 10/18/2017 02:01:18
15028: ASM file '+DATADG1/db/archivelog/2017_09_04/thread_1_seq_73017.554.953855601' not dropped; currently being accessed

提示某些进程正在访问持有相关文件资源，正常来说部署了OGG或者ADG，因为某些原因造成的延迟，这些归档是无法正常删除的。

3.通过ASM lsof查看是否打开了ASM磁盘组的归档文件

Purpose

Lists the open files of the local clients.

Syntax and Description

lsof [--suppressheader] {-G diskgroup|--dbname db| -C instance}

$asmcmd lsof -G datadg1
DB_Name  Instance_Name  Path                                                                   
+ASM  +ASM2          +datadg1/db/archivelog/2017_09_04/thread_1_seq_73017.554.953855601    <== 提示有进程持有在这部分文件句柄
+ASM  +ASM2          +datadg1/db/archivelog/2017_09_04/thread_1_seq_73017.554.953855601 

4.我们通过asm实例的dump直接搜索占有文件资源的进程信息

SO: 0x8b743a68, type: 132, owner: 0x93bf7e50, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
             proc=0xa11d4000, name=ASM file, file=kff2.h LINE:898, pg=0
            (kffil) netnm: +ASM2:+ASM, mapid: 1070986
                    gnum: 1.563140178, fnum: 554.953855601, flgs: 0x182
                    family: 0.0, parent: 0.0
                    opennm: +DATADG1/db/archivelog/2017_09_04/thread_1_seq_73017.554.953855601
                    openflags: 0x82100000
                    fullnm: +datadg1/db/archivelog/2017_09_04/thread_1_seq_73017.554.953855601
                    redun: 0x11, fdflg: 0x0, blksiz: 512, fsiz: 1236992
                    ftype: 4, extsz: [4294967295,0,0]
                    extents: 604, start: 60, count: 544, xp: 0x0
                    base: 0, lxcnt: 0
                    enq: 0x9d82e398, cod: 0x0
                    unlock[0].au: 0, unlock[0].disk: 0, unlock[0].flags 0x0
                    unlock[1].au: 0, unlock[1].disk: 0, unlock[1].flags 0x0
                    unlock[2].au: 0, unlock[2].disk: 0, unlock[2].flags 0x0
                    unlock[3].au: 0, unlock[3].disk: 0, unlock[3].flags 0x0
                    unlock[4].au: 0, unlock[4].disk: 0, unlock[4].flags 0x0
                    unlock[5].au: 0, unlock[5].disk: 0, unlock[5].flags 0x0
                    strpsz: 1048576, strpwdth: 1
                    lnk: 0x0, 0x0
                    aba: 0, 0
                    client pid: 51 osid: 5742
              ----------------------------------------
              SO: 0x9d82e398, type: 8, owner: 0x8b743a68, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
               proc=0xa11d4000, name=enqueue, file=ksq1.h LINE:380, pg=0
              (enqueue) FA-00000001-0000022A    DID: 0000-001A-0010578F
              lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x3
              mode: SS, lock_flag: 0x10, lock: 0x9d82e3f0, res: 0x9d4850f0
              own: 0x9f73be60, sess: 0x9f73be60, proc: 0xa11d4000, prv: 0x9d828db0
              slk: 0x98838060
            ----------------------------------------
            SO: 0x8b69c680, type: 132, owner: 0x93bf7e50, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
             proc=0xa11d4000, name=ASM file, file=kff2.h LINE:898, pg=0
            (kffil) netnm: +ASM2:+ASM, mapid: 1070985
                    gnum: 1.563140178, fnum: 554.953855601, flgs: 0x182
                    family: 0.0, parent: 0.0
                    opennm: +DATADG1/db/archivelog/2017_09_04/thread_1_seq_73017.554.953855601
                    openflags: 0x82100000
                    fullnm: +datadg1/db/archivelog/2017_09_04/thread_1_seq_73017.554.953855601
                    redun: 0x11, fdflg: 0x0, blksiz: 512, fsiz: 1236992
                    ftype: 4, extsz: [4294967295,0,0]
                    extents: 604, start: 60, count: 544, xp: 0x0
                    base: 0, lxcnt: 0
                    enq: 0x9d828d48, cod: 0x0
                    unlock[0].au: 0, unlock[0].disk: 0, unlock[0].flags 0x0
                    unlock[1].au: 0, unlock[1].disk: 0, unlock[1].flags 0x0
                    unlock[2].au: 0, unlock[2].disk: 0, unlock[2].flags 0x0
                    unlock[3].au: 0, unlock[3].disk: 0, unlock[3].flags 0x0
                    unlock[4].au: 0, unlock[4].disk: 0, unlock[4].flags 0x0
                    unlock[5].au: 0, unlock[5].disk: 0, unlock[5].flags 0x0
                    strpsz: 1048576, strpwdth: 1
                    lnk: 0x0, 0x0
                    aba: 0, 0
                    client pid: 39 osid: 5741

可以看到两个进程client pid: 51 osid: 5742，client pid: 39 osid: 5741

ps -ef|grep 5741
dsg       5741 13189  0 Sep05 ?        01:43:54 /dsg/realsync/bin/oxad -startup -n 127.0.0.1:33505 -blen 30M
grid     35361 11105  0 14:47 pts/3    00:00:00 grep 5741
ps -ef|grep 5742
dsg       5742  7326  0 Sep05 ?        01:44:37 /dsg/realsync1/bin/oxad -startup -n 127.0.0.1:35105 -blen 30M
grid     38779 11105  0 14:47 pts/3    00:00:00 grep 574

两个进程为DSG抽取进程，因为文件是7天之前的归档，这里dsg软件占用非正常情况，跟他们技术人员沟通，通过重启了两个抽取进程，文件得以正常删除。