how to recreate acfs filesystem ?

今天西北某电信客户的一个系统有点小问题，无法切换归档日志。通过临时将归档路径都放到本地后，切换ok。检查发现之前的arch目录有点问题，如下：

[root@dtdb1 arch]# ls -ltr
ls: cannot access 1_3883_855680193.dbf: Invalid argument
ls: cannot access 2_2408_855680193.dbf: No such file or directory
ls: cannot access 2_2408_855680193.dbf: No such file or directory
ls: cannot access 1_4776_855680193.dbf: Invalid argument
total 7521476
-????????? ? ?      ?                ?            ? 2_2408_855680193.dbf
-????????? ? ?      ?                ?            ? 2_2408_855680193.dbf
-????????? ? ?      ?                ?            ? 1_4776_855680193.dbf
-????????? ? ?      ?                ?            ? 1_3883_855680193.dbf
drwx------ 2 oracle asmadmin     65536 Nov  5  2014 lost+found
drwxr-xr-x 4 oracle asmadmin      4096 Nov  5  2014 backup1
-rw-r----- 1 oracle asmadmin  23993856 Jan  1 10:06 2_2461_855680193.dbf
-rw-r----- 1 oracle asmadmin 460232704 Jan  1 10:06 1_4858_855680193.dbf
-rw-r----- 1 oracle asmadmin 460146176 Jan  1 12:26 1_4859_855680193.dbf
-rw-r----- 1 oracle asmadmin 460609536 Jan  1 22:05 1_4860_855680193.dbf
......

我们可以看到有几个归档文件的属主、权限等信息都是问号，而且都无法删除。通过VNC登录查看该目录下的这几个文件都不存在，这确实感觉很怪异。通过rm -rf 删除报错，如下所示：

[root@dtdb1 arch]# rm -rf 1_4776_*
rm: cannot remove `1_4776_855680193.dbf': Invalid argument

经过查询发生该arch本质上是acfs集群文件系统，如下：

[grid@dtdb1 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    307196      920                0             920              0             N  ARCHVG/
MOUNTED  NORMAL  N         512   4096  1048576      3105     2179             1035             572              0             Y  CRS/
MOUNTED  EXTERN  N         512   4096  4194304    375752     4572                0            4572              0             N  DATA01/
MOUNTED  EXTERN  N         512   4096  1048576    204797     1902                0            1902              0             N  DATA02/
MOUNTED  EXTERN  N         512   4096  4194304    389104   158476                0          158476              0             N  DATA03/
MOUNTED  EXTERN  N         512   4096  4194304     30724    27380                0           27380              0             N  FRA01/
ASMCMD> cd archvg
ASMCMD> ls
ASMCMD> volinfo -a 
Diskgroup Name: ARCHVG

         Volume Name: ARCHVOL
         Volume Device: /dev/asm/archvol-9
         State: ENABLED
         Size (MB): 306176
         Resize Unit (MB): 32
         Redundancy: UNPROT
         Stripe Columns: 4
         Stripe Width (K): 128
         Usage: ACFS
         Mountpath: /arch

首先将arch下面的文件都先tar 备份一下，然后准备重建下这个acfs文件系统，如下是重建的步骤：
1、删除原diskgroup

[grid@dtdb1 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 3 12:43:55 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> drop diskgroup archvg including contents;
drop diskgroup archvg including contents
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15027: active use of diskgroup "ARCHVG" precludes its dismount

SQL> alter diskgroup archvg dismount force;

Diskgroup altered.

SQL> drop diskgroup archvg  including contents ;
drop diskgroup archvg  including contents
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15027: active use of diskgroup "ARCHVG" precludes its dismount

SQL> alter diskgroup archvg dismount force;

Diskgroup altered.

SQL> drop diskgroup archvg  force including contents ;

Diskgroup dropped.

2、重建diskgroup

SQL> CREATE DISKGROUP archvg EXTERNAL REDUNDANCY DISK 'ORCL:ARCHVOL01' SIZE 306176M  ATTRIBUTE 'compatible.asm' = '11.2', 'compatible.advm' = '11.2'; 

Diskgroup created.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

3、创建卷组

[grid@dtdb1 ~]$ asmcmd volcreate -G archvg -s 299g ARCHVOL
ORA-15032: not all alterations performed
ORA-15041: diskgroup "ARCHVG" space exhausted (DBD ERROR: OCIStmtExecute)
[grid@dtdb1 ~]$ asmcmd volcreate -G archvg -s 297g ARCHVOL 
[grid@dtdb1 ~]$ asmcmd volinfo -a
Diskgroup Name: ARCHVG

         Volume Name: ARCHVOL
         Volume Device: /dev/asm/archvol-396
         State: ENABLED
         Size (MB): 304128
         Resize Unit (MB): 32
         Redundancy: UNPROT
         Stripe Columns: 4
         Stripe Width (K): 128
         Usage: 
         Mountpath:

4、格式acfs文件系统

[grid@dtdb1 ~]$ /sbin/mkfs -t acfs /dev/asm/archvol-396
mkfs.acfs: version                   = 11.2.0.3.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/archvol-396
mkfs.acfs: volume size               = 318901321728
mkfs.acfs: Format complete.

[root@dtdb1 ~]# ls -ltr /dev/asm
total 0
brwxrwx--- 1 root asmadmin 252, 202753 Jan  3 13:17 archvol-396

5、在其他节点激活卷组

[root@dtdb2 u01]# su - grid
[grid@dtdb2 ~]$ ls -ltr /dev/asm
total 0 
[grid@dtdb2 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 3 13:28:51 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup archvg enable volume 'ARCHVOL'; 

Diskgroup altered.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@dtdb2 ~]$ ls -ltr /dev/asm
total 0
brwxrwx--- 1 root asmadmin 252, 202753 Jan  3 13:28 archvol-396

说明：如果节点2不激活，那么无法看到磁盘，需要手工激活。如果是通过asmca图形化界面来创建acfs不会有这个问题。
6、所有节点mount acfs文件系统

[root@dtdb2 u01]# mount -t acfs  /dev/asm/archvol-396  /arch
[root@dtdb2 u01]# df -h
Filesystem            Size  Used Avail Use{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} Mounted on
/dev/mapper/rootvg-lv02
                       12G  450M   11G   4{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /
tmpfs                  30G  891M   29G   3{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /dev/shm
/dev/sda1             388M   38M  331M  11{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /boot
/dev/mapper/rootvg-lv03
                      6.0G  426M  5.2G   8{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /home
/dev/mapper/rootvg-lv07
                      5.0G  168M  4.6G   4{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /opt
/dev/mapper/rootvg-lv04
                      6.0G  203M  5.5G   4{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /tmp
/dev/mapper/datavg-datalv01
                       50G   28G   19G  60{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /u01
/dev/mapper/rootvg-lv06
                      7.9G  4.0G  3.6G  53{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /usr
/dev/mapper/rootvg-lv05
                      7.9G  452M  7.1G   6{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /var
/dev/mapper/rootvg-netbackuplv
                      9.9G  8.2G  1.2G  88{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /usr/openv
/dev/asm/archvol-396  297G  668M  297G   1{39ecd679003247f2ed728ad9c7ed019a369dd84d0731b449c26bf628d3c1a20b} /arch
[root@dtdb2 arch]# chown -R oracle:asmadmin /arch

之前检查/var/log/messages有不少错误，怀疑可能是acfs的问题，通过重建后不再出现任何错误.