11g RAC 在线存储迁移实现 OCR 磁盘组完美替换

JiekeXu之路 2020-09-10

1617

作者 | JiekeXu

来源 | JiekeXu之路（ID: JiekeXu_IT）

转载请联系授权 | (微信ID：xxq1426321293)

大家好，我是 JiekeXu,很高兴又和大家见面了，今天分享上个月处理的案例:11g RAC 在线存储迁移实现 OCR 磁盘组完美替换。本文首发于微信公众号【JiekeXu之路】，欢迎点击上方蓝字关注我吧！

说明：需要将 Oracle 数据库 OCR DATA ARCH 等所有老存储磁盘替换为新存储磁盘，

华为存储替换为 SVC 存储进行存储迁移。数据库新磁盘如下：

rhdisk100-——rhdisk123 共 24 块盘，

rhdisk100、rhdisk101、rhdisk102 这三块准备为 1g OCR 盘，

rhdisk103 是 100g 的 ARCH 归档盘 rhdisk104-106 是 500G 的 ARCH 归档盘；

rhdisk106-——rhdisk123 是 500G 的数据盘 DATA。

操作系统 64 bit AIX 6.1，11204 RAC 环境，存储已划分好新盘，操作系统也已经完成扫盘，

剩下的步骤就是将磁盘添加进数据库然后通过 ASM 的 REBALANCE 功能将其旧盘踢出数据库，进一步踢出操作系统回收存储，即可实现迁移。

一、磁盘检查

--查看磁盘大小（单位M）

jiekeXur1:/dev#bootinfo -s rhdisk100
1024
jiekeXur1:/dev#bootinfo -s rhdisk101
1024
……………………省略部分……………………………
jiekeXur1:/dev#bootinfo -s rhdisk105
512000
jiekeXur1:/dev#bootinfo -s rhdisk123
512000

--检查磁盘属性

--- 查看磁盘的详细信息
lsattr -El hdisk100

jiekeXur1:/dev#lsattr -El hdisk123
PCM             PCM/friend/sddpcm                                   PCM                                     True
PR_key_value    none                                                Reserve Key                             True
algorithm       load_balance                                        Algorithm                               True
clr_q           no                                                  Device CLEARS its Queue on error        True
dist_err_pcnt   0                                                   Distributed Error Percentage            True
dist_tw_width   50                                                  Distributed Error Sample Time           True
flashcpy_tgtvol no                                                  Flashcopy Target Lun                    False
hcheck_interval 60                                                  Health Check Interval                   True
hcheck_mode     nonactive                                           Health Check Mode                       True
location                                                            Location Label                          True
lun_id          0x17000000000000                                    Logical Unit Number ID                  False
lun_reset_spt   yes                                                 Support SCSI LUN reset                  True
max_coalesce    0x40000                                             Maximum COALESCE size                   True
max_transfer    0x40000                                             Maximum TRANSFER Size                   True
node_name       0x500507680c00122f                                  FC Node Name                            False
pvid            none                                                Physical volume identifier              False
q_err           yes                                                 Use QERR bit                            True
q_type          simple                                              Queuing TYPE                            True
qfull_dly       2                                                   delay in seconds for SCSI TASK SET FULL True
queue_depth     20                                                  Queue DEPTH                             True
recoverDEDpath  no                                                  Recover DED Failed Path                 True
reserve_policy  no_reserve                                          Reserve Policy                          True
retry_timeout   120                                                 Retry Timeout                           True
rw_timeout      60                                                  READ/WRITE time out value               True
scbsy_dly       20                                                  delay in seconds for SCSI BUSY          True
scsi_id         0xde00f7                                            SCSI ID                                 False
start_timeout   180                                                 START unit time out value               True
svc_sb_ttl      0                                                   IO Time to Live                         True
timeout_policy  fail_path                                           Timeout Policy                          True
unique_id       33213600507680C80008AC00000000000092104214503IBMfcp Device Unique Identification            False
ww_name         0x500507680c25122f                                  FC World Wide Name                      False

--- 检查权限，660 属组 grid:asmadmin

【排序查看ls -ltr dev |grep rhdisk1*】
ls -l dev/hdisk1*

jiekeXur1:/dev#ls -l dev/rhdisk1*
crw-rw----    1 grid     asmadmin     19,  3 Jul 20 20:30 dev/rhdisk1
crw-rw----    1 grid     asmadmin     19, 14 Jul 20 20:30 dev/rhdisk10
crw-rw----    1 grid     asmadmin     19, 49 Jul 20 20:11 dev/rhdisk100
………………………………………省略部分………………………………………………………
crw-rw----    1 grid     asmadmin     19, 47 Jul 20 20:11 dev/rhdisk109
crw-rw----    1 grid     asmadmin     19,  6 Jul 20 21:35 dev/rhdisk11
crw-rw----    1 grid     asmadmin     19, 40 Jul 20 20:11 dev/rhdisk110

--- 检查PVID

lspv | grep hdisk100

jiekeXur1:/dev#lspv | grep hdisk100
hdisk100        none                                None                        
jiekeXur1:/dev#
jiekeXur1:/dev#lspv | grep hdisk123
hdisk123        none                                None                        
jiekeXur1:/dev#lspv | grep hdisk111
hdisk111        none                                None                        
jiekeXur1:/dev#

--- 检查保留策略

lsattr -E -l hdisk111 | grep reserve_policy  

jiekeXur1:/dev#lsattr -E -l hdisk111 | grep reserve_policy     
reserve_policy  no_reserve                                          Reserve Policy                          True
jiekeXur1:/dev#lsattr -E -l hdisk100 | grep reserve_policy     
reserve_policy  no_reserve                                          Reserve Policy                          True
jiekeXur1:/dev#lsattr -E -l hdisk103 | grep reserve_policy     
reserve_policy  no_reserve                                          Reserve Policy                          True
jiekeXur1:/dev#

--- 查看磁盘是否为共享磁盘

lsattr -El hdisk100 
比对两个主机对应的磁盘号是否一致：unique_id

--- 查看磁盘是否可用

lspv
看 PVID 是否为 none，若为 none 则数据库可用，
再查看数据库当前有没有使用，若没有则可用来扩容 ASM 磁盘组
jiekeXur1:/dev#lspv
hdisk0          00cef2e7a8e8d6b2                    rootvg          active      
…………………………………省略部分输出………………………………………………… 
hdisk121        none                                None                        
hdisk122        none                                None                        
hdisk123        none                                None                                              
hdisk124        00cef2e76c2e3e1d                    nxdumpvg        active      
hdisk125        00cef2e76c2e4837                    nxdumpvg        active      
hdisk126        00cef2e76c2e50e3                    nxdumpvg        active      
hdisk127        00cef2e76c2e5a4c                    nxdumpvg        active      
hdisk128        00cef2e76c2e6343                    nxdumpvg        active      
hdisk129        00cef2e76c2e6c77                    nxdumpvg        active      
hdisk31         00cef2e76c35109c                    rootvg          active

二、数据库检查

---储备知识点
---创建磁盘组语法，添加磁盘组示例，查看均衡时间
CREATE DISKGROUP &diskgroup_name EXTERNAL REDUNDANCY DISK '/dev/rhdiskxx';
create diskgroup ARCH EXTERNAL REDUNDANCY disk '/dev/rhdisk13';

alter diskgroup data add disk '/dev/rhdisk19','/dev/rhdisk20','/dev/rhdisk21';
---调整 ASM 磁盘均衡级别
ALTER DISKGROUP DATA REBALANCE POWER 11;
---查看ASM磁盘均衡时间：
select *  from  v$asm_operation;

---查看 ASM 磁盘使用情况

su - oracle
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Mon Jul 20 21:39:51 2020
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL>  set linesize 600;
 col NAME for a20;
 col PCT_FREE for a30;
 select GROUP_NUMBER,NAME,TYPE,STATE,total_mb/1024 total_gb,free_mb/1024 free_gb,round((free_mb/total_mb)*100,2)||'%' pct_free from v$asm_diskgroup;
SQL> SQL> SQL> 
GROUP_NUMBER NAME                 TYPE   STATE         TOTAL_GB    FREE_GB PCT_FREE
------------ -------------------- ------ ----------- ---------- ---------- ------------------------------
           1 ARCH                 EXTERN CONNECTED         5100 4260.47559 83.54%
           2 DATA                 EXTERN CONNECTED         5000 95.5458984 1.91%
           3 OCR                  NORMAL MOUNTED              3 2.09570313 69.86%
 
 ---查看数据量总大小 1.4 T           
SQL> select sum(bytes)/1024/1024/1024 Total_GB from dba_segments;
  TOTAL_GB
----------
1475.66351

---查看 ASM 磁盘组和 操作系统对应的磁盘名称
set pagesize 100
col PATH for a33
col NAME for a15
col FAILGROUP for a15
select GROUP_NUMBER,free_mb,total_mb,FAILGROUP,disk_number,MOUNT_STATUS,mode_status,STATE,HEADER_STATUS,name,PATH from v$asm_disk order by 4,5; 

GROUP_NUMBER    FREE_MB   TOTAL_MB FAILGROUP       DISK_NUMBER MOUNT_S MODE_ST STATE    HEADER_STATU NAME            PATH
------------ ---------- ---------- --------------- ----------- ------- ------- -------- ------------ --------------- ---------------------------------
           1      78901     102400 ARCH_0000                 0 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0000       dev/rhdisk21
           1     398909     512000 ARCH_0001                 1 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0001       dev/rhdisk11
           1     398869     512000 ARCH_0002                 2 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0002       dev/rhdisk12
           1     398874     512000 ARCH_0003                 3 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0003       dev/rhdisk13
           1     398983     512000 ARCH_0004                 4 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0004       dev/rhdisk14
           1     398889     512000 ARCH_0005                 5 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0005       dev/rhdisk15
           1     398897     512000 ARCH_0006                 6 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0006       dev/rhdisk16
           1     398868     512000 ARCH_0007                 7 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0007       dev/rhdisk17
           1     398932     512000 ARCH_0008                 8 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0008       dev/rhdisk18
           1     398898     512000 ARCH_0009                 9 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0009       dev/rhdisk19
           1     398921     512000 ARCH_0010                10 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0010       dev/rhdisk20
           2       9784     512000 DATA_0000                 0 CACHED  ONLINE  NORMAL   MEMBER       DATA_0000       dev/rhdisk1
           2       9786     512000 DATA_0001                 1 CACHED  ONLINE  NORMAL   MEMBER       DATA_0001       dev/rhdisk10
           2       9782     512000 DATA_0011                11 CACHED  ONLINE  NORMAL   MEMBER       DATA_0011       dev/rhdisk2
           2       9787     512000 DATA_0013                13 CACHED  ONLINE  NORMAL   MEMBER       DATA_0013       dev/rhdisk3
           2       9783     512000 DATA_0014                14 CACHED  ONLINE  NORMAL   MEMBER       DATA_0014       dev/rhdisk4
           2       9786     512000 DATA_0015                15 CACHED  ONLINE  NORMAL   MEMBER       DATA_0015       dev/rhdisk5
           2       9780     512000 DATA_0016                16 CACHED  ONLINE  NORMAL   MEMBER       DATA_0016       /dev/rhdisk6
           2       9785     512000 DATA_0017                17 CACHED  ONLINE  NORMAL   MEMBER       DATA_0017       /dev/rhdisk7
           2       9784     512000 DATA_0018                18 CACHED  ONLINE  NORMAL   MEMBER       DATA_0018       /dev/rhdisk8
           2       9782     512000 DATA_0019                19 CACHED  ONLINE  NORMAL   MEMBER       DATA_0019       /dev/rhdisk9
           3        714       1024 OCR_0000                  0 CACHED  ONLINE  NORMAL   MEMBER       OCR_0000        /dev/rhdisk22
           3        717       1024 OCR_0001                  1 CACHED  ONLINE  NORMAL   MEMBER       OCR_0001        /dev/rhdisk23
           3        715       1024 OCR_0002                  2 CACHED  ONLINE  NORMAL   MEMBER       OCR_0002        /dev/rhdisk24
           0          0          0                           0 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk100
           0          0          0                           1 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk101
           0          0          0                           2 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk102
           0          0          0                           3 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk103
           0          0          0                           4 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk104
           0          0          0                           5 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk105
           0          0          0                           6 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk106
           0          0          0                           7 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk107
           0          0          0                           8 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk108
           0          0          0                           9 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk109
           0          0          0                          10 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk110
           0          0          0                          11 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk111
           0          0          0                          12 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk112
           0          0          0                          13 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk113
           0          0          0                          14 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk114
           0          0          0                          15 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk115
           0          0          0                          16 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk116
           0          0          0                          17 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk117
           0          0          0                          18 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk118
           0          0          0                          19 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk119
           0          0          0                          20 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk120
           0          0          0                          21 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk121
           0          0          0                          22 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk122
           0          0          0                          23 CLOSED  ONLINE  NORMAL   CANDIDATE                    /dev/rhdisk123

三、正式添加磁盘

这里有两点要说明的，首先添加的磁盘总量需要比原来的大一些或者一样大，不能偏小，不然在操作时会报错 ORA-15032、ORA-15250，还有每个磁盘可以比原来的大，即实现小盘换大盘；其次，删除磁盘时使用的是 ARCH_0000 等这样的磁盘名，并不是磁盘路径；最后，在添加磁盘的同时进行删除操作，平衡时间会缩短很多，当遇到数据量几十 T 时均衡时间大概要好几天的时间，很是痛苦。

--- 以下为本次迁移过程中盘符相对应命令，迁移中主要以数据库中
--- 查到的磁盘号为准。即上节中所查的 GROUP_NUMBER 为 0 的磁盘。
-- ARCH 盘
 alter diskgroup ARCH  add disk '/dev/rhdisk103','/dev/rhdisk104','/dev/rhdisk105','/dev/rhdisk106' 
 drop disk 'ARCH_0000','ARCH_0001','ARCH_0002','ARCH_0003','ARCH_0004','ARCH_0005','ARCH_0006','ARCH_0007','ARCH_0008','ARCH_0009','ARCH_0010';
 ALTER DISKGROUP ARCH REBALANCE POWER 10; 
 
 -- DATA盘
 alter diskgroup DATA  add disk '/dev/rhdisk107','/dev/rhdisk108','/dev/rhdisk109','/dev/rhdisk110','/dev/rhdisk111','/dev/rhdisk112','/dev/rhdisk113',
 '/dev/rhdisk114','/dev/rhdisk115','/dev/rhdisk116','/dev/rhdisk117','/dev/rhdisk118','/dev/rhdisk119','/dev/rhdisk120','/dev/rhdisk121','/dev/rhdisk122','/dev/rhdisk123' 
 drop disk 'DATA_0000','DATA_0001','DATA_0011','DATA_0013','DATA_0014','DATA_0015','DATA_0016','DATA_0017','DATA_0018','DATA_0019';
 ALTER DISKGROUP DATA REBALANCE POWER 10; 
 
 --OCR 盘
 alter diskgroup OCR  add disk '/dev/rhdisk100','/dev/rhdisk101','/dev/rhdisk102' drop disk 'OCR_0000','OCR_0001','OCR_0002';

四、生产环境操作示例

22:15:41 SQL> select *  from  v$asm_operation;
no rows selected


---由于删除时写成路径出现如下错误：ORA-15032、ORA-15054
22:15:57 SQL>  alter diskgroup ARCH  add disk '/dev/rhdisk103','/dev/rhdisk104','/dev/rhdisk105','/dev/rhdisk106' drop disk '/dev/rhdisk11','/dev/rhdisk12','/dev/rhdisk13','/dev/rhdisk14','/dev/rhdisk15','/dev/rhdisk16','/dev/rhdisk17','/dev/rhdisk18','/dev/rhdisk19','/dev/rhdisk20','/dev/rhdisk21';
 alter diskgroup ARCH  add disk '/dev/rhdisk103','/dev/rhdisk104','/dev/rhdisk105','/dev/rhdisk106' drop disk '/dev/rhdisk11','/dev/rhdisk12','/dev/rhdisk13','/dev/rhdisk14','/dev/rhdisk15','/dev/rhdisk16','/dev/rhdisk17','/dev/rhdisk18','/dev/rhdisk19','/dev/rhdisk20','/dev/rhdisk21'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15054: disk "/DEV/RHDISK21" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK20" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK19" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK18" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK17" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK16" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK15" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK14" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK13" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK12" does not exist in diskgroup "ARCH"
ORA-15054: disk "/DEV/RHDISK11" does not exist in diskgroup "ARCH"


--- 先添加 ARCH 归档盘的同时删除旧的归档盘，注意虽添加和删除是同时进行的但
--- 还是先完成添加等均衡完才会进行剔除操作只是这个命令同时完成了而已，这个在 alert 日志中有体现。
22:22:07 SQL>  alter diskgroup ARCH  add disk '/dev/rhdisk103','/dev/rhdisk104','/dev/rhdisk105','/dev/rhdisk106' drop disk 'ARCH_0000','ARCH_0001','ARCH_0002','ARCH_0003','ARCH_0004','ARCH_0005','ARCH_0006','ARCH_0007','ARCH_0008','ARCH_0009','ARCH_0010';
Diskgroup altered.
22:22:39 SQL>
22:22:39 SQL> ALTER DISKGROUP ARCH REBALANCE POWER 11; 
Diskgroup altered.
22:23:33 SQL> select *  from  v$asm_operation;
GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------
           1 REBAL      RUN              11         11        492     858623       6758         126
22:23:48 SQL> 


--- 让子弹飞一会吧，我先出去玩等一会儿在看


23:30:20 SQL> select *  from  v$asm_operation;
GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------
           1 REBAL      RUN              11         11     465269     858623       7177          54


23:30:26 SQL>  set linesize 600;
 col NAME for a20;
 col PCT_FREE for a30;
 select GROUP_NUMBER,NAME,TYPE,STATE,total_mb/1024 total_gb,free_mb/1024 free_gb,round((free_mb/total_mb)*100,2)||'%' pct_free from v$asm_diskgroup;
23:30:35 SQL> 23:30:35 SQL> 23:30:35 SQL> 
GROUP_NUMBER NAME                 TYPE         STATE                    TOTAL_GB    FREE_GB PCT_FREE
------------ -------------------- ------------ ---------------------- ---------- ---------- ------------------------------
           1 ARCH                 EXTERN       MOUNTED                      1600 760.456055 47.53%
           2 DATA                 EXTERN       MOUNTED                      5000 95.5458984 1.91%
           3 OCR                  NORMAL       MOUNTED                         3 2.09570313 69.86%


--- 添加数据盘 DATA 的同时踢出旧盘
23:35:36 SQL> alter diskgroup DATA  add disk '/dev/rhdisk107','/dev/rhdisk108','/dev/rhdisk109','/dev/rhdisk110','/dev/rhdisk111','/dev/rhdisk112','/dev/rhdisk113','/dev/rhdisk114','/dev/rhdisk115','/dev/rhdisk116','/dev/rhdisk117','/dev/rhdisk118','/dev/rhdisk119','/dev/rhdisk120','/dev/rhdisk121','/dev/rhdisk122','/dev/rhdisk123' drop disk 'DATA_0000','DATA_0001','DATA_0011','DATA_0013','DATA_0014','DATA_0015','DATA_0016','DATA_0017','DATA_0018','DATA_0019'; 
Diskgroup altered.
23:36:01 SQL> ALTER DISKGROUP DATA REBALANCE POWER 11; 
Diskgroup altered.
23:37:22 SQL> select *  from  v$asm_operation;
GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------
           1 REBAL      RUN              11         11     533407     858623      13871          23
           2 REBAL      WAIT             11
23:37:31 SQL> select GROUP_NUMBER,NAME,TYPE,STATE,total_mb/1024 total_gb,free_mb/1024 free_gb,round((free_mb/total_mb)*100,2)||'%' pct_free from v$asm_diskgroup;
GROUP_NUMBER NAME                 TYPE         STATE                    TOTAL_GB    FREE_GB PCT_FREE
------------ -------------------- ------------ ---------------------- ---------- ---------- ------------------------------
           1 ARCH                 EXTERN       MOUNTED                      1600 760.456055 47.53%
           2 DATA                 EXTERN       MOUNTED                      8500  3595.2832 42.3%
           3 OCR                  NORMAL       MOUNTED                         3 2.09570313 69.86%
-- 添加 OCR 新盘的同时踢出旧盘
23:37:52 SQL> alter diskgroup OCR  add disk '/dev/rhdisk100','/dev/rhdisk101','/dev/rhdisk102' drop disk 'OCR_0000','OCR_0001','OCR_0002';
Diskgroup altered.


--- 查看均衡时间，等待均衡时间为 0 查询结果为空则算是均衡完成，不过估计需要一晚上了
23:39:40 SQL> select *  from  v$asm_operation;
GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------
           1 REBAL      RUN              11         11     563764     858623      10766          27
           2 REBAL      WAIT             11
           3 REBAL      WAIT              1


23:40:02 SQL> select GROUP_NUMBER,NAME,TYPE,STATE,total_mb/1024 total_gb,free_mb/1024 free_gb,round((free_mb/total_mb)*100,2)||'%' pct_free from v$asm_diskgroup;
GROUP_NUMBER NAME                 TYPE         STATE                    TOTAL_GB    FREE_GB PCT_FREE
------------ -------------------- ------------ ---------------------- ---------- ---------- ------------------------------
           1 ARCH                 EXTERN       MOUNTED                      1600 760.456055 47.53%
           2 DATA                 EXTERN       MOUNTED                      8500 3595.52637 42.3%
           3 OCR                  NORMAL       MOUNTED                         3 2.08984375 69.66%
 
--- 这里是截取出来的后台日志，可以看到 ARCH DATA 盘的添加和删除过程。
jiekeXur1:/app/oracle/diag/rdbms/jieke/jieke1/trace$           
Mon Jul 20 22:22:36 2020
SUCCESS: disk ARCH_0011 (11.716583381) added to diskgroup ARCH
SUCCESS: disk ARCH_0012 (12.716583382) added to diskgroup ARCH
SUCCESS: disk ARCH_0013 (13.716583383) added to diskgroup ARCH
SUCCESS: disk ARCH_0014 (14.716583384) added to diskgroup ARCH


Mon Jul 20 23:35:58 2020
SUCCESS: disk DATA_0002 (2.716583385) added to diskgroup DATA
SUCCESS: disk DATA_0003 (3.716583386) added to diskgroup DATA
SUCCESS: disk DATA_0004 (4.716583387) added to diskgroup DATA
SUCCESS: disk DATA_0005 (5.716583388) added to diskgroup DATA
SUCCESS: disk DATA_0006 (6.716583389) added to diskgroup DATA
SUCCESS: disk DATA_0007 (7.716583390) added to diskgroup DATA
SUCCESS: disk DATA_0008 (8.716583391) added to diskgroup DATA
SUCCESS: disk DATA_0009 (9.716583392) added to diskgroup DATA
SUCCESS: disk DATA_0010 (10.716583393) added to diskgroup DATA
SUCCESS: disk DATA_0012 (12.716583394) added to diskgroup DATA
SUCCESS: disk DATA_0020 (20.716583395) added to diskgroup DATA
SUCCESS: disk DATA_0021 (21.716583396) added to diskgroup DATA
SUCCESS: disk DATA_0022 (22.716583397) added to diskgroup DATA
SUCCESS: disk DATA_0023 (23.716583398) added to diskgroup DATA
SUCCESS: disk DATA_0024 (24.716583399) added to diskgroup DATA
SUCCESS: disk DATA_0025 (25.716583400) added to diskgroup DATA
SUCCESS: disk DATA_0026 (26.716583401) added to diskgroup DATA
Mon Jul 20 23:36:36 2020


 Tue Jul 21 00:05:50 2020
NOTE: disk 0 (ARCH_0000) in group 1 (ARCH) is offline for reads
NOTE: disk 1 (ARCH_0001) in group 1 (ARCH) is offline for reads
NOTE: disk 2 (ARCH_0002) in group 1 (ARCH) is offline for reads
NOTE: disk 3 (ARCH_0003) in group 1 (ARCH) is offline for reads
NOTE: disk 4 (ARCH_0004) in group 1 (ARCH) is offline for reads
NOTE: disk 5 (ARCH_0005) in group 1 (ARCH) is offline for reads
NOTE: disk 6 (ARCH_0006) in group 1 (ARCH) is offline for reads
NOTE: disk 7 (ARCH_0007) in group 1 (ARCH) is offline for reads
NOTE: disk 8 (ARCH_0008) in group 1 (ARCH) is offline for reads
NOTE: disk 9 (ARCH_0009) in group 1 (ARCH) is offline for reads
NOTE: disk 10 (ARCH_0010) in group 1 (ARCH) is offline for reads
NOTE: disk 0 (ARCH_0000) in group 1 (ARCH) is offline for writes
NOTE: disk 1 (ARCH_0001) in group 1 (ARCH) is offline for writes
NOTE: disk 2 (ARCH_0002) in group 1 (ARCH) is offline for writes
NOTE: disk 3 (ARCH_0003) in group 1 (ARCH) is offline for writes
NOTE: disk 4 (ARCH_0004) in group 1 (ARCH) is offline for writes
NOTE: disk 5 (ARCH_0005) in group 1 (ARCH) is offline for writes
NOTE: disk 6 (ARCH_0006) in group 1 (ARCH) is offline for writes
NOTE: disk 7 (ARCH_0007) in group 1 (ARCH) is offline for writes
NOTE: disk 8 (ARCH_0008) in group 1 (ARCH) is offline for writes
NOTE: disk 9 (ARCH_0009) in group 1 (ARCH) is offline for writes
NOTE: disk 10 (ARCH_0010) in group 1 (ARCH) is offline for writes
SUCCESS: disk ARCH_0000 (0.716583335) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0001 (1.716583325) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0002 (2.716583326) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0003 (3.716583327) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0004 (4.716583328) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0005 (5.716583329) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0006 (6.716583330) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0007 (7.716583331) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0008 (8.716583332) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0009 (9.716583333) dropped from diskgroup ARCH
SUCCESS: disk ARCH_0010 (10.716583334) dropped from diskgroup ARCH
Tue Jul 21 00:06:49 2020
jiekeXur1:/app/oracle/diag/rdbms/jieke/jieke1/trace$


--- 查看是均衡完毕但有些磁盘还是 DROPPING 中，故需要等 DROPPING 完成，先睡觉吧，白天再看。
SQL>  set linesize 600;
 col NAME for a20;
 col PCT_FREE for a30;
 select GROUP_NUMBER,NAME,TYPE,STATE,total_mb/1024 total_gb,free_mb/1024 free_gb,round((free_mb/total_mb)*100,2)||'%' pct_free from v$asm_diskgroup;
SQL> SQL> SQL> 
GROUP_NUMBER NAME                 TYPE   STATE         TOTAL_GB    FREE_GB PCT_FREE
------------ -------------------- ------ ----------- ---------- ---------- ------------------------------
           1 ARCH                 EXTERN CONNECTED         1600 760.516602 47.53%
           2 DATA                 EXTERN CONNECTED         8500 3595.44629 42.3%
           3 OCR                  NORMAL MOUNTED              3 2.09570313 69.86%
SQL> select * from v$asm_operation;
no rows selected
SQL> set time on
00:26:42 SQL> select * from v$asm_operation;
no rows selected
00:31:56 SQL> set linesize 9999
 set pagesize 100
col PATH for a33
col NAME for a15
col FAILGROUP for a15
select GROUP_NUMBER,free_mb,total_mb,FAILGROUP,disk_number,       MOUNT_STATUS,mode_status,STATE,HEADER_STATUS,name,PATH from v$asm_disk order by 4,5; 
00:32:03 SQL> 00:32:03 SQL> 00:32:03 SQL> 00:32:03 SQL> 00:32:03 SQL> 
GROUP_NUMBER    FREE_MB   TOTAL_MB FAILGROUP       DISK_NUMBER MOUNT_S MODE_ST STATE    HEADER_STATU NAME            PATH
------------ ---------- ---------- --------------- ----------- ------- ------- -------- ------------ --------------- ---------------------------------
           1      48084     102400 ARCH_0011                11 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0011       /dev/rhdisk103
           1     243561     512000 ARCH_0012                12 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0012       /dev/rhdisk104
           1     243561     512000 ARCH_0013                13 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0013       /dev/rhdisk105
           1     243563     512000 ARCH_0014                14 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0014       /dev/rhdisk106
           2      77459     512000 DATA_0000                 0 CACHED  ONLINE  DROPPING MEMBER       DATA_0000       /dev/rhdisk1
           2      77462     512000 DATA_0001                 1 CACHED  ONLINE  DROPPING MEMBER       DATA_0001       /dev/rhdisk10
           2     472183     512000 DATA_0002                 2 CACHED  ONLINE  NORMAL   MEMBER       DATA_0002       /dev/rhdisk107
           2     472185     512000 DATA_0003                 3 CACHED  ONLINE  NORMAL   MEMBER       DATA_0003       /dev/rhdisk108
           2     472184     512000 DATA_0004                 4 CACHED  ONLINE  NORMAL   MEMBER       DATA_0004       /dev/rhdisk109
           2     472185     512000 DATA_0005                 5 CACHED  ONLINE  NORMAL   MEMBER       DATA_0005       /dev/rhdisk110
           2     472185     512000 DATA_0006                 6 CACHED  ONLINE  NORMAL   MEMBER       DATA_0006       /dev/rhdisk111
           2     472184     512000 DATA_0007                 7 CACHED  ONLINE  NORMAL   MEMBER       DATA_0007       /dev/rhdisk112
           2     472185     512000 DATA_0008                 8 CACHED  ONLINE  NORMAL   MEMBER       DATA_0008       /dev/rhdisk113
           2     472185     512000 DATA_0009                 9 CACHED  ONLINE  NORMAL   MEMBER       DATA_0009       /dev/rhdisk114
           2     472184     512000 DATA_0010                10 CACHED  ONLINE  NORMAL   MEMBER       DATA_0010       /dev/rhdisk115
           2      77459     512000 DATA_0011                11 CACHED  ONLINE  DROPPING MEMBER       DATA_0011       /dev/rhdisk2
           2     472184     512000 DATA_0012                12 CACHED  ONLINE  NORMAL   MEMBER       DATA_0012       /dev/rhdisk116
           2      77463     512000 DATA_0013                13 CACHED  ONLINE  DROPPING MEMBER       DATA_0013       /dev/rhdisk3
           2      77462     512000 DATA_0014                14 CACHED  ONLINE  DROPPING MEMBER       DATA_0014       /dev/rhdisk4
           2      77462     512000 DATA_0015                15 CACHED  ONLINE  DROPPING MEMBER       DATA_0015       /dev/rhdisk5
           2      77458     512000 DATA_0016                16 CACHED  ONLINE  DROPPING MEMBER       DATA_0016       /dev/rhdisk6
           2      77460     512000 DATA_0017                17 CACHED  ONLINE  DROPPING MEMBER       DATA_0017       /dev/rhdisk7
           2      77461     512000 DATA_0018                18 CACHED  ONLINE  DROPPING MEMBER       DATA_0018       /dev/rhdisk8
           2      77460     512000 DATA_0019                19 CACHED  ONLINE  DROPPING MEMBER       DATA_0019       /dev/rhdisk9
           2     472184     512000 DATA_0020                20 CACHED  ONLINE  NORMAL   MEMBER       DATA_0020       /dev/rhdisk117
           2     472184     512000 DATA_0021                21 CACHED  ONLINE  NORMAL   MEMBER       DATA_0021       /dev/rhdisk118
           2     472184     512000 DATA_0022                22 CACHED  ONLINE  NORMAL   MEMBER       DATA_0022       /dev/rhdisk119
           2     472184     512000 DATA_0023                23 CACHED  ONLINE  NORMAL   MEMBER       DATA_0023       /dev/rhdisk120
           2     472184     512000 DATA_0024                24 CACHED  ONLINE  NORMAL   MEMBER       DATA_0024       /dev/rhdisk121
           2     472184     512000 DATA_0025                25 CACHED  ONLINE  NORMAL   MEMBER       DATA_0025       /dev/rhdisk122
           2     472183     512000 DATA_0026                26 CACHED  ONLINE  NORMAL   MEMBER       DATA_0026       /dev/rhdisk123
           3        715       1024 OCR_0003                  3 CACHED  ONLINE  NORMAL   MEMBER       OCR_0003        /dev/rhdisk100
           3        715       1024 OCR_0004                  4 CACHED  ONLINE  NORMAL   MEMBER       OCR_0004        /dev/rhdisk101
           3        716       1024 OCR_0005                  5 CACHED  ONLINE  NORMAL   MEMBER       OCR_0005        /dev/rhdisk102
           0          0          0                           0 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk11
           0          0          0                           1 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk12
           0          0          0                           2 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk13
           0          0          0                           3 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk14
           0          0          0                           4 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk15
           0          0          0                           5 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk16
           0          0          0                           6 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk17
           0          0          0                           7 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk18
           0          0          0                           8 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk19
           0          0          0                           9 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk20
           0          0          0                          10 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk21
           0          0          0                          11 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk22
           0          0          0                          12 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk23
           0          0          0                          13 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk24
48 rows selected.


--- 第二天白天查看 STATE 全部变成 NORMAL 状态了。且新盘 100——123 已全部添加进 ASM，旧盘可以直接踢出操作系统了。
09:34:11 SQL> select GROUP_NUMBER,free_mb,total_mb,FAILGROUP,disk_number,       MOUNT_STATUS,mode_status,STATE,HEADER_STATUS,name,PATH from v$asm_disk order by 4,5;
GROUP_NUMBER    FREE_MB   TOTAL_MB FAILGROUP       DISK_NUMBER MOUNT_S MODE_ST STATE    HEADER_STATU NAME            PATH
------------ ---------- ---------- --------------- ----------- ------- ------- -------- ------------ --------------- ---------------------------------
           1      48040     102400 ARCH_0011                11 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0011       /dev/rhdisk103
           1     243341     512000 ARCH_0012                12 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0012       /dev/rhdisk104
           1     243343     512000 ARCH_0013                13 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0013       /dev/rhdisk105
           1     243344     512000 ARCH_0014                14 CACHED  ONLINE  NORMAL   MEMBER       ARCH_0014       /dev/rhdisk106
           2     216576     512000 DATA_0002                 2 CACHED  ONLINE  NORMAL   MEMBER       DATA_0002       /dev/rhdisk107
           2     216576     512000 DATA_0003                 3 CACHED  ONLINE  NORMAL   MEMBER       DATA_0003       /dev/rhdisk108
           2     216576     512000 DATA_0004                 4 CACHED  ONLINE  NORMAL   MEMBER       DATA_0004       /dev/rhdisk109
           2     216576     512000 DATA_0005                 5 CACHED  ONLINE  NORMAL   MEMBER       DATA_0005       /dev/rhdisk110
           2     216576     512000 DATA_0006                 6 CACHED  ONLINE  NORMAL   MEMBER       DATA_0006       /dev/rhdisk111
           2     216576     512000 DATA_0007                 7 CACHED  ONLINE  NORMAL   MEMBER       DATA_0007       /dev/rhdisk112
           2     216577     512000 DATA_0008                 8 CACHED  ONLINE  NORMAL   MEMBER       DATA_0008       /dev/rhdisk113
           2     216577     512000 DATA_0009                 9 CACHED  ONLINE  NORMAL   MEMBER       DATA_0009       /dev/rhdisk114
           2     216577     512000 DATA_0010                10 CACHED  ONLINE  NORMAL   MEMBER       DATA_0010       /dev/rhdisk115
           2     216576     512000 DATA_0012                12 CACHED  ONLINE  NORMAL   MEMBER       DATA_0012       /dev/rhdisk116
           2     216576     512000 DATA_0020                20 CACHED  ONLINE  NORMAL   MEMBER       DATA_0020       /dev/rhdisk117
           2     216576     512000 DATA_0021                21 CACHED  ONLINE  NORMAL   MEMBER       DATA_0021       /dev/rhdisk118
           2     216577     512000 DATA_0022                22 CACHED  ONLINE  NORMAL   MEMBER       DATA_0022       /dev/rhdisk119
           2     216577     512000 DATA_0023                23 CACHED  ONLINE  NORMAL   MEMBER       DATA_0023       /dev/rhdisk120
           2     216576     512000 DATA_0024                24 CACHED  ONLINE  NORMAL   MEMBER       DATA_0024       /dev/rhdisk121
           2     216576     512000 DATA_0025                25 CACHED  ONLINE  NORMAL   MEMBER       DATA_0025       /dev/rhdisk122
           2     216576     512000 DATA_0026                26 CACHED  ONLINE  NORMAL   MEMBER       DATA_0026       /dev/rhdisk123
           3        715       1024 OCR_0003                  3 CACHED  ONLINE  NORMAL   MEMBER       OCR_0003        /dev/rhdisk100
           3        715       1024 OCR_0004                  4 CACHED  ONLINE  NORMAL   MEMBER       OCR_0004        /dev/rhdisk101
           3        716       1024 OCR_0005                  5 CACHED  ONLINE  NORMAL   MEMBER       OCR_0005        /dev/rhdisk102
           0          0          0                           0 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk11
           0          0          0                           1 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk12
           0          0          0                           2 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk13
           0          0          0                           3 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk14
           0          0          0                           4 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk15
           0          0          0                           5 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk16
           0          0          0                           6 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk17
           0          0          0                           7 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk18
           0          0          0                           8 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk19
           0          0          0                           9 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk20
           0          0          0                          10 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk21
           0          0          0                          11 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk22
           0          0          0                          12 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk23
           0          0          0                          13 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk24
           0          0          0                          14 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk1
           0          0          0                          15 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk10
           0          0          0                          16 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk2
           0          0          0                          17 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk3
           0          0          0                          18 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk4
           0          0          0                          19 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk5
           0          0          0                          20 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk6
           0          0          0                          21 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk7
           0          0          0                          22 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk8
           0          0          0                          23 CLOSED  ONLINE  NORMAL   FORMER                       /dev/rhdisk9
48 rows selected.

白天早上查看后台日志中有从磁盘组 DATA 中删除的记录，上面夜间日志中有添加成功的记录，验证了前面说的同事添加删除的命令并不是同时完成的结果。

Tue Jul 21 06:37:34 2020
NOTE: disk 0 (DATA_0000) in group 2 (DATA) is offline for reads
NOTE: disk 1 (DATA_0001) in group 2 (DATA) is offline for reads
NOTE: disk 11 (DATA_0011) in group 2 (DATA) is offline for reads
NOTE: disk 13 (DATA_0013) in group 2 (DATA) is offline for reads
NOTE: disk 14 (DATA_0014) in group 2 (DATA) is offline for reads
NOTE: disk 15 (DATA_0015) in group 2 (DATA) is offline for reads
NOTE: disk 16 (DATA_0016) in group 2 (DATA) is offline for reads
NOTE: disk 17 (DATA_0017) in group 2 (DATA) is offline for reads
NOTE: disk 18 (DATA_0018) in group 2 (DATA) is offline for reads
NOTE: disk 19 (DATA_0019) in group 2 (DATA) is offline for reads
NOTE: disk 0 (DATA_0000) in group 2 (DATA) is offline for writes
NOTE: disk 1 (DATA_0001) in group 2 (DATA) is offline for writes
NOTE: disk 11 (DATA_0011) in group 2 (DATA) is offline for writes
NOTE: disk 13 (DATA_0013) in group 2 (DATA) is offline for writes
NOTE: disk 14 (DATA_0014) in group 2 (DATA) is offline for writes
NOTE: disk 15 (DATA_0015) in group 2 (DATA) is offline for writes
NOTE: disk 16 (DATA_0016) in group 2 (DATA) is offline for writes
NOTE: disk 17 (DATA_0017) in group 2 (DATA) is offline for writes
NOTE: disk 18 (DATA_0018) in group 2 (DATA) is offline for writes
NOTE: disk 19 (DATA_0019) in group 2 (DATA) is offline for writes
SUCCESS: disk DATA_0000 (0.716583336) dropped from diskgroup DATA
SUCCESS: disk DATA_0001 (1.716583337) dropped from diskgroup DATA
SUCCESS: disk DATA_0011 (11.716583338) dropped from diskgroup DATA
SUCCESS: disk DATA_0013 (13.716583339) dropped from diskgroup DATA
SUCCESS: disk DATA_0014 (14.716583340) dropped from diskgroup DATA
SUCCESS: disk DATA_0015 (15.716583341) dropped from diskgroup DATA
SUCCESS: disk DATA_0016 (16.716583342) dropped from diskgroup DATA
SUCCESS: disk DATA_0017 (17.716583343) dropped from diskgroup DATA
SUCCESS: disk DATA_0018 (18.716583344) dropped from diskgroup DATA
SUCCESS: disk DATA_0019 (19.716583345) dropped from diskgroup DATA

五、操作系统踢盘问题

当 AIX 岗系统管理人员踢盘时发现有如下两块盘占用，无法删除，报错0514-062 divice is busy.

jiekeXur1:/#rmdev -dl hdisk7
Method error (/usr/lib/methods/ucfgdevice):
        0514-062 Cannot perform the requested function because the
                 specified device is busy.
jiekeXur1:/#rmdev -dl hdisk2
Method error (/usr/lib/methods/ucfgdevice):
        0514-062 Cannot perform the requested function because the
                 specified device is busy.
                 
--- 使用 fuser 查看占用磁盘的进程。


jiekeXur1:/#fuser /dev/rhdisk2
/dev/rhdisk2:  6881888
nxywywk3r1:/#fuser /dev/rhdisk7
/dev/rhdisk7:  6881888                 


---后记---
---后续系统中由于磁盘总数过小，添加的同时进行删除时报错 ORA-15250 
SQL> alter diskgroup DATA drop disk 'DATA_0094','DATA_0095','DATA_0096','DATA_0097','DATA_0098','DATA_0099','DATA_0100',
  2  'DATA_0101','DATA_0102','DATA_0103','DATA_0104','DATA_0105','DATA_0106','DATA_0107';
alter diskgroup DATA drop disk 'DATA_0094','DATA_0095','DATA_0096','DATA_0097','DATA_0098','DATA_0099','DATA_0100',
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15250: insufficient diskgroup space for rebalance completion


SQL> select * from gv$asm_operation;


   INST_ID GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
---------- ------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------
         2            3 REBAL      WAIT              1
         1            3 REBAL      RUN               1          1       7521   31409443       5513        5695

查看发现 rhdisk2 和 rhdisk2 对应 DATA_0011 和 DATA_0017，对应的连接进程为数据库本地连接 LOCAL=YES 进程，此进程非数据库核心进程可通过操作系统命令 kill -9 杀掉，然后即可踢出磁盘，实现存储迁移，即 RAC ASM 磁盘组在线迁移。

jiekeXur1:/#ps -ef | grep 6881888
  oracle  6881888        1   0   Feb 27      -  8:23 oraclenjieke1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
    root  7799346 10944560   0 23:54:31  pts/2  0:00 grep 6881888  
jiekeXur1:/home/oracle$
jiekeXur1:/home/oracle$ps -ef | grep 6881888
  oracle  6881888        1   0   Feb 27      -  8:23 oraclenjieke1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
  oracle 12845802 13893668   0 23:59:15  pts/2  0:00 grep 6881888
jiekeXur1:/home/oracle$
jiekeXur1:/home/oracle$ps -ef | grep  LOCAL=YES
    grid  6750240        1   0   Feb 27      -  9:50 oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
    grid  6947056        1   0   Feb 27      -  0:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
    grid  7667838        1   0   Feb 27      -  0:36 oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
  oracle 13631510 13893668   0 00:00:17  pts/2  0:00 grep LOCAL=YES
  oracle  7864796        1   0   Feb 27      - 10:14 oracle+ASM1_asmb_jieke1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
    grid 11927844        1   0   Jun 30      -  0:01 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
  oracle  6881888        1   0   Feb 27      -  8:23 oraclejieke1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
  oracle  7864888        1   0   Feb 27      -  0:00 oraclejieke1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
  oracle 10486466        1   0 23:57:29      -  0:00 oracle+ASM1_pz99_jieke1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
    grid 10617408        1   0   Jun 29      -  0:01 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
  oracle 11272858        1   0   May 08      -  0:45 oracle+ASM1_o008_jieke1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
jiekeXur1:/home/oracle$kill -9 6881888
jiekeXur1:/home/oracle$ps -ef | grep 6881888
  oracle 11010646 13893668   0 00:01:47  pts/2  0:00 grep 6881888
jiekeXur1:/#rmdev -dl hdisk7
jiekeXur1:/#rmdev -dl hdisk2

该问题处理完后，即实现了 ASM 存储在线迁移，不需要单独进行 OCR 磁盘的替换，只需操作系统能够扫到盘，能够正常添加到 ASM 即可，和平时添加 ASM 磁盘组几乎没有任何区别。加油，让我们一起努力做更好的自己,今天就到这里了，祝小伙伴们新的一天愉快，愿天下所有教师节日快乐，身体健康，工作顺利！