文档课题：rac环境模拟vote盘和data盘磁盘头损坏的修复.

系统：centos 7.9 64位

数据库：oracle 11.2.0.4 64位

环境：rac (两节点)

1、磁盘组信息

1.1、系统信息

[root@hisdb1 ~]# cat /etc/*release

CentOS Linux release 7.9.2009 (Core)

NAME="CentOS Linux"

VERSION="7 (Core)"

ID="centos"

ID_LIKE="rhel fedora"

VERSION_ID="7"

PRETTY_NAME="CentOS Linux 7 (Core)"

ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:centos:centos:7"

HOME_URL="https://www.centos.org/"

BUG_REPORT_URL="https://bugs.centos.org/"

 

CENTOS_MANTISBT_PROJECT="CentOS-7"

CENTOS_MANTISBT_PROJECT_VERSION="7"

REDHAT_SUPPORT_PRODUCT="centos"

REDHAT_SUPPORT_PRODUCT_VERSION="7"

 

CentOS Linux release 7.9.2009 (Core)

CentOS Linux release 7.9.2009 (Core)

1.2、磁盘信息

SQL> select group_number,name,path,state,total_mb,free_mb from v$asm_disk where name is not null order by path;

 

GROUP_NUMBER NAME   PATH     STATE      TOTAL_MB    FREE_MB

------------ --------------- -------------------- -------- ---------- ----------

           2 DATA02   ORCL:DATA02          NORMAL        10239       6662

           1 DATA03   ORCL:DATA03          NORMAL        20479      13765

           3 DATA04   ORCL:DATA04          NORMAL        10239       9843

SQL> select group_number,name,type,total_mb,free_mb from v$asm_diskgroup;

 

GROUP_NUMBER NAME      TYPE     TOTAL_MB    FREE_MB

------------ --------------- ------ ---------- ----------

           1 DATA            EXTERN      20479      13765

           2 FRA             EXTERN      10239       6662

           3 OCRBK          EXTERN      10239       9843

[root@hisdb1 disks]# pwd

/dev/oracleasm/disks

[root@hisdb1 disks]# ll /dev/oracleasm/disks/*

brw-rw---- 1 grid asmadmin 8, 17 Dec 27 20:27 /dev/oracleasm/disks/DATA01

brw-rw---- 1 grid asmadmin 8, 33 Dec 27 20:27 /dev/oracleasm/disks/DATA02

brw-rw---- 1 grid asmadmin 8, 49 Dec 27 20:27 /dev/oracleasm/disks/DATA03

brw-rw---- 1 grid asmadmin 8, 65 Dec 27 20:27 /dev/oracleasm/disks/DATA04

说明：以上DATA04对应vote盘，DATA03对应data盘.

2、vote盘

模拟vote盘的损坏以及修复.

2.1、拷贝数据

--从/dev/oracleasm/disks/DATA04拷贝1个8k的块到/home/grid/data04.dd

[grid@hisdb1 disks]$ dd if=/dev/oracleasm/disks/DATA04 of=/home/grid/data04.dd bs=8192 count=1

1+0 records in

1+0 records out

8192 bytes (8.2 kB) copied, 0.000340858 s, 24.0 MB/s

[grid@hisdb1 ~]$ ll data04.dd

-rw-r--r-- 1 grid oinstall 8192 Dec 27 21:32 data04.dd

--借助kfed读取/dev/oracleasm/disks/DATA04磁盘头信息.

[grid@hisdb1 ~]$ kfed read /dev/oracleasm/disks/DATA04 text=data04.txt

[grid@hisdb1 ~]$ head data04.txt

kfbh.endian:                          1 ; 0x000: 0x01

kfbh.hard:                          130 ; 0x001: 0x82

kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD

kfbh.datfmt:                          1 ; 0x003: 0x01

kfbh.block.blk:                       0 ; 0x004: blk=0

kfbh.block.obj:              2147483648 ; 0x008: disk=0

kfbh.check:                  3855329304 ; 0x00c: 0xe5cba818

kfbh.fcn.base:                        0 ; 0x010: 0x00000000

kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000

kfbh.spare1:                          0 ; 0x018: 0x00000000

2.2、损坏磁盘

--破坏votedisk磁盘组的磁盘

[grid@hisdb1 ~]$ dd if=/dev/zero of=/dev/oracleasm/disks/DATA04 bs=8192 count=1

1+0 records in

1+0 records out

8192 bytes (8.2 kB) copied, 0.000128522 s, 63.7 MB/s

[grid@hisdb1 ~]$ kfed read /dev/oracleasm/disks/DATA04 | head

kfbh.endian:                          0 ; 0x000: 0x00

kfbh.hard:                            0 ; 0x001: 0x00

kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt:                          0 ; 0x003: 0x00

kfbh.block.blk:                       0 ; 0x004: blk=0

kfbh.block.obj:                       0 ; 0x008: file=0

kfbh.check:                           0 ; 0x00c: 0x00000000

kfbh.fcn.base:                        0 ; 0x010: 0x00000000

kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000

kfbh.spare1:                          0 ; 0x018: 0x00000000

2.3、异常重现

--重启集群

[root@hisdb1 ~]# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'hisdb1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hisdb1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.OCRBK.dg' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.heal.db' on 'hisdb1'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.hisdb1.vip' on 'hisdb1'

CRS-2677: Stop of 'ora.FRA.dg' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.heal.db' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'hisdb1'

CRS-2677: Stop of 'ora.hisdb1.vip' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.crsd' on 'hisdb2'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hisdb2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.cvu' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.OCRBK.dg' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.heal.db' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.oc4j' on 'hisdb2'

CRS-2677: Stop of 'ora.cvu' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.hisdb2.vip' on 'hisdb2'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'hisdb2'

CRS-2677: Stop of 'ora.FRA.dg' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.heal.db' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'hisdb2'

CRS-2677: Stop of 'ora.hisdb2.vip' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.OCRBK.dg' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb1'

CRS-2677: Stop of 'ora.asm' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'hisdb1'

CRS-2677: Stop of 'ora.oc4j' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.ons' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'hisdb1'

CRS-2677: Stop of 'ora.net1.network' on 'hisdb1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hisdb1' has completed

CRS-2677: Stop of 'ora.crsd' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.evmd' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb1'

CRS-2677: Stop of 'ora.evmd' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.OCRBK.dg' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb2'

CRS-2677: Stop of 'ora.asm' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'hisdb2'

CRS-2677: Stop of 'ora.ons' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'hisdb2'

CRS-2677: Stop of 'ora.net1.network' on 'hisdb2' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hisdb2' has completed

CRS-2677: Stop of 'ora.crsd' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.evmd' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb2'

CRS-2677: Stop of 'ora.evmd' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.asm' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'hisdb1'

CRS-2677: Stop of 'ora.ctssd' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'hisdb1'

CRS-2677: Stop of 'ora.cssd' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.asm' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'hisdb2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'hisdb2'

CRS-2677: Stop of 'ora.cssd' on 'hisdb2' succeeded

[root@hisdb1 ~]# /u01/app/11.2.0/grid/bin/crsctl start cluster -all

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hisdb2'

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hisdb1'

CRS-2676: Start of 'ora.cssdmonitor' on 'hisdb2' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'hisdb2'

CRS-2676: Start of 'ora.cssdmonitor' on 'hisdb1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'hisdb1'

CRS-2672: Attempting to start 'ora.diskmon' on 'hisdb2'

CRS-2676: Start of 'ora.diskmon' on 'hisdb2' succeeded

CRS-2672: Attempting to start 'ora.diskmon' on 'hisdb1'

CRS-2676: Start of 'ora.diskmon' on 'hisdb1' succeeded

……

说明：此时会一直hang住，因为损坏的是投票盘，集群无法启动.

2.4、相关告警

--ocssd.log不断报如下错误：

2022-12-27 22:17:15.937: [    CSSD][3821278976]clssnmvDiskVerify: Successful discovery of 0 disks

2022-12-27 22:17:15.937: [    CSSD][3821278976]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery

2022-12-27 22:17:15.937: [    CSSD][3821278976]clssnmvFindInitialConfigs: No voting files found

2022-12-27 22:17:15.937: [    CSSD][3821278976](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds

2022-12-27 22:17:15.996: [    CSSD][3823675136]clssscSelect: cookie accept request 0x7fa0d80845c0

2022-12-27 22:17:15.996: [    CSSD][3823675136]clssscevtypSHRCON: getting client with cmproc 0x7fa0d80845c0

2022-12-27 22:17:15.996: [    CSSD][3823675136]clssgmRegisterClient: proc(4/0x7fa0d80845c0), client(358/0x7fa0d8071230)

2022-12-27 22:17:15.996: [    CSSD][3823675136]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7fa0d80845c0) client(0x7fa0d8071230)

2022-12-27 22:17:15.996: [    CSSD][3823675136]clssgmDiscEndpcl: gipcDestroy 0x5976

2022-12-27 22:17:16.329: [    CSSD][3823675136]clssscSelect: cookie accept request 0x7fa0d8099e80

2022-12-27 22:17:16.329: [    CSSD][3823675136]clssscevtypSHRCON: getting client with cmproc 0x7fa0d8099e80

2022-12-27 22:17:16.329: [    CSSD][3823675136]clssgmRegisterClient: proc(5/0x7fa0d8099e80), client(357/0x7fa0d8071230)

2022-12-27 22:17:16.329: [    CSSD][3823675136]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7fa0d8099e80) client(0x7fa0d8071230)

2022-12-27 22:17:16.329: [    CSSD][3823675136]clssgmDiscEndpcl: gipcDestroy 0x598c

2022-12-27 22:17:16.998: [    CSSD][3823675136]clssscSelect: cookie accept request 0x7fa0d80845c0

2022-12-27 22:17:16.998: [    CSSD][3823675136]clssscevtypSHRCON: getting client with cmproc 0x7fa0d80845c0

2022-12-27 22:17:16.998: [    CSSD][3823675136]clssgmRegisterClient: proc(4/0x7fa0d80845c0), client(359/0x7fa0d8071230)

2022-12-27 22:17:16.998: [    CSSD][3823675136]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x7fa0d80845c0) client(0x7fa0d8071230)

-- alerthisdb1.log报错如下

[grid@hisdb1 hisdb1]$ tail -5000f alerthisdb1.log

每隔15s如下错误

[cssd(7816)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/hisdb1/cssd/ocssd.log

2.5、恢复vote磁盘

[grid@hisdb1 ~]$ kfed repair /dev/oracleasm/disks/DATA04

说明：修复成功后，集群恢复正常.

3、data盘

模拟data盘的损坏和修复.

3.1、拷贝数据

[grid@hisdb1 ~]$ dd if=/dev/oracleasm/disks/DATA03 of=/home/grid/data03.dd bs=8192 count=1 

1+0 records in

1+0 records out

8192 bytes (8.2 kB) copied, 0.000373797 s, 21.9 MB/s

[grid@hisdb1 ~]$ kfed read /dev/oracleasm/disks/DATA03 | head

kfbh.endian:                          1 ; 0x000: 0x01

kfbh.hard:                          130 ; 0x001: 0x82

kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD

kfbh.datfmt:                          1 ; 0x003: 0x01

kfbh.block.blk:                       0 ; 0x004: blk=0

kfbh.block.obj:              2147483648 ; 0x008: disk=0

kfbh.check:                  3875939376 ; 0x00c: 0xe7062430

kfbh.fcn.base:                        0 ; 0x010: 0x00000000

kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000

kfbh.spare1:                          0 ; 0x018: 0x00000000

3.2、损坏磁盘

[grid@hisdb1 ~]$ dd if=/dev/zero of=/dev/oracleasm/disks/DATA03 bs=8192 count=1

1+0 records in

1+0 records out

8192 bytes (8.2 kB) copied, 0.000199175 s, 41.1 MB/s

[grid@hisdb1 ~]$ kfed read /dev/oracleasm/disks/DATA03 | head

kfbh.endian:                          0 ; 0x000: 0x00

kfbh.hard:                            0 ; 0x001: 0x00

kfbh.type:                            0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt:                          0 ; 0x003: 0x00

kfbh.block.blk:                       0 ; 0x004: blk=0

kfbh.block.obj:                       0 ; 0x008: file=0

kfbh.check:                           0 ; 0x00c: 0x00000000

kfbh.fcn.base:                        0 ; 0x010: 0x00000000

kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000

kfbh.spare1:                          0 ; 0x018: 0x00000000

3.3、异常重现

[root@hisdb1 ~]# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.crsd' on 'hisdb2'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hisdb1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.cvu' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.oc4j' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.OCRBK.dg' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'hisdb1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hisdb2'

CRS-2673: Attempting to stop 'ora.heal.db' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.OCRBK.dg' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.heal.db' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'hisdb2'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.hisdb2.vip' on 'hisdb2'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'hisdb1'

CRS-2677: Stop of 'ora.cvu' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.hisdb1.vip' on 'hisdb1'

CRS-2677: Stop of 'ora.heal.db' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'hisdb2'

CRS-2677: Stop of 'ora.heal.db' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'hisdb1'

CRS-2677: Stop of 'ora.hisdb2.vip' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.hisdb1.vip' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.oc4j' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.OCRBK.dg' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb2'

CRS-2677: Stop of 'ora.OCRBK.dg' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb1'

CRS-2677: Stop of 'ora.asm' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'hisdb2'

CRS-2677: Stop of 'ora.asm' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.ons' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'hisdb2'

CRS-2677: Stop of 'ora.net1.network' on 'hisdb2' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hisdb2' has completed

CRS-2673: Attempting to stop 'ora.ons' on 'hisdb1'

CRS-2677: Stop of 'ora.ons' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'hisdb1'

CRS-2677: Stop of 'ora.net1.network' on 'hisdb1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hisdb1' has completed

CRS-2677: Stop of 'ora.crsd' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.evmd' on 'hisdb2'

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb2'

CRS-2677: Stop of 'ora.crsd' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.evmd' on 'hisdb1'

CRS-2673: Attempting to stop 'ora.asm' on 'hisdb1'

CRS-2677: Stop of 'ora.evmd' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.evmd' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'hisdb2' succeeded

CRS-2677: Stop of 'ora.asm' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'hisdb2'

CRS-2677: Stop of 'ora.asm' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'hisdb1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'hisdb1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'hisdb1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'hisdb2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'hisdb2'

CRS-2677: Stop of 'ora.cssd' on 'hisdb1' succeeded

CRS-2677: Stop of 'ora.cssd' on 'hisdb2' succeeded

[root@hisdb1 ~]# /u01/app/11.2.0/grid/bin/crsctl start cluster -all 

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hisdb1'

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hisdb2'

CRS-2676: Start of 'ora.cssdmonitor' on 'hisdb1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'hisdb1'

CRS-2676: Start of 'ora.cssdmonitor' on 'hisdb2' succeeded

CRS-2672: Attempting to start 'ora.diskmon' on 'hisdb1'

CRS-2672: Attempting to start 'ora.cssd' on 'hisdb2'

CRS-2672: Attempting to start 'ora.diskmon' on 'hisdb2'

CRS-2676: Start of 'ora.diskmon' on 'hisdb1' succeeded

CRS-2676: Start of 'ora.diskmon' on 'hisdb2' succeeded

CRS-2676: Start of 'ora.cssd' on 'hisdb1' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'hisdb1'

CRS-2676: Start of 'ora.cssd' on 'hisdb2' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'hisdb1'

CRS-2672: Attempting to start 'ora.ctssd' on 'hisdb2'

CRS-2676: Start of 'ora.ctssd' on 'hisdb2' succeeded

CRS-2672: Attempting to start 'ora.evmd' on 'hisdb2'

CRS-2676: Start of 'ora.ctssd' on 'hisdb1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'hisdb2'

CRS-2672: Attempting to start 'ora.evmd' on 'hisdb1'

CRS-2676: Start of 'ora.evmd' on 'hisdb2' succeeded

CRS-2676: Start of 'ora.evmd' on 'hisdb1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'hisdb1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'hisdb1'

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'hisdb2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'hisdb2'

CRS-2676: Start of 'ora.asm' on 'hisdb1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'hisdb1'

CRS-2676: Start of 'ora.asm' on 'hisdb2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'hisdb2'

CRS-2676: Start of 'ora.crsd' on 'hisdb1' succeeded

CRS-2676: Start of 'ora.crsd' on 'hisdb2' succeeded

说明：集群能成功开启，但无法打开实例，因为实例的相关数据文件全在data磁盘组.

3.4、相关异常

[grid@hisdb1 ~]$ sqlplus / as sysasm

 

SQL*Plus: Release 11.2.0.4.0 Production on Tue Dec 27 22:46:21 2022

 

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

 

 

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Real Application Clusters and Automatic Storage Management options

 

SQL> col name for a20

SQL> col path for a40

SQL> set line 160

SQL> select name,total_mb,usable_file_mb,state from v$asm_diskgroup;

 

NAME                   TOTAL_MB USABLE_FILE_MB STATE

-------------------- ---------- -------------- -----------

FRA                       10239           6624 MOUNTED

OCRBK                     10239           9843 MOUNTED

SQL> alter diskgroup data mount;

alter diskgroup data mount

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "DATA" cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

说明：可以看到data磁盘无法挂载.

[grid@hisdb1 hisdb1]$ tail -5000f alerthisdb1.log

2022-12-27 22:46:18.033:

[crsd(10020)]CRS-2807:Resource 'ora.DATA.dg' failed to start automatically.

2022-12-27 22:46:18.033:

[crsd(10020)]CRS-2807:Resource 'ora.DATA.dg' failed to start automatically.

2022-12-27 22:46:18.033:

[crsd(10020)]CRS-2807:Resource 'ora.heal.db' failed to start automatically.

2022-12-27 22:46:18.033:

[crsd(10020)]CRS-2807:Resource 'ora.heal.db' failed to start automatically.

说明：集群告警日志如上.

SQL> select group_number,name,path,state,total_mb,free_mb from v$asm_disk;

 

GROUP_NUMBER NAME   PATH  STATE      TOTAL_MB    FREE_MB

------------ -------------------- --------------- -------- ---------- ----------

           0            ORCL:DATA01     NORMAL            0          0

           0            ORCL:DATA03     NORMAL            0          0

           2 DATA02     ORCL:DATA02     NORMAL        10239       6624

           3 DATA04     ORCL:DATA04     NORMAL        10239       9843

[grid@hisdb2 hisdb2]$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS      

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

               ONLINE  OFFLINE      hisdb1                                      

               ONLINE  OFFLINE      hisdb2                                      

ora.FRA.dg

               ONLINE  ONLINE       hisdb1                                      

               ONLINE  ONLINE       hisdb2                                      

ora.LISTENER.lsnr

               ONLINE  ONLINE       hisdb1                                      

               ONLINE  ONLINE       hisdb2                                      

ora.OCRBK.dg

               ONLINE  ONLINE       hisdb1                                      

               ONLINE  ONLINE       hisdb2                                      

ora.asm

               ONLINE  ONLINE       hisdb1                   Started             

               ONLINE  ONLINE       hisdb2                   Started            

ora.gsd

               OFFLINE OFFLINE      hisdb1                                      

               OFFLINE OFFLINE      hisdb2                                       

ora.net1.network

               ONLINE  ONLINE       hisdb1                                      

               ONLINE  ONLINE       hisdb2                                      

ora.ons

               ONLINE  ONLINE       hisdb1                                       

               ONLINE  ONLINE       hisdb2                                      

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       hisdb2                                      

ora.cvu

      1        ONLINE  ONLINE       hisdb1                                      

ora.heal.db

      1        ONLINE  OFFLINE                               Instance Shutdown  

      2        ONLINE  OFFLINE                               Instance Shutdown  

ora.hisdb1.vip

      1        ONLINE  ONLINE       hisdb1                                      

ora.hisdb2.vip

      1        ONLINE  ONLINE       hisdb2                                      

ora.oc4j

      1        ONLINE  ONLINE       hisdb1                                      

ora.orcl.db

      1        OFFLINE OFFLINE                               Instance Shutdown  

      2        OFFLINE OFFLINE                               Instance Shutdown  

ora.scan1.vip

      1        ONLINE  ONLINE       hisdb2       

说明：集群状态显示异常，heal数据库无法开启.

3.5、恢复data磁盘

[grid@hisdb1 ~]$ kfed repair /dev/oracleasm/disks/DATA03

[grid@hisdb1 ~]$ sqlplus / as sysasm

 

SQL*Plus: Release 11.2.0.4.0 Production on Tue Dec 27 22:54:47 2022

 

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

 

 

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Real Application Clusters and Automatic Storage Management options

 

SQL> alter diskgroup data mount;

 

Diskgroup altered.

SQL> select group_number,name,path,state,total_mb,free_mb from v$asm_disk;

 

GROUP_NUMBER NAME   PATH           STATE      TOTAL_MB    FREE_MB

------------ --------------- ------------------------- -------- ---------- ----------

           0          ORCL:DATA01        NORMAL            0          0

           2 DATA02  ORCL:DATA02        NORMAL        10239       6618

           1 DATA03  ORCL:DATA03        NORMAL        20479      13765

           3 DATA04  ORCL:DATA04        NORMAL        10239       9843

说明：data磁盘修复成功后，集群恢复正常.

 

参考文档：

https://www.modb.pro/db/22060

https://blog.csdn.net/jycjyc/article/details/106275991

https://blog.51cto.com/lhrbest/2699983