文档课题：向现有集群环境添加新磁盘组——ASMLIB场景.

系统：centos 7.9 64位

数据库：oracle 11.2.0.4 64位

环境：rac(双节点) + dg + asmlib创建磁盘组

1、新增磁盘

节点1添加1块10g磁盘，节点2添加节点1新增的磁盘.

1.1、磁盘分区

节点1对新增磁盘分区.

[root@hisdb1 ~]# lsblk -p

NAME                        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT

/dev/sda                      8:0    0  100G  0 disk 

├─/dev/sda1                   8:1    0    1G  0 part  /boot

└─/dev/sda2                   8:2    0   99G  0 part 

  ├─/dev/mapper/centos-root 253:0    0   95G  0 lvm   /

  └─/dev/mapper/centos-swap 253:1    0    4G  0 lvm   [SWAP]

/dev/sdb                      8:16   0   10G  0 disk 

├─/dev/sdb1                   8:17   0   10G  0 part 

└─/dev/mapper/ocr           253:2    0   10G  0 mpath

  └─/dev/mapper/ocr1        253:8    0   10G  0 part 

/dev/sdc                      8:32   0   10G  0 disk 

├─/dev/sdc1                   8:33   0   10G  0 part 

└─/dev/mapper/fra           253:3    0   10G  0 mpath

  └─/dev/mapper/fra1        253:6    0   10G  0 part 

/dev/sdd                      8:48   0   20G  0 disk 

├─/dev/sdd1                   8:49   0   20G  0 part 

└─/dev/mapper/data          253:5    0   20G  0 mpath

  └─/dev/mapper/data1       253:7    0   20G  0 part 

/dev/sde                      8:64   0   10G  0 disk 

└─/dev/mapper/mpatha        253:4    0   10G  0 mpath

/dev/sr0                     11:0    1  4.4G  0 rom   /run/media/grid/CentOS 7 x86_64

说明：/dev/sde为新增磁盘.

[root@hisdb1 ~]# fdisk /dev/sde

Welcome to fdisk (util-linux 2.23.2).

 

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

 

Device does not contain a recognized partition table

Building a new DOS disklabel with disk identifier 0x1b499d05.

 

Command (m for help): n

Partition type:

   p   primary (0 primary, 0 extended, 4 free)

   e   extended

Select (default p):

Using default response p

Partition number (1-4, default 1):

First sector (2048-20971519, default 2048):

Using default value 2048

Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):

Using default value 20971519

Partition 1 of type Linux and of size 10 GiB is set

 

Command (m for help): w

The partition table has been altered!

 

Calling ioctl() to re-read partition table.

Syncing disks.

[root@hisdb1 ~]# partprobe

[root@hisdb2 ~]# partprobe

[root@hisdb1 ~]# lsblk -p

NAME                        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT

/dev/sda                      8:0    0  100G  0 disk 

├─/dev/sda1                   8:1    0    1G  0 part  /boot

└─/dev/sda2                   8:2    0   99G  0 part 

  ├─/dev/mapper/centos-root 253:0    0   95G  0 lvm   /

  └─/dev/mapper/centos-swap 253:1    0    4G  0 lvm   [SWAP]

/dev/sdb                      8:16   0   10G  0 disk 

├─/dev/sdb1                   8:17   0   10G  0 part 

└─/dev/mapper/ocr           253:2    0   10G  0 mpath

  └─/dev/mapper/ocr1        253:8    0   10G  0 part 

/dev/sdc                      8:32   0   10G  0 disk 

├─/dev/sdc1                   8:33   0   10G  0 part 

└─/dev/mapper/fra           253:3    0   10G  0 mpath

  └─/dev/mapper/fra1        253:6    0   10G  0 part 

/dev/sdd                      8:48   0   20G  0 disk 

├─/dev/sdd1                   8:49   0   20G  0 part 

└─/dev/mapper/data          253:5    0   20G  0 mpath

  └─/dev/mapper/data1       253:7    0   20G  0 part 

/dev/sde                      8:64   0   10G  0 disk 

├─/dev/sde1                   8:65   0   10G  0 part 

└─/dev/mapper/mpatha        253:4    0   10G  0 mpath

  └─/dev/mapper/mpatha1     253:9    0   10G  0 part 

/dev/sr0                     11:0    1  4.4G  0 rom   /run/media/grid/CentOS 7 x86_64

1.2、asmlib判断

判断磁盘组配置方式.

[grid@hisdb1 ~]$ ls -ltr /dev/oracleasm/disks

total 0

brw-rw---- 1 grid asmadmin 8, 17 Nov  5 16:11 DATA01

brw-rw---- 1 grid asmadmin 8, 49 Nov  5 16:11 DATA03

brw-rw---- 1 grid asmadmin 8, 33 Nov  5 16:11 DATA02

说明：磁盘组配置方式为asmlib.

1.3、多路径

1.3.1、多路径显示

[root@hisdb1 ~]# multipath -ll

mpatha (36000c29087f5e11b96a53b025d0e8d60) dm-4 VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:4:0 sde 8:64 active ready running

fra (36000c2935008c75902f7cbbe7ee6cb1a) dm-3 VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:2:0 sdc 8:32 active ready running

data (36000c294842ccadd084847980127e55a) dm-5 VMware, ,VMware Virtual S

size=20G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:3:0 sdd 8:48 active ready running

ocr (36000c29d0b8efae055435249dec3dfad) dm-2 VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:1:0 sdb 8:16 active ready running

1.3.2、获取wwid

获取新磁盘wwid.

[root@hisdb1 disks]# /usr/lib/udev/scsi_id -g -u /dev/sde

36000c29087f5e11b96a53b025d0e8d60

1.3.3、配置multipath

双节点向/etc/multipath.conf文件添加以下内容.

  multipath {

  wwid "36000c29087f5e11b96a53b025d0e8d60"

  alias orcbk

  }

1.3.4、激活multipath

说明：双节点激活multipath多路径.

[root@hisdb1 ~]# multipath -F

[root@hisdb1 ~]# multipath -v2

create: ocr (36000c29d0b8efae055435249dec3dfad) undef VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=undef

`-+- policy='service-time 0' prio=1 status=undef

  `- 0:0:1:0 sdb 8:16 undef ready running

create: fra (36000c2935008c75902f7cbbe7ee6cb1a) undef VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=undef

`-+- policy='service-time 0' prio=1 status=undef

  `- 0:0:2:0 sdc 8:32 undef ready running

create: data (36000c294842ccadd084847980127e55a) undef VMware, ,VMware Virtual S

size=20G features='0' hwhandler='0' wp=undef

`-+- policy='service-time 0' prio=1 status=undef

  `- 0:0:3:0 sdd 8:48 undef ready running

create: orcbk (36000c29087f5e11b96a53b025d0e8d60) undef VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=undef

`-+- policy='service-time 0' prio=1 status=undef

  `- 0:0:4:0 sde 8:64 undef ready running

[root@hisdb1 ~]# multipath -ll

fra (36000c2935008c75902f7cbbe7ee6cb1a) dm-4 VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:2:0 sdc 8:32 active ready running

data (36000c294842ccadd084847980127e55a) dm-6 VMware, ,VMware Virtual S

size=20G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:3:0 sdd 8:48 active ready running

orcbk (36000c29087f5e11b96a53b025d0e8d60) dm-8 VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:4:0 sde 8:64 active ready running

ocr (36000c29d0b8efae055435249dec3dfad) dm-2 VMware, ,VMware Virtual S

size=10G features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 0:0:1:0 sdb 8:16 active ready running

[root@hisdb1 ~]# lsblk -p

NAME                        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT

/dev/sda                      8:0    0  100G  0 disk 

├─/dev/sda1                   8:1    0    1G  0 part  /boot

└─/dev/sda2                   8:2    0   99G  0 part 

  ├─/dev/mapper/centos-root 253:0    0   95G  0 lvm   /

  └─/dev/mapper/centos-swap 253:1    0    4G  0 lvm   [SWAP]

/dev/sdb                      8:16   0   10G  0 disk 

├─/dev/sdb1                   8:17   0   10G  0 part 

└─/dev/mapper/ocr           253:2    0   10G  0 mpath

  └─/dev/mapper/ocr1        253:3    0   10G  0 part 

/dev/sdc                      8:32   0   10G  0 disk 

├─/dev/sdc1                   8:33   0   10G  0 part 

└─/dev/mapper/fra           253:4    0   10G  0 mpath

  └─/dev/mapper/fra1        253:5    0   10G  0 part 

/dev/sdd                      8:48   0   20G  0 disk 

├─/dev/sdd1                   8:49   0   20G  0 part 

└─/dev/mapper/data          253:6    0   20G  0 mpath

  └─/dev/mapper/data1       253:7    0   20G  0 part 

/dev/sde                      8:64   0   10G  0 disk 

├─/dev/sde1                   8:65   0   10G  0 part 

└─/dev/mapper/orcbk         253:8    0   10G  0 mpath

  └─/dev/mapper/orcbk1      253:9    0   10G  0 part 

/dev/sr0                     11:0    1  4.4G  0 rom   /run/media/grid/CentOS 7 x86_64

2、创建新磁盘组

2.1、新建asm磁盘

节点1创建新asm磁盘。

[root@hisdb1 ~]# oracleasm createdisk data04 /dev/mapper/orcbk1

Writing disk header: done

Instantiating disk: done

双节点验证

[root@hisdb1 ~]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

[root@hisdb1 ~]# oracleasm listdisks

DATA01

DATA02

DATA03

DATA04

[root@hisdb2 ~]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

Instantiating disk "DATA04"

[root@hisdb2 ~]# oracleasm listdisks

DATA01

DATA02

DATA03

DATA04

2.2、创建新磁盘组

开启xmanager，节点1创建新磁盘组ocrbk.

[grid@hisdb1 ~]$ export DISPLAY=192.168.133.1:0.0

[grid@hisdb1 ~]$ asmca

3、验证

[grid@hisdb1 ~]$ asmcmd

ASMCMD> lsdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512   4096  1048576     20479    16490                0           16490              0             N  DATA/

MOUNTED  EXTERN  N         512   4096  1048576     10239     8957                0            8957              0             N  FRA/

MOUNTED  EXTERN  N         512   4096  1048576     10239     9843                0            9843              0             Y  OCR/

MOUNTED  EXTERN  N         512   4096  1048576     10239    10144                0           10144              0             N  OCRBK/

SQL> select group_number,path,voting_file from v$asm_disk

 

GROUP_NUMBER PATH                   V

------------ ------------------------- -

           3 ORCL:DATA01               Y

           2 ORCL:DATA02               N

           1 ORCL:DATA03               N

           4 ORCL:DATA04               N