multipath配置及udev绑定

Openfiler安装配置multipath多路径作为Oracle ASM磁盘

一、软件准备

VMware? Workstation 15 Pro
openfileresa-2.99.2-x86_64-disc1.iso
rhel-server-7.7-x86_64-dvd.iso
二、Openfiler配置

在VMworkstation上Openfiler系统已经安装完成并且配置了两块块物理网卡用于实现存储多路径，相应的IP地址已经配置规划；

安装参考文档：https://www.modb.pro/doc/656
在虚拟机上添加虚拟硬盘

ocr_voting 5120M

redo 5120M

data 5120M

arch 5120M
在openfiler主机上扫盘

echo '- - -' >/sys/class/scsi_host/host0/scan

echo '- - -' >/sys/class/scsi_host/host1/scan

echo '- - -' >/sys/class/scsi_host/host2/scan

root@localhost host0]# lsblk
创建硬盘分区，配置raid，本测试由于本地空间不足，不配置raid；

Volumes-Block Devices,创建gpt类型磁盘分区

创建卷组

Volumes-Volume Groups

rac_arch_vg 4.75 GB

rac_data_vg 4.75 GB

rac_redo_vg 4.75 GB

rac_ocr+voting_vg 4.75 GB
创建逻辑卷

Volumes-Add Volumes

启动iSCSI目标服务器

点击 iSCSI Target 后面的 Enable 和 Start ，把 iSCSI 打开，并设置自启动

创建iSCSI Targets，并配置访问控制列表

Volumes-iSCSI Targets

Add new iSCSI Target

LUNs mapped to target: iqn.2006-01.com.openfiler:tsn.813052860478

Network Access Configuration

在下方的网络访问配置里把两块网卡的地址段都配置进去，要配置地址段，不要配置地址Name 可以根据实际需要填写即可

三、客户端配置

两个客户端主机分别配置IP

nmcli conn show

nmcli dev st

nmcli conn add type ethernet ifname ens38 con-name ens38

nmcli conn mod ens38 ipv4.address 192.168.37.204 gw4 192.168.37.1 ipv4.method manual autoconnect yes
配置本地YUM源

--挂载系统镜像

mount -t iso9660 -o loop /dev/sr0 /media/

--设置开机启动，修改/etc/rc.d/rc.local文件

chmod +x /etc/rc.d/rc.local

echo "mount -t iso9660 -o loop /dev/sr0 /media/" >> /etc/rc.d/rc.local

--配置yum配置文件

touch /etc/yum.repos.d/os.repo
echo "[os]" >>/etc/yum.repos.d/os.repo
echo "name=install iso" >>/etc/yum.repos.d/os.repo
echo "baseurl=file:///media/">>/etc/yum.repos.d/os.repo
echo "enabled=1">>/etc/yum.repos.d/os.repo
echo "gpgcheck=0">>/etc/yum.repos.d/os.repo

--清理软件源

yum clean all

--生成元数据缓

yum makecache
安装iscsi-initiator-utils.x86_64.0.6.2.0.874-11.el7

yum install iscsi-initiator*

rpm -qa | grep iscsi-initiator
用iscsiadm命令检查网络存储服务器上的所有可用目标

iscsiadm -m discovery -t sendtargets -p 192.168.0.202
对扫描到的ISCSI识别存储进行登陆

iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.813052860478 -p 192.168.0.202:3260 -l

iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.813052860478 -p 192.168.37.202:3260 -l
修改iscsi软件开机扫描信息

vi /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.2006-01.com.openfiler:tsn.813052860478
查看ISCSI当前登陆会话是否正常

iscsiadm -m session
安装mulpiath多路径软件

--查看是否安装多路径软件包

rpm -qa | grep device-mapper-multipath

--如未安装，则安装device-mapper-multipath软件包

yum install device-mapper-multipath
设置开机自启动

systemctl enable multipathd.service

--确认是否已设置开机启动

systemctl list-unit-files |grep multipath
检查multipath模块是否加载成功，如未加载则进行初始化

--检查内核模块加载情况

lsmod|grep dm_multipath

--加载内核模块

modprobe dm-multipath

modprobe dm-round-robin
生成多路径配置文件,并启动multipathd服务

mpathconf --enable

systemctl start multipathd.service
查看并配置multipath配置文件

multipathd show config

–查看wwid

cat /etc/multipath/bindings

mpatha 14f504e46494c4552526674326c752d494e6e732d424c4c74

mpathb 14f504e46494c45527141734c56542d743130322d42374f50

mpathc 14f504e46494c45526c49643830372d726154482d54414247

mpathd 14f504e46494c45524467383979392d6a4630322d6e465052

--配置文件

vi /etc/multipath.conf 添加如下：

multipaths{
multipath{
wwid 14f504e46494c4552526674326c752d494e6e732d424c4c74
alias asm_ocr1
path_grouping_policy multibus
}
multipath{
wwid 14f504e46494c45527141734c56542d743130322d42374f50
alias asm_redo1
path_grouping_policy multibus
}
multipath{
wwid 14f504e46494c45526c49643830372d726154482d54414247
alias asm_data1
path_grouping_policy multibus
}
multipath{
wwid 14f504e46494c45524467383979392d6a4630322d6e465052
alias asm_arch1
path_grouping_policy multibus
}
}
重启multipathd服务，并查看

systemctl restart multipathd.service

multipath -ll

[root@localhost etc]# multipathd show paths
hcil dev dev_t pri dm_st chk_st dev_st next_check
2:0:0:0 sda 8:0 -1 undef undef unknown orphan
33:0:0:0 sdc 8:32 1 active ready running .......... 1/20
33:0:0:1 sde 8:64 1 active ready running .......... 1/20
33:0:0:2 sdg 8:96 1 active ready running .......... 1/20
33:0:0:3 sdi 8:128 1 active ready running .......... 1/20
34:0:0:0 sdb 8:16 1 active ready running .......... 1/20
34:0:0:1 sdd 8:48 1 active ready running .......... 1/20
34:0:0:2 sdf 8:80 1 active ready running .......... 1/20
34:0:0:3 sdh 8:112 1 active ready running .......... 1/20
[root@localhost etc]# multipathd show status
path checker states:
up 9

paths: 8
busy: False

ls -lrt /dev/mapper/asm*
lrwxrwxrwx. 1 root root 7 1月 6 02:41 /dev/mapper/asm_redo1 -> ../dm-3
lrwxrwxrwx. 1 root root 7 1月 6 02:41 /dev/mapper/asm_ocr1 -> ../dm-2
lrwxrwxrwx. 1 root root 7 1月 6 02:41 /dev/mapper/asm_data1 -> ../dm-4
lrwxrwxrwx. 1 root root 7 1月 6 02:41 /dev/mapper/asm_arch1 -> ../dm-5
将multipath.conf文件复制到第二个客户端节点，重启服务

scp multipath.conf 192.168.0.204:/etc

systemctl restart multipathd.service
对共享磁盘分区，采用 GPT 格式的分区以进行分区对齐

[root@localhost mapper]# parted asm_ocr1
GNU Parted 3.1
使用 /dev/dm-2
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) h
align-check TYPE Ncheck partition N for TYPE(min|opt) alignment
help [COMMAND] print general help, or help on COMMAND
mklabel,mktable LABEL-TYPE create a new disklabel (partition table)
mkpart PART-TYPE [FS-TYPE] START END make a partition
name NUMBER NAME name partition NUMBER as NAME
print [devices|free|list,all|NUMBER] display the partition table, available devices, free space, all found partitions, or a particular partition
quit exit program
rescue START END rescue a lost partition near START and END

resizepart NUMBER ENDresize partition NUMBER
rm NUMBERdelete partition NUMBER
select DEVICEchoose the device to edit
disk_set FLAG STATE change the FLAG on selected device
disk_toggle [FLAG] toggle the state of FLAG on selected device
set NUMBER FLAG STATEchange the FLAG on partition NUMBER
toggle [NUMBER [FLAG]] toggle the state of FLAG on partition NUMBER
unit UNITset the default unit to UNIT
version display the version number and copyright information of GNU Parted
(parted) mklabel
新的磁盘标签类型？ gpt
(parted) mkpart primary 8192s -1
(parted) p
Model: Linux device-mapper (multipath) (dm)
Disk /dev/dm-2: 5100MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name 标志
1 4194kB 5099MB 5095MB primary
在第二节点更新分区表

kpartx -a /dev/mapper/asm_ocr1

ls -lrt /dev/mapper/asm*
使用udev方式配置ASM磁盘

查询磁盘的dm_uuid

--查询DM_UUID
udevadm info --query=all --name=/dev/mapper/asm_ocr1p1 |grep -i DM_UUID

--批量获得DM_UUID

for i in asm_ocr1p1 asm_redo1p1 asm_data1p1 asm_arch1p1;
do
printf "%s %s\n" "$i" "$(udevadm info --query=all --name=/dev/mapper/$i |grep -i dm_uuid)";
done
创建新的udev规则文件

touch /etc/udev/rules.d/99-oracle-asmdevices.rules

--根据查到的dm_uuid配置规则如下：

KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c4552526674326c752d494e6e732d424c4c74",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c45527141734c56542d743130322d42374f50",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c45526c49643830372d726154482d54414247",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c45524467383979392d6a4630322d6e465052",OWNER="grid",GROUP="asmadmin",MODE="0660"
保存99-oracle-asmdevices.rules文件，重新加载udev生效

udevadm control --reload-rules

udevadm trigger --type=devices --action=change
查询dm-device对应的磁盘分区

for i in asm_ocr1p1 asm_redo1p1 asm_data1p1 asm_arch1p1;
do
printf "%s %s\n" "$i" "$(ls -ll /dev/mapper/$i)";
done
测试规则文件是否生效，使用udevadm test

udevadm test /sys/block/dm-6
确认分区对应的dm设备权限是否正确

[root@localhost dev]# ls -lrt dm-6