vmware-linux-multipath多路径实验-构建iscsi模拟环境用于搭建oracle rac

multipath多路径实验-构建iscsi模拟环境
1.网络环境：
客户端和服务端分别配置两个网卡，分别为两个网段。

2.软件环境：
服务端：
[root@cy01 /]# yum -y install scsi-target-utils
客户端：
[root@rac1 ~]# yum -y install iscsi-initiator-utils

3.模拟存储加盘
[root@cy01 ~]# pvcreate /dev/sdd
[root@cy01 /]# vgcreate ora /dev/sdd
[root@cy01 /]# lvcreate -L 199g -n lvora ora

4.配置ISCSI服务器
vi /etc/tgt/targets.conf 配置文件 ，内容如下：
<target iqn.2014-07.dev.iscsi-target:iscsidisk> #target名字自定义
backing-store /dev/sdb1 #/dev/sdb1分区名称
</target>

<target iqn.2022-09.dev.iscsi-target:iscsidisk>
backing-store /dev/ora/lvora
</target>

重新启动服务并设置开机自启：
[root@cy01 /]# service tgtd restart
[root@cy01 /]# chkconfig tgtd on
[root@cy01 /]# chkconfig --list --如果没有就设置systemctl
[root@cy01 /]# systemctl enable tgtd
systemctl list-unit-files|grep tgtd

查询lun相关信息：
[root@cy01 /]# tgt-admin --show
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 213675 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /dev/ora/lvora
Backing store flags:
Account information:
ACL information:
ALL
查询使用端口：
[root@cy01 /]# netstat -tlunp|grep tgt
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 14602/tgtd
tcp6 0 0 :::3260 :::* LISTEN 14602/tgtd

5.配置ISCSI客户端
配置服务开机自启：
[root@rac1 ~]# systemctl status iscsid.service
[root@rac1 ~]# systemctl enable iscsid.service
systemctl list-unit-files|grep iscsid.service

使用iscsiadm命令扫描服务端的LUN
[root@rac1 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.9.35
192.168.9.35:3260,1 iqn.2022-09.dev.iscsi-target:iscsidisk
[root@rac1 ~]# iscsiadm -m discovery -t sendtargets -p 2.0.9.35
2.0.9.35:3260,1 iqn.2022-09.dev.iscsi-target:iscsidisk

查询/var/lib/iscsi/nodes/下文件：
[root@rac1 ~]# ll -R /var/lib/iscsi/nodes/
/var/lib/iscsi/nodes/:
total 0
drw------- 4 root root 56 Sep 22 15:01 iqn.2022-09.dev.iscsi-target:iscsidisk

'/var/lib/iscsi/nodes/iqn.2022-09.dev.iscsi-target:iscsidisk':
total 0
drwx------ 2 root root 21 Sep 22 14:59 192.168.9.35,3260,1
drwx------ 2 root root 21 Sep 22 15:01 2.0.9.35,3260,1

'/var/lib/iscsi/nodes/iqn.2022-09.dev.iscsi-target:iscsidisk/192.168.9.35,3260,1':
total 4
-rw------- 1 root root 2153 Sep 22 14:59 default

'/var/lib/iscsi/nodes/iqn.2022-09.dev.iscsi-target:iscsidisk/2.0.9.35,3260,1':
total 4
-rw------- 1 root root 2145 Sep 22 15:01 default

[root@rac1 ~]# iscsiadm -m node
192.168.9.35:3260,1 iqn.2022-09.dev.iscsi-target:iscsidisk
2.0.9.35:3260,1 iqn.2022-09.dev.iscsi-target:iscsidisk

登录target：
[root@rac1 ~]# iscsiadm -m node -T iqn.2022-09.dev.iscsi-target:iscsidisk --login
Logging in to [iface: default, target: iqn.2022-09.dev.iscsi-target:iscsidisk, portal: 192.168.9.35,3260]
Logging in to [iface: default, target: iqn.2022-09.dev.iscsi-target:iscsidisk, portal: 2.0.9.35,3260]
Login to [iface: default, target: iqn.2022-09.dev.iscsi-target:iscsidisk, portal: 192.168.9.35,3260] successful.
Login to [iface: default, target: iqn.2022-09.dev.iscsi-target:iscsidisk, portal: 2.0.9.35,3260] successful.

[root@rac1 ~]# fdisk -l
Disk /dev/sdh: 199 GiB, 213674622976 bytes, 417333248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdg: 199 GiB, 213674622976 bytes, 417333248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
重启服务器之后发现依然存在

6.客户端配置多路径
[root@rac1 ~]# ll /var/lib/iscsi/nodes/
total 0
drw------- 4 root root 56 Sep 22 15:01 iqn.2022-09.dev.iscsi-target:iscsidisk

multipath安装：
[root@rac1 ~]# yum -y install device-mapper device-mapper-multipath
创建multipath配置文件：
[root@rac1 ~]# /sbin/mpathconf --enable
添加配置策略：
[root@rac1 ~]# vi /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
--find_multipaths yes --多路径忽略本地磁盘
重启服务
[root@rac1 ~]# service multipathd restart
Redirecting to /bin/systemctl restart multipathd.service

绑定多路径wwid
注意：没有绑定路径uuid时，客户端重启可能会导致盘符发生变化

查询多路径状态：
[root@rac1 ~]# multipath -ll
mpatha (360000000000000000e00000000010001) dm-3 IET,VIRTUAL-DISK
size=199G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 3:0:0:1 sdh 8:112 active ready running
`- 4:0:0:1 sdg 8:96 active ready running
#[root@rac1 ~]# ll /dev/mapper/mpatha
[root@rac2 ~]# ll /dev/mapper/asm_data01
lrwxrwxrwx 1 root root 7 Sep 23 08:59 /dev/mapper/mpatha -> ../dm-3
#[root@rac1 ~]# cat /etc/multipath/bindings
[root@rac2 ~]# cat /etc/multipath/wwids
# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# Format:
# alias wwid
#
mpatha 360000000000000000e00000000010001
[root@rac1 ~]# ll /dev/disk/by-id/
lrwxrwxrwx 1 root root 10 Sep 23 08:59 scsi-360000000000000000e00000000010001 -> ../../dm-3
[root@rac1 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Sep 23 08:59 ip-192.168.9.35:3260-iscsi-iqn.2022-09.dev.iscsi-target:iscsidisk-lun-1 -> ../../sdh
lrwxrwxrwx 1 root root 9 Sep 23 08:59 ip-2.0.9.35:3260-iscsi-iqn.2022-09.dev.iscsi-target:iscsidisk-lun-1 -> ../../sdg
获取uuid（存储为wwid，linux为uuid，unix为wwid）
[root@rac1 ~]# /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdh
360000000000000000e00000000010001
[root@rac1 ~]# /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdg
360000000000000000e00000000010001

排除sda配置路径：
[root@rac1 ~]# vi /etc/multipath.conf
blacklist {
devnode "^sda"
}

multipaths {
multipath {
wwid "360000000000000000e00000000010001"
alias asm-data01
}
}
激活多路径：
[root@rac1 ~]# multipath -F --刷新未使用的多路径
[root@rac1 ~]# multipath -v2 --重新扫描磁盘
create: asm_ocr01 (360000000000000000e00000000010001) undef IET,VIRTUAL-DISK
size=199G features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
|- 3:0:0:1 sdh 8:112 undef ready running
`- 4:0:0:1 sdg 8:96 undef ready running
[root@rac1 ~]# multipath -ll --再次查询
asm_ocr01 (360000000000000000e00000000010001) dm-3 IET,VIRTUAL-DISK
size=199G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 3:0:0:1 sdh 8:112 active ready running
`- 4:0:0:1 sdg 8:96 active ready running
#重载multipath
#[root@rac1 ~]# systemctl restart multipathd

修改文件99-my-asmdevices.rules
[root@rac1 ~]# vi /etc/udev/rules.d/99-my-asmdevices.rules
KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="360000000000000000e00000000010001", RUN+="/bin/sh -c 'mknod /dev/asmdiskg b $major $minor; chown grid:asmadmin /dev/asmdiskg; chmod 0660 /dev/asmdiskg'"

使用如下命令重新加载UDEV生效
/sbin/udevadm control --reload-rules
/sbin/udevadm trigger --type=devices --action=change

查询是否生效：
[root@rac1 ~]# ll /dev/asm*