案例说明:
通过openfiler虚拟机模拟iscsi server建立存储共享,测试多路径(multipath)配置。
测试架构:
iscsi server网络配置:
一、通过firefox浏览器访问openfiler服务配置
1、连接服务配置失败
2、修改TLS认证级别
3、连接访问openfiler配置服务
二、配置openfiler iscsi服务
1、启动iscsi target服务
2、创建共享卷
1)创建物理卷
2)创建卷组
3)创建逻辑卷
逻辑卷信息:
3、iscsi target配置
1)创建lun
2)映射逻辑卷
3)配置允许访问存储的客户端
允许访问的客户端信息:
4)配置network ACL
注释信息:
4、查看openfiler服务信息
如下所示,openfiler服务端已经启动iscsi server服务,访问端口为3260:
二、客户端访问配置
1、发现iscsi server服务
[root@node202 network-scripts]# iscsiadm -m discovery -t sendtargets -p 192.168.1.11:3260
192.168.1.11:3260,1 iqn.2006-01.com.openfiler:tsn.a979784803e4
192.168.2.11:3260,1 iqn.2006-01.com.openfiler:tsn.a979784803e4
192.168.1.11:3260,1 iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299
192.168.2.11:3260,1 iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299
可以看到,openfiler返回了2个target,每个target 2条路径. 执行命令后,在/var/lib/iscsi/目录生成了如下文件:
[root@node202 192.168.1.11,3260]# pwd
/var/lib/iscsi/send_targets/192.168.1.11,3260[root@node202 192.168.1.11,3260]# ls -lh
total 4.0K
lrwxrwxrwx 1 root root 83 Dec 31 17:50 iqn.2006-01.com.openfiler:tsn.a979784803e4,192.168.1.11,3260,1,default -> /var/lib/iscsi/nodes/iqn.2006-01.com.openfiler:tsn.a979784803e4/192.168.1.11,3260,1
lrwxrwxrwx 1 root root 83 Dec 31 17:50 iqn.2006-01.com.openfiler:tsn.a979784803e4,192.168.2.11,3260,1,default -> /var/lib/iscsi/nodes/iqn.2006-01.com.openfiler:tsn.a979784803e4/192.168.2.11,3260,1
lrwxrwxrwx 1 root root 83 Dec 31 17:50 iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299,192.168.1.11,3260,1,default -> /var/lib/iscsi/nodes/iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299/192.168.1.11,3260,1
lrwxrwxrwx 1 root root 83 Dec 31 17:50 iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299,192.168.2.11,3260,1,default -> /var/lib/iscsi/nodes/iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299/192.168.2.11,3260,1
-rw------- 1 root root 585 Dec 31 17:50 st_config
2、客户端获取存储信息
如下所示,通过lsblk查看客户端的存储信息,在登陆iscsi server后,存储信息增加了共享存储:
[root@node202 192.168.1.11,3260]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 102.9G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 102.4G 0 part├─centos-root 253:0 0 50G 0 lvm /├─centos-swap 253:1 0 3G 0 lvm [SWAP]└─centos-home 253:2 0 49.3G 0 lvm /home
sdb 8:16 0 60.6G 0 disk
sdc 8:32 0 512M 0 disk
sdd 8:48 0 60.6G 0 disk
sde 8:64 0 128M 0 disk
sdf 8:80 0 128M 0 disk
sdg 8:96 0 10.7G 0 disk
sdh 8:112 0 2.2G 0 disk
sdi 8:128 0 128M 0 disk
sr0 11:0 1 1024M 0 rom# 登录iscsi server访问共享存储[root@node202 192.168.1.11,3260]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.a979784803e4 -p 192.168.1.11 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.a979784803e4, portal: 192.168.1.11,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.a979784803e4, portal: 192.168.1.11,3260] successful.
[root@node202 192.168.1.11,3260]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.a979784803e4 -p 192.168.2.11 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.a979784803e4, portal: 192.168.2.11,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.a979784803e4, portal: 192.168.2.11,3260] successful.
[root@node202 192.168.1.11,3260]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 102.9G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 102.4G 0 part├─centos-root 253:0 0 50G 0 lvm /├─centos-swap 253:1 0 3G 0 lvm [SWAP]└─centos-home 253:2 0 49.3G 0 lvm /home
sdb 8:16 0 60.6G 0 disk
sdc 8:32 0 512M 0 disk
sdd 8:48 0 60.6G 0 disk
sde 8:64 0 128M 0 disk
sdf 8:80 0 128M 0 disk
sdg 8:96 0 10.7G 0 disk
sdh 8:112 0 2.2G 0 disk
sdi 8:128 0 128M 0 disk
sdj 8:144 0 5G 0 disk
sdk 8:160 0 5.2G 0 disk
sdl 8:176 0 5G 0 disk
sdm 8:192 0 5.2G 0 disk
sr0 11:0 1 1024M 0 rom[root@node202 192.168.1.11,3260]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299 -p 192.168.1.11 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299, portal: 192.168.1.11,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299, portal: 192.168.1.11,3260] successful.
[root@node202 192.168.1.11,3260]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299 -p 192.168.2.11 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299, portal: 192.168.2.11,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.b9ae2d3b9299, portal: 192.168.2.11,3260] successful.[root@node202 192.168.1.11,3260]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 102.9G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 102.4G 0 part├─centos-root 253:0 0 50G 0 lvm /├─centos-swap 253:1 0 3G 0 lvm [SWAP]└─centos-home 253:2 0 49.3G 0 lvm /home
sdb 8:16 0 60.6G 0 disk
sdc 8:32 0 512M 0 disk
sdd 8:48 0 60.6G 0 disk
sde 8:64 0 128M 0 disk
sdf 8:80 0 128M 0 disk
sdg 8:96 0 10.7G 0 disk
sdh 8:112 0 2.2G 0 disk
sdi 8:128 0 128M 0 disk
sdj 8:144 0 5G 0 disk
sdk 8:160 0 5.2G 0 disk
sdl 8:176 0 5G 0 disk
sdm 8:192 0 5.2G 0 disk
sr0 11:0 1 1024M 0 rom
三、配置客户端多路径访问
1、查看存储设备SCSI_ID
如下所示,磁盘sdj和sdl具有相同的scsi_id,sdk和sdm具有相同的scsi_id:
[root@node202 192.168.1.11,3260]# udevadm info /sys/class/block/sdj |grep -i scsi
S: disk/by-id/scsi-14f504e46494c4552486f683057302d7a3446562d47513369
S: disk/by-path/ip-192.168.1.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-14f504e46494c4552486f683057302d7a3446562d47513369 /dev/disk/by-path/ip-192.168.1.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-0
E: ID_BUS=scsi
E: ID_PATH=ip-192.168.1.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-0
E: ID_PATH_TAG=ip-192_168_1_11_3260-iscsi-iqn_2006-01_com_openfiler_tsn_a979784803e4-lun-0
E: ID_SCSI=1
E: ID_SCSI_SERIAL=Hoh0W0-z4FV-GQ3i[root@node202 192.168.1.11,3260]# udevadm info /sys/class/block/sdk |grep -i scsi
S: disk/by-id/scsi-14f504e46494c4552374d4a7233742d5546444e2d42693539
S: disk/by-path/ip-192.168.1.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-1
E: DEVLINKS=/dev/disk/by-id/scsi-14f504e46494c4552374d4a7233742d5546444e2d42693539 /dev/disk/by-path/ip-192.168.1.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-1
E: ID_BUS=scsi
E: ID_PATH=ip-192.168.1.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-1
E: ID_PATH_TAG=ip-192_168_1_11_3260-iscsi-iqn_2006-01_com_openfiler_tsn_a979784803e4-lun-1
E: ID_SCSI=1
E: ID_SCSI_SERIAL=7MJr3t-UFDN-Bi59[root@node202 192.168.1.11,3260]# udevadm info /sys/class/block/sdl |grep -i scsi
S: disk/by-id/scsi-14f504e46494c4552486f683057302d7a3446562d47513369
S: disk/by-path/ip-192.168.2.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-14f504e46494c4552486f683057302d7a3446562d47513369 /dev/disk/by-path/ip-192.168.2.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-0
E: ID_BUS=scsi
E: ID_PATH=ip-192.168.2.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-0
E: ID_PATH_TAG=ip-192_168_2_11_3260-iscsi-iqn_2006-01_com_openfiler_tsn_a979784803e4-lun-0
E: ID_SCSI=1
E: ID_SCSI_SERIAL=Hoh0W0-z4FV-GQ3i[root@node202 192.168.1.11,3260]# udevadm info /sys/class/block/sdm |grep -i scsi
S: disk/by-id/scsi-14f504e46494c4552374d4a7233742d5546444e2d42693539
S: disk/by-path/ip-192.168.2.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-1
E: DEVLINKS=/dev/disk/by-id/scsi-14f504e46494c4552374d4a7233742d5546444e2d42693539 /dev/disk/by-path/ip-192.168.2.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-1
E: ID_BUS=scsi
E: ID_PATH=ip-192.168.2.11:3260-iscsi-iqn.2006-01.com.openfiler:tsn.a979784803e4-lun-1
E: ID_PATH_TAG=ip-192_168_2_11_3260-iscsi-iqn_2006-01_com_openfiler_tsn_a979784803e4-lun-1
E: ID_SCSI=1
E: ID_SCSI_SERIAL=7MJr3t-UFDN-Bi59
2、启动multipathd服务
# 创建multipath.conf空文件
[root@node202 192.168.1.11,3260]# touch /etc/multipath.conf# 启动multipathd服务
[root@node202 192.168.1.11,3260]# systemctl start multipathd.service
[root@node202 192.168.1.11,3260]# systemctl status multipathd.service
● multipathd.service - Device-Mapper Multipath Device ControllerLoaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)Active: active (running) since Tue 2024-12-31 18:12:57 CST; 3s agoProcess: 5057 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)Process: 5054 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)Process: 5048 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)Main PID: 5060 (multipathd)Tasks: 9Memory: 3.2MCGroup: /system.slice/multipathd.service└─5060 /sbin/multipathdDec 31 18:12:57 node202 multipathd[5060]: VBOX_HARDDISK_VB4d5f890b-fdc168fc: ignoring map
Dec 31 18:12:57 node202 multipathd[5060]: VBOX_HARDDISK_VB4709e33e-0e72190a: load table [0 127001664 mu...6 1]
Dec 31 18:12:57 node202 multipathd[5060]: 14f504e46494c4552486f683057302d7a3446562d47513369: load table...6 1]
Dec 31 18:12:57 node202 multipathd[5060]: 14f504e46494c4552374d4a7233742d5546444e2d42693539: load table...2 1]
Dec 31 18:12:57 node202 multipathd[5060]: VBOX_HARDDISK_VB4709e33e-0e72190a: event checker started
Dec 31 18:12:57 node202 multipathd[5060]: 14f504e46494c4552486f683057302d7a3446562d47513369: event chec...rted
Dec 31 18:12:57 node202 multipathd[5060]: 14f504e46494c4552374d4a7233742d5546444e2d42693539: event chec...rted
Dec 31 18:12:57 node202 multipathd[5060]: path checkers start up
Dec 31 18:12:57 node202 multipathd[5060]: dm-3: remove map (uevent)
Dec 31 18:12:57 node202 multipathd[5060]: dm-3: remove map (uevent)
Hint: Some lines were ellipsized, use -l to show in full.
3、查看多路径配置
[root@node202 192.168.1.11,3260]# multipath -ll
14f504e46494c4552486f683057302d7a3446562d47513369 dm-4 OPNFILER,VIRTUAL-DISK
size=5.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 6:0:0:0 sdj 8:144 active ready running
`-+- policy='service-time 0' prio=1 status=enabled`- 7:0:0:0 sdl 8:176 active ready running
14f504e46494c4552374d4a7233742d5546444e2d42693539 dm-5 OPNFILER,VIRTUAL-DISK
size=5.2G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 6:0:0:1 sdk 8:160 active ready running
`-+- policy='service-time 0' prio=1 status=enabled`- 7:0:0:1 sdm 8:192 active ready running
VBOX_HARDDISK_VB4709e33e-0e72190a dm-3 ATA ,VBOX HARDDISK
size=61G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active`- 3:0:0:0 sdb 8:16 active ready running[root@node202 192.168.1.11,3260]# ls -lh /dev/mapper/
total 0
lrwxrwxrwx 1 root root 7 Dec 31 18:12 14f504e46494c4552374d4a7233742d5546444e2d42693539 -> ../dm-5
lrwxrwxrwx 1 root root 7 Dec 31 18:12 14f504e46494c4552486f683057302d7a3446562d47513369 -> ../dm-4
lrwxrwxrwx 1 root root 7 Dec 31 14:39 centos-home -> ../dm-2
lrwxrwxrwx 1 root root 7 Dec 31 14:39 centos-root -> ../dm-0
lrwxrwxrwx 1 root root 7 Dec 31 14:39 centos-swap -> ../dm-1
crw------- 1 root root 10, 236 Dec 31 14:39 control
lrwxrwxrwx 1 root root 7 Dec 31 18:12 VBOX_HARDDISK_VB4709e33e-0e72190a -> ../dm-3[root@node202 192.168.1.11,3260]# ls -lh /dev/dm-5
brw-rw---- 1 root disk 253, 5 Dec 31 18:12 /dev/dm-5
[root@node202 192.168.1.11,3260]# ls -lh /dev/dm-4
brw-rw---- 1 root disk 253, 4 Dec 31 18:12 /dev/dm-4