iscsi offload on rh el based servers guide?

Gianluca Cecchi gianluca.cecchi at gmail.com
Fri May 14 10:40:16 CDT 2010


Hello,
I found several guides (both Red Hat and Dell) about iscsi configuration,
but it seems to me some doubts remain about configuration itself and
functionality when using iscsi offload....

I can test two M610 blades, where I have RH EL 5.5 and RH EL 6 beta.
Both are x86_64 systems.
Below information from rh el 5.5 server
lspci gives:
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet (rev 20)
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet (rev 20)
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711
10-Gigabit PCIe
05:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711
10-Gigabit PCIe

I was able to configure the two 10Gbit interfaces to connect to several
volumes on a Dell EQL 6010XV storage array, using sw based iscsi.
Also, I was able to configure dm-multipath so that each volume is seen by
two paths.

Some raw performance data show not more than 130-140MB/s (both by using
single devices and multipath devices) where other Windows based blades,
configured with Broadcom offload, reach more than 300MB/s on the same
storage array.

Trying to configure offload scsi on rh el I have these doubts:

My 10Gb BCM57711 card has two mac addresses (actually I have two of these
cards in total, to bind to the EQL).
00:10:18:58:E8:F8 base eth card for network
00:10:18:58:E8:F9 for iscsi offload (not seen from ifconfig, for example)

1) Do I have to configure an IP for both eth card and for the iscsi offload
mac? If so must they be different?
Or if I use offload, the nic is not assigned an ip at all?

2) Do the nic has to be configured anyway to start in
/etc/sysconfig/network-scripts/ifcfg-eth0?
With something like this for example if without an ip:
# Broadcom Corporation NetXtreme II BCM57711 10-Gigabit PCIe
DEVICE=eth0
HWADDR=00:10:18:58:E8:F8
ONBOOT=yes
BOOTPROTO=static
TYPE=Ethernet
MTU=9000

3) Can I set MTU=9000 if using offload scsi? Is it supported?
If I don't have instead to set the nic to start at boot, where to put the
mtu parameter?

4) I configured this way the iscsi offload iface, under
/var/lib/iscsi/ifaces dir:

# cat bnx2i.00:10:18:58:e8:f9
# BEGIN RECORD 2.0-871
iface.iscsi_ifacename = bnx2i.00:10:18:58:e8:f9
iface.ipaddress = 10.10.100.178
iface.hwaddress = 00:10:18:58:e8:f9
iface.transport_name = bnx2i
# END RECORD

and this for the nic associated:
[root at orasvi2 ifaces]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:10:18:58:E8:F8
          inet addr:10.10.100.174  Bcast:10.10.100.255  Mask:255.255.255.0
          inet6 addr: fe80::210:18ff:fe58:e8f8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:1683 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:155976 (152.3 KiB)  TX bytes:4562 (4.4 KiB)
          Interrupt:114 Memory:dc800000-dcffffff

I have this results:
[root at orasvi2 ifaces]# iscsiadm -m node -P1
Target:
iqn.2001-05.com.equallogic:0-8a0906-97d4b5e06-596000000264bc83-blg9-vol3
Portal: 10.10.100.30:3260,1
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Name: bnx2i.00:10:18:58:e8:fb
Target:
iqn.2001-05.com.equallogic:0-8a0906-8904b5e06-b66000000204bc83-blg9-vol1
Portal: 10.10.100.30:3260,1
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Name: bnx2i.00:10:18:58:e8:fb
Target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2
Portal: 10.10.100.30:3260,1
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Name: bnx2i.00:10:18:58:e8:fb

But actually I can connect only with one card and only to one node..... ;-(
[root at orasvi2 ifaces]# iscsiadm -m session -P1
Target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2
Current Portal: 10.10.100.32:48140,1
Persistent Portal: 10.10.100.30:3260,1
**********
Interface:
**********
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Transport: bnx2i
Iface Initiatorname: iqn.1994-05.com.redhat:ab59ecbce6c
Iface IPaddress: <empty>
Iface HWaddress: 00:10:18:58:e8:f9
Iface Netdev: eth0
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE

Note the
Iface IPaddress: <empty> ...

Just a basic test shows, for this single out of three volumes:
 dd if=/dev/sdb of=/dev/null bs=10240k count=9500

[root at orasvi2 ~]# vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
wa st
 1  1     24 139100 46577740  43056    0    0  3513     2  203  533  0  0 97
 2  0
 2  0     24 139660 46576220  42552    0    0 108117    23 3196 6280  0  2
87 11  0
 1  1     24 140576 46575508  42676    0    0 120064     0 3496 6508  0  2
87 11  0
 1  1     24 137440 46580264  42196    0    0 117888     0 3410 6429  0  2
87 11  0
 2  0     24 141512 46577092  41888    0    0 134741    17 3956 6979  0  2
87 11  0
 1  1     24 138568 46580280  41944    0    0 117376     0 3508 6458  0  2
87 11  0
 0  1     24 137620 46581736  41404    0    0 124629     0 3740 6088  0  2
87 11  0
 0  1     24 137148 46582524  41144    0    0 116651    11 3512 5570  0  2
87 11  0
 0  1     24 141428 46578336  40524    0    0 130560    32 3879 6844  0  2
87 11  0
 0  1     24 145376 46574348  41268    0    0 118997     0 3590 6512  0  2
87 11  0
 0  1     24 136468 46583624  40696    0    0 119467     0 3590 5904  0  2
87 11  0

Logs I can see:
May 14 16:35:10 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:10 orasvi2 iscsid: Received iferror -19
May 14 16:35:10 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-19,11)
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.01]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.01]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 iscsid: Received iferror -101
May 14 16:35:11 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-101,11)
May 14 16:35:11 orasvi2 iscsid: Received iferror -19
May 14 16:35:11 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-19,11)
May 14 16:35:11 orasvi2 iscsid: Received iferror -101
May 14 16:35:11 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-101,11)
May 14 16:35:12 orasvi2 kernel: bnx2i [05:00.01]: ISCSI_INIT passed
May 14 16:35:12 orasvi2 iscsid: Received iferror -101
May 14 16:35:12 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-101,11)
May 14 16:35:13 orasvi2 kernel:  connection1:0: detected conn error (1011)
May 14 16:35:13 orasvi2 iscsid: Login authentication failed with target
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2
May 14 16:35:15 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:17 orasvi2 kernel:  connection1:0: bnx2i: conn update - MBL
0x40000 FBL 0x10000MRDSL_I 0x40000 MRDSL_T 0x10000
May 14 16:35:17 orasvi2 kernel:   Vendor: EQLOGIC   Model: 100E-00
Rev: 4.3
May 14 16:35:17 orasvi2 kernel:   Type:   Direct-Access
 ANSI SCSI revision: 05
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: 419450880 512-byte hdwr
sectors (214759 MB)
May 14 16:35:17 orasvi2 kernel: sdb: Write Protect is off
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: drive cache: write through
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: 419450880 512-byte hdwr
sectors (214759 MB)
May 14 16:35:17 orasvi2 kernel: sdb: Write Protect is off
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: drive cache: write through
May 14 16:35:17 orasvi2 kernel:  sdb: unknown partition table
May 14 16:35:17 orasvi2 kernel: sd 2:0:0:0: Attached scsi disk sdb
May 14 16:35:17 orasvi2 multipathd: sdb: add path (uevent)
May 14 16:35:17 orasvi2 kernel: sd 2:0:0:0: Attached scsi generic sg3 type 0
May 14 16:35:17 orasvi2 kernel: device-mapper: multipath round-robin:
version 1.0.0 loaded
May 14 16:35:17 orasvi2 multipathd: vol2: load table [0 419450880 multipath
1 queue_if_no_path 0 1 1 round-robin 0 1 1 8:16 10]
May 14 16:35:17 orasvi2 multipathd: vol2: event checker started
May 14 16:35:17 orasvi2 multipathd: dm-3: add map (uevent)
May 14 16:35:17 orasvi2 multipathd: dm-3: devmap already registered
May 14 16:35:17 orasvi2 iscsid: connection1:0 is operational now
May 14 16:35:17 orasvi2 iscsid: Could not write to
/sys/bus/scsi/devices/2:0:0:0/queue_depth. Invalid permissions.
May 14 16:35:17 orasvi2 iscsid: Could not queue depth for LUN 0 err 13.

and
# multipath -l
vol2 (36090a068e0b5c49483bc3402000090df) dm-3 EQLOGIC,100E-00
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 2:0:0:0 sdb 8:16  [active][undef]


If I now try to connect with the other card ending fb mac .175 ip for eth1
and .179 for the iscsi part):
# iscsiadm --mode node --portal 10.10.100.30:3260 -I bnx2i.00:10:18:58:e8:fb
--login
Logging in to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-97d4b5e06-596000000264bc83-blg9-vol3,
portal: 10.10.100.30,3260]
Logging in to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-8904b5e06-b66000000204bc83-blg9-vol1,
portal: 10.10.100.30,3260]
Logging in to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2,
portal: 10.10.100.30,3260]
iscsiadm: Could not login to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-97d4b5e06-596000000264bc83-blg9-vol3,
portal: 10.10.100.30,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not login to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-8904b5e06-b66000000204bc83-blg9-vol1,
portal: 10.10.100.30,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not login to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2,
portal: 10.10.100.30,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)

with this in messages:
May 14 17:28:59 orasvi2 kernel: bnx2i [05:00.01]: ISCSI_INIT passed
May 14 17:29:00 orasvi2 last message repeated 2 times
May 14 17:29:00 orasvi2 iscsid: Received iferror -101
May 14 17:29:00 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-101,11)
May 14 17:29:00 orasvi2 iscsid: Received iferror -101
May 14 17:29:00 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-101,11)
May 14 17:29:00 orasvi2 iscsid: Received iferror -101
May 14 17:29:00 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 (-101,11)

Any insight or guide?

M610 fw information:
Bios 2.0.13
network firmware (from NETW_FRMW_LX_R259547.BIN applied)
Release Date:
February 12, 2010
Default Log File Name: R259547

Thanks in advance for any help and/or pointers,
Gianluca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20100514/7ca666de/attachment-0001.htm 


More information about the Linux-PowerEdge mailing list