SAS 6/iR write performance depends on kernel version, blade server m600
Dominique LALOT
dom.lalot at gmail.com
Wed May 28 07:24:33 CDT 2008
Just some news and may be the last of that thread.
We followed that link:
http://pocitace.tomasek.cz/SAS5iRperf/
Get the lsiutil program and found much more possibilities than playng
with BIOS, we then activated write cache.
The kernel said: Write cache enabled, but still complains about support
DPO or FUA
We did some tests, write performance still stay low 70MB/s. May be it
has to do with DPO/FUA
As a conclusion:
Dell hardware is good, but the latest linux kernel above 2.6.18 are
not able to get full advantage of the hardware. I guess, it's probably
due to a better asking of firmware capabilities (I've seen a thread, but
forget it). But, (I presume) the default firmware parameters are not
very good and we fall back to very slow writes. I have no time to look
and patch the kernel..
On new blade m600: 70 MB/sec (last kernels) instead of 550MB/sec (2.6.18)
On old blades 1855 with 2.6.18: 236MB/sec
We were lucky to find an openvz kernel compiled with 64G for 2.6.18. But
what support will we get from gold if we don't follow the rules? Having
support or performance, that's our new problem
Hey, is there people at Dell or RedHat working on that problem? I know,
that's there is something between kernel developpers and hardware, but
if there's some benchmarck playing with Dell hardware on linux, there
will be some very bad reports, and I suppose that Dell wants to sell
hardware. For me, if nothing new arrives, I will post in french
academics mailing lists to other sysadmin to take care of the new I/O
and hardware. So that should become also a Dell problem.
-- Previous mail (I don't get mails I sent to the list..)
Hi,
Just to answer to myself, I give some more information, as we spent days
on this problem:
We tested Redhat AS 5 2.6.18, SUSE 2.6.16, Ubuntu 2.6.24 and sent
reports for Dell support...
We just confirm that all the kernels above 2.6.18 are working very badly
with sas 6. (2.6.22, 2.6.24 at least)
We checked the drivers, even getting them from LSI directly, same problem
Write cache is disabled, throuput fails to 10 percent of normal speed
(70MB vs 600MB). We hope that somebody will solve the problem without
waiting for a new redhat release (they will be obliged to get rid of
such problem)
More than 2 weeks for two persons to search..
Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
lspci:
SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express
Fusion-MPT SAS (rev 08)
So bad..
Also related:
http://lists.us.dell.com/pipermail/linux-precision/2007-June/001186.html
https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.20/+bug/185127
m1000e + 9 new blades waiting to get in production.. What solution can
we take?, I've no idea. We don't like redhat and the limited filesystem
support (ext3). On the debian/ubuntu side, we need speed, 8GB memory
recognition and openvz, so we have to compile a working kernel!
dell at bobich.net a écrit :
> More likely it's down to one of the patches that RH apply that makes
> it work better.
>
> RH put a lot of effort into making sure hardware from major vendors
> works as well as it should with their distro. To expect similar effort
> to go into smaller and less commercial efforts is misguided.
> Conversely, expecting support for more than RH (and possibly, at a
> push, SuSE in some parts of the world) from hardware vendors is a bit
> much to expect considering the amount of effort required.
>
> For the sake of education and diversity, unpack the RH source RPM and
> see just how many patches they roll into the vanilla kernel tree. That
> should give you the idea of just how much vendor effort goes into it.
>
> In general, I only ever use a non vendor supplied kernel when I have
> no choice. Luckily, that doesn't happen very often.
>
> Gordan
>
> On Mon, 19 May 2008, Dominique LALOT wrote:
>
>> Hello,
>>
>> That's quite a long time, I'm "playing" around a performance problem.
>>
>> We just acquire some new blades and I discovered something strange
>> affecting the new SAS 6
>> subsystem.
>>
>> I've just done a disk dump like this:
>> debian:~# dd if=/dev/zero of=test.cdrom bs=10k count=60000
>> 60000+0 records in
>> 60000+0 records out
>> 614400000 bytes (614 MB) copied, 1.12691 seconds, 545 MB/s
>>
>> Well not bad? But, etch has no good standard kernel if you have 8GB
>> of RAM and want to run
>> vservers. So I compiled a new kernel
>> 2.6.22.19-vs2.2.0.7
>> I ran the same test, and I fell to 70MB/s !!
>> So I installed kernel from backports, not better...
>>
>> Well, may be have a look to Ubuntu server as Dell (we hope) will
>> support it on our servers.
>> On Ubuntu 8.04 and 2.6.24-16-server same problem slow speed
>>
>> Well: installed redhat AS5 and check:
>> kernel 2.6.18 and performance OK
>>
>> So: Is 2.6.18 a "magic" kernel?
>>
>> On a good kernel (2.6.18)
>> debian:~# uname -a
>> Linux debian 2.6.18-6-vserver-686 #1 SMP Thu Apr 24 10:53:03
>> UTC 2008 i686
>> GNU/Linux
>>
>> SCSI subsystem initialized
>> Fusion MPT base driver 3.04.01
>> Copyright (c) 1999-2005 LSI Logic Corporation
>> Fusion MPT SAS Host driver 3.04.01
>> ACPI: PCI Interrupt 0000:08:00.0[A] -> GSI 16 (level, low) ->
>> IRQ 177
>> mptbase: Initiating ioc0 bringup
>> ioc0: SAS1068E: Capabilities={Initiator}
>> PCI: Setting latency timer of device 0000:08:00.0 to 64
>> scsi0 : ioc0: LSISAS1068E, FwRev=00143000h, Ports=1, MaxQ=511,
>> IRQ=177
>> Vendor: SEAGATE Model: ST9146802SS Rev: S229
>> Type: Direct-Access ANSI SCSI
>> revision: 05
>> Vendor: SEAGATE Model: ST9146802SS Rev: S229
>> Type: Direct-Access ANSI SCSI
>> revision: 05
>> Vendor: Dell Model: VIRTUAL DISK Rev: 1028
>> Type: Direct-Access ANSI SCSI
>> revision: 05
>> SCSI device sda: 285155328 512-byte hdwr sectors (146000 MB)
>> sda: Write Protect is off
>> sda: Mode Sense: 03 00 00 08
>> SCSI device sda: drive cache: write through
>> SCSI device sda: 285155328 512-byte hdwr sectors (146000 MB)
>> sda: Write Protect is off
>> sda: Mode Sense: 03 00 00 08
>> SCSI device sda: drive cache: write through
>> sda: sda1 sda2 < sda5 >
>> sd 0:1:0:0: Attached scsi disk sda
>>
>> On a "bad kernel"
>> root at ubuntu-test:~# dd if=/dev/zero of=test.cdrom bs=10k
>> count=60000
>> 60000+0 records in
>> 60000+0 records out
>> 614400000 bytes (614 MB) copied, 8,39494 s, 73,2 MB/s
>> root at ubuntu-test:~# uname -a
>> Linux ubuntu-test 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00
>> UTC 2008 i686
>> GNU/Linux
>>
>> [ 96.845125] Fusion MPT base driver 3.04.06
>> [ 96.845127] Copyright (c) 1999-2007 LSI Corporation
>> [ 96.850189] Fusion MPT SAS Host driver 3.04.06
>> [ 96.942269] ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 20
>> (level, low) -> IRQ
>> 21
>> [ 99.599489] ioc0: LSISAS1068E B3: Capabilities={Initiator}
>> [ 115.985384] sd 4:1:0:0: [sdb] 285155328 512-byte hardware
>> sectors (146000 MB)
>> [ 115.985610] sd 4:1:0:0: [sdb] Write Protect is off
>> [ 115.985613] sd 4:1:0:0: [sdb] Mode Sense: 03 00 00 08
>> [ 115.986031] sd 4:1:0:0: [sdb] Write cache: disabled, read
>> cache: enabled,
>> doesn't support DPO or FUA
>> [ 115.986494] sd 4:1:0:0: [sdb] 285155328 512-byte hardware
>> sectors (146000 MB)
>> [ 115.986721] sd 4:1:0:0: [sdb] Write Protect is off
>> [ 115.986724] sd 4:1:0:0: [sdb] Mode Sense: 03 00 00 08
>> [ 115.987144] sd 4:1:0:0: [sdb] Write cache: disabled, read
>> cache: enabled,
>> doesn't support DPO or FUA
>> [ 115.987148] sdb: sdb1 sdb2 < sdb5 >
>> [ 116.006731] sd 4:1:0:0: [sdb] Attached SCSI disk
>> [ 116.006773] sd 4:1:0:0: Attached scsi generic sg4 type 0
>>
>> What we see is a bad negociation between kernel and BIOS about the
>> device capabilities
>>
>> It's evident that write caching is not enabled on some kernel. Is
>> there a specific patch or
>> kernel parameters (I checked /boot/config files without success)
>>
>> We tried sdparm to change write caching without success too!
>>
>> Any ideas?
>>
>> Thanks
>>
>> Dominique
>>
>>
>>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Linux-PowerEdge mailing list
> Linux-PowerEdge at dell.com
> http://lists.us.dell.com/mailman/listinfo/linux-poweredge
> Please read the FAQ at http://lists.us.dell.com/faq
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20080528/38011632/attachment.htm
More information about the Linux-PowerEdge
mailing list