SAS 6/iR write performance depends on kernel version, blade server m600

Dominique LALOT dom.lalot at gmail.com
Thu May 22 11:05:36 CDT 2008


Hi,

Just to answer to myself, I give some more information, as we spent days 
on this problem:

We tested Redhat AS 5 2.6.18, SUSE 2.6.16, Ubuntu 2.6.24 and sent 
reports for Dell support...
We just confirm that all the kernels above 2.6.18 are working very badly 
with sas 6. (2.6.22, 2.6.24 at least)

We checked the drivers, even getting them from LSI directly, same problem

Write cache is disabled, throuput fails to 10 percent of normal speed 
(70MB vs 600MB). We hope that somebody will solve the problem without 
waiting for a new redhat release (they will be obliged to get rid of 
such problem)
More than 2 weeks for two persons to search..

Write cache: disabled, read cache: enabled,  doesn't support DPO or FUA

lspci:
SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express 
Fusion-MPT SAS (rev 08)

So bad..

Also related:
http://lists.us.dell.com/pipermail/linux-precision/2007-June/001186.html
https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.20/+bug/185127

m1000e + 9 new blades waiting to get in production.. What solution can 
we take?, I've no idea. We don't like redhat and the limited filesystem  
support (ext3). On the debian/ubuntu side, we need speed, 8GB memory 
recognition and  openvz, so we have to compile a working kernel!

dell at bobich.net a écrit :
> More likely it's down to one of the patches that RH apply that makes 
> it work better.
>
> RH put a lot of effort into making sure hardware from major vendors 
> works as well as it should with their distro. To expect similar effort 
> to go into smaller and less commercial efforts is misguided. 
> Conversely, expecting support for more than RH (and possibly, at a 
> push, SuSE in some parts of the world) from hardware vendors is a bit 
> much to expect considering the amount of effort required.
>
> For the sake of education and diversity, unpack the RH source RPM and 
> see just how many patches they roll into the vanilla kernel tree. That 
> should give you the idea of just how much vendor effort goes into it.
>
> In general, I only ever use a non vendor supplied kernel when I have 
> no choice. Luckily, that doesn't happen very often.
>
> Gordan
>
> On Mon, 19 May 2008, Dominique LALOT wrote:
>
>> Hello,
>>
>> That's quite a long time, I'm "playing" around a performance problem.
>>
>> We just acquire some  new blades and  I discovered something strange 
>> affecting the new SAS 6
>> subsystem.
>>
>> I've just done a disk dump like this:
>> debian:~# dd if=/dev/zero of=test.cdrom  bs=10k count=60000
>> 60000+0 records in
>> 60000+0 records out
>> 614400000 bytes (614 MB) copied, 1.12691 seconds, 545 MB/s
>>
>> Well not bad? But, etch has no good standard kernel if you have 8GB 
>> of RAM and want to run
>> vservers. So I compiled a new kernel
>> 2.6.22.19-vs2.2.0.7
>> I ran the same test, and I fell to 70MB/s !!
>> So I installed kernel from backports, not better...
>>
>> Well, may be have a look to Ubuntu server as Dell (we hope) will 
>> support it on our servers.
>> On Ubuntu 8.04 and 2.6.24-16-server same problem slow speed
>>
>> Well: installed redhat AS5 and check:
>> kernel 2.6.18 and performance OK
>>
>> So: Is 2.6.18 a "magic" kernel?
>>
>> On a good kernel (2.6.18)
>>       debian:~# uname -a
>>       Linux debian 2.6.18-6-vserver-686 #1 SMP Thu Apr 24 10:53:03 
>> UTC 2008 i686
>>       GNU/Linux
>>
>>       SCSI subsystem initialized
>>       Fusion MPT base driver 3.04.01
>>       Copyright (c) 1999-2005 LSI Logic Corporation
>>       Fusion MPT SAS Host driver 3.04.01
>>       ACPI: PCI Interrupt 0000:08:00.0[A] -> GSI 16 (level, low) -> 
>> IRQ 177
>>       mptbase: Initiating ioc0 bringup
>>       ioc0: SAS1068E: Capabilities={Initiator}
>>       PCI: Setting latency timer of device 0000:08:00.0 to 64
>>       scsi0 : ioc0: LSISAS1068E, FwRev=00143000h, Ports=1, MaxQ=511, 
>> IRQ=177
>>         Vendor: SEAGATE   Model: ST9146802SS       Rev: S229
>>         Type:   Direct-Access                      ANSI SCSI 
>> revision: 05
>>         Vendor: SEAGATE   Model: ST9146802SS       Rev: S229
>>         Type:   Direct-Access                      ANSI SCSI 
>> revision: 05
>>         Vendor: Dell      Model: VIRTUAL DISK      Rev: 1028
>>         Type:   Direct-Access                      ANSI SCSI 
>> revision: 05
>>       SCSI device sda: 285155328 512-byte hdwr sectors (146000 MB)
>>       sda: Write Protect is off
>>       sda: Mode Sense: 03 00 00 08
>>       SCSI device sda: drive cache: write through
>>       SCSI device sda: 285155328 512-byte hdwr sectors (146000 MB)
>>       sda: Write Protect is off
>>       sda: Mode Sense: 03 00 00 08
>>       SCSI device sda: drive cache: write through
>>        sda: sda1 sda2 < sda5 >
>>       sd 0:1:0:0: Attached scsi disk sda
>>
>> On a "bad kernel"
>>       root at ubuntu-test:~# dd if=/dev/zero of=test.cdrom  bs=10k 
>> count=60000
>>       60000+0 records in
>>       60000+0 records out
>>       614400000 bytes (614 MB) copied, 8,39494 s, 73,2 MB/s
>>       root at ubuntu-test:~# uname -a
>>       Linux ubuntu-test 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 
>> UTC 2008 i686
>>       GNU/Linux
>>
>>       [   96.845125] Fusion MPT base driver 3.04.06
>>       [   96.845127] Copyright (c) 1999-2007 LSI Corporation
>>       [   96.850189] Fusion MPT SAS Host driver 3.04.06
>>       [   96.942269] ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 20 
>> (level, low) -> IRQ
>>       21
>>       [   99.599489] ioc0: LSISAS1068E B3: Capabilities={Initiator}
>>       [  115.985384] sd 4:1:0:0: [sdb] 285155328 512-byte hardware 
>> sectors (146000 MB)
>>       [  115.985610] sd 4:1:0:0: [sdb] Write Protect is off
>>       [  115.985613] sd 4:1:0:0: [sdb] Mode Sense: 03 00 00 08
>>       [  115.986031] sd 4:1:0:0: [sdb] Write cache: disabled, read 
>> cache: enabled,
>>       doesn't support DPO or FUA
>>       [  115.986494] sd 4:1:0:0: [sdb] 285155328 512-byte hardware 
>> sectors (146000 MB)
>>       [  115.986721] sd 4:1:0:0: [sdb] Write Protect is off
>>       [  115.986724] sd 4:1:0:0: [sdb] Mode Sense: 03 00 00 08
>>       [  115.987144] sd 4:1:0:0: [sdb] Write cache: disabled, read 
>> cache: enabled,
>>       doesn't support DPO or FUA
>>       [  115.987148]  sdb: sdb1 sdb2 < sdb5 >
>>       [  116.006731] sd 4:1:0:0: [sdb] Attached SCSI disk
>>       [  116.006773] sd 4:1:0:0: Attached scsi generic sg4 type 0
>>
>> What we see is a bad negociation between kernel and BIOS about the 
>> device capabilities
>>
>> It's evident that write caching is not enabled on some kernel. Is 
>> there a specific patch or
>> kernel parameters (I checked /boot/config files without success)
>>
>> We tried sdparm to change write caching without success too!
>>
>> Any ideas?
>>
>> Thanks
>>
>> Dominique
>>
>>
>>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Linux-PowerEdge mailing list
> Linux-PowerEdge at dell.com
> http://lists.us.dell.com/mailman/listinfo/linux-poweredge
> Please read the FAQ at http://lists.us.dell.com/faq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20080522/a474b71d/attachment.htm 


More information about the Linux-PowerEdge mailing list