MD3000 has same bottleneck as MD1000!

[justin]

This June topic:
http://lists.us.dell.com/pipermail/linux-poweredge/2007-June/031502.html

shows that the MD1000 connected to a single SAS 5/E card cannot do more
than about 250mb/sec write speed no matter how many spindles one
throws at a LUN ...

I've been playing around (seemingly endlessly) with a new MD3000 fully
populated with SAS 146gb 15k drives, and two SAS 5/E cards connected
to a 1950 (PCI-X bus).
I find the same bottleneck even though I have two SAS 5/E cards, and 8
cores and 64bit SUSE 10.2
I think it might be the MD3000 controllers (there are two of them) that are
the bottleneck. The SAS card is an LSI 1068 and the spec sheet shows it
should be able to go over 1 gigabyte/sec.

Adding more disks to a 6 disk RAID10 LUN in an MD3000 does not
increase total read or write speed in any benchmark I can find..
Having the two HBAs did not improve much if anything over having one
card even though I tried  testing a md0 over two LUNs one allocated to
each controller. Going to a big RAID0 LUN hardly broke 350mb/sec
on IOZONE read.

Orion (oracle benchmark tool) maxes out at 400mb/sec total throughput with
100% read tests, and 250mb/sec total with 50:50 read write mix.  (Orion
uses the block devices so doesn't have filesystem layer). The cpu
on the host is hardly ticking over.

A read of the md3000 marketing boasts of agg 1400 mb/sec capacity and a very
high IOP max, but it is phrased such that this is probably out of the 
512mb cache,
not sustained. For read purposes, the cache is useless compared to linux 
read
cache. For sustained writes, it is useless too.

Even if I create 1 LUN per physical disk then strap them all together 
using mdadm
i can't improve nett throughput from the above figures.

Upgrading the SuSE 10.2 mptsas driver to the one Dell supplies, does not 
change
the results either.

Why have 14 drives that can read at 90mb/sec and write at
80mb/sec let alone any expansion arrays, 4 SAS cables at 300mb/sec each, and
four HBAs (two hosts) in the end the performance ceiling on sustained 
i/o appears
to be little more than the capacity of just one sas cable? A controller 
bottleneck
doesn't bode well for  random I/O over two hosts to two different disk 
groups
either (i'll be testing that asap).

-Justin