Redundant NFS storage setup (part 3) : The disappointing PERC5/E

[Jimmy Christensen]

Hi,

I would suggest testing with a stripe size of 64kb instead of 128kb. We 
are currently testing our HP servers with a P800 with similar 
dissapointment, but found out that anything above a stripe size of 64kb 
absolutely kills the performance (ofcourse no one would tell us that). I 
believe it's based on a similar chip that the PERC5/E has.

Jimmy

Matthias Saou wrote:
> Hi,
>
> I'm in the final stages of the file server setup I've already
> mentioned twice on the list.
>
> Right now, I'm still evaluating the production hardware :
> - PowerEdge 1950 III with 16GB RAM and dual Quad Core E5410
> - PowerVault MD1000 with 15 1TB SATAII drives
>
> I initially configured the PERC5/E with a single RAID5 array of 14
> disks and 1 global hot spare. I was VERY disappointed with the
> overall I/O performance of the volume. So I tried with RAID10. Again,
> performance was mediocre at best, including simple reads.
>
> A few numbers. I use RHEL5 x86_64 and XFS for the filesystem :
>
> Non cached 10GB file read :
>  * RAID5 : 255MB/s
>  * RAID10 : 224MB/s (???)
>  * LVM : 306MB/s
>  * RAID5 software : 329MB/s (during rebuild! dd taking 100% of one core)
>  * RAID6 software : 319MB/s (during rebuild)
>
> Cached 10GB file read (2nd read of the same file) :
>  * RAID5 : 434MB/s
>  * RAID10 : 437MB/s
>  * RAID5 software : 463MB/s (dd taking 100% of one core)
>
> Non cached 2 x 10GB files read :
>  * RAID5 : 2 x 36MB/s
>  * RAID10 : 2 x 31MB/s (ouch!)
>  * LVM : 273 & 147MB/s
>  * RAID5 software : 2 x 160MB/s (during rebuild)
>  * RAID6 software : 2 x 160MB/s (during rebuild)
>
> Notes :
>  * Stripe size used is 128kB for the PERC volumes (the maximum).
>  * Chunk size used is 128kB for the software RAID.
>  * The RAID5 and RAID10 tests were done after the volume initialization
> was over (it took 4 to 5h approx.)
>  * LVM was a test where all disks were configured as a singled volume
> each in the PERC, like for the software RAID, but they were all in a
> single non-redundant volume group with 15 stripes, in order to test
> "raw" parallel performance as much as possible.
>  * Individual disks get 80MB/s reads with hdparm -t when idle.
>  * I can't seem to go over 320-330MB/s with dd as it maxes out one CPU
> core, but I can get to a sustained read of 500MB/s on software RAID with
> cat... but it also maxes out a core at that point.
>  * Single file read or write was okay with hardware RAID, but with 2 or
> more concurrent accesses, performance dropped DANGEROUSLY. I really
> can't figure out why, other than the PERC5/E being a lousy RAID
> controller.
>  * I tried with various options in the PERC. Write Through made things
> even slower, so did Read Ahead. Adaptive Read Ahead seemed to make very
> little difference. Disabling the "Cache" for the volumes also decreased
> performance significantly. The numbers I've put are the best I could
> achieve...
>
> My conclusion is that the PERC5/E is plain lousy. I recall others
> already complaining about MD1000 performance on this list... maybe they
> were also using a PERC5/E. Right now, I'm replicating 1TB of random
> data split in 100 10G files four times in parallel on the 12TB (once
> formatted etc.) software RAID6 volume. The sustained speed for the last
> few hours has been 40 to 60MB/s for both read and write, which is WAY
> BETTER than anything I can expect from the PERC5/E RAID. FWIW,
> md0_raid5 is taking ~20% of one CPU core.
>
> I've read all of the PERC and MD1000 docs I could come across, so I
> really don't think I'm missing anything obvious. Maybe something a
> little more obscure... but right now, it seems that Linux's software
> RAID will be the way to go, as the performance gain will be worth it.
> The only downside I can think of in my case is hot swapping the drives,
> which might not be as easy... I still need to test that.
>
> I hope this might help others... comments are welcome, especially about
> similar experiences!
>
> Matthias
>
>