Redundant NFS storage setup (part 3) : The disappointing PERC5/E

[Matthias Saou]

Hi,

I'm in the final stages of the file server setup I've already
mentioned twice on the list.

Right now, I'm still evaluating the production hardware :
- PowerEdge 1950 III with 16GB RAM and dual Quad Core E5410
- PowerVault MD1000 with 15 1TB SATAII drives

I initially configured the PERC5/E with a single RAID5 array of 14
disks and 1 global hot spare. I was VERY disappointed with the
overall I/O performance of the volume. So I tried with RAID10. Again,
performance was mediocre at best, including simple reads.

A few numbers. I use RHEL5 x86_64 and XFS for the filesystem :

Non cached 10GB file read :
 * RAID5 : 255MB/s
 * RAID10 : 224MB/s (???)
 * LVM : 306MB/s
 * RAID5 software : 329MB/s (during rebuild! dd taking 100% of one core)
 * RAID6 software : 319MB/s (during rebuild)

Cached 10GB file read (2nd read of the same file) :
 * RAID5 : 434MB/s
 * RAID10 : 437MB/s
 * RAID5 software : 463MB/s (dd taking 100% of one core)

Non cached 2 x 10GB files read :
 * RAID5 : 2 x 36MB/s
 * RAID10 : 2 x 31MB/s (ouch!)
 * LVM : 273 & 147MB/s
 * RAID5 software : 2 x 160MB/s (during rebuild)
 * RAID6 software : 2 x 160MB/s (during rebuild)

Notes :
 * Stripe size used is 128kB for the PERC volumes (the maximum).
 * Chunk size used is 128kB for the software RAID.
 * The RAID5 and RAID10 tests were done after the volume initialization
was over (it took 4 to 5h approx.)
 * LVM was a test where all disks were configured as a singled volume
each in the PERC, like for the software RAID, but they were all in a
single non-redundant volume group with 15 stripes, in order to test
"raw" parallel performance as much as possible.
 * Individual disks get 80MB/s reads with hdparm -t when idle.
 * I can't seem to go over 320-330MB/s with dd as it maxes out one CPU
core, but I can get to a sustained read of 500MB/s on software RAID with
cat... but it also maxes out a core at that point.
 * Single file read or write was okay with hardware RAID, but with 2 or
more concurrent accesses, performance dropped DANGEROUSLY. I really
can't figure out why, other than the PERC5/E being a lousy RAID
controller.
 * I tried with various options in the PERC. Write Through made things
even slower, so did Read Ahead. Adaptive Read Ahead seemed to make very
little difference. Disabling the "Cache" for the volumes also decreased
performance significantly. The numbers I've put are the best I could
achieve...

My conclusion is that the PERC5/E is plain lousy. I recall others
already complaining about MD1000 performance on this list... maybe they
were also using a PERC5/E. Right now, I'm replicating 1TB of random
data split in 100 10G files four times in parallel on the 12TB (once
formatted etc.) software RAID6 volume. The sustained speed for the last
few hours has been 40 to 60MB/s for both read and write, which is WAY
BETTER than anything I can expect from the PERC5/E RAID. FWIW,
md0_raid5 is taking ~20% of one CPU core.

I've read all of the PERC and MD1000 docs I could come across, so I
really don't think I'm missing anything obvious. Maybe something a
little more obscure... but right now, it seems that Linux's software
RAID will be the way to go, as the performance gain will be worth it.
The only downside I can think of in my case is hot swapping the drives,
which might not be as easy... I still need to test that.

I hope this might help others... comments are welcome, especially about
similar experiences!

Matthias

-- 
Clean custom Red Hat Linux rpm packages : http://freshrpms.net/
Fedora release 8 (Werewolf) - Linux kernel 2.6.23.8-63.fc8
Load : 0.98 1.00 0.98