Tuning a RAID60 array for best performance.

[Nicolas Chipaux]

Hello,

My setup : PERC6/E adapter with one MD1120 enclosure on each connector full of 24 physical drives.
We manage this server with MegaCLI SAS RAID Management Tool  Ver 2.00.11 March 
10, 2008 and OMSA 5.4.0. For best performance, we switch Linux scheduler to cfq.

We try to get the best performance with a correct security on the RAID volume. 
That's why we choose RAID60. We are using orion (Oracle Database Load Simul.)
to benchmark random read/write in different data size.

On the RAID60 device under Debian Linux, we got :

# ./o -run advanced -testname t -num_disks 1 -type rand -matrix detailed \
        -size_small 8 -size_large 2048 -verbose 
Small IO size: 8 KB
Large IO size: 2048 KB
IO Types: Small Random IOs, Large Random IOs

Maximum Large MBPS=203.30 @ Small=0 and Large=2
Maximum Small IOPS=821 @ Small=5 and Large=0
Minimum Small Latency=5.75 @ Small=1 and Large=0


If we setup 2 RAID6 devices (still using Debian), we got : 

# ./o -testname 2vol -run normal -num_disks 2 
Small IO size: 8 KB
Large IO size: 1024 KB
IO Types: Small Random IOs, Large Random IOs

Maximum Large MBPS=189.81 @ Small=0 and Large=4
Maximum Small IOPS=1241 @ Small=10 and Large=0
Minimum Small Latency=7.68 @ Small=4 and Large=0


The latency is higher be we obtain a 50% performance bump. As RAID60 add a 
layer of stripping between the spans, how can IOPS decrease ?

Then, we wanted to test using different stripe size in the RAID6 span of the 
RAID60 array. As OMSA web interface doesn't provide such a feature, we try to 
use MegaCli. Let's just say that the documentation is pretty not existent.

MegaCli -CfgSpanAdd -r60 -Array1 [ 0:0, 0:1, 0:2, 0:3, 0:4, 0:5, 0:6, 0:7, \
0:8, 0:9, 0:10, 0:11, 0:12, 0:13, 0:14, 0:15, 0:16, 0:17, 0:18, 0:19, 0:20, \
0:21, 0:22  ] -Array2 [ 1:0, 1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 1:7, 1:8, 1:9, \
1:10, 1:11, 1:12, 1:13, 1:14, 1:15, 1:16, 1:17, 1:18, 1:19, 1:20, 1:21, \
1:22 ] WB ADRA Cached NoCachedBadBBU -sz 64 -sz 64 -strpsz 2 -a0

2 span of 23 physical drives in RAID60 array. We think "-sz" option could be \
span stripe size and -strpsz arrays stripe size, but can't get it working.

Does anyone successfully created a RAID60 array using MegaCli tools ?
How can we increase performance on Perc6/E for a RAID60 array ?



Regards.

-- 
Nicolas Chipaux
GANDI - http://www.gandi.net