PERC3/DC, PERC4/DC Write Performance
Richard, WhidbeyNet NOC
dellnoc at whidbey.net
Fri Jan 2 13:03:00 CST 2004
We purchased four dual-2.8Ghz 2650s, with PERC3/DC cards, along with
two PowerVault 220S arrays (Ultra320 version), to create two storage
clusters as recommended by Dell Power Solutions and Red Hat Enterprise.
Cluster testing went relatively smoothly. Failover worked under both
RHEL AS 2.1 and 3.0. However, we noticed writes to the shared storage
were slower than expected. So, we went back to a basic configuration
with factory defaults to diagnose the problem:
With a single 36gb-10K-U320 disk in the PV220S, connected to one 2650,
on RedHat 9 installed by the Open Manage CD (v7.5), no clustered SCSI,
a sequential write could be done at 20 MB/s.
Changing to "WRITETHRU" mode on a single drive, required by clustering,
sequential writes dropped to 10 MB/s. On a 5-disk RAID5, writes fell to
In comparison, we can write to a PE2650 single internal disk, with the
on-board Adaptec AIC-7899, at 35 MB/s:
sync; time dd ibs=1048576 obs=1048576 count=1024 if=/dev/zero
1024+0 records in 1024+0 records out
real 0m25.274s user 0m2.690s sys 0m11.730s
real 0m3.645s user 0m0.000s sys 0m0.040s
(1024 MB/28.9 seconds = 35.4 MB/s)
We tried the following to improve PERC3/DC writes, with little effect:
* Check logs and settings:
No failures or events in the BIOS, "messages", or megamon/raid monitor.
Disk, channel and "megaraid" all report U160 speed, proper termination.
* Change the defaults on the PERC3/DC (to):
Write Policy = WRBACK (WTHRU dropped peformance by 40%)
Read Policy = ADAPTIVE (tried NONE, READ AHEAD)
Cache Policy = DIRECTIO (tried CACHEDIO, which gave a small gain)
BIOS: Enabled (DISABLED, required by clustering)
Stripe size: 64k (others had no effect)
* Update all BIOS and Firmware:
PV220S revision E.14, Ultra320 Version
PE2650 version A15 through A17
PERC3/DC Dell firmware 1.92 (BIOS 3.31)
(also tried LSI Elite 1600 Firmware C170 through 111Q)
* Update software Megaraid driver
Tried versions 1.18h, 1.18j and the latest 2.00.9
* Change OS:
Tested with RedHat AS 2.1, RedHat 9.0, RedHat AS 3.0.
* Change disk models:
Atlas U320 IV 10K 36Gb are included. Also tried IBM Ultrastar U320 10K
* Change channels and cables:
Configured PV220S on Channel 1 instead of 0. New LVD cables tried.
* Change slots:
Moved the PERC3/DC from slot 1 (66mhz shared) to slot 3 (133mhz
* Change enclosures and bus modes:
Tested with a second PV220S in shared, split-bus and clustered modes.
Also tried with the internal backplane, without a PV220S. Same results.
* Change PowerEdges:
Two seperate PowerEdges tested individually.
After speaking with several Dell technicians, and escalating the issue,
we exchanged the PERC3/DC's for PERC4/DC's.
With a PERC4/DC, factory-defaulted sequential writes to a single disk
(without writethrough), went from 20 MB/s to nearly 40 MB/s. However,
just by switching to "writethrough", they dropped to 10 MB/s (8 MB/s on
We then spoke with an Advanced Software Support technician, who was
able to forward our results to the Dell Hardware Escalation team. They
concluded the following:
"The server and power vault are working as designed and we do not
expect a solution to the performance difference on this configuration."
We hope that Dell chooses to more truthfully advertise the performance
tradeoff of the cluster configuration they recommend. If more accurate
information were to be provided prior to sale, customers would no
longer be suprised by this issue.
More information about the Linux-PowerEdge