Perc5/i: very poor write performance

John Workman johnw at voyageurweb.com
Mon Oct 24 14:25:31 CDT 2011


Perc5/i - Poor write performance

I have 3 PowerEdges that all exhibit this exact problem: two PE1950s and 
a PE2950. All have Dell PERC5/i with firmware 5.2.2-0072

The problem is that the write performance is very poor, and while 
writing, read latency is EXTREMELY poor.

These tests are run on a PE1950 with two 146GB 15krpm SAS drives in 
RAID1 (ST3146855SS)

# dd if=/dev/zero bs=1M count=1024 oflag=direct of=zeroes
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 27.0182 s, 39.7 MB/s

# dd if=/dev/zero bs=1M count=1024 of=zeroes
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 26.6586 s, 40.3 MB/s

While dd is writing, read IO performance is extremely poor, doing 'ls' 
in a directory can take more than 5 seconds. This is the worst part of 
this problem.

Also, the kernel reports the PERC VD as having write-cache off and no 
DPO or FUA:

[    1.618625] sd 0:2:0:0: [sda] 285474816 512-byte logical blocks: (146 
GB/136 GiB)
[    1.618892] sd 0:2:0:0: [sda] Write Protect is off
[    1.618956] sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
[    1.619064] sd 0:2:0:0: [sda] Write cache: disabled, read cache: 
enabled, doesn't support DPO or FUA
[    1.619670]  sda: sda1 sda2
[    1.620626] sd 0:2:0:0: [sda] Attached SCSI disk

I've tried:

    * Various kernels with different versions of megaraid_sas, including
      latest Ubuntu in x86_32 and x86_64.
    * Deleting my Virtual Disk in the controller BIOS and recreating the
      RAID
    * different IO schedulers, noop, cfq, deadline, etc.
    * Different cache policy settings in controller: writeback, write
      through, forced write back, etc.


I seem to remember having good write performance on these systems. The 
only change I've made to them that could have possibly caused this was 
upgrading the firmware to 5.2.2-0072, I believe from 5.2.1-0066. 
Downgrading this will be my next attempt, but it looks like 5.2.2-0072 
has been out for a few years so I kind of doubt this is the issue.

No matter what I do, I can't seem to get any better than ~50MB/s write 
throughput, I believe I should be seeing at least 100MB/s for 15k RPM 
SAS drives in RAID1. And I certainly shouldn't be seeing huge IO read 
latency while write streaming to the disks.

If any of you have any insight into this problem, I'd be very grateful 
for any suggestions. Thanks.


# omreport storage controller
 Controller  PERC 5/i Integrated (Embedded)

Controllers
ID                                            : 0
Status                                        : Ok
Name                                          : PERC 5/i Integrated
Slot ID                                       : Embedded
State                                         : Ready
Firmware Version                              : 5.2.2-0072
Minimum Required Firmware Version             : Not Applicable
Driver Version                                : 00.00.04.01

Minimum Required Driver Version               : Not Applicable
Number of Connectors                          : 2
Rebuild Rate                                  : 30%
BGI Rate                                      : 30%
Check Consistency Rate                        : 30%
Reconstruct Rate                              : 30%
Alarm State                                   : Not Applicable
Cluster Mode                                  : Not Applicable
SCSI Initiator ID                             : Not Applicable
Cache Memory Size                             : 256 MB
Patrol Read Mode                              : Auto
Patrol Read State                             : Stopped
Patrol Read Rate                              : 30%
Patrol Read Iterations                        : 0
Abort check consistency on error              : Not Applicable
Allow Revertible Hot Spare and Replace Member : Not Applicable
Auto replace member on predictive failure     : Not Applicable
Load balance                                  : Not Applicable


# omreport storage vdisk
List of Virtual Disks in the System

Controller PERC 5/i Integrated (Embedded)
ID                  : 0
Status              : Ok
Name                : vd0
State               : Ready
Progress            : Not Applicable
Layout              : RAID-1
Size                : 136.13 GB (146163105792 bytes)
Device Name         : /dev/sda
Bus Protocol        : SAS
Media               : HDD
Read Policy         : No Read Ahead
Write Policy        : Write Back
Cache Policy        : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy   : Enabled

BBU seems good:
# ./MegaCli -AdpBbuCmd -aAll                                        
                                                                                     

BBU status for Adapter: 
0                                                           

BatteryType: TBBU
Voltage: 4070 mV
Current: 0 mA   
Temperature: 16 C
Firmware Status: 00000000

Battery state:

GasGuageStatus:
  Fully Discharged        : No
  Fully Charged           : Yes
  Discharging             : Yes
  Initialized             : Yes
  Remaining Time Alarm    : No
  Remaining Capacity Alarm: No
  Discharge Terminated    : No
  Over Temperature        : No
  Charging Terminated     : No
  Over Charged            : No

Relative State of Charge: 99 %
Charger Status: Complete     
Remaining Capacity: 523 mAh  
Full Charge Capacity: 528 mAh
isSOHGood: Yes               

BBU Capacity Info for Adapter: 0

Relative State of Charge: 99 %
Absolute State of charge: 28 %
Remaining Capacity: 523 mAh
Full Charge Capacity: 528 mAh
Run time to empty: 65535 Min
Average time to empty: 65535 Min
Average Time to full: 65535 Min
Cycle Count: 36
Max Error: 0 %
Remaining Capacity Alarm: 190 mAh
Remaining Time Alarm: 10 Min


BBU Design Info for Adapter: 0

Date of Manufacture: 2,13/ 2007
Design Capacity: 1900 mAh
Design Voltage: 3700 mV
Specification Info: 49
Serial Number: 2830
Pack Stat Configuration: 0xe4bc
Manufacture Name: SANYO
Device Name: DLU8735
Device Chemistry: LION


BBU Properties for Adapter: 0

Auto Learn Period: 7776000 Sec
Next Learn time: 374471629 Sec
Learn Delay Interval:0 Hours
Auto-Learn Mode: Enabled

-- 
John Workman
VoyageurWeb Engineering
P.O. Box 205 Mankato, MN 56002-0205
(507) 344-2280
http://www.voyageurweb.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20111024/88096a84/attachment.html 


More information about the Linux-PowerEdge mailing list