Slow RAID IO on PE2970

Jason Ede J.Ede at birchenallhowden.co.uk
Fri Mar 13 08:13:52 CDT 2009


We've got a PE2970 running openfiler 2.3 and 750GB SATA drives and the disk access is temperamental to say the least.

(Incidentally managed to get MegaCli on it for RAID monitoring)

We've also got an old PE830 running the same version of openfiler that seems much more responsive. I think there is something funny with the RAID, but not sure what. It seems that although the drives aren't under heavy disk IO load the utilisation goes through the roof and performance becomes very, very sluggish.

The drives are in a RAID5 array and iostat reports:

[root at BHLSAN MegaCli]# iostat -d -k -x 5 3
Linux 2.6.26.8-1.0.11.smp.gcc3.4.x86_64 (BHLSAN)        03/13/2009

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.01   0.42  0.08  0.21    1.74    5.00     0.87     2.50    22.95     0.01   37.01  10.71   0.31
sda1         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     8.32     0.00   14.68   8.76   0.00
sda2         0.01   0.42  0.08  0.21    1.73    5.00     0.87     2.50    22.96     0.01   37.03  10.72   0.31
sdb          0.75   0.61  7.67 16.36  141.02 1520.80    70.51   760.40    69.14     0.30   12.41   3.91   9.39
sdb1         0.75   0.61  7.67 16.36  141.02 1520.80    70.51   760.40    69.14     0.30   12.41   3.91   9.39
dm-0         0.00   0.00  7.68 16.97  140.28 1520.80    70.14   760.40    67.39     0.31   12.64   3.81   9.39

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb          0.00   1.80 27.40 65.20  318.40 6392.40   159.20  3196.20    72.47     6.02   65.56  10.49  97.12
sdb1         0.00   1.80 27.40 65.20  318.40 6392.40   159.20  3196.20    72.47     6.02   65.56  10.49  97.12
dm-0         0.00   0.00 27.20 66.40  313.60 6387.40   156.80  3193.70    71.59     6.60   71.12  10.38  97.12

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   2.59  0.00  0.40    0.00   23.95     0.00    11.98    60.00     0.00    2.00   2.00   0.08
sda1         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2         0.00   2.59  0.00  0.40    0.00   23.95     0.00    11.98    60.00     0.00    2.00   2.00   0.08
sdb          0.00   1.60 25.95 63.27  469.46 6591.62   234.73  3295.81    79.14     3.05   34.28  10.89  97.17
sdb1         0.00   1.60 25.95 63.27  469.46 6591.62   234.73  3295.81    79.14     3.05   34.28  10.89  97.17
dm-0         0.00   0.00 25.95 64.07  469.46 6281.84   234.73  3140.92    75.00     3.07   34.20  10.79  97.17


Info on the disks...

./MegaCli64 -PDList -aALL

Adapter #0

Enclosure Device ID: 8
Slot Number: 0
Device Id: 0
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 698.636 GB [0x575466f0 Sectors]
Non Coerced Size: 698.136 GB [0x574466f0 Sectors]
Coerced Size: 698.125 GB [0x57440000 Sectors]
Firmware state: Online
SAS Address(0): 0x1221000000000000
Connected Port Number: 0
Inquiry Data: S13UJDWQ723377      SAMSUNG HD753LJ                         1AA01113
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Foreign State: None
Device Speed: Unknown
Link Speed: Unknown
Media Type: Hard Disk Device

Enclosure Device ID: 8
Slot Number: 1
Device Id: 1
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 698.636 GB [0x575466f0 Sectors]
Non Coerced Size: 698.136 GB [0x574466f0 Sectors]
Coerced Size: 698.125 GB [0x57440000 Sectors]
Firmware state: Online
SAS Address(0): 0x1221000001000000
Connected Port Number: 1
Inquiry Data: S13UJDWQ723368      SAMSUNG HD753LJ                         1AA01113
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Foreign State: None
Device Speed: Unknown
Link Speed: Unknown
Media Type: Hard Disk Device

Enclosure Device ID: 8
Slot Number: 2
Device Id: 2
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 698.636 GB [0x575466f0 Sectors]
Non Coerced Size: 698.136 GB [0x574466f0 Sectors]
Coerced Size: 698.125 GB [0x57440000 Sectors]
Firmware state: Online
SAS Address(0): 0x1221000002000000
Connected Port Number: 2
Inquiry Data: S13UJDWQ723376      SAMSUNG HD753LJ                         1AA01113
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Foreign State: None
Device Speed: Unknown
Link Speed: Unknown
Media Type: Hard Disk Device

Extracted from ./MegaCli64 -AdpAllInfo -aAll

                Device Present
                ================
Virtual Drives    : 1
  Degraded        : 0
  Offline         : 0
Physical Devices  : 4
  Disks           : 3
  Critical Disks  : 0
  Failed Disks    : 0

                Supported Adapter Operations
                ================
Rebuild Rate                    : Yes
CC Rate                         : Yes
BGI Rate                        : Yes
Reconstruct Rate                : Yes
Patrol Read Rate                : Yes
Alarm Control                   : Yes
Cluster Support                 : No
BBU                             : Yes
Spanning                        : Yes
Dedicated Hot Spare             : Yes
Revertible Hot Spares           : No
Foreign Config Import           : Yes
Self Diagnostic                 : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares               : Yes
Deny SCSI Passthrough           : No
Deny SMP Passthrough            : No
Deny STP Passthrough            : No
Support Security                : No

                Supported VD Operations
                ================
Read Policy          : Yes
Write Policy         : Yes
IO Policy            : Yes
Access Policy        : Yes
Disk Cache Policy    : Yes
Reconstruction       : Yes
Deny Locate          : No
Deny CC              : No
Allow Ctrl Encryption: No

                Supported PD Operations
                ================
Force Online                            : Yes
Force Offline                           : Yes
Force Rebuild                           : Yes
Deny Force Failed                       : No
Deny Force Good/Bad                     : No
Deny Missing Replace                    : No
Deny Clear                              : No
Deny Locate                             : No
Disable Copyback                        : No
Enable Copyback on SMART                : No
Enable Copyback to SSD on SMART Error   : No
Enable SSD Patrol Read                  : No

                Error Counters
                ================
Memory Correctable Errors   : 0
Memory Uncorrectable Errors : 0

                Cluster Information
                ================
Cluster Permitted     : No
Cluster Active        : No

                Default Settings
                ================
Phy Polarity                     : 0
Phy PolaritySplit                : 0
Background Rate                  : 30
Stripe Size                      : 64kB
Flush Time                       : 4 seconds
Write Policy                     : WB
Read Policy                      : None
Cache When BBU Bad               : Disabled
Cached IO                        : No
SMART Mode                       : Mode 6
Alarm Disable                    : No
Coercion Mode                    : 128MB
ZCR Config                       : IDSEL
Dirty LED Shows Drive Activity   : No
BIOS Continue on Error           : No
Spin Down Mode                   : None
Allowed Device Type              : SAS/SATA Mix
Allow Mix in Enclosure           : Yes
Allow HDD SAS/SATA Mix in VD     : No
Allow SSD SAS/SATA Mix in VD     : No
Allow HDD/SSD Mix in VD          : No
Allow SATA in Cluster            : No
Max Chained Enclosures           : 1
Disable Ctrl-R                   : No
Enable Web BIOS                  : No
Direct PD Mapping                : No
BIOS Enumerate VDs               : No
Restore Hot Spare on Insertion   : No
Expose Enclosure Devices         : No
Maintain PD Fail History         : No
Disable Puncturing               : No
Zero Based Enclosure Enumeration : No
PreBoot CLI Enabled              : Yes
LED Show Drive Activity          : No
Cluster Disable                  : Yes
SAS Disable                      : No
Auto Detect BackPlane Enable     : SGPIO/i2c SEP
Use FDE Only                     : No
Enable Led Header                : No


Any idea on what I can check/change?

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20090313/b5ecc3d9/attachment-0001.htm 


More information about the Linux-PowerEdge mailing list