Slow PERC4 performance

Nick Stephens nick at ceiva.com
Tue May 23 13:28:14 CDT 2006


I have also verified that we are running the latest firmware on both
controllers.  Any ideas?

Is this something I should be contacting my support rep about?  My
initial fear is that the limited linux support might not cover this
however at this point I cannot tell if it is a driver or a hardware
issue.

Nick

-----Original Message-----
From: linux-poweredge-bounces at dell.com
[mailto:linux-poweredge-bounces at dell.com] On Behalf Of Nick Stephens
Sent: Monday, May 22, 2006 10:39 AM
To: linux-poweredge at dell.com
Subject: Slow PERC4 performance 

Hi all, 

We are setting up our 3rd PE1850 + PV220s linux fileserver, but this
newest one has slightly different raid hardware and seems to be running
much slower than our previous installations.  Our 2 other systems like
this are also PE1850s, but utilize PERC 4/Di and PERC 3/DC.  This new
1850 uses a PERC 4/DC and a PERC 4e/Si.  Also, the older boxes are
running Redhat 9, whereas I am attempting to use Fedora Core 5 on the
new one.

Everything on the system seems to go ok until it comes to the disk
access on the PV220s.  Mind you, this fileserver is 2x as large as the
previous ones (1.7TB), but the amount of disk space in use is exactly
the same as the box I am testing it against, so I don't believe that
should factor in.  

I noticed the problem during our peak hours when smbd is at its highest
load transferring files to our windows servers.  Whereas the older boxes
would achieve a light load on them, we never ran into issues where they
would stop serving.  On the new box it reaches a similar load and then
simply stops serving files, causing other servers in the chain to lock
up with requests not being met.  I have tried the latest version of
samba to ensure that it wasn't a direct problem  with it, but it seems
to not be limited to samba, but rather to anything that accesses that
disk.  To this end it makes me think that I either have a bad raid
configuration (though it is seemingly the same as the other 2 I have
running), bad drivers, or possibly bad hardware.  

Thank you so much for you help!

Here are some tests that I have run to compare:

(this is a simple find script that walks the disk looking for files not
chowned to a user, and then chowns them appropriately)

For the sake of this test, I created a directory of 24gb on the PERC
4/DC filesystems, and on the PERC 4e/Si filesystem.  You will see that
it is much faster access on the 'local' filesystem vs the PV220s.

PV220s find script
----------------------
# time ./findjacked2.sh
real    4m22.550s
user    0m0.720s
sys     0m3.928s

LOCAL find script
-----------------------
# time ./findjacked2.sh
real    1m47.667s
user    0m21.733s
sys     1m29.174s


PV220s df 
------------
[root at blizzard bin]# du -sh /usr/local/ceiva/kfile/NICK/large/
24G     /usr/local/ceiva/kfile/NICK/large/


'LOCAL' df
-------------
[root at blizzard bin]# du -sh /usr/local/NICK/large/
24G     /usr/local/NICK/large/


NEW 1850 release / kernel information 
---------------------------------------
Fedora Core release 5 (Bordeaux)
Linux blizzard 2.6.16-1.2111_FC5smp #1 SMP Thu May 4 21:35:09 EDT 2006
i686 i686 i386 GNU/Linux

NEW 1850 LSMOD
---------------------
Module                  Size  Used by
ipv6                  246561  18 
nfsd                  204497  17 
exportfs                9665  1 nfsd
autofs4                22981  1 
i2c_dev                12741  0 
i2c_core               24769  1 i2c_dev
nfs                   198061  1 
lockd                  59465  3 nfsd,nfs
nfs_acl                 7745  2 nfsd,nfs
sunrpc                140925  13 nfsd,nfs,lockd,nfs_acl
dm_mirror              24081  0 
joydev                 13441  0 
dm_mod                 55129  1 dm_mirror
video                  18885  0 
button                 10577  0 
battery                13253  0 
ac                      8901  0 
lp                     16265  0 
parport_pc             29413  0 
parport                38281  2 lp,parport_pc
floppy                 61829  0 
nvram                  12617  0 
sg                     36317  0 
e1000                 103853  0 
ehci_hcd               33229  0 
uhci_hcd               33233  0 
e752x_edac             13889  0 
edac_mc                18069  1 e752x_edac
ext3                  121417  4 
jbd                    56917  1 ext3
megaraid_mbox          33745  5 
megaraid_mm            14829  1 megaraid_mbox
sd_mod                 20417  7 
scsi_mod              129001  3 sg,megaraid_mbox,sd_mod


NEW 1850 MEGARAID INFORMATION
---------------------------------------

                *******Information Of Logical Drive 0*******

          Logical Drive : 0( Adapter: 0 ):  Status: OPTIMAL
        ---------------------------------------------------
        SpanDepth :01     RaidLevel: 5  RdAhead : Adaptive  Cache:
DirectIo
        StripSz   :064KB   Stripes  : 13  WrPolicy: WriteBack

        Logical Drive 0 : SpanLevel_0 Disks
        Chnl  Target  StartBlock   Blocks      Physical Target Status
        ----  ------  ----------   ------      ----------------------
        0      00    0x00000000   0x1113e000   ONLINE
        0      01    0x00000000   0x1113e000   ONLINE
        0      02    0x00000000   0x1113e000   ONLINE
        0      03    0x00000000   0x1113e000   ONLINE
        0      04    0x00000000   0x1113e000   ONLINE
        0      05    0x00000000   0x1113e000   ONLINE
        0      08    0x00000000   0x1113e000   ONLINE
        0      09    0x00000000   0x1113e000   ONLINE
        0      10    0x00000000   0x1113e000   ONLINE
        0      11    0x00000000   0x1113e000   ONLINE
        0      12    0x00000000   0x1113e000   ONLINE
        0      13    0x00000000   0x1113e000   ONLINE
        0      14    0x00000000   0x1113e000   ONLINE


OLD 1850 MEGARAID INFORMATION
---------------------------------------
                *******Information Of Logical Drive 0*******

          Logical Drive : 0( Adapter: 1 ):  Status: OPTIMAL
        ---------------------------------------------------
        SpanDepth :01     RaidLevel: 5  RdAhead : Adaptive  Cache:
DirectIo
        StripSz   :064KB   Stripes  : 12  WrPolicy: WriteBack

        Logical Drive 0 : SpanLevel_0 Disks
        Chnl  Target  StartBlock   Blocks      Physical Target Status
        ----  ------  ----------   ------      ----------------------
        0      00    0x00000000   0x0887c000   ONLINE
        0      01    0x00000000   0x0887c000   ONLINE
        0      02    0x00000000   0x0887c000   ONLINE
        0      03    0x00000000   0x0887c000   ONLINE
        0      04    0x00000000   0x0887c000   ONLINE
        0      08    0x00000000   0x0887c000   ONLINE
        0      09    0x00000000   0x0887c000   ONLINE
        0      10    0x00000000   0x0887c000   ONLINE
        0      11    0x00000000   0x0887c000   ONLINE
        0      12    0x00000000   0x0887c000   ONLINE
        0      13    0x00000000   0x0887c000   ONLINE
        0      14    0x00000000   0x0887c000   ONLINE

NEW 1850 EXT3 INFORMATION
--------------------------------
Filesystem volume name:   /
Last mounted on:          <not available>
Filesystem UUID:          7e47c2ba-e1b4-4823-a2ca-356512b4cc2f
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              17088192
Block count:              17075079
Reserved block count:     853753
Free blocks:              9231830
Free inodes:              16594724
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32736
Inode blocks per group:   1023
Filesystem created:       Wed May 17 07:08:50 2006
Last mount time:          Thu May 18 17:21:57 2006
Last write time:          Thu May 18 17:21:57 2006
Mount count:              5
Maximum mount count:      -1
Last checked:             Wed May 17 07:08:50 2006
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
First orphan inode:       1578710
Default directory hash:   tea
Directory Hash Seed:      387ddc56-0acc-49b1-9085-242344739098
Journal backup:           inode blocks


OLD 1850 EXT3 INFORMATION
--------------------------------
Filesystem volume name:   iFile
Last mounted on:          <not available>
Filesystem UUID:          7add9ee6-3965-48bb-a8a7-22286cbdfa79
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery
sparse_super
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              196804608
Block count:              196780177
Reserved block count:     0
Free blocks:              18851111
Free inodes:              171640778
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   1024
Filesystem created:       Fri Feb  4 16:22:31 2005
Last mount time:          Wed Apr 12 20:32:07 2006
Last write time:          Wed Apr 12 20:32:07 2006
Mount count:              8
Maximum mount count:      25
Last checked:             Tue Feb  8 18:27:29 2005
Check interval:           15552000 (6 months)
Next check after:         Sun Aug  7 19:27:29 2005
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal UUID:             <none>
Journal inode:            8
Journal device:           0x0000
First orphan inode:       0


NEW 1850 DMESG RAID INFO
--------------------------------------
SCSI subsystem initialized
megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.7 (Release Date: Mon Nov 14 12:27:22 EST 2005)
megaraid: probe new device 0x1000:0x1960:0x1028:0x0518: bus 2:slot
12:func 0
ACPI: PCI Interrupt 0000:02:0c.0[A] -> GSI 39 (level, low) -> IRQ 19
megaraid: fw version:[351X] bios version:[1.10]
scsi0 : LSI Logic MegaRAID driver
scsi[0]: scanning scsi channel 0 [Phy 0] for non-raid devices
  Vendor: DELL      Model: PV22XS            Rev: E.17
  Type:   Processor                          ANSI SCSI revision: 03
scsi[0]: scanning scsi channel 1 [Phy 1] for non-raid devices
input: ImPS/2 Generic Wheel Mouse as /class/input/input1
scsi[0]: scanning scsi channel 2 [virtual] for logical drives
  Vendor: MegaRAID  Model: LD 0 RAID5 1678G  Rev: 351X
  Type:   Direct-Access                      ANSI SCSI revision: 02
SCSI device sda: 3438182400 512-byte hdwr sectors (1760349 MB)
sda: Write Protect is off
sda: Mode Sense: 00 00 00 00
sda: asking for cache data failed
sda: assuming drive cache: write through
SCSI device sda: 3438182400 512-byte hdwr sectors (1760349 MB)
sda: Write Protect is off
sda: Mode Sense: 00 00 00 00
sda: asking for cache data failed
sda: assuming drive cache: write through
 sda: sda1
sd 0:2:0:0: Attached scsi disk sda
megaraid: probe new device 0x1028:0x0013:0x1028:0x016c: bus 2:slot
14:func 0
ACPI: PCI Interrupt 0000:02:0e.0[A] -> GSI 46 (level, low) -> IRQ 20
megaraid: fw version:[521X] bios version:[H430]
scsi1 : LSI Logic MegaRAID driver
scsi[1]: scanning scsi channel 0 [Phy 0] for non-raid devices
  Vendor: PE/PV     Model: 1x2 SCSI BP       Rev: 1.0 
  Type:   Processor                          ANSI SCSI revision: 02
scsi[1]: scanning scsi channel 1 [virtual] for logical drives
  Vendor: MegaRAID  Model: LD 0 RAID1   69G  Rev: 521X
  Type:   Direct-Access                      ANSI SCSI revision: 02
SCSI device sdb: 143114240 512-byte hdwr sectors (73274 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 00 00 00
sdb: asking for cache data failed
sdb: assuming drive cache: write through
SCSI device sdb: 143114240 512-byte hdwr sectors (73274 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 00 00 00
sdb: asking for cache data failed
sdb: assuming drive cache: write through
 sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 >
sd 1:1:0:0: Attached scsi disk sdb

_______________________________________________
Linux-PowerEdge mailing list
Linux-PowerEdge at dell.com
http://lists.us.dell.com/mailman/listinfo/linux-poweredge
Please read the FAQ at http://lists.us.dell.com/faq



More information about the Linux-PowerEdge mailing list