2950 write performance issues

[McDougall, Marshall (FSH)]

>-----Original Message-----
>From: linux-poweredge-bounces at dell.com 
>[mailto:linux-poweredge-bounces at dell.com] On Behalf Of 
>McDougall, Marshall (FSH)
>Sent: Friday, June 08, 2007 3:08 PM
>To: Derek Yuen; linux-poweredge at dell.com
>Subject: RE: 2950 write performance issues
>
>
>>-----Original Message-----
>>From: Derek Yuen [mailto:derek.yuen at utoronto.ca] 
>>Sent: Friday, June 08, 2007 2:41 PM
>>To: McDougall, Marshall (FSH)
>>Subject: Re: 2950 write performance issues
>>
>>
>>Hi Marshall,
>>
>>Could you provide some metrics as to what you'd consider as 
>"poor write
>>performance" (perhaps writeKB, IOPs ?)
>>
>>
>>
>>Thanks,
>>Derek
>>--
>>Derek Yuen
>>Computing & Network Services
>>University of Toronto
>>
>>
>>
>>
>>On Fri, 8 Jun 2007, McDougall, Marshall (FSH) wrote:
>>
>>> I have several identical 2950's that have very inconsistent write
>>> characteristics.  They all have 8GB ram, 6 x 146 GB SAS drives in a
>>> single raid10 config. The bios is at 1.3.7, ESM backplane at 
>>1.05, A01
>>> and PERC at 1.03.10-0216.  Each server is running RHEL4U5, 
>Sybase ASE
>>> 12.5.4.  I have one server that chugs along at an acceptable level,
>>> compared to a 2550, and rest have write statistics that are 
>up to 100
>>> times worse depending on if it's an Insert, Commit, Delete, etc.
>>>
>>> I have compared the RAID config, with megacli, in all cases 
>>and they are
>>> identical.  I am running out of ideas.  I would appreciate any fresh
>>> approaches that are offered. Hell, I'll even take old stale 
>>ones just to
>>> make sure I haven't missed anything :)  Thanks.
>>>
>>> Regards, Marshall
>>>
>>
> Here's an example.  Sorry for the formatting, but you get the idea.
>
>Regards, Marshall
>
> TEST                      Srvr1   Srvr2
>Insert1K.out     Time	   0.040   4.224
>Copy1K.out       Time	   0.007   0.064
>JoinPK1K.out     Time	   0.007   0.010
>JoinNK1K.out     Time	   0.030   0.033
>CommitSmallTX1K.out      Time   0.070   4.260
>CommitBigTX1K.out        Time   0.043   0.284
>Update1K.out     Time	   0.007   0.090
>Delete1K.out     Time	   0.004   0.043
>Insert10K.out    Time	   0.390   42.246
>Copy10K.out      Time	   0.060   0.440
>JoinPK10K.out    Time	   0.083   0.080
>JoinNK10K.out    Time	   0.186   0.467
>CommitSmallTX10K.out     Time   0.723   42.703
>CommitBigTX10K.out       Time   0.423   2.813
>Update10K.out    Time	   0.017   0.800
>Delete10K.out    Time	   0.026   0.404
>Insert100K.out   Time	   4.987   417.410
>Copy100K.out     Time	   0.470   4.393
>JoinPK100K.out   Time	   0.827   0.820
>JoinNK100K.out   Time	   1.980   4.613
>CommitSmallTX100K.out    Time   7.120   441.480
>CommitBigTX100K.out      Time   4.500   29.320
>Update100K.out   Time	   0.143   7.967
>Delete100K.out   Time	   0.193   6.560
>CreateLargeVol.out       Time A   0.003   0.074
>CreateLargeVol.out       Time B   0.007   0.216
>CreateLargeVol.out       Time C   0.010   0.110
>CreateLargeVol.out       Time D   0.053   0.897
>CreateLargeVol.out       Time E   0.550   5.647
>CreateLargeVol.out       Time F   6.047   51.980
>CreateLargeVol.out       Time G   6.106   64.223
>CopyLargeVol.out         Time A   0.000   0.026
>CopyLargeVol.out         Time B   0.003   0.027
>CopyLargeVol.out         Time C   0.000   0.037
>CopyLargeVol.out         Time D   0.017   0.063
>CopyLargeVol.out         Time E   0.136   0.190
>CopyLargeVol.out         Time F   1.517   1.523
>CopyLargeVol.out         Time G   1.553   1.600  
>
>_______________________________________________
>Linux-PowerEdge mailing list
>Linux-PowerEdge at dell.com
>http://lists.us.dell.com/mailman/listinfo/linux-poweredge
>Please read the FAQ at http://lists.us.dell.com/faq
>

I keep looking for something I can identify as the problem here.  I just
noticed that memory usage is drastically different in these machines.
Outside of adjusting the shared memory for these boxes, they are the
same.  Can anyone enlighten me as to why there is such a radical
difference in memory usage?  What can I look at to tune?

Server 1 - bad:
[root at fsh1166db01x]# free
             total       used       free     shared    buffers
cached
Mem:       8310532    8294836      15696          0       6280
8144720
-/+ buffers/cache:     143836    8166696
Swap:     16386292        208   16386084

Server 2 - good:
[root at fsh1166db02x]# free
             total       used       free     shared    buffers
cached
Mem:       8310532    2952540    5357992          0     125464
2679676
-/+ buffers/cache:     147400    8163132
Swap:     16386292        208   16386084

Thanks.

Regards, Marshall