Performance/stability problems on PowerEdge 1950

Harry Roberts harry at telappliant.com
Tue Dec 19 10:13:26 CST 2006


Hi,
 We tried forcing use of the CFQ I/O scheduler when using standard 2.6.18.1 and it made no noticable change to performance, and still ran into the problems.

 It's gotten to the bang-head-on-wall stage already, we've gone through all the fairly obvious possibilities.

 For example changing from journal writes to writeback mode for ext3, and switching from writethrough mode to writeback mode on the PERC5 controller helped somewhat. Before those changes system load was much higher during the same stress situations and tests, often climbing to 30+. But we still encountered the stability problems at around the same time no matter what the reported system load was.

 We've tried simulating the same pattern of writes using some Perl scripts to do ~20 small writes per second, and starting up 20-40 of them at once but were unable to replicate it.

 The full environment we've got setup is to replicate the problems (discovered in a live environment) is as such:
  1) Dell 1950 server with Asterisk 1.2.x - call recording
  2) Dell 1850 server with Asterisk 1.2.x - answering test calls & playing test messages in a loop back to callers.
  3) ~30 IP phones

 Calls are being placed from the IP phones which are routed to the Dell 1850 via the Dell 1950, the 1950 is recording both channels of audio (in & out) for every call. The amount of data being written really isn't much (8kb/s per channel, so 500-600kb/s excluding overheads).

 Dstat was reporting that between 1-1.5mb was being written to disk every couple of seconds, so buffering's definitely going on and I don't think the controller it'self should be being hammered with write requests.

 Again, any insights would be helpful, but we've spent a good few weeks on this already trying various things out. The good side is that we've found out quite a bit about performance tuning :)

 Thanks
 - Harry 

-----Original Message-----
From: linux-poweredge-bounces at dell.com [mailto:linux-poweredge-bounces at dell.com] On Behalf Of Robin Humble
Sent: 18 December 2006 21:50
To: linux-poweredge at dell.com
Subject: Re: Performance/stability problems on PowerEdge 1950

On Mon, Dec 18, 2006 at 10:49:57AM -0000, Harry Roberts wrote:
>  The only thing we've found so far to stop the issue is running RedHat's customized 2.6.9 kernel, I know it won't really help, but if it's an all-or-nothing situation it's either that or re-install with RedHat EL.

which elevator are you using with your kernels?
the default is usually 'anticipatory' which is good for desktop, but apparently doesn't perform well under multiple client loads.
  http://www.redhat.com/magazine/008jun05/features/schedulers/

>OTOH, is anybody here using SAS based 1950s with CentOS under really high disk load?

we have one of those machines.
do you have a high disk load benchmark that will trigger the problem?
perhaps multiple bonnie++'s? iozone? or a small test code?

cheers,
robin

_______________________________________________
Linux-PowerEdge mailing list
Linux-PowerEdge at dell.com
http://lists.us.dell.com/mailman/listinfo/linux-poweredge
Please read the FAQ at http://lists.us.dell.com/faq

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

--
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.15.23/591 - Release Date: 17/12/2006
 

-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.15.24/592 - Release Date: 18/12/2006
 



More information about the Linux-PowerEdge mailing list