Performance/stability problems on PowerEdge 1950
gao at schrodinger.com
Tue Dec 26 12:21:36 CST 2006
Have you tried reiserfs filesystem? Reiserfs is much better with
handling lots of small files.
Harry Roberts wrote:
> We tried forcing use of the CFQ I/O scheduler when using standard 18.104.22.168 and it made no noticable change to performance, and still ran into the problems.
> It's gotten to the bang-head-on-wall stage already, we've gone through all the fairly obvious possibilities.
> For example changing from journal writes to writeback mode for ext3, and switching from writethrough mode to writeback mode on the PERC5 controller helped somewhat. Before those changes system load was much higher during the same stress situations and tests, often climbing to 30+. But we still encountered the stability problems at around the same time no matter what the reported system load was.
> We've tried simulating the same pattern of writes using some Perl scripts to do ~20 small writes per second, and starting up 20-40 of them at once but were unable to replicate it.
> The full environment we've got setup is to replicate the problems (discovered in a live environment) is as such:
> 1) Dell 1950 server with Asterisk 1.2.x - call recording
> 2) Dell 1850 server with Asterisk 1.2.x - answering test calls & playing test messages in a loop back to callers.
> 3) ~30 IP phones
> Calls are being placed from the IP phones which are routed to the Dell 1850 via the Dell 1950, the 1950 is recording both channels of audio (in & out) for every call. The amount of data being written really isn't much (8kb/s per channel, so 500-600kb/s excluding overheads).
> Dstat was reporting that between 1-1.5mb was being written to disk every couple of seconds, so buffering's definitely going on and I don't think the controller it'self should be being hammered with write requests.
> Again, any insights would be helpful, but we've spent a good few weeks on this already trying various things out. The good side is that we've found out quite a bit about performance tuning :)
> - Harry
> -----Original Message-----
> From: linux-poweredge-bounces at dell.com [mailto:linux-poweredge-bounces at dell.com] On Behalf Of Robin Humble
> Sent: 18 December 2006 21:50
> To: linux-poweredge at dell.com
> Subject: Re: Performance/stability problems on PowerEdge 1950
> On Mon, Dec 18, 2006 at 10:49:57AM -0000, Harry Roberts wrote:
>> The only thing we've found so far to stop the issue is running RedHat's customized 2.6.9 kernel, I know it won't really help, but if it's an all-or-nothing situation it's either that or re-install with RedHat EL.
> which elevator are you using with your kernels?
> the default is usually 'anticipatory' which is good for desktop, but apparently doesn't perform well under multiple client loads.
>> OTOH, is anybody here using SAS based 1950s with CentOS under really high disk load?
> we have one of those machines.
> do you have a high disk load benchmark that will trigger the problem?
> perhaps multiple bonnie++'s? iozone? or a small test code?
> Linux-PowerEdge mailing list
> Linux-PowerEdge at dell.com
> Please read the FAQ at http://lists.us.dell.com/faq
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.1.409 / Virus Database: 268.15.23/591 - Release Date: 17/12/2006
More information about the Linux-PowerEdge