PE2550 & RH7.2 vs PE2650 & RH7.3

[Darrin Hyrup]

Hey folks,

We've just deployed a number of new 2.0GHz Xeon dual processor PE2650 
servers running RH7.3 (kernel 2.4.18-5) with 2GB DDR RAM (4x 512MB.)  We 
also have a number of installed  1.26GHz P3 dual processor PE2550 servers 
running in the same basic configuration except the PE2550s are running 
RH7.2 (kernel 2.4.9-34).  Both server types are using the same hard drives 
under the on-board RAID enabler.

In our implementation, we're running the same exact software on both 
hardware configurations, and on the new PE2650s the servers are straining 
to keep up... the processors are railed at 100% cpu utilization most of the 
time... even when they're supposed to be idle, the processes themselves 
randomly spike up to 20-30% cpu utilization every few seconds.  On the 
older hardware, the processes idle at 0% cpu load (and stay pretty much 
constantly at 0% while idle, not spiking up) and under load rarely exceed 
75% cpu utilization.  This is under the same conditions.

What I'm wondering is if there is some kind of CPU hogging bug in the more 
recent kernels that could be attributed to this, or if perhaps there is 
just some kind of difference in how the systems perform.  Given the much 
more powerful processors, faster bus speeds, and faster RAM, there is no 
reason why the new machines should perform so poorly compared to the older 
boxes.

I recall having a similar problem back a few months ago when we updated to 
one of the earlier 2.4.1x kernels... we reverted back to the 2.4.9 kernel 
and the problems went away, but I later heard that there was some kind of 
interrupt issue that was draining system resources, so I assumed that was 
the cause for the major efficiency loss with that kernel upgrade.  The 
performance hits we're seeing are very similar if not identical.

Hopefully, we're not alone in this observation.  If anyone has any theories 
and/or suggestions as to what we can do to eliminate this slowdown it would 
definitely be appreciated!  We may have to go back to the PE2550s (or at 
least try to get RH7.2 installed somehow) if we can't find a solution... 
I'd hate to do that since both the new hardware and new OS have a lot of 
advantages over the old.

Thanks,

Darrin Hyrup
Mythic Entertainment