disk i/o latency on 1850 4e/si

[J Potter]

Hi List,

Has anyone else encountered high latency for disk i/o on a PE 1850?  
What was the solution?

We have a server that's been running fine for several years, and over  
the past few months, i/o throughput has gotten dramatically worse.  
I've been unable to track it down to any particular change.

We're running CentOS 4.7, relatively standard config. I've compared  
bonnie++ output between this server and another 1850 running CentOS 4.  
(See "vs" column below.) omreport shows both systems as having the  
same write through settings on the raid controller. I've not been able  
to find any clues in system logs or omreport logs. Both servers are a  
raid-1 config, although the one with performance issues is running  
LVM. Both servers use ext3 as the filesystem.

Sequential Output:
	Latency K/sec Per Char:  1379ms  vs  41ms  (33x)
	Latency K/sec Block:        3015ms  vs  846ms (3x)

Sequential Create:
	files /sec:		1926   vs   22816 (11x)
	Latency:			334ms  vs  0.2ms (1670x)

Unfortunately, the server is in production, so it's not so easy to  
isolate things. The test results will be skewed because of other  
processes running on the production instance.

Thoughts? I'm stumped.

-Jeff