RHEL 4U6 x86_64 freezed by heavy I/O on PE2950

[roderick.castillo at metanomics.de]

Hello

>> reproducibly renders the server unusuable for a period of time [
>> ... ] But at times, the operation finishes in 2 seconds! Only
>> trying a simple "ls -l" afterwards puts the server in the
>> mentioned "frozen" state. This is so because at times I/O
>> operations are completely buffered

>You have just discovered the obvious :-).  That is, yet another
>case where some part of the Linux IO subsystem seems to have been
>designed with astonishingly misguided naivety (to use euphemisms).

>> and apparently the issue arises only when actually flushing to
>> disk. This server has 16 GB memory.

You are addressing the aspect of buffer sizing. It turns
out, this is not the issue here.

>> I don't know how to experiment with vm parameters in this
>> version of RedHat (Nahant).

>Thanks to your using a recentish RHEL4 you only really need to
>worry about the 'sysctl' parameter 'vm/max_queue_depth' (very
>curiously it is not yet in RHEL5 I think, but it may have been
>added). It is in megabytes and tells the flusher to start writing
>out after those many dirty megabytes have accumulated. A value
>like 100-500MB, depending on your appetite for risk and the speed
>of your storage, may be appropriate.

This is an interesting point indeed. I changed vm.max_queue_depth to
100 MB with almost no perceptible difference. Then I changed it to
16 MB, with the result that the server no longer "freezed", the load
was much lower. Then I changed the parameter to 8 MB and found out
that the server was quite responsive during the copy operation. But
in all cases the operation was very slow (throughput of about 1 MB/s).

So I was able to investigate what was happening because with
max_queue_depth=8 the server did not "freeze" any more. Using
iostat I found out, that transfers took place only every 4 or
5 seconds at about 4 MB/s and the rest of the time the server
was quiescent, the device showed %util of 102% (=100%) due
to iowait, and the avgqu-sz was about 2500.

So transfers were done in an intermittent way. The only advantage
of lowering this parameter is that the server stays responsive,
because the flushing is more distributed of spread in time, and
the load is kept low.

Now I am working with Dell to tune the settings of the PERC5i
controller. There is a real brake here that you cannot fix with
vm parameters.

>Anyhow there is a discussion of some aspects of this here:

>  http://www.sabi.co.uk/blog/0707jul.html#070701
>  http://www.sabi.co.uk/blog/0707jul.html#070701b

That refers the buffer sizing. I have references indicating
a more general problem:

http://lists.us.dell.com/pipermail/linux-poweredge/2006-June/026098.html 
http://www.mydatabasesupport.com/forums/linux-misc/268098-very-high-iowait.html

Regards
Rick