R515/H700 high iowait
mark.nelson at inktank.com
Wed Jul 18 07:15:45 CDT 2012
On 7/18/12 3:43 AM, Pavel Mateja wrote:
>> Hi Guys,
>> I was wondering if anyone has had any problems either on the R515 or
>> with the H700 card where there are very high io wait times when multiple
>> writers write to the same raid group. Basically we first noticed this
>> due to inconsistent buffered IO performance under heavy IO load. We saw
>> IO wait times as high as 6+ seconds.
>> After that we started testing with fio. What we noticed is that if
>> writing to say a large raid0 with 7-8 drives we could achieve 800MB/s,
>> but only with large (256MB) IOs using direct IO and a single "job" (ie
>> fio process). Simply increasing the number of jobs to 2 reduced
>> performance to 80MB/s and dramatically increased IO wait times. The
>> controller has a BBU and the raid array was configured with adaptive
>> readahead and writeback cache.
>> One thought is that perhaps this is not a problem with the raid
>> controller, but some kind of preemption issue with the io scheduler. At
>> once point early on I tried switching from cfq to deadline with little
>> apparent change, though I have not yet done exhaustive tests. I will
>> try to record blktrace output soon to see if that provides any insight
>> into what is going on.
>> In the mean time, has anyone else seen anything like this?
> what about filesystem? Ext3 with barrier enabled?
This is with XFS and barriers were left enabled. We also tested having
fio write directly to the device.
More information about the Linux-PowerEdge