R515/H700 high iowait

Mark Nelson mark.nelson at inktank.com
Wed Jul 18 07:15:45 CDT 2012


On 7/18/12 3:43 AM, Pavel Mateja wrote:
>> Hi Guys,
>>
>> I was wondering if anyone has had any problems either on the R515 or
>> with the H700 card where there are very high io wait times when multiple
>> writers write to the same raid group.  Basically we first noticed this
>> due to inconsistent buffered IO performance under heavy IO load.  We saw
>> IO wait times as high as 6+ seconds.
>>
>> After that we started testing with fio.  What we noticed is that if
>> writing to say a large raid0 with 7-8 drives we could achieve 800MB/s,
>> but only with large (256MB) IOs using direct IO and a single "job" (ie
>> fio process).  Simply increasing the number of jobs to 2 reduced
>> performance to 80MB/s and dramatically increased IO wait times.  The
>> controller has a BBU and the raid array was configured with adaptive
>> readahead and writeback cache.
>>
>> One thought is that perhaps this is not a problem with the raid
>> controller, but some kind of preemption issue with the io scheduler.  At
>> once point early on I tried switching from cfq to deadline with little
>> apparent change, though I have not yet done exhaustive tests.  I will
>> try to record blktrace output soon to see if that provides any insight
>> into what is going on.
>>
>> In the mean time, has anyone else seen anything like this?
>
> Hi,
> what about filesystem? Ext3 with barrier enabled?
>

Hi Pavel,

This is with XFS and barriers were left enabled.  We also tested having 
fio write directly to the device.

Thanks,
Mark



More information about the Linux-PowerEdge mailing list