R515/H700 high iowait

Mark Nelson mark.nelson at inktank.com
Tue Jul 17 11:53:09 CDT 2012


Hi Wayne,

Our standard setup is:

megaraid_sas: 00.00.06.14-rc1
Firmware Rev: 2.10

Those specific results were taken after upgrading to the latest LSI 2.13 
firmware to see if it would help.  I can rerun the tests on another node 
with 2.10 or a newer Dell firmware if desired.

Mark

On 07/17/2012 11:45 AM, Wayne_Weilnau at Dell.com wrote:
> What are your driver and firmware versions?
>
> Wayne Weilnau
> Systems Management Technologist
> Dell | OpenManage Software Development
>
> Please consider the environment before printing this email.
>
> Confidentiality Notice | This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, immediately contact the sender by reply e-mail and destroy all copies of the original message.
>
>
> -----Original Message-----
> From: linux-poweredge-bounces-Lists On Behalf Of Mark Nelson
> Sent: Tuesday, July 17, 2012 11:40 AM
> To: G.Bakalarski at icm.edu.pl
> Cc: linux-poweredge-Lists
> Subject: Re: R515/H700 high iowait
>
> Hi GB,
>
> Sorry, I should have included more info earlier.
>
> OS: Ubuntu 12.04, Kernel 3.4 (We believe we've had the same issue with
> 11.10 and earlier kernels as well though).
>
> Disk Type: Western Digital RE SAS drives, model: WD1000FYYG.
>
> Block size: 64k.
>
> IO pattern: Sequential writes.
>
> Here's an example of the fio commands we used to try and track this down:
>
> XFS on 7 drive raid0, direct IO:
>> sudo fio --rw=write --bs=256M --direct=1 --size=64G --runtime=300 --name=foo
> Result: ~680MB/s, 98% of latencies are 500ms or lower.
>
>
> XFS on 7 drive raid0, buffered IO:
>> sudo fio --rw=write --bs=256M --direct=0 --size=64G --runtime=300 --name=foo
> Result: ~46MB/s, 81.5% of latencies are 2000ms or higher.
>
>
> No FS on 7 drive raid0, direct IO, 1 job:
>> fio --rw=write -ioengine=sync --size=64G --runtime=300 --name=/dev/sdb --numjobs=1 --direct=1 --bs=256M --iodepth=16
> Result: ~793MB/s, 100% of latencies are 500ms or lower.
>
>
> No FS on 7 drive raid0, direct IO, 2 jobs:
>> fio --rw=write -ioengine=sync --size=64G --runtime=300 --name=/dev/sdb --numjobs=2 --direct=1 --bs=256M --iodepth=16
> Result: ~95MB/s, 100% of latencies are 2000ms or higher.
>
> Also, this problem also appears on smaller raid0 arrays as well which is
> where we first saw it.  It's not as easy to tell what is going on though
> as the performance delta isn't as great.
>
> Mark
>
> On 07/17/2012 10:51 AM, G.Bakalarski at icm.edu.pl wrote:
>> What OS type? what disks (SAS 10k, 15k, 7.2k)?
>> You wrote RAID0 ... Array block size 64k ???
>> Very strange
>>
>> Maybe directIO is a clue? Which type of directIO do you use?
>>
>> What is IO pattern - sequential or random or unknown ???
>>
>> GB
>>
>>> I was wondering if anyone has had any problems either on the R515 or
>>> with the H700 card where there are very high io wait times when multiple
>>> writers write to the same raid group.  Basically we first noticed this
>>> due to inconsistent buffered IO performance under heavy IO load.  We saw
>>> IO wait times as high as 6+ seconds.
>>>
>>> After that we started testing with fio.  What we noticed is that if
>>> writing to say a large raid0 with 7-8 drives we could achieve 800MB/s,
>>> but only with large (256MB) IOs using direct IO and a single "job" (ie
>>> fio process).  Simply increasing the number of jobs to 2 reduced
>>> performance to 80MB/s and dramatically increased IO wait times.  The
>>> controller has a BBU and the raid array was configured with adaptive
>>> readahead and writeback cache.
>>>
>>> One thought is that perhaps this is not a problem with the raid
>>> controller, but some kind of preemption issue with the io scheduler.  At
>>> once point early on I tried switching from cfq to deadline with little
>>> apparent change, though I have not yet done exhaustive tests.  I will
>>> try to record blktrace output soon to see if that provides any insight
>>> into what is going on.
>>>
>>> In the mean time, has anyone else seen anything like this?
>>>
>>> Mark
>>>
>>> _______________________________________________
>>> Linux-PowerEdge mailing list
>>> Linux-PowerEdge at dell.com
>>> https://lists.us.dell.com/mailman/listinfo/linux-poweredge
>>>
>>
>>
>
>


-- 
Mark Nelson
Performance Engineer
Inktank



More information about the Linux-PowerEdge mailing list