R515/H700 high iowait

Wayne_Weilnau at Dell.com Wayne_Weilnau at Dell.com
Tue Jul 17 11:45:27 CDT 2012


What are your driver and firmware versions?

Wayne Weilnau
Systems Management Technologist
Dell | OpenManage Software Development 

Please consider the environment before printing this email.

Confidentiality Notice | This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, immediately contact the sender by reply e-mail and destroy all copies of the original message.


-----Original Message-----
From: linux-poweredge-bounces-Lists On Behalf Of Mark Nelson
Sent: Tuesday, July 17, 2012 11:40 AM
To: G.Bakalarski at icm.edu.pl
Cc: linux-poweredge-Lists
Subject: Re: R515/H700 high iowait

Hi GB,

Sorry, I should have included more info earlier.

OS: Ubuntu 12.04, Kernel 3.4 (We believe we've had the same issue with 
11.10 and earlier kernels as well though).

Disk Type: Western Digital RE SAS drives, model: WD1000FYYG.

Block size: 64k.

IO pattern: Sequential writes.

Here's an example of the fio commands we used to try and track this down:

XFS on 7 drive raid0, direct IO:
> sudo fio --rw=write --bs=256M --direct=1 --size=64G --runtime=300 --name=foo
Result: ~680MB/s, 98% of latencies are 500ms or lower.


XFS on 7 drive raid0, buffered IO:
> sudo fio --rw=write --bs=256M --direct=0 --size=64G --runtime=300 --name=foo
Result: ~46MB/s, 81.5% of latencies are 2000ms or higher.


No FS on 7 drive raid0, direct IO, 1 job:
> fio --rw=write -ioengine=sync --size=64G --runtime=300 --name=/dev/sdb --numjobs=1 --direct=1 --bs=256M --iodepth=16
Result: ~793MB/s, 100% of latencies are 500ms or lower.


No FS on 7 drive raid0, direct IO, 2 jobs:
> fio --rw=write -ioengine=sync --size=64G --runtime=300 --name=/dev/sdb --numjobs=2 --direct=1 --bs=256M --iodepth=16
Result: ~95MB/s, 100% of latencies are 2000ms or higher.

Also, this problem also appears on smaller raid0 arrays as well which is 
where we first saw it.  It's not as easy to tell what is going on though 
as the performance delta isn't as great.

Mark

On 07/17/2012 10:51 AM, G.Bakalarski at icm.edu.pl wrote:
> What OS type? what disks (SAS 10k, 15k, 7.2k)?
> You wrote RAID0 ... Array block size 64k ???
> Very strange
>
> Maybe directIO is a clue? Which type of directIO do you use?
>
> What is IO pattern - sequential or random or unknown ???
>
> GB
>
>> I was wondering if anyone has had any problems either on the R515 or
>> with the H700 card where there are very high io wait times when multiple
>> writers write to the same raid group.  Basically we first noticed this
>> due to inconsistent buffered IO performance under heavy IO load.  We saw
>> IO wait times as high as 6+ seconds.
>>
>> After that we started testing with fio.  What we noticed is that if
>> writing to say a large raid0 with 7-8 drives we could achieve 800MB/s,
>> but only with large (256MB) IOs using direct IO and a single "job" (ie
>> fio process).  Simply increasing the number of jobs to 2 reduced
>> performance to 80MB/s and dramatically increased IO wait times.  The
>> controller has a BBU and the raid array was configured with adaptive
>> readahead and writeback cache.
>>
>> One thought is that perhaps this is not a problem with the raid
>> controller, but some kind of preemption issue with the io scheduler.  At
>> once point early on I tried switching from cfq to deadline with little
>> apparent change, though I have not yet done exhaustive tests.  I will
>> try to record blktrace output soon to see if that provides any insight
>> into what is going on.
>>
>> In the mean time, has anyone else seen anything like this?
>>
>> Mark
>>
>> _______________________________________________
>> Linux-PowerEdge mailing list
>> Linux-PowerEdge at dell.com
>> https://lists.us.dell.com/mailman/listinfo/linux-poweredge
>>
>
>


-- 
Mark Nelson
Performance Engineer
Inktank

_______________________________________________
Linux-PowerEdge mailing list
Linux-PowerEdge at dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge



More information about the Linux-PowerEdge mailing list