R515/H700 high iowait

Mark Nelson mark.nelson at inktank.com
Tue Jul 17 11:39:42 CDT 2012

Hi GB,

Sorry, I should have included more info earlier.

OS: Ubuntu 12.04, Kernel 3.4 (We believe we've had the same issue with 
11.10 and earlier kernels as well though).

Disk Type: Western Digital RE SAS drives, model: WD1000FYYG.

Block size: 64k.

IO pattern: Sequential writes.

Here's an example of the fio commands we used to try and track this down:

XFS on 7 drive raid0, direct IO:
> sudo fio --rw=write --bs=256M --direct=1 --size=64G --runtime=300 --name=foo
Result: ~680MB/s, 98% of latencies are 500ms or lower.

XFS on 7 drive raid0, buffered IO:
> sudo fio --rw=write --bs=256M --direct=0 --size=64G --runtime=300 --name=foo
Result: ~46MB/s, 81.5% of latencies are 2000ms or higher.

No FS on 7 drive raid0, direct IO, 1 job:
> fio --rw=write -ioengine=sync --size=64G --runtime=300 --name=/dev/sdb --numjobs=1 --direct=1 --bs=256M --iodepth=16
Result: ~793MB/s, 100% of latencies are 500ms or lower.

No FS on 7 drive raid0, direct IO, 2 jobs:
> fio --rw=write -ioengine=sync --size=64G --runtime=300 --name=/dev/sdb --numjobs=2 --direct=1 --bs=256M --iodepth=16
Result: ~95MB/s, 100% of latencies are 2000ms or higher.

Also, this problem also appears on smaller raid0 arrays as well which is 
where we first saw it.  It's not as easy to tell what is going on though 
as the performance delta isn't as great.


On 07/17/2012 10:51 AM, G.Bakalarski at icm.edu.pl wrote:
> What OS type? what disks (SAS 10k, 15k, 7.2k)?
> You wrote RAID0 ... Array block size 64k ???
> Very strange
> Maybe directIO is a clue? Which type of directIO do you use?
> What is IO pattern - sequential or random or unknown ???
> GB
>> I was wondering if anyone has had any problems either on the R515 or
>> with the H700 card where there are very high io wait times when multiple
>> writers write to the same raid group.  Basically we first noticed this
>> due to inconsistent buffered IO performance under heavy IO load.  We saw
>> IO wait times as high as 6+ seconds.
>> After that we started testing with fio.  What we noticed is that if
>> writing to say a large raid0 with 7-8 drives we could achieve 800MB/s,
>> but only with large (256MB) IOs using direct IO and a single "job" (ie
>> fio process).  Simply increasing the number of jobs to 2 reduced
>> performance to 80MB/s and dramatically increased IO wait times.  The
>> controller has a BBU and the raid array was configured with adaptive
>> readahead and writeback cache.
>> One thought is that perhaps this is not a problem with the raid
>> controller, but some kind of preemption issue with the io scheduler.  At
>> once point early on I tried switching from cfq to deadline with little
>> apparent change, though I have not yet done exhaustive tests.  I will
>> try to record blktrace output soon to see if that provides any insight
>> into what is going on.
>> In the mean time, has anyone else seen anything like this?
>> Mark
>> _______________________________________________
>> Linux-PowerEdge mailing list
>> Linux-PowerEdge at dell.com
>> https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Mark Nelson
Performance Engineer

More information about the Linux-PowerEdge mailing list