R515/H700 high iowait

Dennis Jacobfeuerborn dennisml at conversis.de
Wed Jul 18 11:59:18 CDT 2012


On 07/18/2012 05:35 PM, Mark Nelson wrote:
> On 07/18/2012 08:52 AM, Dennis Jacobfeuerborn wrote:
>> On 07/18/2012 03:37 PM, Mark Nelson wrote:
>>> On 7/18/12 7:26 AM, G.Bakalarski at icm.edu.pl wrote:
>>>>
>>>>> This is with XFS and barriers were left enabled.  We also tested having
>>>>> fio write directly to the device.
>>>>
>>>> Hi Mark,
>>>>
>>>> Did you enable any virtualization settings ???
>>>> Does your Ubuntu runs on bare metal hardware without
>>>> any xen, citrix or whatever???
>>>>
>>>> Did you changes anything in BIOS setting eg.
>>>> IOMMU on (DMA virtualisation) ? What is your
>>>> C states setting? Is BIOS optimized for Performance
>>>> (CPU seting)?
>>>>
>>>> Have you monitored IO behaviour with e.g. iostat -xzk 2 ???
>>>> What was a size of queue length? Did it change during tests?
>>>> Have you tested IO with another tools like e.g. iozone or others ??
>>>>
>>>> Bests,
>>>>
>>>> GB
>>>>
>>>>
>>>
>>> Hi GB,
>>>
>>> Looks like virtualization technology is on in the bios, but DMA
>>> virtualization is off.  OS is running on bare metal.
>>>
>>> C1E is on, which should probably be disabled.  CPU Power and Performance
>>> Management is set to Maximum Performance.
>>>
>>> While I did not run iostat specifically, I did run collectl with the
>>> disk subsystem on during some of the tests.
>>>
>>> Here's an example when things are bad (ie multiple writers):
>>>
>>>> #<---------reads---------><---------writes---------><--------averages-------->  Pct
>>>> #Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
>>>> 11:18:40 sdb              0      0    0    0   24964      0  196  127     127   142   725      5   99
>>>> 11:18:41 sdb              0      0    0    0   22272      0  174  128     128   142   925      5   99
>>>> 11:18:42 sdb              0      0    0    0   30464      0  238  128     128   145   716      4   99
>>>
>>> And when things are good (256MB requests, DirectIO, one writer):
>>>
>>>> #<---------reads---------><---------writes---------><--------averages-------->  Pct
>>>> #Time     Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
>>>> 11:29:32 sdb              0      0    0    0  737920      0 5765  128     128   119    20      0   87
>>>> 11:29:33 sdb              0      0    0    0  798464      0 6238  128     128   126    20      0   95
>>>> 11:29:34 sdb              0      0    0    0  798976      0 6242  128     128   126    20      0   94
>>>
>>
>> May I ask which tool you use to get this output? This particular way to
>> display these numbers looks quite nice.
>>
>> Regards,
>>    Dennis
>>
>> _______________________________________________
>> Linux-PowerEdge mailing list
>> Linux-PowerEdge at dell.com
>> https://lists.us.dell.com/mailman/listinfo/linux-poweredge
> 
> Hi Dennis,
> 
> This is a tool written by Mark Seger at HP called Collectl.  It can be 
> used to monitor statistics for disk, network, cpu, memory, lustre, IB, 
> and other things too.  It also supports per process and slab monitoring.
> 
> The above statistics can be generated by using the disk subsystem with a 
> filter to specifically look at sdb:
> 
> collectl -sD -oT --dskfilt sdb
> 
> The webpage is here:
> 
> http://collectl.sourceforge.net/
> 

Thanks for the pointer. I'm going to take a closer especially since it's
also available via epel so there is really no excuse not to use it on RHEL
based systems.

Regards,
  Dennis



More information about the Linux-PowerEdge mailing list