Time-honoured problem : RHEL3 iowait performance

[Robin Humble]

On Sun, Feb 18, 2007 at 08:00:59PM +0000, Brendan Heading wrote:
>Aaron wrote:
>> This is probably not a logistical option, but you might try creating a 
>> non-lvm volume and test the performance on it using the same controller 
>> and same model of disks and raid configuration.  I am just throwing that 
>> out there in the event that you have enough channels/drive bays to do 
>> this.  Should it work out, you could always rsync the data over to the 
>> new volume.
>I tried this and the results are quite profound; it appears that the 
>local disk LVM disk area goes considerably slower than the local disk 
>non-LVM version.
>
>Does anyone know why this might be ?

maybe.
lvm sets a pathetically low read-ahead on the block device.
what does this tell you for your raw and lvm block device? eg.
  blockdev --getra /dev/vol_md0/1
  blockdev --getra /dev/md0

when I set the read-ahead on the lvm to the same value as the non-lvm it
goes at very close to the same speed. I find the below to be a good
(large file) value for both:
  blockdev --setra 16384 <block dev>

tested on software raid0 and raid5. on a raw scsi disk lvm isn't
noticably slower.
I haven't tested the setra theory with hardware raid.

cheers,
robin