[Linux-PowerEdge] Extremely poor performance with LVM vs. RAW disc
L. A. Walsh, Tlinx Solutions
dell at tlinx.org
Mon Nov 5 07:39:12 CST 2012
Gregory Gulik wrote:
> I am working on setting up a large RAID setup for a variety of
> functions that aren't very performance critical. I'm trying to use LVM
> on the filesystem to allow me to carve up the space for different
> functions with the flexibility to adjust things later if necessary.
> However I'm finding the performance of the LVM based filesystem is
> many times slower than the raw partition. I guess I expect some
> overhead but not 10x.
> Specifics of my test configuration:
> Dell PowerEdge 1950 with 12G RAM
> Dell PERC 6/E RAID controller
> Dell MD1000 Controller
> 8 x Seagate 2TB SATA drives in RAID-6 config (otherwise using default
> settings in OpenManage)
> Operating System is CentOS 6.3
> Total space available is 12TB and partitioned using parted as follows:
> Disk /dev/sdb: 12.0TB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Number Start End Size File system Name Flags
> 1 17.4kB 10.0TB 10000GB data1 lvm
> 2 10.0TB 12.0TB 1999GB ext4 data3
> I then created a 2TB ext4 filesystem mounted on /data2 using /dev/sdb2
> physical partition and created a 2TB logical volume using the LVM
> partition and also formatted it with ext4 mounted on /data2
> Then when I compare basic read/write performance the difference is
> shocking. Using:
> time sh -c "dd if=/dev/zero of=ddfile bs=8k count=2000000 && sync"
> time dd if=ddfile of=/dev/null bs=8k
> Below are the average of several attempts
> Raw partition:
> Read: 48.3 secs
> Write: 70.2 secs
> LVM partition:
> Read: 288.3 secs
> Write: 701.6 secs\
Couple things are in play here.
1) you need to start your pv's on a stripe unit boundary for large I/O
For optimal smaller I/O, you want to start it on a stripe-width
boundary. This is a parameter
you give to pvcreate. (data offset). So I have 12 datax64k = 768k -- I
start my 1st pv @ 768k.
Then you layer a vg on top of that -- it has a segment size -- you want
that to be a multiple of
your stripwidth for optimal I/O (I messed that up.. it's a
strip-multple, but not a stripe-width
boundary.... so I get lower perf on some small writes than I might
otherwise. Too much of a pain
Then your lv's have a start position and chunksize -- you want those to
multiple out as well.
Then your specify your strip width and size to as file system param (or
at least you do on
file systems that support RAID -- like XFS)... I assume, since you are
using ext4, that it also
has such params.
While the OS knows the layout of disks in the pv -- it doesn't know
about the lvm layers on top of
that... so the file systems generally don't even try to guess about
proper alignment -- that's all manual
AFAIK...(at least on linux)...
If you want to test disk speed -- you should pre-allocate the file as
contiguous (you can
do this on xfs... dunnow about ext4)....then when you 'dd', use direct
I/O -- and use
nocreat,notrunc options on 'dd', so you are reading/writing to the same
area and aren't
exercising the file-system's ability to allocate space, nor the OS's
Doing those things, should get you alot closer to your theoreticals...
Hopefully I explained enough...if not, feel free to ask more...
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Linux-PowerEdge