[Linux-PowerEdge] Extremely poor performance with LVM vs. RAW disc
bond.masuda at jlbond.com
Mon Nov 5 11:19:24 CST 2012
The other thing to consider is the hardware architecture and how you
make use of it. The MD1000 and PERC6/E setup has 4 lanes from the
controller to the MD1000 head unit, which uses multipliers to make those
4 lanes into 4x4=16 lanes, of which only 15 are used. Let's call them
drives 0 thru 14. If you try to access drives 0-3, or 4-7, or 8-11, or
12-14 within those sets, you'll have contention at the lane multiplier.
A RAID-0 over 0,4,8,12 is going to be faster than a RAID-0 over 0,1,2,3.
These results I've verified and tested before, so it's not just
This is only useful of course, if you have the liberty to choose to use
only certain subsets and running in unified mode (vs split mode).
On Mon, 2012-11-05 at 11:09 -0600, Gregory Gulik wrote:
> Thanks for this great info.
> How would I find out if the stripes line up and if they don't how does
> one go about fixing them?
> I'm looking at the mkfs.ext4 man page and is it the "-b block_size"
> option or something else?
> BTW, in experimenting further over the weekend I found the PERC 6/E
> firmware was out of date so I updated it and now performance is
> significantly better. That said I want to make sure all settings are
> as optimal as they could be before we put real data on the server.
> On 11/5/12 7:39 AM, L. A. Walsh, Tlinx Solutions wrote:
> > ----
> > Couple things are in play here.
> > 1) you need to start your pv's on a stripe unit boundary for large
> > I/O performance.
> > For optimal smaller I/O, you want to start it on a stripe-width
> > boundary. This is a parameter
> > you give to pvcreate. (data offset). So I have 12 datax64k = 768k
> > -- I start my 1st pv @ 768k.
> > Then you layer a vg on top of that -- it has a segment size -- you
> > want that to be a multiple of
> > your stripwidth for optimal I/O (I messed that up.. it's a
> > strip-multple, but not a stripe-width
> > boundary.... so I get lower perf on some small writes than I might
> > otherwise. Too much of a pain
> > to reformat.
> > Then your lv's have a start position and chunksize -- you want those
> > to multiple out as well.
> > Then your specify your strip width and size to as file system param
> > (or at least you do on
> > file systems that support RAID -- like XFS)... I assume, since you
> > are using ext4, that it also
> > has such params.
> > While the OS knows the layout of disks in the pv -- it doesn't know
> > about the lvm layers on top of
> > that... so the file systems generally don't even try to guess about
> > proper alignment -- that's all manual
> > AFAIK...(at least on linux)...
> > If you want to test disk speed -- you should pre-allocate the file
> > as contiguous (you can
> > do this on xfs... dunnow about ext4)....then when you 'dd', use
> > direct I/O -- and use
> > nocreat,notrunc options on 'dd', so you are reading/writing to the
> > same area and aren't
> > exercising the file-system's ability to allocate space, nor the OS's
> > buffer system.
> > Doing those things, should get you alot closer to your
> > theoreticals...
> > Hopefully I explained enough...if not, feel free to ask more...
> > Linda
> Gregory Gulik
> email | delivered
> REACHMAIL Inc. | Main 888-947-3224 | 208 S.
> LaSalle | Suite1427 | Chicago, IL 60604
> ggulik at reachmail.com | Direct 312-229-0132 |
> Linux-PowerEdge mailing list
> Linux-PowerEdge at dell.com
JL Bond Consulting / www.JLBond.com
Email: bond.masuda at JLBond.com
This message contains confidential information and is intended only for
the individual(s) named. If you are not the named addressee you should
not disseminate, distribute or copy this e-mail. Please notify the
sender immediately by e-mail if you have received this e-mail by mistake
and delete this e-mail from your system. E-mail transmission cannot be
guaranteed to be secure or error-free as information could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
contain viruses. The sender therefore does not accept liability for any
errors or omissions in the contents of this message, which arise as a
result of e-mail transmission.
More information about the Linux-PowerEdge