Preventing I/O starvation on MD1000s triggered by a failed disk.
bond.masuda at jlbond.com
Mon Aug 23 23:33:24 CDT 2010
On Mon, 2010-08-23 at 23:01 -0500, Chris Adams wrote:
> I assume you are talking about the case where one drive in a RAID-5
> fails, and while rebuilding, a bad block is discovered on another drive
> (which results in at least that block being unrecoverable, and possibly
> turning the whole array offline, depending on the RAID controller).
That is correct and is the scenario I'm referring to. On the PERC 5/6
controllers, I've seen various behaviors. If the read returns an error
related to the journal of the file system, I've seen the kernel mount
the array read-only. One or two attempts to read the unreadable block
usually just returns I/O error messages and that's it. If an application
or person makes attempts several to read the same blocks that are
unreadable, this usually results in the PERC 5/6 marking that disk as
failed, and hence making the entire array unavailable; at which point
all I/O to that array stops. That is, at least, been my observations.
> Linux software RAID has a way to validate an array. Newer RHEL and
> Fedora (and others I expect) run this check weekly and report any
> faults. Is there a way to do the same thing on the hardware RAID
> controllers (either internal cards or external arrays like EqualLogic)?
Yes, I believe that can be accomplished by issuing a consistency check.
This is done with the following command:
omconfig storage vdisk action=checkconsistency controller=# vdisk=#
I run this once a month on all my RAID-5 arrays that use 500GB or larger
disks. It was also recommended to me by a Dell technician.
However, what I actually do in practice aside, I sometimes wonder if
this is a good thing or not? If I recall correctly, an error occurs
about every 12TB read per disk. Every time I do a consistency check, the
entire array is read and so it would seem that I'm increasing my chances
of the eventual and inevitable encounter with a URE? On the other hand,
if there is a URE, I would rather find out about it before the RAID-5
becomes degraded so I can actually recover.
Either way, having encountered numerous UREs now, and often during a
degraded array rebuild, I think the more effective solution is simply to
go with RAID-6 rather than relying on regular consistency checks.
More information about the Linux-PowerEdge