Preventing I/O starvation on MD1000s triggered by a failed disk.

John LLOYD jal at mdacorporation.com
Tue Aug 24 11:27:52 CDT 2010


...<snip>...

> Yes, I believe that can be accomplished by issuing a consistency check.
> This is done with the following command:
> 
> omconfig storage vdisk action=checkconsistency controller=# vdisk=#
> 
> I run this once a month on all my RAID-5 arrays that use 500GB or
> larger
> disks. It was also recommended to me by a Dell technician.

Your controller does Patrol Reads automatically, so a manual run should not be necessary unless Patrol Read is turned off.


> 
> However, what I actually do in practice aside, I sometimes wonder if
> this is a good thing or not? If I recall correctly, an error occurs
> about every 12TB read per disk. Every time I do a consistency check,

Huh?  Disk error rates are given as bits read without error.  That is bits, not blocks.  Blocks are self-correcting for many errors (obviously not all otherwise disks would never fail).  A single-bit error on a block is corrected because the drive records redundancy info with each block.  But what you get at the drive SAS interface is what counts.

For example Seagate says "Unrecoverable Data Errors are specified at less than 1 sector in error per 10**16 bits transferred."  (Cheetah 15k SAS drives) Ten to the sixteenth is a lot more than a dozen terabytes.

Of course, that is theory.  Practise appears to be different!


> the
> entire array is read and so it would seem that I'm increasing my
> chances
> of the eventual and inevitable encounter with a URE? On the other hand,
> if there is a URE, I would rather find out about it before the RAID-5
> becomes degraded so I can actually recover.
> 
> Either way, having encountered numerous UREs now, and often during a
> degraded array rebuild, I think the more effective solution is simply
> to
> go with RAID-6 rather than relying on regular consistency checks.
> 
> -Bond

I would suggest that RAID-5 or RAID-6 is not acceptable when a rebuild takes enough time to put data at risk from another failure.  We (almost) always use RAID-10 as a result of this logic.  Disks are very inexpensive when compared with the cost of replacing or recovering from lost data.


--John



> 
> 
> 
> ------------------------------
> 
> Message: 7
> Date: Tue, 24 Aug 2010 14:56:03 +1000
> From: Jeff Ewing <jewing at aconex.com>
> Subject: Re: Preventing I/O starvation on MD1000s triggered by a
> 	failed disk.
> To: Bond Masuda <bond.masuda at jlbond.com>
> Cc: linux-poweredge at dell.com
> Message-ID: <20100824045603.GC8459 at wsjewing.yarra.acx>
> Content-Type: text/plain; charset=us-ascii
> 
> On Mon, Aug 23, 2010 at 08:46:03PM -0700, Bond Masuda wrote:
> > Do you know if you encountered a URE? I've been running into URE's on
> >
> I have now exported the controller log from the controller that failed.
> I didnt see a URE around the time the that my NFS requests stopped
> (around 11:38)(rebuild was actually 19% complete):
> 
> 08/16/10 11:37:32: EVT#47438-08/16/10 11:37:32: 103=Rebuild progress on
> PD 33(e0x32/s14) is 18.98%(3262s)^M
> 08/16/10 11:40:10: EVT#47439-08/16/10 11:40:10: 103=Rebuild progress on
> PD 33(e0x32/s14) is 19.98%(3420s)^M
> 08/16/10 11:42:48: EVT#47440-08/16/10 11:42:48: 103=Rebuild progress on
> PD 33(e0x32/s14) is 20.98%(3578s)^M
> 08/16/10 11:45:26: EVT#47441-08/16/10 11:45:26: 103=Rebuild progress on
> PD 33(e0x32/s14) is 21.98%(3736s)^M
> 08/16/10 11:48:04: EVT#47442-08/16/10 11:48:04: 103=Rebuild progress on
> PD 33(e0x32/s14) is 22.98%(3894s)^M
> 08/16/10 11:50:42: EVT#47443-08/16/10 11:50:42: 103=Rebuild progress on
> PD 33(e0x32/s14) is 23.98%(4052s)^M
> 08/16/10 11:53:20: EVT#47444-08/16/10 11:53:20: 103=Rebuild progress on
> PD 33(e0x32/s14) is 24.98%(4210s)^M
> 
> 
> There were errors later, my collegue tried to set the rebuild rate to
> 5% to bring the server back:
> 
> 08/16/10 13:22:54: EVT#47479-08/16/10 13:22:54: 103=Rebuild progress on
> PD 33(e0x32/s14) is 58.96%(9584s)^M
> 08/16/10 13:25:31: EVT#47480-08/16/10 13:25:31: 103=Rebuild progress on
> PD 33(e0x32/s14) is 59.96%(9741s)^M
> 08/16/10 13:27:06: NCQ Mode value is not valid or not found, return
> default^M
> 08/16/10 13:27:06: EVT#47481-08/16/10 13:27:06:  40=Rebuild rate
> changed to 5%^M
> 08/16/10 13:31:12: mfiIsr: idr=00000020^M
> 08/16/10 13:31:12: Driver detected possible FW hang, halting FW.^M
> 08/16/10 13:31:12: Pending Command Details:^M
> 
> 
> 
> > Were there any kernel messages when the I/O stopped? Have you dumped
> the
> > log from the controller? If not, that's where I would start
> looking....
> 
> 
> There were alot of megasas messages in the debug log at the time the
> NFS requests stopped:
> 
> Aug 16 11:38:06 nas2 kernel: sd 1:2:0:0: megasas: RESET -356143413
> cmd=2a retries=0
> Aug 16 11:38:06 nas2 kernel: megasas: [ 0]waiting for 4 commands to
> complete
> Aug 16 11:38:07 nas2 kernel: megasas: reset successful
> 
> These were in dmesg also:
> 
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc4 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc5 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc6 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc7 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc0 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc1 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc2 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc3 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc4 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc5 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc6 not
> found!
> mptctldrivers/message/fusion/mptctl.c::mptctl_ioctl() @596 - ioc7 not
> found!
> sd 1:2:0:0: megasas: RESET -356143413 cmd=2a retries=0
> megasas: [ 0]waiting for 4 commands to complete
> megasas: reset successful
> sd 1:2:0:0: megasas: RESET -356143737 cmd=2a retries=0
> megasas: [ 0]waiting for 5 commands to complete
> megasas: reset successful
> sd 1:2:0:0: megasas: RESET -356143749 cmd=2a retries=0
> megasas: [ 0]waiting for 5 commands to complete
> megasas: reset successful
> sd 1:2:0:0: megasas: RESET -356143771 cmd=2a retries=0
> megasas: [ 0]waiting for 4 commands to complete
> megasas: reset successful
> sd 1:2:0:0: megasas: RESET -356143781 cmd=2a retries=0
> megasas: [ 0]waiting for 5 commands to complete
> 
> 
> 
> Thank you.
> 
> Jeff Ewing
> 
> 
> 
> 
> ------------------------------
> 
> _______________________________________________
> Linux-PowerEdge mailing list
> Linux-PowerEdge at dell.com
> https://lists.us.dell.com/mailman/listinfo/linux-poweredge
> Please read the FAQ at http://lists.us.dell.com/faq
> 
> End of Linux-PowerEdge Digest, Vol 74, Issue 39
> ***********************************************



More information about the Linux-PowerEdge mailing list