[Linux-PowerEdge] Intermittent i/o error while reading the data from Hitachi Storage for M610

Gopala_Rao at Dell.com Gopala_Rao at Dell.com
Thu Feb 21 01:55:27 CST 2013


Hi Team,

Getting I/O buffer error and not able to figure out whether multipathing/driver/firmware /HBA card issue for the subject mentioned error.

Issue                                     : Intermittent i/o error while reading the data from Hitachi  Storage
Server                                   : M610
FC Card                                 : Qlogic FC8 Mezzanine Card QME2572 on Mezzanine C1
Operating System            : RHEL 5.7

Current HBA card driver version: 8.03.07.03.05.07


Does anyone ever experienced this problem?

What could be the cause for this error messages?

Any  help on this would be highly appreciated.

Dmesg logs as follows.


qla2xxx 0000:04:00.0: qla2xxx_eh_host_reset: reset succeeded
qla2xxx 0000:04:00.1: qla2xxx_eh_bus_reset: reset succeeded
sd 3:0:0:18: scsi: Device offlined - not ready after error recovery
sd 3:0:0:18: rejecting I/O to offline device
qla2xxx 0000:04:00.1: scsi(4:0:16): ADAPTER RESET ISSUED.
qla2xxx 0000:04:00.1: Performing ISP error recovery - ha= ffff81203e0d84f8.
qla2xxx 0000:04:00.1: LIP reset occured (f7f7).
qla2xxx 0000:04:00.1: LOOP UP detected (4 Gbps).
qla2xxx 0000:04:00.1: qla2xxx_eh_host_reset: reset succeeded
sd 4:0:0:16: scsi: Device offlined - not ready after error recovery
sd 4:0:0:16: scsi: Device offlined - not ready after error recovery
sd 4:0:0:16: scsi: Device offlined - not ready after error recovery
sd 4:0:0:16: scsi: Device offlined - not ready after error recovery
sd 4:0:0:16: scsi: Device offlined - not ready after error recovery
sd 4:0:0:16: scsi: Device offlined - not ready after error recovery
sd 4:0:0:19: scsi: Device offlined - not ready after error recovery
sd 4:0:0:16: rejecting I/O to offline device
sd 4:0:0:16: rejecting I/O to offline device
sd 4:0:0:16: rejecting I/O to offline device
sd 4:0:0:16: rejecting I/O to offline device
sd 4:0:0:16: SCSI error: return code = 0x070e0000
end_request: I/O error, dev sdj, sector 1048976
sd 4:0:0:16: SCSI error: return code = 0x070e0000
end_request: I/O error, dev sdj, sector 1049088
sd 4:0:0:19: SCSI error: return code = 0x070e0000
end_request: I/O error, dev sdm, sector 1049080
sd 4:0:0:18: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdl, sector 6736
Buffer I/O error on device dm-9, logical block 3178
lost page write due to I/O error on dm-9
sd 3:0:0:16: SCSI error: return code = 0x00010000
Aborting journal on device dm-9.
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 1049080
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 2120
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 1048976
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 1049088
Buffer I/O error on device dm-9, logical block 524351
lost page write due to I/O error on dm-9
Buffer I/O error on device dm-9, logical block 841
lost page write due to I/O error on dm-9
Buffer I/O error on device dm-9, logical block 524290
lost page write due to I/O error on dm-9
Buffer I/O error on device dm-9, logical block 524291
lost page write due to I/O error on dm-9
Buffer I/O error on device dm-9, logical block 524352
lost page write due to I/O error on dm-9
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
end_request: I/O error, dev sdb, sector 384
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 131472
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 1048960
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 3604880
Buffer I/O error on device dm-9, logical block 0
lost page write due to I/O error on dm-9
Buffer I/O error on device dm-9, logical block 1
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 384
Buffer I/O error on device dm-9, logical block 0
lost page write due to I/O error on dm-9
EXT3-fs error (device dm-9) in ext3_dirty_inode: Journal has aborted
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 384
Buffer I/O error on device dm-9, logical block 0
lost page write due to I/O error on dm-9
EXT3-fs error (device dm-9) in ext3_dirty_inode: Journal has aborted
EXT3-fs error (device dm-9) in ext3_dirty_inode: Journal has aborted
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 384
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 384
ext3_abort called.
EXT3-fs error (device dm-9): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
lost page write due to I/O error on dm-9
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-9) in ext3_reserve_inode_write: Journal has aborted
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 1315488
Aborting journal on device dm-10.
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 1314248
printk: 6 messages suppressed.
Buffer I/O error on device dm-10, logical block 1545
lost page write due to I/O error on dm-10
ext3_abort called.
EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
qla2xxx 0000:04:00.0: Unable to read SFP data (102/a0/0).
qla2xxx 0000:04:00.1: Unable to read SFP data (102/a0/0).
qla2xxx 0000:04:00.1: LIP reset occured (f7f7).
qla2xxx 0000:04:00.0: LOOP DOWN detected (0 0 0 0).
qla2xxx 0000:04:00.0: LOOP UP detected (4 Gbps).
sddlmaa:<6>sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
Dev sddlmaa: unable to read RDB block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sddlmaa, logical block 0
unable to read partition table
sdk : READ CAPACITY failed.
sdk : status=0, message=00, host=1, driver=00
sdk : sense not available.
sdk: Write Protect is off
sdk: Mode Sense: 00 00 00 00
sdk: asking for cache data failed
sdk: assuming drive cache: write through
sdk: detected capacity change from 16866017280 to 0
sddlmab:<3>Buffer I/O error on device sddlmab, logical block 0
Dev sddlmab: unable to read RDB block 0
unable to read partition table
sdl : READ CAPACITY failed.
sdl : status=0, message=00, host=1, driver=00
sdl : sense not available.
sdl: Write Protect is off
sdl: Mode Sense: 00 00 00 00
sdl: asking for cache data failed
sdl: assuming drive cache: write through
sdl: detected capacity change from 16866017280 to 0
sddlmac:Dev sddlmac: unable to read RDB block 0
unable to read partition table
sddlmad:<6>sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
Dev sddlmad: unable to read RDB block 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
sd 3:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sde, sector 0
unable to read partition table
sdn : READ CAPACITY failed.
sdn : status=0, message=00, host=1, driver=00
sdn : sense not available.
sdn: Write Protect is off
sdn: Mode Sense: 00 00 00 00
sdn: asking for cache data failed
sdn: assuming drive cache: write through
sdn: detected capacity change from 16866017280 to 0
sddlmae:Dev sddlmae: unable to read RDB block 0
unable to read partition table
sdo : READ CAPACITY failed.
sdo : status=0, message=00, host=1, driver=00
sdo : sense not available.
sdo: Write Protect is off
sdo: Mode Sense: 00 00 00 00
sdo: asking for cache data failed
sdo: assuming drive cache: write through
sdo: detected capacity change from 16866017280 to 0
sddlmaf:Dev sddlmaf: unable to read RDB block 0
unable to read partition table
sdp : READ CAPACITY failed.
sdp : status=0, message=00, host=1, driver=00
sdp : sense not available.
sdp: Write Protect is off
sdp: Mode Sense: 00 00 00 00
sdp: asking for cache data failed
sdp: assuming drive cache: write through
sdp: detected capacity change from 16866017280 to 0
sddlmag:Dev sddlmag: unable to read RDB block 0
unable to read partition table
sdq : READ CAPACITY failed.
sdq : status=0, message=00, host=1, driver=00
sdq : sense not available.
sdq: Write Protect is off
sdq: Mode Sense: 00 00 00 00
sdq: asking for cache data failed
sdq: assuming drive cache: write through
sdq: detected capacity change from 16866017280 to 0
sddlmah:Dev sddlmah: unable to read RDB block 0
unable to read partition table
printk: 62 messages suppressed.
Buffer I/O error on device dm-12, logical block 1129
lost page write due to I/O error on dm-12
Buffer I/O error on device dm-12, logical block 1129
lost page write due to I/O error on dm-12
Buffer I/O error on device dm-12, logical block 1129
lost page write due to I/O error on dm-12
Buffer I/O error on device dm-12, logical block 1129
lost page write due to I/O error on dm-12
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 30409088
Buffer I/O error on device dm-12, logical block 0
lost page write due to I/O error on dm-12
sd 3:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 20971904
Buffer I/O error on device dm-11, logical block 0
lost page write due to I/O error on dm-11
ext3_abort called.
EXT3-fs error (device dm-9): ext3_put_super: Couldn't clean up the journal
ext3_abort called.
EXT3-fs error (device dm-10): ext3_put_super: Couldn't clean up the journal
qla2xxx 0000:04:00.0: Unable to read SFP data (102/a0/0).
qla2xxx 0000:04:00.1: Unable to read SFP data (102/a0/0).
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-10, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-11, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-12, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-11, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-9, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
qla2xxx 0000:04:00.1: LOOP DOWN detected (0 0 0 0).
qla2xxx 0000:04:00.1: LOOP UP detected (4 Gbps).
qla2xxx 0000:04:00.0: Unable to read SFP data (102/a0/0).
qla2xxx 0000:04:00.1: Unable to read SFP data (102/a0/0).
qla2xxx 0000:04:00.1: LOOP DOWN detected (2 0 0 0).
rport-4:0-0: blocked FC remote port time out: saving binding
sd 4:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdj, sector 1061248
sd 4:0:0:16: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdj, sector 1069440
sd 4:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdm, sector 36304
qla2xxx 0000:04:00.1: LOOP UP detected (4 Gbps).
qla2xxx 0000:04:00.0: Unable to read SFP data (102/a0/0).
qla2xxx 0000:04:00.1: Unable to read SFP data (102/a0/0).
qla2xxx 0000:04:00.1: LOOP DOWN detected (2 0 0 0).
rport-4:0-0: blocked FC remote port time out: saving binding
sd 4:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdm, sector 1382120
sd 4:0:0:19: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdm, sector 1049080
qla2xxx 0000:04:00.1: LOOP UP detected (4 Gbps).
opcmona[20395]: segfault at 0000000040000000 rip 0000003375c728ae rsp 000000004214aca0 error


Regards
Gopala Rao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20130221/a24ee2ea/attachment-0001.html 


More information about the Linux-PowerEdge mailing list