PE 1950 and PERC5/i Critical error message

Mon Jul 27 04:37:26 CDT 2009

Hi all,

This morning we had a PE1950 go into a "zombie" state, where we could log in, but could not execute any commands, the OS having put all the filesystems in Read-Only mode.
We had someone reboot it, and going through the logs we saw this message appear in the alertlog during the reboot (and via SNMP traps) :

Severity      : Critical
ID            : 2336
Date and Time : Mon Jul 27 09:42:33 2009
Category      : Storage Service
Description   : Controller event log: Fatal firmware error: Line 1029 in ../../raid/verdeMain.c :  Controller 0 (PERC 5/i Integrated) 

Current firmware rev is : 5.2.1-0067
The OS is : Centos 5.3 x86_64 

Uname -a output : Linux vp-ast-01 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Dmesg output during boot :

megasas: 00.00.04.01-RH1 Thu July 10 09:41:51 PST 2008
megasas: 0x1028:0x0015:0x1028:0x1f03: bus 2:slot 14:func 0
GSI 19 sharing vector 0x52 and IRQ 19
ACPI: PCI Interrupt 0000:02:0e.0[A] -> GSI 78 (level, low) -> IRQ 82
megasas: FW now in Ready state
scsi0 : LSI SAS based MegaRAID driver
  Vendor: SEAGATE   Model: ST3146855SS       Rev: S527
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: SEAGATE   Model: ST3146855SS       Rev: S527
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: DP        Model: BACKPLANE         Rev: 1.05
  Type:   Enclosure                          ANSI SCSI revision: 05
  Vendor: DELL      Model: PERC 5/i          Rev: 1.03
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 285474816 512-byte hdwr sectors (146163 MB)
sda: Write Protect is off
sda: Mode Sense: 1f 00 00 08
SCSI device sda: drive cache: write back
SCSI device sda: 285474816 512-byte hdwr sectors (146163 MB)
sda: Write Protect is off
sda: Mode Sense: 1f 00 00 08
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 >

The TTY_00000000.log shows :

07/24/09  4:55:48: *** DATA ABORT exception: sf=a00ffe34
07/24/09  4:55:48: CPSR=800000d3: N=1,Z=0,C=0,V=0,Q=0   I=1 F=1 T=0 MODE=13(SVC)
07/24/09  4:55:48:  r0=00000140   r1=00000000   r2=00000000   r3=a0c2f400
07/24/09  4:55:48:  r4=a07c6c00   r5=a0c2f400   r6=00000260   r7=00000000
07/24/09  4:55:48:  r8=a0000013   r9=00000000  r10=00000000  r11=a00ffeb0
07/24/09  4:55:48: r12=00000000   lr=a081ce00   pc=a081ce14
EVT#06149-07/24/09  4:55:48:  15=Fatal firmware error: Line 1029 in ../../raid/verdeMain.c

[0]: fp=a00ffdbc, lr=a0898cf8  -  abort_data+114
[1]: fp=a00ffe0c, lr=a000ac78  -  dbits+1788264
[2]: fp=a00ffe30, lr=a000a84c  -  dbits+1787e38
[3]: fp=a00ffe78, lr=a081ce14  -  DM_ProcessMsg+60
[4]: fp=a00ffeb0, lr=a088654c  -  BmpCmdCreateAndFire+120
[5]: fp=a00ffed8, lr=a08509c8  -  flush_cache+3f8
[6]: fp=a00fff5c, lr=a087dc9c  -  set_state+194
[7]: fp=a00fff90, lr=a087d9c8  -  raid_task+318
[8]: fp=a00fffb8, lr=a08978b0  -  main+3b0
[9]: fp=a00fffe4, lr=a0895fe0  -  c_start+30
[10]: fp=a00ffffc, lr=9e8804cc  -  _start+6c
[11]: fp=a0018350, lr=a0006204  -  dbits+17837f0
[12]: fp=a00183fc, lr=4a8  -  000004a8
MonTask: line 1029 in file ../../raid/verdeMain.c
INTCTL=16c00000:1003dcf, IINTSRC=0:0, FINTSRC=0:1002100, CPSR=600000d3, sp=a00ffb00
MegaMon> ������������������������������������������������������������������������������������������������������������������
���������������������������������������������������������������������������������������������������������������������������
������������������

I reckon this server has some old firmware, so we will update that tonight, but I wanted to know if this was a known issue in this firmware version and if it had been fixed.

Thanks for any clues,

Robert von Bismarck
Senior Systems Engineer

VTX SERVICES SA
Une société du groupe VTX Telecom
================================================================
Tél : 021 721 11 11 - Mobile : 079 541 37 28 - VoIP VTX 062 503 79 05
Av. de Lavaux 101 - 1009 Pully
http://www.vtx.ch <blocked::http://www.vtx.ch/>  - robert.von bismarck at vtx-telecom.ch <blocked::mailto:bismarck at vtx-telecom.ch> 
----------------------------------------------------------------
VTX, votre partenaire telecom proche de vous !
================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20090727/f8fa1f17/attachment.htm 