EMC Powerpath / RHEL 4 u6 / sd* devices high load

[Robert Jaroszuk]

Hi there.

I have a 3-node Oracle 10g RAC cluster on RedHat 4 update 6 (2.6.9-67)
with an EMC storage connected through FC.
The problem is, that after some time, a number of sdX devices seems to
have a 100% load:

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s   wkB/s   avgrq-sz avgqu-sz   await  svctm  %util
sda          0.02   0.02  0.66 53.87   33.93  431.09    16.97   215.54  8.53    59.78   22.09  18.70 101.98
sdb          0.01   0.00  0.02  0.00    0.07    0.00     0.03   0.00    3.07     0.00    2.41   0.06   0.00
sdc          0.01   0.00  0.02  0.00    0.07    0.00     0.03   0.00    3.07     0.00    2.64   0.10   0.00
.
.
sdx          0.01   0.00  2.15  1.02    5.43    1.02     2.71   0.51    2.04    85.60    0.37 322.26 101.89
sdy          0.01   0.00  0.71  0.00    4.93    0.00     2.46   0.00    6.94     1.02    0.74 1435.65 101.94

But server was NOT used at that time. It does nothing. It is
impossible, that this %util was caused by real load.

I noticed that the problem lies somewhere on kernel level, because the
number of I/Os currently in /proc/diskstats is very huge.

   8    0 sda 32652 1095 1682597 110960 2678799 774 21436528 59785187
4294966435 50633734 2909368627
  65  112 sdx 106545 306 269352 30959 50429 0 50429 27440 4294967295
50588021 4244391545

It is the third field from the right (4294966435 and 4294967295).
Notice that this number is 232 - 861 and 232 - 1.

I also use EMC PowerPath (EMCpower.LINUX-5.0.0-157) and I think that
this problem may be caused
by this software. Those devices behave correctly unless I load kernel
modules from emc. After that it's just
a matter of minutes when devices have 100%load in iostat.

Unfortunately, Dell and EMC support told me that they do not support
PowerPath for RH update 6 (only for u4),
so I can downgrade my systems (unacceptable for me) or I can pis^H^H^H
go away...

Maybe you have any idea what can I do with that?


-- 
... Robert Jaroszuk ...
GCS/IT/O d? s: a- C++ ULB++++$ P+ L++++$ E- W++ K- N+ DI+ V-
w M- PS+ PE Y(+) PGP-(+++) t-- 5? X R !tv b++>++++ D- y+ G++
. http://zim.iq.pl/ . RJ735-RIPE . http://zim.iq.pl/photo/ .
.. The superior warrior wins without fighting -- Sun Tzu. ..

	-> New photos: http://zim.iq.pl/photo/