What to do when OM stops working?

[Trond Hasle Amundsen]

"Flaherty, Patrick" <pflaherty at wsi.com> writes:

>> On occasion Openmanage stops working, on seemingly random times and on
>> random servers. Omreport will show output like this:
>> 
>>   # omreport chassis memory
>>   Memory Information
>>   
>>   Error : Memory object not found
>> 
>> Similar errors for all other components. Sometimes it helps to restart
>> the services ('srvadmin-services restart'), but most often it 
>> does not.
>> Only thing that seems to help is to power off the server. The servers
>> are running OM 5.4.0 on RHEL4 and RHEL5. The problem applies to
>> different poweredge models.
>> 
>> Have any of you experienced the same, and if so, do you have a better
>> solution than powering off the server?
>
> I think it might be an ipmi bug/incompatibility/gremlin/evil spirit.
> Seen a similar bug on a bunch of different models and patch levels for
> `omreport chassis`. 

Yep, I was thinking the same. It happens rarely, but often enough to
become annoying with 100+ servers :/

> Try :
>  #this command stops omsa, start ipmi, and starts omsa
>  srvadmin-services.sh stop && service ipmi start && srvadmin-services.sh
> start

Doesn't help if OM is completely uncooperative, which unfortunately does
happen on occasion..

> On a side note, most of the monitoring scripts I've seen that run
> omreport directly don't catch this condition. I modified mine to error
> out if too few lines come back from omreport. You could also make a sudo
> rule to allow your monitoring user to run `srvadmin-services.sh status`,
> but that seemed like more work.

Our monitoring solution (Nagios/check_openmanage) will also catch this
error, fortunately :)

BTW thanks for the response, nice to see that I'm not the only one
seeing this problem. Perhaps OM 5.5 will perform better.

Cheers,
-- 
Trond Hasle Amundsen <t.h.amundsen at usit.uio.no>
Gruppe for basis systemdrift (BSD), SAPP, USIT
Tel. +47 22840058 (office)