datamgr page allocation failure

[Rich Bishop]

We have a number of m905 blades which have been working fine for about 9 months
now. We use Dell's omreport to monitor them via nagios. The machines are running
RHEL4 64bit with the latest version of omsa (5.5).

Over the weekend our DBAs upgraded Oracle on some of these blades to 10.2.0.4
(from some older version of Oracle 10). Since this upgrade I'm seeing lots of
page allocation failures datamgr3:

Jun  8 08:29:27 bluebird kernel: dsm_sa_datamgr3: page allocation failure. order:4, mode:0xd1
Jun  8 08:29:27 bluebird kernel: 
Jun  8 08:29:27 bluebird kernel: Call Trace:<ffffffff8015fa55>{__alloc_pages+789} <ffffffff8015f101>{__free_pages_ok+240} 
Jun  8 08:29:27 bluebird kernel:        <ffffffff8012105d>{dma_alloc_pages+125} <ffffffff80121284>{dma_alloc_coherent+97} 
Jun  8 08:29:27 bluebird kernel: <ffffffffa00b3e02>{:megaraid_sas:megasas_mgmt_ioctl_fw+464} 
Jun  8 08:29:27 bluebird kernel: <ffffffffa00b4298>{:megaraid_sas:megasas_mgmt_compat_ioctl+410} 
Jun  8 08:29:27 bluebird kernel:        <ffffffff801a1570>{compat_sys_ioctl+235} <ffffffff80126437>{cstar_do_call+27} 
Jun  8 08:29:27 bluebird kernel:        
Jun  8 08:29:27 bluebird kernel: Mem-info:
Jun  8 08:29:27 bluebird kernel: Node 3 DMA per-cpu: empty
Jun  8 08:29:27 bluebird kernel: Node 3 Normal per-cpu:
Jun  8 08:29:27 bluebird kernel: cpu 0 hot: low 32, high 96, batch 16
Jun  8 08:29:27 bluebird kernel: cpu 0 cold: low 0, high 32, batch 16
Jun  8 08:29:27 bluebird kernel: cpu 1 hot: low 32, high 96, batch 16
Jun  8 08:29:27 bluebird kernel: cpu 1 cold: low 0, high 32, batch 16
.
.
.


I see this on all the machines that had the Oracle upgrade (and only those
machines). I assume that the new version of oracle is consuming more resources,
but I've not managed to track down the problem. The system has plenty of free
RAM. We were close to max semaphores, so I increased semmni from 128 to 192, but
it had no obvious effect.

I'd greatly appreciate any suggestions.

Thanks,

Rich