datamgr page allocation failure

[John LLOYD]

> On Wed, Jun 10, 2009 at 12:26:48AM -0500, Matt Domsch wrote:
> > On Tue, Jun 09, 2009 at 10:44:54AM -0400, Rich Bishop wrote:
> > > We have a number of m905 blades which have been working 
> fine for about 9 months
> > > now. We use Dell's omreport to monitor them via nagios. 
> The machines are running
> > > RHEL4 64bit with the latest version of omsa (5.5).
> > > 
> > > Over the weekend our DBAs upgraded Oracle on some of 
> these blades to 10.2.0.4
> > > (from some older version of Oracle 10). Since this 
> upgrade I'm seeing lots of
> > > page allocation failures datamgr3:
> > > 
> > > Jun  8 08:29:27 bluebird kernel: dsm_sa_datamgr3: page 
> allocation failure. order:4, mode:0xd1
> > 
> > An order 4 allocation means the driver is trying to 
> allocate 2^4 (16)
> > consecutive pages of RAM (thus 64KB) for an operation, but 
> the kernel
> > memory is fragmented ot the point where that many consecutive pages
> > are not available.
> > 
> > Was there a kernel change during the Oracle upgrade?
> 
> 
> Thanks for the response. There were no kernel changes. Is 
> there anything I can
> do to debug this or prevent it from happening?
> 
> Thanks,
> 
> Rich


The general advice is to increase min_free_kbytes, generally to about
6000 to 8000 per core. Default is around 15,000 or 16,000 for all cores,
whether you have one or 16, depending on distro.

e.g. for permanent change

echo "vm.min_free_kbytes = 35000" >> /etc/sysctl.conf

and for immediate change

echo 35000 > /proc/sys/vm/min_free_kbytes


--John