R610 + Debian Lenny 32-bit bigmem kernel => incorrect 3.5GB memory limit

[Stephen Dowdy]

We have a couple R610s which are currently running Debian Lenny
32-bit bigmem (PAE enabled) kernels.  They have 8GB of RAM, but
the kernel, with PAE enabled, is only showing 3.5GB.

(we have yet to successfully get Debian Lenny AMD64 FAI to install
these systems. They hang during the network install. (not sure yet
if this is still a bnx2 issue, but i think we've addressed that in
the FAI setup)

(btw, we're also having issues with Debian Lenny amd64 with stock
2.6.26 kernel installing on the Precision T3500 -- "nobody cared"
Interrupt processing failures cause the system to fail -- though a
handbuilt 2.6.29 installed kernel seems to work fine there, so
we're definitely having issues with 11Gen/Nehalem Dell h/w -- sigh)


I have a script which i use to detect PAE presence and compare the
DMI response to what the kernel is presenting...

--------------------------------------------------
r610-1:~# DEBUG=1 ./check_PAE
ssi_cpu_cap_lm=1
ssi_cpu_cap_nx=1
ssi_cpu_cap_pae=1
ssi_cpu_flags=' fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx rdtscp lm constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 sse4_2 popcnt lahf_lm ida '
ssi_kernel_id=2.6.26-2-686-bigmem
ssi_mem_size_mb=3584
ssi_os_bits=32
ssi_os_mem_leftover=3446611968
ssi_os_pae_enabled=1
ssi_os_pagesize=4096
ssi_os_physpages=841458
ssi_swap_size_mb=8197
> 64-bit capable CPU
> CPU supports No-Exec (NX)
> CPU supports Physical Address Extensions (PAE)
DEBUG:  sys_memcount (from DMI) = 8192
DEBUG:  We have 4GB or more of RAM
WARN: DMI (8192)  and system (3584) differ on memory count
DEBUG:  PAE is enabled
--------------------------------------------------

As you can see, while the system has 8GB of physical memory
installed, the kernel is only showing 3.5GB.

'dmesg' indicates the problem:

[    0.000000] BIOS-provided physical RAM map:
[    0.000000] Ooops! Too many entries in the memory map!
(many repeats of this line, which appear to come from e820.c BIOS query)
[    0.000000]  BIOS-e801: 0000000000000000 - 000000000009f000 (usable)
[    0.000000]  BIOS-e801: 0000000000100000 - 00000000cf690000 (usable)
[    0.000000] 2422MB HIGHMEM available.
[    0.000000] 896MB LOWMEM available.
[    0.000000] found SMP MP-table at [c00fe710] 000fe710
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] Entering add_active_range(0, 0, 849552) 0 entries of 256 used
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA             0 ->     4096
[    0.000000]   Normal       4096 ->   229376
[    0.000000]   HighMem    229376 ->   849552
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[1] active PFN ranges
[    0.000000]     0:        0 ->   849552
[    0.000000] On node 0 totalpages: 849552
[    0.000000]   DMA zone: 32 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 4064 pages, LIFO batch:0
[    0.000000]   Normal zone: 1760 pages used for memmap
[    0.000000]   Normal zone: 223520 pages, LIFO batch:31
[    0.000000]   HighMem zone: 4846 pages used for memmap
[    0.000000]   HighMem zone: 615330 pages, LIFO batch:31
[    0.000000]   Movable zone: 0 pages used for memmap


849552 (pages in 'node 0') * 4096 (pagesize) ~= 3.5GB.

Is this likely a BIOS problem, a Linux 2.6.26 kernel problem or
something else? If anyone has experienced this problem, the "nobody
cared" Interrupt errors on the T3500 (another BIOS issue?), or has
successfully installed and is running Debian Lenny 64-bit stock
distribution operationally (even with bnx2 non-free install), i'd
appreciate hearing from you.

thanks,
--stephen
-- 
Stephen Dowdy  -  Systems Administrator  -  NCAR/RAL
303.497.2869   -  sdowdy at ucar.edu        -  http://www.ral.ucar.edu/~sdowdy/