16 TB Disk Arrays and ext3fs woes

[Sam Flory]

Fischer, Carl wrote:
>    I'm running a 16 TB disk array on a PE2950 with a PERC 5/E
> (megaraid_sas kernel module 3.09-1).  I expanded a pre-existing 3.5 TB
> partition to fill the volume (using LVM and resizefs).  Now, the volume
> mounts just fine, but only root can write to it.  Everyone else gets a
> "disk full" error.  I don't see any hardware errors in
> /var/log/messages.  My suspicion is that the filesystem thinks that the
> 13.5 additional TB are reserved for root, but I tried adjusting that
> parameter with tune2fs and had no luck.  

  There are a lot of things that can go wrong with a resize.  I'd backup
everything up, and do a new mke2fs with a 4 or 8k block size. What block
size are you using now? (I think you are limited to a 4k block size on
x86 hardware even with 64 bit kernels.) If tune2fs says you're using
2048 as a block you just need to start over.

IE: "mke2fs -j -b 4096 -t largefile"

>    Even with 64-bit CentOS installed, fsck.ext3 runs for a long time,
> then seg-faults (only the seg-fault shows up in /var/log/messages).

  Likely you ran out memory, or some other resource.

> I've compiled and installed the latest e2progs, with the same results.
> Not sure if this is a known software issue or some kind of hardware
> issue.  

 It's a bit of both.  The x86 arch imposes a 4k block size limit in most
cases.

> Any suggestions would be appreciated.  I've read online that XFS might
> work better for large disk arrays.  Does anyone have experience here?

  XFS works much better on large arrays.  You may have some issues if
you are running a 32 bit kernel.  You could also use jfs, or reiserfs.
In nearly every case you want to be sure you have the following:
1)64 bit kernel, and user space
2)4K or larger blocksize.