16 TB Disk Arrays and ext3fs woes

[Kuba Ober]

On Monday 06 August 2007, Fischer, Carl wrote:
>    I'm running a 16 TB disk array on a PE2950 with a PERC 5/E
> (megaraid_sas kernel module 3.09-1).  I expanded a pre-existing 3.5 TB
> partition to fill the volume (using LVM and resizefs).  Now, the volume
> mounts just fine, but only root can write to it.  Everyone else gets a
> "disk full" error.  I don't see any hardware errors in
> /var/log/messages.  My suspicion is that the filesystem thinks that the
> 13.5 additional TB are reserved for root, but I tried adjusting that
> parameter with tune2fs and had no luck.

Using ext2/3 on anything beyond maybe a terabyte is asking for trouble. ext3 
is relatively lean and mean, and while it will handle almost anything you 
throw at it, it may well be slow while doing so. Fsck will take ages, that's 
a known design tradeoff. Myself I wouldn't use ext3 for anything but mostly 
static OS files (stuff that goes into /bin, /sbin, /lib, /usr if this is a 
relatively "small" system).

Presumably you'll have millions of files on your multi-terabyte volume, and 
this will keep ext2 slow.

XFS or ZFS would be the way to go.

Cheers, Kuba