[Fwd: RHEL connundrum with df and du]

[Frank Warnke]

On Mon, 2006-11-06 at 06:00 -0600, sankarshan.mukhopadhyay at gmail.com
wrote:
> 
> Message: 3
> Date: Mon, 06 Nov 2006 10:43:32 +0530
> From: Sankarshan Mukhopadhyay <sankarshan.mukhopadhyay at gmail.com>
> Subject: [Fwd: RHEL connundrum with df and du]
> To: linux-poweredge at dell.com
> Cc: haven at thehavennet.org.uk
> Message-ID: <454EC47C.2090400 at gmail.com>
> Content-Type: text/plain; charset=iso-8859-1
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Fwd-ing to the Dell Poweredge list for a possible faster response
> 
> :SM
> 
> - -------- Original Message --------
> Subject: RHEL connundrum with df and du
> Date: Thu, 2 Nov 2006 16:13:34 -0000 (UTC)
> From: Simon Alman <haven at thehavennet.org.uk>
> Reply-To: haven at thehavennet.org.uk
> To: ext3-users at redhat.com
> 
> Hi All
> 
> I am having an issue with disk space and since it is happening using
> an
> ext3 formatted partition I felt that this would be the most
> appropriate
> list to post to.
> 
> The problem is this; I have access to two RHEL systems (one running
> RHEL3
> and one running
> RHEL4). Both are in the same company and both exhibit the same problem
> that I have not seen anywhere else before.
> 
> Both have full / partitions. df shows this to be the case so it must
> be
> true ... right ?
> 
> Well du disagrees, on one system with a full 20GB / parition that df
> shows
> to be full, df can only find 3.6GB of files.
> 
> So off to the redhat FAQ I went and found this:
> http://kbase.redhat.com/faq/FAQ_35_5209.shtm
> 
> Great that explains the problem precisely ... except it didn't help.
> No
> processes were holding large deleted files in a locked state.
> 
> So I looked at inodes thinking that they may have all been used up ...
> they haven't df shows only 5% inode usage.
> 
> I forced an fsck run on the partition on reboot and neither this nor
> the
> reboot helped, fsck shows clean and after the reboot things were still
> broken.
> 
> So I'm sat here trying to explain to the client why their 5k worth of
> dell
> server is currently no much use to them (I can't install anything on
> it
> due to the space issue ...).
> 
> Has anyone come across anything similar before ? I've worked with
> linux
> for six years and this is a new one on me ... and twice in the same
> company. I suspect major kernel b0rkage since both systems use Dell's
> RHEL
> build for the specific model of server but proving it is beyond me
> right
> now.
> 
> Any help/advice would be very gratefully appreciated.
> 
> Kind regards
> 
> Simon Alman

We had a similar experience when a device (removable SATA drive) was not
mounted before the data was copied to the mount point which was then
just a subdirectory under /.

In our case we unmounted the data filesystems (including NFS and Samba
mounts) and did a du in each mount point until we found the problem.