>16tb filesystems on linux

Andrew Robert Nicols andrew.nicols at luns.net.uk
Fri Aug 27 03:18:35 CDT 2010

Hi Nick,

On 26 August 2010 18:26, Nick Stephens <nick at ceiva.com> wrote:

> I am very interested in ZFS, but it seems like it will never make it (in
> a stable fashion) into the linux world at this rate.

We're primarily a Debian shop but we've dabbled with ZFS. It's really pretty
good and it's fault tolerance is really reassuring.
I'd avoid OpenSolaris though - since Oracle took over shop, active
development on it has all  but stopped. Solaris 10 is actually pretty
reasonable these days. There should be a new release out in the next 2 weeks
from what I recall.

We run Solaris 10 and ZFS in a RAIDz2 configuration on a SunFire X4500. The
X4500 has 48 x 500Gb disks and, after losing 2 for mirrored boot disks,
we're left with 46 disks. We've calculated that the optimum number of disks
in a zpool is 11, so we have four zpools of 11 disks, and two hot spares
which will be used by the first two disks to fail in the zpool.
The X4500 also has six SATA controllers and we've calculated a zpool
configuration such that we could potentially drop any one controller and
there would be no service interruption.
So of our 22Tb of raw space (after dropping another 2 disks for hot spares),
we get about 16Tb of usable space. You can run in Raidz instead of Raidz2
and, on our configuration, you'd have 18Tb usable space, but you'd lose the
ability to drop a controller. Obviously, a lot of this is irrelevant for a

Our disk usage is primarily lots of small files, with various really large
files (zip files for backup, database backups, etc) too.

The other *really* nice feature of ZFS is it's snapshot ability. While LVM
snapshotting is possible, let's face it, it's pants. With zfs snapshot, you
can have a virtually unlimited number of snapshots, you can promote
snapshots and mount them elsewhere (read only), delve into any snapshot
(handy for restores), and send snapshots over the wire to another server
(very useful for backups).
On our X4500, we've got 4313 snapshots at present. These are hourly
snapshots for the last three months or so, and then daily snapshots since
February. We recently purged snapshtos from September 2007 because we just
didn't see the point in keeping them any longer.

As I say, we're primarily a Debian shop and Solaris did used to feel like a
bit of a thorn in the side but things have improved. The package management
in Solaris blows compared to Debian and creating packages can be painful.
The community packaged software effort on the other hand is really good.
There are two main projects - OpenCSW and Blastwave. They did used to be the
same project but forked about 18 months ago. We use OpenCSW for various
reasons of preference. It's really worth using one or other if you do go for
the Solaris route.

The zfs-discuss mailing list on Open Solaris used to be really handy for
questions on ZFS but I haven't taken part in it for some time.

Hope this is of some help,


Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 04311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.us.dell.com/pipermail/linux-poweredge/attachments/20100827/a8ea7f02/attachment-0001.htm 

More information about the Linux-PowerEdge mailing list