Copying a large file system (again)

[Tino Schwarze]

On Mon, Dec 03, 2007 at 05:40:05PM +0100, Tino Schwarze wrote:

> > > Thanks, it looks like I'll need to look for some more memory for the
> > > source machine. Maybe I'll try plugging the SATA RAID into the
> > > destination machine directly (a PE1800 with more RAM) and copy locally,
> > > skipping the network, then maybe rsync will be an option since most of
> > > the files are already in place, just the main pool is still missing.
> > 
> > Does your destination machine have sufficient memory?  I'm not
> > hard-stuck on rsync, but one of the niceties of running rsync over SSH
> > is that you can do it from either end.  The side that runs the rsync
> > command is the one where all the resources are allocated; the other
> > just gets an SSH connection and a lot of disk i/o.
> 
> Ah, this is good. :-) I'll try that and see how long (and how much
> memory) it takes for rsync to build the file list.

Oh well, rsync is building the file list. It's at 5.7 million now and
consumes 660 meg of RAM on the destination side and 512 meg on the
source machine.  BTW: 9 million files is the actual number of files,
most of them have got a lot of hardlinks, so rsync will see a lot more -
I don't know how many... so the client will probably run out of memory
and start swapping.

I'll probably hack up a perl script to get lists of files separated by
hardlink count, then see what I get as a result, then decide how to
continue.

Thanks for the support,

Tino.

-- 
www.craniosacralzentrum.de
www.lebensraum11.de
www.spiritualdesign-chemnitz.de

Tino Schwarze * Parkstraße 17h * 09120 Chemnitz