Copying a large file system (again)

[Tom Rockwell]

How about breaking the rsync into steps --- instead of doing the whole 
directory tree at once, do the top level or two manually and then fill 
in the branches in a few or several chunks.

-Tom



Tino Schwarze wrote:
> On Mon, Dec 03, 2007 at 08:56:16AM -0700, RB wrote:
>   
>>> Thanks, it looks like I'll need to look for some more memory for the
>>> source machine. Maybe I'll try plugging the SATA RAID into the
>>> destination machine directly (a PE1800 with more RAM) and copy locally,
>>> skipping the network, then maybe rsync will be an option since most of
>>> the files are already in place, just the main pool is still missing.
>>>       
>> Does your destination machine have sufficient memory?  I'm not
>> hard-stuck on rsync, but one of the niceties of running rsync over SSH
>> is that you can do it from either end.  The side that runs the rsync
>> command is the one where all the resources are allocated; the other
>> just gets an SSH connection and a lot of disk i/o.
>>     
>
> Ah, this is good. :-) I'll try that and see how long (and how much
> memory) it takes for rsync to build the file list.
>
>   
>> Failing that, GNU TAR may well work better on less RAM.  S-TAR is
>> nice, but it *might* have a memory leak or something that causes it to
>> utilize so much memory.  The problem with so many archival programs is
>> that they really want to keep an in-memory list of all the files
>> they've processed and/or are going to process.  It's not so much naive
>> as presumptive, but that rant doesn't help you much.
>>     
>
> I suppose, you need to keep a list of all files in memory if you want to
> track hardlinks correctly. Hm, the machine swapping heavily now and does
> hardly transfer anything now. Time to give rsync a try.
>
> Thanks to all!
>
> Tino.
>
>