Copying a large file system (again)

[Jonathan Dill]

Kuba Ober wrote:
> If you have two PowerEdges nearby and in your control, there's no reason not 
> to use a dedicated gigabit link between the two, without any encryption. SSH 
> is dog-slow and takes up CPU unnecessarily when all you do is move data 
> between adjacent machines (if that's what you do).
>   
Excellent idea, I have done that before with a 2nd NIC and direct 
crossover connection between two servers and forget about encryption.
> I guess you can try contacting Joerg Schilling, there may be an option (or a 
> test version somewhere) where this problem is fixed. I admit that I've not
> dealt much with star and over 150GB.
>
> You may want to run star multiple times, each time transferring a part of the 
> tree.
>   
The issue is getting the hard links set up right, that might not work in 
multiple passes.  In one pass, that may cause transfer programs to use 
more memory since they may need to keep a table of which files are hard 
links to the same file at the fs level.  900 MB is really not that bad, 
with rsync I needed about 3 GB to move the 300 GB pool to new physical 
drives in the same server.  Probably best way to do chunks at a time is 
have some extra space, forget about hard links, then run BackupPC_link 
after each pass as someone mentioned.

If you have huge amounts of extra disk space and can take the filesystem 
offline while you do this, you could also use partimage to make an image 
of the whole filesystem then restore the image on the target.  The 
partimage image could be dumped to an NFS or CIFS / SMB share on a 3rd 
server then grabbed from the same share on the real target.  Either you 
will need to have partimage split into 2 GB chunks, or NFSv3 over a 
filesystem that supports huge files.

If you are decomissioning the old server and bringing up a new one, the 
drives aren't on hardware RAID, and you have enough I/O channels on the 
new server, I'd just physically move the drives to the new server, at 
least temporarily.  NB though with some hardware RAID, physically moving 
the drives can trash the whole RAID when you set up the RAID on the new 
system.

If you aren't using hardware RAID and have enough I/O channels in the 
source server, you could install the target drives temporarily and copy 
locally, or use an external USB drive as an intermediate step.

Jonathan