Lessons Learned from Migrating 17TB of Data

I finally pulled the trigger on some new hard drives for my home NAS. I am migrating from a 5U Server down two a small desktop size NAS. Ultimately this removes the need for my 42U standing rack.

I did this transfer a year or so ago when I did a full rebuild of my server but forgot to take any notes on the processes that I used. Instant regret. I remembered utilizing Rsync to do the actual transfer and I assumed that I mounted both the existing NAS to an NFS share and the new NAS through NFS. Both these mounts would reside inside a throwaway virtual machine on my application server.

I used the following Rsync command to start.

rsync --ignore-existing -ahzrvvv --progress {Source} {Destination}

To break this down a little bit:

–ignore-existing: This will ignore any existing files that copy over

-a: Archive flag. This preserves my data structure

-h: Human readable. If this flag exists for a command, use it. It makes things much easier to use.

-z: Compression. There are a bunch of different compression options for Rsync. This one does enough for me.

-r: This makes Rsync copy files recursively through the directories

-vvv: I put triple verbose on because I was having so many issues.

–progress: This will show the number of files and the progress of the file that is currently being copied. Especially useful when copying large files.

Now, my command changed over time but ultimately this is what I ended on. My source and destination were set to the respective NFS mounts and I hit [enter] to start the transfer. I left it running on the console of my Virtual Machine and walked away after I saw a handful of successful transfers. Assuming everything was going fine I went about my day as 17TB is going to take a while.

A few hours later I decided to check in on my transfer and saw that it had gotten stuck on a file after only 37KB of data transfer! Frustrated, I restarted the process. Only to see the same results later on.

After updating, downgrading, and modifying my command structure I came to the realization that there must be an issue with transferring between to NFS shares.

I am still researching why this happens but to me, it seems as though when the transfer starts the files are brought into a buffer somewhere within the Linux filesystem which gets maxed out causing the file transfer to stall. Almost as if the buffer can’t send the new files fast enough.

When I switched the transfer to utilize SSH instead of NFS to NFS the transfer completed successfully.

If someone has some information regarding how this works I would love to learn more.