Mdthbs

I am dealing with transferring large files from one machine to another (600GB+) and I'm tarring them up using

tar -cpvzf file.tar.gz -C PATH_TO_DIR DIR

Once finished with the tarring process, the following is done:

split -d -b 2G file.tar.gz file_part_

This creates a bunch of file_part_00, file_part_01, ... until the whole file is split into 2GB chunks. Before transferring the file, I loop through each part in the directory the tar was split and collect their md5 hashes using an equivalent to:

md5sum PART_NAME >> list_md5.start

Once each part has been hashed, I do the following:

sort -u list_md5.start

(This sorts them and remove duplicates, just to be safe ya know)

The parts are then transferred one by one in the order they're in the list_md5.start. Once they arrive on the other computer, their md5 hash is collected using the same method but in a different list let's call it list_md5_2.start. After the transfer, before putting the parts back together, I run the following:

diff list_md5.start list_md5_2.start

If no difference is found, I continue to the next part. Otherwise, I give up and delete all the parts. When it comes to putting them back together I do the following:

cat file_part_* > file.tar.gz.incomplete

(The incomplete is there because I have a watchdog waiting to untar any .tar.gz it comes across).
Once the cat is done, the file is renamed using:

mv file.tar.gz.incomplete file.tar.gz

At this point, the watchdog detects it and untars it using:

tar -C DEST -xzv file.tar.gz --totals --unlink-first --recursive-unlink

At this point, I get an error I can't debug:

Tar Failed 2

gzip: stdin: unexpected end of file

tar: Unexpected EOF in archive

tar: Unexpected EOF in archive

tar: Error is not recoverable: exiting now

 /PATH/TO/DEST

After untarring, the tar is removed regardless if it failed or not (No point in keeping large files that failed to untar).

It is worth noting that sometimes the md5sum don't match up which also results in stopping the process (this is checked before the cat assembling step).

I have tried ensuring the names were not invalid. I've tried changing the part size to smaller sizes. I've tried manually going through the process and still either got an issue with a mismatch in md5sum or the EOF error.

This is all done on Ubuntu machines which have both been updated (No update pending).

Does anyone have an idea as to how to solve this issue?

Rsync is a free software utility for Unix- and Linux-like systems that copies files and directories from one host to another.

Use rsync to transfer file from one system to other.
You can use screen and the start rsync then deattach screen

Rsync is considered to be a lightweight application because file transfers are incremental -- after the initial full transfer, only bits in files that have been changed are transferred. Rsynch is often used to provide offsite backups by syncing data to a remote machine outside a firewall. It is also used for mirroring Web sites.