Tracking the Progress of rsync Transfers

In my first introductory rsync post, How To Synchronize Directories with Rsync, I’ve shown you the most basic approach to syncing two directories up. Today, I’d like to show you another useful thing you can do with [rsync][rsync-command.

Just to remind you all, rsync is a remote synchronisation tool. This means that its primary use lies in the filed of synchronising files and directories between remote Unix systems, but I feel that it’s really important for you to understand the basics before moving on to more advanced stuff.

The beauty of rsync is that its syntax and command line options are exactly the same for local and remote directories synchronisations, so it’s not like you’ll have to learn it all from scratch when I show you the real world usage of rsync in my next post. rsync experiments setup

Just use the following commands to re-create your rsync playground for this post:

ubuntu$ rm -rf /tmp/dir1 /tmp/dir2
ubuntu$ mkdir /tmp/dir1 /tmp/dir2
ubuntu$ cd /tmp
ubuntu$ echo "original file 1" > dir1/file1
ubuntu$ echo "original file 2" > dir1/file2
ubuntu$ echo "original file 3" > dir1/file3
ubuntu$ cp dir1/file1 dir2

It will probably be more convenient for you to just download the same commands in form of a script though, so here’s the link: rsync-setup.sh

Tracking rsync progress

The larger the directories you’re synchronising with rsync, the longer it will take for the command to finish. Depending on the scale of your task, it can be minutes, long hours or even days before you get the synchronisation complete. With this in mind, the importance of progress tracking with rsync should be obvious to you.

Here’s how you make rsync report its progress: just use the –progress command line option (in addition to the command line I introduced you to last time):

ubuntu$ rsync -avz --stats --progress /tmp/dir1/ /tmp/dir2
building file list ...
4 files to consider
file2
		  16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
file3
		  16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)

Number of files: 4
Number of files transferred: 2
Total file size: 48 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 59
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 193
Total bytes received: 64

sent 193 bytes  received 64 bytes  514.00 bytes/sec
total size is 48  speedup is 0.19

Why you should track the progress of rsyncs

The key differences are that you first get a regularly updated message about building a file list. When you run into hundreds of thousands of files, this becomes quite useful:

building file list ...
4 files to consider

Another way to track progress is to see the transfer progress for each of the involved files. For thousands tiny files it won’t be all that impressive a feature, but if you’re transferring huge files between remote systems, a dynamic transfer progress for each file will be exactly what you need:

file2
		  16 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/4)
file3
		  16 100%   15.62kB/s    0:00:00 (xfer#2, to-check=0/4)

That’s it, hope you like this new option. I promise to tell you more some other time!

See also

• rsync command
• How to compare directories in Unix
• Synchronize Directories in Unix with rsync command
• List subdirectories of a Unix directory
• Finding large files and directories
• Advanced Unix commands