Linux Rsync examples for Backup and Remote Sync
As name hint that rsync is used sync data from one machine to another machines. So rsync is nice OpenSource tools used worldwide in Linux/Unix setups to take backup of various different files,directories,file-systems,application-data etc like backup of regular files, link files, dat files, database files, mail files etc.
In this post we will try to understand various options that can used with rsync that could help to take backup in various scenarios that can make our life easier.
So this post is regarding various rsync examples for backup.
Rsync capable to copy data changes between source and destination,this is called as rsync works on “Delta transfer algorithm” so we can copy only updated data from Source towards destination, this way this tool is consider as very good tool to take backup of various kinds of files and file-systems.
So through rsync we don’t need to copy complete data again, it is capable to copy incremental new data.
We mostly get rsync available on all sort of Unix/Linux machines.Let’s see name of packages and installation method of some of major distributions used in Linux/Unix Servers and Desktop environments.
Installation of rsync on Ubuntu Machine.
Installation of any packages on Unix/Linux machine through any packages manager can be done through superuser root or sudo privileges users only
Ubuntu Machines has very flexible package manager with large number features called apt-get
For installing rsync on Ubuntu machine we have rsync package that could install with below command
sudo apt-get install rsync
Installation of rsync on RHEL/CentOS Machines
RedHat Machines also has very wide featured available packages manager which allow to install packages in very easy and flexible way called yum, we can use it to install rsync packages like down below command.
yum install rsync
During this post, we will see various options provided with rsync that can used to take advantages from rsync command on daily operations.
Rsync command is fairly simple command with logical options used that easy to understand, if you get basic concept for transferring file from any source towards destination.
#Rsync Syntax rsync options source destination #Remote sync pull rsync options user@remote-source:source-directory local-Destination #Remote sync push rsync options local-source user@remote-destination-host:Directory
So if we are using same users working on both machines then we need not provide user name while using rsync command, but in case we like as another user then sure we need user name like above.
Below are most commonly used rsync options command to sync data between two hosts.which i am using in my daily Linux/Unix Stuff
- -v, --verbose increase verbosity
- -q, --quiet suppress non-error messages
- -a, --archive archive mode. equals -rlptgoD (no -H,-A,-X)
- -r, --recursive recurse into directories
- -u, --update skip files that are newer on the receiver
- -d, --dirs transfer directories without recursing
- -l, --links copy symlinks as symlinks
- -L, --copy-links transform symlink into referent file/dir
- -H, --hard-links preserve hard links
- -p, --perms preserve permissions
- -A, --acls preserve ACLs (implies -p)
- -X, --xattrs preserve extended attributes
- -o, --owner preserve owner (super-user only)
- -g, --group preserve group
- -n, --dry-run perform a trial run with no changes made
- --delete delete extraneous files from dest dirs
- -z, --compress compress file data during the transfer
Although there are various other options provided with rsync, you can check them rsync man pages. But before on same, let see some serious stuff in rsync with some examples.
Copy data from local to remote
Copy data from local machine to remote machine on designated destination directory.
[root@srv14 ~]# rsync -avz /data/images/ srv31:/data-backup/ root@srv31's password: sending incremental file list ./ centos5/ centos5/CentOS_BuildTag centos5/EULA centos5/GPL
In above options we are sync some files over network from one machine to another with archive, verbose and compress options. Archive mode itself used contain many options (-rlptgoD), like it contain many options that we don’t carry in case we used archive options with rsync command.
These options are some of very base of every sync that should always included.
-r -- Sync Recursive -l -- sync Link file as well -p -- With permission attributes as well -t -- preserve modification time as well -g -- sync with same group ownership -o -- sync with same ownership
Above provided list of options are some of must used with any sync and all of these cover in -a rsync option
See progress and speed while rsync data
We can can see what is progress of of sync through which data is moving to local or another host while rsync working on our data.
This will provide good human readable format in which we can see progress of rsync on which current file is working on, it also provide rate on which data is floating from source to destination either it is remote or local machine, like below
[root@srv31 ~]# rsync -av /data/iso/* 192.168.0.14:/data-backup/ --progress email@example.com's password: sending incremental file list CentOS-5.9-x86_64-bin-DVD-1of2.iso 4684441600 100% 40.83MB/s 0:01:49 (xfer#1, to-check=275/276)
Instead of –progress , we can use -P (capital p), this would include progress status with rsync command.
Use ssh(Secure shell) for rsync
In case you like to secure ssh for copy, you can also do this with -e ssh option with ssh like below.
rsync -av -e ssh source-IP:/data/ /data/
In above command we doing sync with one remote source directory towards with ssh, which is more secure and safe for file copy when we working in non-trusted network.
Delete files from destination if not in source during sync
Sometime we have scenario in which our current source have some increased data but also deleted some of files as compared to last backup in destination, so to make it exact copy of it, we need copy new and delete some of old files.
For same, we have –delete option in rsync command which copy data but also delete files those are missing in source in comparison of source.
rsync -avz --delete /data remote-destination:/data/
Through above way we can delete files in destination which were deleted within source after last backup taken.So we can be sure that while taking incremental backup, we have remove deleted files. Although we have many option with –delete, like below.
--delete delete extraneous files from dest dirs --delete-before receiver deletes before xfer, not during --delete-during receiver deletes during the transfer --delete-delay find deletions during, delete after --delete-after receiver deletes after transfer, not during --delete-excluded also delete excluded files from dest dirs --ignore-missing-args ignore missing source args without error --delete-missing-args delete missing source args from destination
File maximum and minimum size while transfer
There is possibility in rsync, that we can ask to limit file size during file transfer, like below command we can limit maximum and minimum file size that allow to transfer during transfer.
Syntax used for rsync options.
–max-size=SIZE don’t transfer any file larger than SIZE
–min-size=SIZE don’t transfer any file smaller than SIZE
rsync -avz --max-size="10M" /data/ remote-destination-IP:/data/ or rsync -avz --min-size="512K" remote-source-IP:/hometech/ssirohi/ /hometech/ssirohi/
Don’t modified updated file at destination
Some time we make some changes in files in destination and we don’t want to remove and get revert back as it saved in source file after transfer. So in rsync we have options for that as well, syntax for same mentioned below
-u, --update This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.)
rsync -avzu remote-destination:/data/ /data/
Crating and copy link from source to destination
This is really good approach to deal with link while transfer through rsync. Below are syntax for same which we can use in rsync command.
-l, --links When symlinks are encountered, recreate the symlink on the destination. -L, --copy-links When symlinks are encountered, the item that they point to (the referent) is copied, rather than the symlink. -H, --hard-links This tells rsync to look for hard-linked files in the source and link together the corresponding files on the destination. Without this option, hard-linked files in the source are treated as though they were separate files.
Copy same permission on destination same as source machine
In Linux we have permission those are used to maintain file and directory access through users and groups. We can save same permission as we have in source.Syntax for rsync used for permission preservation for this is like below.
-p, --perms This option causes the receiving rsync to set the destination permissions to be the same as the source permissions. -A, --acls This option causes rsync to update the destination ACLs to be the same as the source ACLs.
Run dry run
This is really good options to test and understand things better before actually transfer data from source to destination. Through this we can run see operation details that going to happened if we actually work.
I personally used this options to know space required to get transfer all of data on destination and only run actual transfer if we have enough space on destination machine.
rsync -avz --dry-run Remote-IP:/data/ data/
Set transfer speed limt
During rsync we can set transfer limit, like rsync will not used more than mentioned speed for rsync.This is good options to limit and save bandwidth for other machine in network.
This limit is used to mentioned with –bwlimit options in KB/S, So in case we need to apply it in MB or GB, we have calculate it in KB and mentioned during running this command, like below.
rsync -avz --bwlimit="5120" Remote-destination:/data/ /backup/
In above command, we tried to limit transfer rate upto 5MB/S during rsync. This is helpful to prevent our over all bandwidth, so we will not used it all during production hours.
See compare file between source and destination
This option is used to see how destination and source file has different from each other and will identify file difference from both sides.
For same we can use -i option with rsync which will introduce complete new column like below.
root@srv32:~# rsync -avzi /etc/ srv17:/backup/ root@srv17's password: sending incremental file list .d..t...... ./ <f.st...... hosts sent 66,576 bytes received 256 bytes 19,094.86 bytes/sec total size is 2,719,476 speedup is 40.69
We can see there is new column which define difference between source and destination, there are some alphabet to decode what is difference in between.
- d: indicates change in destination file
- f: indicates a file
- t: indicates change in timestamps
- s: indicates change in size
Exclude some files or directories during rsync
Sometime we need to exclude some file from rsync which is part of directory which we rsync. So for such case we have exclude option in rsync command, like below
rsync -axhzSHA --exclude=/www/html/wp-content/cache/log/* --exclude=/www/html/wp-content/cache/object/* /etc/httpd /var/www /backup/tmp/apache --delete
Actually if you see it closely, these exclude path mentioned relative to path source machine path.
Although we still have many options left in rsync But we also cover various examples mentioned in this post, that explain why rsync is used world wide in many data backup with various types of data backup.