Data Replication Using rsync

Having just discussed replication in Linux -- what it is, how it can be used and how it's not the same as a backup -- it's time to tackle a simple example of one of the replication tools: rsync. You will be surprised how easy it is to use rsync to replicate data to a second storage pool.

Replication Review

Recently we discussed the importance of data replication for situations ranging from mission critical environments to home users. Replication ensures that you have a copy of your current data on a separate storage environment (secondary system) so that if you lose the first system (primary system), you still have access to the data.

In general there are two types of replication: synchronous and asynchronous. Synchronous replication, as the name implies, means that the primary storage and secondary storage are kept exactly the same. Any data writes (or deletes) have to complete on both the primary and secondary storage pools before returning to the application allowing it to continue. This means that the data in the data pools is an exact match.

Asynchronous replication allows any writes (or deletes) from an application to finish on the primary storage pool. Then the data is copied from the primary storage pool to the secondary storage pool, typically outside of the application I/O path. This means that there can be some data difference between the two pools at any instance in time. The amount of data difference you can tolerate is up to you (your requirements) but you can shrink that data difference to something fairly small and tolerable.

But one of the most important aspects of data replication that you shouldn’t forget is that data replication and data backup are not the same thing. A backup can keep prior versions of data so that you can effectively go back in time over the life of the data to get prior versions. On the other hand, replication keeps a replica (copy) of the current data. You can get the most current state of the data from a backup but it will only be as accurate as when the backup was made. Replicas are much more recent so they will, in general, capture the data changes since the last backup. The question you need to answer is when do I need replication?

There is no universal answer to that question. You need to examine your data requirements and determine how important having the latest copy of the data is to your mission and the importance of the accessibility of that data. During your examinations you should also weave in discussions about off-site disaster recovery for your data center (or your home system).

In the previous article about replication, two replication techniques for Linux were discussed – DRBD and rsync. DRBD is a kernel based replication method that automatically replicates all data from the primary storage pool to a secondary storage pool without the user or administrator having to intervene. On the the hand, rsync is a file based approach allowing you to selectively replicate directory trees so that you don’t have to replicate an entire file system. However, rsync mush be invoked manually so it’s not as automatic as DRBD increasing the possible data differences between the primary and secondary storage pools. But many times the flexibility of replicating only portions of the data to one or more secondary storage pools make it a popular choice despite the possibility of increased data differences.

Since rsync is so flexible and is file based, in this article I want to show a simple rsync example to illustrate what it can do for data replication.

Simple example using rsync

I want to present a simple rsync example to illustrate the basic steps involved in getting to replicate a directory tree. One of the advantages of rsync is that you don’t have to make a copy of the entire file system – you can just make a copy of a specific directory tree and even specific file types or file names. This makes it incredibly flexible since you can now replicate directory trees to various secondary storage servers as needed and/or focus on certain types of files or directory trees.

For this article I will be using rsync to replicate data from my laptop to my main storage box at home. The overall concept is that I come home from being on the road, I fire up the laptop on the home network and my data gets replicated to my home server. In addition, I will only be replicating a specific directory from the laptop to the home server since that is where I keep all the data I modify while I travel.

Let’s start by defining terms within the rsync framework. The first two systems or terms that we need to define are the rsync server and the rsync client. One would think that the traditional client/server terms would apply, but a common source of confusion in rsync is that the rsync server does not necessarily have to be the system that has the original copy of the data and the rsync client does not have to be the recipient of the data. To better understand rsync, remember that there is a distinction between roles and processes in rsync. So to make sure we understand all the terms used in this article, there are four terms we’ll be using (taken from this link).


  • Client: This is a role within rsync where the client initiates the synchronization.
  • Server: This is a role within rsync and refers to the remote process (system) that clients connect to either within a local transfer, a remote shell, or with a network socket.
  • Sender: This is a role and a process within rsync and applies to the particular rsync process that has access to the original files being synchronized. So the “sender” process reads the data and sends it to the “receiver” process.
  • Receiver: This is a role and a process within rsync and describes the receiving process that receives the updates to the data and writes it to the storage device (i.e. the secondary storage pool).

To be honest, things can get a little confusing between roles and processes but I like to keep things simple. So for this example where I’m replicating data from my laptop to my home server I will define the home server as the rsync server, and the laptop as the rsync client and the laptop is the rsync sender since it has the original data and the home server is the rsync receiver. The idea is that when I plug in the laptop to my home network, my data is automatically replicated to my home server so I have a copy in case the laptop dies.

A reasonably good tutorial to use to start learning rsync is here. It is a bit old (1999) but it has a very good overview of rsync and explains things fairly well. There are other tutorials that cover useful topics such as how to use ssh with rsync or using stunnel with rsync.

For the rsync command used for my simple scenario let’s start with the simple example in the “everythinglinux” tutorial. Here is the script I used to perform the rsync that is taken from the article and adapted to my situation (notice that there are few changes).

rsync --verbose --progress --stats --compress --rsh=/usr/bin/ssh \
  --recursive --times --perms --links --delete \
  --exclude "*bak" \
  /home/laytonjb/Documents/* 192.168.1.8:/data/laytonj/rsync_test

Comments on "Data Replication Using rsync"

rct

I don’t usually rant about articles, but it seems like this article was overly padded to make a minimum word limit for submission. Many of the options used could have been replaced by using rsync -a.

The paragraph: “rsh=/usr/bin/ssh: This option tells rsync what to use for the remote shell [ ... ] but I used the old school option of rsh=, is really a waste of space and reader’s time. As the fine manual page says “a modern rsync uses ssh for its communications”. Given that this is Linux magazine, modern == all.

Reply
grabur

Rsync is fantastic. Just to add that I always run rsync first with the dry-run option. Just in case!

Reply
beerse

With cygwin comes a perfect implementation of rsync (its from the same source). Hence it also works for msDos/msWindows based machines. I even use it on the sd-cards in my camera: just compare it on the local machine between the sd-card (current drive) and the pictures folder. It only copies the new pictures. Much faster with current multi-gb cards.

Be noted however: The fat filesystem only has a 2 second time-stamp resolution. And msWindows timestamps varies around daylight-savingtime changes. Hence use “–modify-window=3602″
And the filename specification differs somehow.

Reply
bhepple

But wait! there’s more!!

rsync _can_ be used to give multi-versioned backup – ie to efficiently backup files with all versions visible at once. This uses the --link-dest option of rsync to make hard links between unchanged files in successive backups – an unchanged file is only stored once. It’s a kind of deduplication at the file level.

This works very well for the typical case of frequent changes to small files. Less well for very large files such as media.

Here’s my backup script that I run daily at minimal cost in time or space: http://bhepple.freeshell.org/oddmuse/wiki.cgi/backup-copy

Reply
laytonjb

Thanks every one for your feedback on rsync. I really appreciate the feedback that you have given – it helps everyone.

Thanks again!

Jeff

Reply
uclicktu

Nice article Jeff. For those of us that are cutting our teeth on linux it is very useful for the total explanation of using rsync. Thanks again!

Reply

Does rsync allows master-master configuration ?

Reply

It may possibly common excite your ft to jog a further stat mi for any dynamic lumination around excess fat bedding. Its designed with quite a few nike absolutely free perform 3 kind subtle back-up by using nylon uppers advent. It is really intended to have which will barefooted-sense boasting it has the mobility.
Air Jordan 13 http://outletsaler.allmyblog.com/1-cheap-air-jordan-13-shoes.html

Reply

yu0IlK tcvuhyqccnrd, [url=http://pwyekeibaixo.com/]pwyekeibaixo[/url], [link=http://vtcdlpkhlatt.com/]vtcdlpkhlatt[/link], http://clrmlhdsuydo.com/

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>