dcsimg

Data Replication Using rsync

Having just discussed replication in Linux -- what it is, how it can be used and how it's not the same as a backup -- it's time to tackle a simple example of one of the replication tools: rsync. You will be surprised how easy it is to use rsync to replicate data to a second storage pool.

Replication Review

Recently we discussed the importance of data replication for situations ranging from mission critical environments to home users. Replication ensures that you have a copy of your current data on a separate storage environment (secondary system) so that if you lose the first system (primary system), you still have access to the data.

In general there are two types of replication: synchronous and asynchronous. Synchronous replication, as the name implies, means that the primary storage and secondary storage are kept exactly the same. Any data writes (or deletes) have to complete on both the primary and secondary storage pools before returning to the application allowing it to continue. This means that the data in the data pools is an exact match.

Asynchronous replication allows any writes (or deletes) from an application to finish on the primary storage pool. Then the data is copied from the primary storage pool to the secondary storage pool, typically outside of the application I/O path. This means that there can be some data difference between the two pools at any instance in time. The amount of data difference you can tolerate is up to you (your requirements) but you can shrink that data difference to something fairly small and tolerable.

But one of the most important aspects of data replication that you shouldn’t forget is that data replication and data backup are not the same thing. A backup can keep prior versions of data so that you can effectively go back in time over the life of the data to get prior versions. On the other hand, replication keeps a replica (copy) of the current data. You can get the most current state of the data from a backup but it will only be as accurate as when the backup was made. Replicas are much more recent so they will, in general, capture the data changes since the last backup. The question you need to answer is when do I need replication?

There is no universal answer to that question. You need to examine your data requirements and determine how important having the latest copy of the data is to your mission and the importance of the accessibility of that data. During your examinations you should also weave in discussions about off-site disaster recovery for your data center (or your home system).

In the previous article about replication, two replication techniques for Linux were discussed – DRBD and rsync. DRBD is a kernel based replication method that automatically replicates all data from the primary storage pool to a secondary storage pool without the user or administrator having to intervene. On the the hand, rsync is a file based approach allowing you to selectively replicate directory trees so that you don’t have to replicate an entire file system. However, rsync mush be invoked manually so it’s not as automatic as DRBD increasing the possible data differences between the primary and secondary storage pools. But many times the flexibility of replicating only portions of the data to one or more secondary storage pools make it a popular choice despite the possibility of increased data differences.

Since rsync is so flexible and is file based, in this article I want to show a simple rsync example to illustrate what it can do for data replication.

Simple example using rsync

I want to present a simple rsync example to illustrate the basic steps involved in getting to replicate a directory tree. One of the advantages of rsync is that you don’t have to make a copy of the entire file system – you can just make a copy of a specific directory tree and even specific file types or file names. This makes it incredibly flexible since you can now replicate directory trees to various secondary storage servers as needed and/or focus on certain types of files or directory trees.

For this article I will be using rsync to replicate data from my laptop to my main storage box at home. The overall concept is that I come home from being on the road, I fire up the laptop on the home network and my data gets replicated to my home server. In addition, I will only be replicating a specific directory from the laptop to the home server since that is where I keep all the data I modify while I travel.

Let’s start by defining terms within the rsync framework. The first two systems or terms that we need to define are the rsync server and the rsync client. One would think that the traditional client/server terms would apply, but a common source of confusion in rsync is that the rsync server does not necessarily have to be the system that has the original copy of the data and the rsync client does not have to be the recipient of the data. To better understand rsync, remember that there is a distinction between roles and processes in rsync. So to make sure we understand all the terms used in this article, there are four terms we’ll be using (taken from this link).


  • Client: This is a role within rsync where the client initiates the synchronization.
  • Server: This is a role within rsync and refers to the remote process (system) that clients connect to either within a local transfer, a remote shell, or with a network socket.
  • Sender: This is a role and a process within rsync and applies to the particular rsync process that has access to the original files being synchronized. So the “sender” process reads the data and sends it to the “receiver” process.
  • Receiver: This is a role and a process within rsync and describes the receiving process that receives the updates to the data and writes it to the storage device (i.e. the secondary storage pool).

To be honest, things can get a little confusing between roles and processes but I like to keep things simple. So for this example where I’m replicating data from my laptop to my home server I will define the home server as the rsync server, and the laptop as the rsync client and the laptop is the rsync sender since it has the original data and the home server is the rsync receiver. The idea is that when I plug in the laptop to my home network, my data is automatically replicated to my home server so I have a copy in case the laptop dies.

A reasonably good tutorial to use to start learning rsync is here. It is a bit old (1999) but it has a very good overview of rsync and explains things fairly well. There are other tutorials that cover useful topics such as how to use ssh with rsync or using stunnel with rsync.

For the rsync command used for my simple scenario let’s start with the simple example in the “everythinglinux” tutorial. Here is the script I used to perform the rsync that is taken from the article and adapted to my situation (notice that there are few changes).

rsync --verbose --progress --stats --compress --rsh=/usr/bin/ssh \
  --recursive --times --perms --links --delete \
  --exclude "*bak" \
  /home/laytonjb/Documents/* 192.168.1.8:/data/laytonj/rsync_test

Comments on "Data Replication Using rsync"

No matter if ?ome one searches for ?is essential t?ing, thu? he/s?? wants to bbe av?ilable that ?n detail, so
t?at thing is maintained ovesr ?ere.

Also visit myy web site – auger bit

Hey there j?st wante? to give you a b?ief heads up and let you know
? fe? of the pictures a?en’t loading properly.
I’m not ?ure wh? b?t ? t?ink ?ts ? linking issue.
?’v? tried it in t?? ?ifferent web browsers aand ?oth ?ho? t??
s?me outcome.

Herre ?s my web pa?e auger bit

Fanastic blog! D? ?o? hav? aany tips for aspiring writers?
?’m hoping t? start my oown blog ?oon ?ut I’m? little lost ?n everything.
Wo?ld you propose starting w?th a free platform
?ike WordPress ?r go for a paid option? ?here ?re so many choices ?ut there that ?’m totally overtwhelmed ..

?ny ideas? Kudos!

Feel free t? surf to? myy webssite tractor post hole digger

Im grateful for the article post.Thanks Again. Will read on…

Thanks again for the article.Thanks Again.

I gotta bookmark this site it seems extremely helpful very beneficial

I quite like reading through a post that can make people think. Also, many thanks for permitting me to comment!

I truly appreciate this blog post.Really looking forward to read more. Really Cool.

Great article.Much thanks again. Much obliged.

Fantastic post.Really looking forward to read more. Really Cool.

Hello very cool web site!! Guy .. Beautiful .. Amazing ..
I’ll bookmark your web site and take the feeds additionally?

I’m glad to search out a lot of helpful info right here within the submit,
we want develop extra techniques on this regard, thank you
for sharing. . . . . .

Thanks-a-mundo for the blog post.Much thanks again. Really Great.

I truly appreciate this blog article.Really thank you! Cool.

I appreciate you sharing this article.Much thanks again. Awesome.

Say, you got a nice article.Really looking forward to read more.

Awesome article post.Thanks Again. Keep writing.

Thank you for some other wonderful post. The place else could anyone get that kind of info in such an ideal means of writing? I have a presentation next week, and I am at the search for such info.

VIO???I??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????I??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

I do believe all of the concepts you’ve presented in your post.
They are very convincing and can certainly work. Still, the posts are too quick for beginners.
Could you please lengthen them a little from subsequent time?
Thanks for the post.

Im thankful for the blog.Much thanks again. Much obliged.

I truly appreciate this blog post.Thanks Again. Awesome.

Appreciate you sharing, great post.Much thanks again. Much obliged.

“Im obliged for the blog article.Really thank you! Great.”

That would be the finish of this post. Here you will uncover some web-sites that we feel you will appreciate, just click the hyperlinks.

What do all these sexy ads mean for those who play the game.
So how does this very promising breast enlargement
method works. This gives the breasts a healthier and more natural look.

Please stop by the websites we comply with, including this one, because it represents our picks through the web.

Very neat blog post.Thanks Again. Awesome.

Here is a superb Weblog You might Locate Fascinating that we encourage you to visit.

Very couple of web sites that take place to become in depth below, from our point of view are undoubtedly properly really worth checking out.

Every as soon as in a though we decide on blogs that we read. Listed beneath are the most recent web-sites that we opt for.

We came across a cool site that you just may possibly take pleasure in. Take a search when you want.

Always a huge fan of linking to bloggers that I enjoy but don?t get quite a bit of link adore from.

“I really appreciate this post. I have been looking everywhere for this! Thank goodness I found it on Bing. You??™ve made my day! Thank you again”

“Thanks for your post. I would love to say that a health insurance dealer also works well with the benefit of the coordinators of a group insurance. The health insurance agent is given a long list of benefits desired by someone or a group coordinator. What any broker really does is seek out individuals or perhaps coordinators which often best match up those desires. Then he provides his tips and if all sides agree, this broker formulates binding agreement between the two parties.”

Here is an excellent Blog You might Discover Interesting that we encourage you to visit.

Check below, are some absolutely unrelated internet websites to ours, nonetheless, they may be most trustworthy sources that we use.

Here is an excellent Weblog You might Come across Interesting that we encourage you to visit.

Im thankful for the blog article.Thanks Again. Want more.

Always a massive fan of linking to bloggers that I appreciate but don?t get a lot of link adore from.

We like to honor many other world-wide-web web sites on the internet, even though they aren?t linked to us, by linking to them. Below are some webpages worth checking out.

We came across a cool web page which you may appreciate. Take a search when you want.

Always a major fan of linking to bloggers that I adore but do not get a great deal of link adore from.

Please take a look at the web-sites we stick to, which includes this one particular, as it represents our picks from the web.

Here are a number of the web pages we advise for our visitors.

Here are some of the web-sites we suggest for our visitors.

Wonderful story, reckoned we could combine a few unrelated data, nonetheless genuinely really worth taking a look, whoa did a single discover about Mid East has got a lot more problerms also.

Very couple of sites that take place to be detailed beneath, from our point of view are undoubtedly properly worth checking out.

Here is an excellent Weblog You might Uncover Interesting that we encourage you to visit.

The details mentioned within the article are some of the most effective accessible.

Usually posts some extremely exciting stuff like this. If you are new to this site.

Leave a Reply