dcsimg

On-line Backups: Flexible Enough for Home & the Office

Backups are a technology or process that everyone -- everyone! -- needs to consider. This article looks at some on-line backup options for Linux that can apply to the spectrum of home to enterprise-class users.

Last week’s column featured a discussion about using Linux software RAID (md) to provide additional data protection in addition to backups. Md can be used for RAID-1 (mirroring) or RAID-5 or RAID-6 to give extra data protection in the event of the loss of a drive. But the primary data protection scheme for almost all situations is a backup. This is true for home systems or laptops, up to very large server farms (there are some questions about doing backups on very large size storage systems but that’s a subject for another day).

This article focuses a bit more on backups for home users and Small and Medium Business (SMB). In particular, this article looks at the concept of on-line backups. That is, backups to a storage farm on the Internet. However, this discussion is not limited to home users and SMB’s but can be applied to most any company or storage need.

Taxonomy of Backups

It is well beyond the scope of this article to discuss the fundamentals of system backups, but for the sake of consistency, let’s do a quick review of backups. A backup is a simple concept of making a copy of data that can then be used to either restore a complete file system in the event that it crashes, or to restore individual files in the event that files are erased (accidentally or otherwise). The complexity behind backups comes from the details of how this process occurs and how it it managed.

While Wikipedia is not the most authoritative source of information, the basic concepts of backups are explained very well and serve as a good starting point for explaining the details of backups including management. It breaks down backups into a easily understood hierarchy:


  • It starts by discussing “Data Repository Models” which are a description of “how” the data is stored. For example,


    • Unstructured
    • Full + incremental (arguably the most common)
    • Continuous data protection

  • A discussion of “Storage Media” for backups


    • Magnetic tape
    • Floppy drives (for the older crowd)
    • Hard drives
    • Optical drives
    • Solid-state drives
    • Remote (on-line) storage

  • Techniques for managing the data repository


    • On-line
    • Off-line
    • Near-line
    • Off-site, off-line vault
    • Backup site of disaster recovery (DR) site

  • Then there is a discussion about selecting and manipulating data that is to be backed up including how to handle live data and metadata


    • Copying files
    • Partial files
    • Filesystem dump
    • Identification of changes
    • Versioning file system

  • There is a discussion about the manipulation of data and dataset optimization (this is where the concept of de-duplication comes in). It discusses the concepts of:

    • Compression
    • De-duplication
    • Encryption
    • Multi-plexing
    • Refactoring
    • Staging

  • Finally there is a section that talks about managing the backup process. It discusses such topics as:


    • Objectives

      • Recovery Point Objective (RPO)
      • Recovery Time Objective (RTO)
      • Data security

    • Limitations (what limitations are there in the overall process):

      • Backup window
      • Performance impact
      • Costs of hardware, software, labor
      • Network Bandwidth

    • Implementation

      • Scheduling
      • Authentication
      • Chain of trust

    • Measuring the process (making sure the backup scheme worked)

      • Backup validation
      • Reporting
      • Logging
      • Validation
      • Monitored Backup


The discussion is very thorough and logical. It is well worth reading even if you are doing backups today.

This article will focus on the backup media from the above taxonomy of backups. In particular it will focus on on-line (remote) backups.

On-line Storage Concepts

The idea of storing backups on-line is really not a difficult one. The fundamental idea is to store data, either full backups, or incremental data, on a remote storage server that is located somewhere on the Internet. You can then use this stored data in whatever manner you see fit. For example, it could be used for restoring systems in the event of a storage failure, or restoring individual files that have been erased. As long as the system requiring the data has access to the Internet, the data can be retrieved.

One additional concept for on-line backups is sharing files. One user can copy their files to the on-line backup and then other users, sharing the same backup, can download the files as needed. You can also extend this concept to something called synchronization. That is, making sure you have the latest files on several of your systems (i.e. the same data is on all of the systems). For example, you can use an on-line backup to upload your important files from your desktop and then download them to your laptop when you are on the road.

Since storage is not free, on-line backups are commercial enterprises with varying cost structures. There are really two main types of backup services


  • Services that use standard Linux tools such as rsync

  • Services that use a custom tool, sometimes web-based

Which approach is better depends upon you, your preferences, your processes, and the cost structure you are comfortable with.

On-line Backup Services Using Standard Linux Tools

The most common tool used for this type of on-line backup is rsync. Rsync is a file copying tool that can be used on local hosts or across remote hosts using a remote shell or a rsync daemon. A key feature of rsync is that it can send just the differences between two files rather than the entire file, saving bandwidth and time. This makes rsync a very popular tool for backups.

S3rsync.com

One service that is available that uses rsync is “rsync to Amazon’s S3″ by S3rsync.com. Amazon has developed a “cloud” storage product called Amazon S3 (Simple Storage Service). S3 is used by a number of companies because of the relatively low cost nature and also because it can be expanded at any time without having to buy hardware. The cost structure of S3 is based on three aspects:


  • Capacity (how much are you storing)
  • Data transfer (how much data is transferred in and out)
  • Requests (getting and putting data)

The rates are also based on monthly usage patterns.

While S3 sounds really great there a couple of caveats to it’s usage. First, S3 does not have a guarantee about not losing data but given the size and scope of S3, losing data is not likely. In addition, to access S3 you need to use Amazon’s tools so they are not easily integrated into existing processes and tools. Moreover, the S3 storage protocol allows you to only ‘get” or “put” data so you cannot modify a file that is stored in S3. If you modify even 1 bit in the file, you have to download the entire file, modify it, and upload the entire file again, leading to extra cost and burning up bandwidth between S3 and your servers.

S3rsync.com has developed a simple way to integrate rsync with S3. The basic concept is to use rsync to S3rsync.com’s servers which then use S3 for their storage. What R3sync.com has done is to put their rsync servers inside the Amazon EC2 facility to allow easy access to S3. All you need is rsync (comes with every Linux distribution) and ssh. In addition there is currently a 10GB limit on buckets (see the S3 site for more information on buckets).

As of the writing of this article, S3rsync.com charges $0.05 per hour in addition to the standard S3 charges.

rsync.net

Another on-line backup service that uses rsync is rsync.net. It is a very popular service, primarily because of it’s support (i.e. engineers man the phones for the most part). Currently, rsync.net supports:


  • rsync/scp/sftp

  • unison

  • rdiff-backup

  • It also supports certain commands using ssh

  • You can mount the file system using sshfs

  • An encryption tool called duplicity that uses rsync but also performs GPG encryption in the process

  • Remote subversion repositories
  • Remote data dumps of MySQL and PostgreSQL databases


Rsync.net also does not have a traffic charge so you can move as much data into and out of the file system as you need. Also, rsync.net allows multiple users to share one account (great for work groups and/or file synchronization).

Pricing for rsync.net as of this writing starts at $1.20/GB per month up to about 50GB ($60/month). Above 50GB there is a discount. If you want to have redundant copies of the data in two different locations, the price is $2.10/GB per month for up to 50GB’s ($110/month).

On-line Backup Services Using Custom Tools

The second class of backup services uses a custom tool or is web based. Some of them may also take a different approach in that the files that are to be backed up have to be in the same directory (this approach works better for work groups and file synchronization). A key feature of this class is that many of them offer free backups up to a certain limit (not a bad way to test the backup services).

Comments on "On-line Backups: Flexible Enough for Home & the Office"

Appreciating the dedication youu put into your blog and detailed information you provide.
It’s nice to come across a blog every once in a while that isn’t
the same outdated rehashed information.Excellent read! I’ve
bookmarked your site and I’m including your RSS feeds to my Google account.

my web blog … Ambassade de pologne 0 paris

What’s up Dear, are you actually visiting thios weeb site on a
regular basis, if so then you will absolutely get fastidious experience.

My page :: imvu credits hack online 2014

“Our ultimate goal is to establish a pro structure for heroes of the storm code competition, but with the game in closed beta, it’s still too early to discuss what form that might ultimately take.You will also
seee the percentage of the game you haave completed at the top of the screen on this menu.

SEO for apps takes work, and probably seems likoe a boring chore for creatijve types who rock app development.

Awwesome article.

Feeel free to surf to my web blpg earning free imvu credits

When some one searches for his necessary thing, so he/she desires to
be available that in detail, thus that thing is maintained over here.

My web-site; game Hacks trainer candy crush saga

constantly i used to read smaller articles that
also clear their motive, and that is also happening with this piece of writing wwhich I am reading at this time.

Feel free to visit my web site: tuniszie pologne 78 (http://www.foundationkey.com)

It’s going to be ending of mine day, except bedfore ending I am reading this fantastic piece off writing to increase my knowledge.

Feel free to vijsit my web blog; war Thunder Esp

I really like your blog.. very nice colors & theme. Did you make
this website yourself or did you hire someone to do it
for you? Plz answer back as I’m looking to create my own blog and would like
to find out where u got this from. thanks

This post is wortgh everyone’s attention. Wheen can I ind out more?

Take a looik at my web blog; war Thunder cheat

I have been browsing online more than 3 hours today, yet
I never found any interesting article like yours. It is pretty worth
enough for me. In my opinion, if all website owners and bloggers made good content as you did, the net will be much more useful than ever before.

Fastidious answers in return of this query with genuine arguments and explaining all
concerning that.

It’s a pity you don’t have a donate button! I’d most certainly donate
to this superb blog! I suppose for now i’ll setttle for bookmarking aand adeing your RSS
feed to my Google account. I look forward to fresh updates and will talk
about this site with my Facebook group. Talk soon!

my website … Pologne xix

Origem: Wikipédia, a enciclopédia livre.
Livre Negocio Criado dia 17 de abril de 2015 por José marcos ribeiro, técnico em telecomunicações e estudante de hipnose

José marcos Ribeiro teve a ideia de criar o site http://www.livrenegocio.com apos perceber
que seu pais estava passando por momentos difíceis, com uma grave crise financeira

jose marcos ribeiro percebeu que apesar de haver milhares de sites de compras
coletivas pessoas comuns não poriam iniciar suas vendas
na internet,
o grande problema que a maioria dos logistas encontravam
erá a demora no repasse do dinheiro

José marcos ribeiro criou o Livrenegocio ou Livre Negocio, a empresa repassa o dinheiro para o
vendedor imediatamente apos o comprador receber o produto que comprou assim o vendedor pode repor o seu
estoque e continuar vendendo.

A ideia do livre Negocio é promover a compra e venda
interna em nosso pais pois todos querem vender aquele celular usado ou augo
novo Porem presisam do dinheiro em tempo abil para pagar suas contas ou usar o seu dinheiro para outras compras

Não que eu não gosto do mercadolivre porem você
pode vender também em sites alternativos que estão crescendo mundialmente eu encontrei
um site na internet que superou minhas expectativas este site http://www.livrenegocio.com tem uma boa clientela pois as quem compra
la esta acostumado a comprar produtos usados de baixo valor e
uma coisa que me agradou muito é o repasse se o cliente
pagar com boleto debito ou deposito eles repassam o valar em até 2 horas
apos confirmar a entrega do produto

Great article! That is the kind of info that are supposed to be
shared across the internet. Disgrace on the
search engines for no longer positioning this publish upper!
Come on over and discuss with my web site . Thank you =)

Also visit my weblog :: best fiends hack tool

Your mode of describing the whole thing in this paragraph is in fact good, every one be capable
of easily understand it, Thanks a lot.

Thanks for any other informative site. The place else could I get that
kind of info written in such an ideal means?
I’ve a mission that I’m just now running on, and I’ve been at the look out for such info.

I love reading through ɑn article tҺаt wiill make men аnd women think.
&#5034lso, thank у&#1086u fօr permitting mе tߋ ϲomment!

Frank Breinling iѕ a recognized expeгt in Ꭺffilіatе Мaгкеtіng.
Тɦе еntгерrеneuгіaⅼ օѡneг օf, ѕaʏ, a $2Ꮇ, $5Ϻ օг $20Ӎ bᥙѕіness ѡillⅼ аlmoѕt ѕսгᥱⅼʏ ƅᥱ ɑ νeгʏ іntгіǥսіǥ ρeгsоn. Ƭherᥱ аге sоmе tһіngs ⲣеοрlе faіl tо tɦink ονег аnd ߋne οf thеѕе іs dеaіng աitɦh mοneʏ.

my ԝеvlօɡ :: http://Www.Keysol.Info

THANK YOU for sharing this with us. I downloaded Genesis last night and
will be playing it at work.

Appreciating the time and energy you put into your website and
detailed information you present. It’s good to come across a blog every once in a while
that isn’t the same outdated rehashed material. Excellent read!
I’ve bookmarked your site and I’m adding your RSS feeds to
my Google account.

Hi! I’m at work surfing around your blog from my new iphone 3gs!
Just wanted to say I love reading through your blog and look
forward to all your posts! Carry on the superb work!

Thank you for the good writeup. It if truth be told was a enjoyment account
it. Look advanced to more added agreeable from you!
By the way, how could we communicate?

I’m going to go over a few tips that really have given tthe experts
an upper edge and is detrimental to heroes of the storm Gold code success in each game.
Draw a KWL chart on the board, and divide it into three rows.
Items now have thneir item levvel listed and a plzyer can see the average gear score
on his character tab.

Thanks for one’s marvelous posting! I certainly enjoyed reading it, you may be a great author.I will be sure to bookmark your blog and may cone back iin the foreseeable future.
I want to encourage you to ultimatesly continue your greatt work,
have a nice morning!

my blog :: Idol software

I think the admin of this web page is in fact working hard
in favor of his web site, for the reason that
here every information is quality based stuff.

Hello there! This article couldn’t be written any better!
Going through this article reminds me of my previous roommate!
He constantly kept preaching about this. I most certainly will send this information to him.
Fairly certain he’ll have a very good read. Many thanks for sharing!

I do not even know the way I stopped up here, but I assumed this publish was good.
I don’t understand who you might be however certainly you’re going to a famous blogger when you aren’t already.
Cheers!

My brotherr recommended I might like this web site.

He was totally right. This post acdtually made
my day. You cann’t imagine simply how much time I hhad spent for this information! Thanks!

my web page: war thunder aimbot

Wow! Finally I got a weblog from where I can really obtain valuable
data regarding my study and knowledge.

I’m curious to find out what blog system you have been utilizing?
I’m having some minor security problems with my latest blog and I would like to
find something more safeguarded. Do you have any suggestions?

Dreamѕ become as viviԀ as еνᥱг, and үоu ѡіlⅼ ѡаҝе
ᥙр fᥱᥱling ϲomρⅼеtеⅼү геfгеѕɦeɗ.
Տо ɡеneгаllү,
еat meat, fiѕһ, frᥙіt аnd ᴠеgetаƅlᥱѕ.

If ʏоᥙ mᥙѕt ѕnaсқ іn thе
еvеning Һoᥙrs,
ріcκ а Ԁᥱfіnitᥱ tіmе, mayЬе tеn РᎷ fοг tҺᥱ smaⅼⅼ tгеɑt.
Sоmеtіme Ⅰ аm ѕᥱaгcҺing іn tһе mігrⲟr аnd Ƅеǥіn оf
ϲгʏіng. Hеге aгᥱ a fᥱѡ tɦіngѕ уоᥙ сan ехреct ߋn tҺе HCG Dіᥱt.

Ⲏеrе іѕ
mу աеƄ pagе; hcg diet plan

Hello mates, its enormous piece of writing concerning tutoringand entirely defined, keep it up all the time.

Everyone loves it when people get together and share
thoughts. Great blog, continue the good work!

Way cool! Some very valid points! I appreciate you
writing this write-up and also the rest of the website is really good.

Thanks designed for sharing such a good idea, post is
fastidious, thats why i have read it completely

You’ve made some decent points there. I
checked on the web for additional information about the issue and found
most individuals will go along with your views on this web
site.

My spouse and I stumbled over here different page and thought I
may as well check things out. I like what I see so now
i am following you. Look forward to looking over your web
page for a second time.

Nice post. I used to be checking continuously this blog and I’m impressed!
Extremely helpful info specially the remaining part
:) I care for such info much. I was seeking this
particular info for a very lengthy time. Thank you and good luck.

I have been browsing online more than three hours today,
yet I never found any attention-grabbing article like yours.
It is beautiful price sufficient for me. Personally, if all webmasters and
bloggers made excellent content material as you did,
the web will probably be much more useful than ever before.

Excellent way of explaining, and good post to obtain facts regarding my
presentation focus, which i am going to convey in academy.

Inspiring story there. What happened after?
Take care!

Write more, thats all I have to say. Literally, it seems as though you
relied on the video to make your point. You obviously know what
youre talking about, why waste your intelligence on just posting videos to your weblog when you could
be giving us something enlightening to read?

Superb blog! Do you have any hints for aspiring writers?
I’m hoping to start my own website soon but I’m a little lost on everything.
Would you suggest starting with a free platform like WordPress
or go for a paid option? There are so many choices out there that I’m totally confused ..
Any ideas? Bless you!

I know this site gives quality depending articles or reviews and additional material, is there any other website
which provides such things in quality?

Hello there, I discovered your site by the use of Google while
searching for a comparable subject, your website
got here up, it looks great. I have bookmarked it in my google bookmarks.

Hello there, simply was alert to your blog via Google, and located that it is really informative.

I am going to watch out for brussels. I will be grateful
should you continue this in future. A lot of people will probably be
benefited out of your writing. Cheers!

I’m really loving the theme/design of your website.
Do you ever run into any browser compatibility issues?

A small number of my blog audience have complained about my site not working correctly in Explorer but looks great in Opera.
Do you have any tips to help fix this problem?

Good replies in return of this difficulty with genuine arguments and describing everything concerning that.

It’s an remarkable article designed for all the internet
people; they will take advantage from it I am sure.

Right here is the right website for everyone
who hopes to understand this topic. You know a whole lot its almost hard to argue with you (not that I really will need to…HaHa).
You certainly put a fresh spin on a subject which has been written about for years.

Excellent stuff, just great!

It’s an awesome paragraph in favor of all the online visitors; they will take benefit from it I am
sure.

Leave a Reply