dcsimg

Creating a NAS Box with an Existing System

Standalone Network Attached Storage (NAS) servers provide file level storage to heterogeneous clients, enabling shared storage. This article presents the basics of NAS units (NFS servers) and how you can create one from an existing system.

There are many times when you need shared storage, a common file system or the ability to have data to be easily shared between clients (even ones that might be heterogeneous). Ideally you want something standards-based so you can share data with Linux, BSD, OSX, or even Windows systems. Fortunately there is a standard for shared storage: Network File System (NFS). In fact, it is the only standard file system and is the driving force behind the creation of NAS (Network Attached Storage) devices in widespread use today.

This article takes a quick peak at NFS and NAS devices and presents simple steps for getting NFS going on almost any existing server you might have. Since NFS is the driving force behind this, let’s start with a quick review of NFS.

NFS

NFS was the first widespread file system that allowed distributed systems to share data effectively. The fact that it’s the only standard network file system bears repeating. NFS comes with virtually every *nix distribution known and you can also get clients for other operating systems such as Windows.

Basically NFS allows you to take a server with some attached storage and “export” it to or “share” it with, a group of clients. These clients can then all access the same file system and share data. The storage on the server actually shares files to the clients rather than just provide raw storage in the case of a SAN (Storage Area Network). This means that the storage on the server has to have been formatted with a file system such as ext3, ext4, xfs, jfs, reiserfs, etc.

NFS is a fairly easy protocol to follow. All information, data and metadata, flows through a file server. This is commonly referred to as an “in-band” data flow model shown in Figure One below.

NFS_model.jpg
Figure 1: In-Band Data Flow Model (Courtesy of Panasas)

Notice that the file server touches and manages all data and metadata. This model does make storage systems a bit easier to configure and monitor since you have to worry about a single system. In addition it has narrow, well defined failure modes. Some drawbacks of the architecture include an obvious bottleneck for performance, problems with load balancing, and security is a function of the server node and not the protocol (this situation means that security features can be all over the map).

The general data flow in NFS is fairly simple. When a client makes a file request to a NFS file system it has “mounted”, the mount daemon transfers the request to the NFS server, which then accesses the file on the local file system. The data is the transferred from the NFS server to the requesting node, typically using TCP. Notice that NFS is “file” based. That is, when a data request is made, it is done on a file, not blocks of data or a byte range. This is why we say that NFS is a file based protocol.

For more details of how NFS works, Figure 2 below illustrates the stack layout of NFS

NFS_Stack.jpg
Figure 2: NFS Protocol Stack (Courtesy of Panasas)

The top portion (in pink) is the client which has an application that makes an IO request that goes to the system call interface. If the file system is NFS based then the request is sent through the network to the server (the center portion in blue labeled “Server”). The request is sent to the user component of the file system which then communicates with the storage component of the file system. Notice that the box to the right labeled, “NVRAM” is an optional component that some vendors use to speed up operations through a cache.

Finally the server communicates with the the sector/LBA interface to the blocks managed on the storage device which then retrieves the data from either the cache or the platters. Then the data is passed back up the stack to the client application. To the client application the file systems looks and behaves as though it is a local file system.

There is much more work in developing a good understanding of NFS but the point of this article is to discuss taking an existing box and making a NAS box. The next section talks about how to “activate” NFS on a server, effectively creating a NAS box. There are a number of very good howto’s around the web on configuring and starting NFS so this article will just be a quick summary and not a comprehensive howto (i.e. some details will be left out).

Starting up NFS

To better explain the steps for configuring NFS on an existing server, I’ll use a server that I’ve used in past articles. The highlights of the server test system used in this article are:

  • CentOS 5.4
  • GigaByte MAA78GM-US2H motherboard
  • An AMD Phenom II X4 920 CPU
  • 8GB of memory (DDR2-800)
  • Linux 2.6.30 kernel (with reiser4 patches only)
  • The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ultra ATA/100)
  • /home is on a Seagate ST1360827AS
  • There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb, for the file system, which in this case is ext4. The second hard drive, /dev/sdc was used for the journal portion of the file system. It was partitioned to the correct size and only that partition was used for the journal (/dev/sdc1).

The first step in creating a NAS box from an existing server is to simply configure and start up NFS. I will assume that you have installed NFS on your system including the NFS server component (it is beyond the scope of this article to walk through the various distributions to explain how to do it). One way to check if NFS is installed on the server is if the file /etc/exports exists (it may not have anything in the file, but the file should exist). In addition, on CentOS or RHEL, you can also check if the NFS server is installed by looking for the file /usr/sbin/rpc.nfsd.

[laytonjb@test64 ]$ ls -s /usr/sbin/rpc.nfsd
20 /usr/sbin/rpc.nfsd

For rpm based systems you might also try the following:

[laytonjb@home8 ]$ rpm -qa | grep -i nfs
system-config-nfs-1.2.8-1
nfs-utils-1.0.6-46

(this is an old CentOS 4.1 system so the versions numbers will definitely not match anything newer). At this point let’s assume that the server components of NFS are installed.

Typically I next start NFS on the server. A quick way to check if NFS is running is to use the command, “rpcinfo -p” which reports the RPC (remote procedure call) information on the system. If NFS server has not been started, you will see something like the following.

[root@test64 ]# /usr/sbin/rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    774  status
    100024    1   tcp    777  status

The most important thing in this output is that the “portmapper” is running which is an important part of NFS. If yout don’t see portmapper in the list, please read your documentation on how to get it running.

To start the NFS server, or really the server portion of NFS, depends upon your distribution to some degree. On the test CentOS system it was simply accomplished by the following command.

[root@test64 ]# /etc/init.d/nfs start

We then run “rpcinfo -p” again to see if NFS was started.

[root@test64 ]# /usr/sbin/rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    774  status
    100024    1   tcp    777  status
    100011    1   udp    645  rquotad
    100011    2   udp    645  rquotad
    100011    1   tcp    648  rquotad
    100011    2   tcp    648  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100021    1   udp  42049  nlockmgr
    100021    3   udp  42049  nlockmgr
    100021    4   udp  42049  nlockmgr
    100021    1   tcp  54256  nlockmgr
    100021    3   tcp  54256  nlockmgr
    100021    4   tcp  54256  nlockmgr
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp    669  mountd
    100005    1   tcp    672  mountd
    100005    2   udp    669  mountd
    100005    2   tcp    672  mountd
    100005    3   udp    669  mountd
    100005    3   tcp    672  mountd

Notice that you see NFS listed and that NFS v2, NFS v3, and NFS v4 were started (look at the second column). Also notice that the UDP and TCP protocols for NFS are listed as well. The program mountd, is the NFS mount daemon.

There are a number of daemons needed for NFS to operate. The /etc/init.d/nfs script started everything for us but in general the daemons we need are:

  • rpc.nfsd (server)
  • rpc.lockd (common to both server and client)
  • rpc.statd (common to both server and client)
  • rpc.mountd (common to both server and client)
  • rpc.rquotad (server)

Exporting Data

The next step is to tell the NFS server what directories can be shared with other systems (clients). In the vocabulary of NFS, this is called “exporting” directories and the specific directories are called “exports.” Now that NFS is started, let’s configure it to export a directory on the server.

To do this, we edit the file /etc/exports by defining the directories to be exported and their properties. The typical entry in /etc/exports looks like the following:

directory machine1(option11,option12) machine2(option21,option22)

where:

  • directory is the server directory you want to export to clients. It can be a single directory or an entire drive. However, it has to be formatted with a file system. Each line in /etc/exports is a separate directory.
  • machine1, machine2 are the names of clients you wish to share data with. For example they could be listed using system names such as, client1 or client2 (be sure the clients are listed in /etc/hosts or via NIS), or they could be listed by IP address such as 192.168.1.8. You can also used the wild card, “*”, to indicate any client. It is HIGHLY recommended that you list every client here to help ensure that no “rogue” clients can mount the exported directory. It is a pain to maintain the list of clients, particularly if it is a long list, but this little bit of security can help with unsophisticated attacks. On the other hand, if you are behind a big juicy firewall and are confident in it’s abilities and that no one will cause any hard either intentional or by accident, then you can just list the machines as “*” which means all clients (many times HPC clusters will do this because the cluster is on a private network which is usually behind a big juicy firewall).
  • (option11, options12, …) are a list of options used for exporting the directory. There are a number of options that can be used and it’s beyond the scope of this article to present then all. However, some of the more important ones are:

    • ro which stands for read-only. So the server can export directories that are read-only so the clients cannot write to them.
    • rw which stands for read-write. This means that the clients can read and write to the exported directory.
    • no_root_squash which means that user “root” on the client machine will have the same level of access to the files on the system as user “root” on the server. Again, there are a number of security issues surrounding this and it is recommended that you do NOT use no_root_squash on the clients.
    • sync which tells NFS to wait until the data has been flushed to the storage device before returning. The other option is async which allows the NFS server to return to the client application before the data has been sent to the device (i.e. it may just be in cache somewhere). It is recommended that you use the “sync” option to ensure that the data has been written to permanent storage. However, there is a performance penalty for this option with NFS file systems mounted using the “sync” option being slower than “async.”

There are many ways in which you can export directories to clients. It all depends upon what you want to accomplish and how you want to accomplish it. For example, you could export a directory on the server that contains applications that the clients might need as shown below in a sample line from /etc/exports.

/opt 192.168.1.8(ro)

In this case the server is exporting (sharing) the directory, /opt which contains some applications, to a client, 192.168.1.8, which can mount it read-only (no writing). This is a fairly common way of installing applications on a single server and sharing them with other clients.

One of the biggest uses of NFS is for home directories. It is fairly easy and common to put user’s home directories on a single server and then mount it on the clients. Such an entry in /etc/exports might look like the following:

/home 192.168.1.8(rw)

In this case, /home is mounted as read-write on the client 192.168.1.8 since we want users to be able to write to their own home directory. If you want, you can get more granular and specify each user on a line in /etc/exports and control which clients that user will have a home directory. This gives you some control at the expense of extra work on the NFS server along with some additional complexities. For example, you might want to do something such as the following.

/home/laytonjb 192.168.1.8(rw)
/home/test 192.168.1.65(rw)

In this case the first user, laytonjb, is exported to client 192.168.1.8 and the second user, test, is exported to client 192.168.1.65. This gives you fairly fine control over what is exported and to which machines it is exported. Coupling this with a fairly good way to update /etc/fstab on the clients gives the administrator great control over data access.

Again, NFS security is not a focus of this article, but to help yourself, it is recommended that you use the files /etc/hosts.allow and /etc/hosts.deny. They are strictly not necessary but it does give the administrator a bit more control over the NFS configuration. The first file can be used to define which clients are allowed to use services on the host machine and the second file lists which clients are denied access to certain services on the host. While this link is a bit old, it does explain how to use these files to better protect your system.

For the simple example in this article I have just made a single entry in /etc/exports.

/mnt/home1 192.168.1.8(rw)

It exports a directory called /home1 to a client, 192.168.1.8, and allows read and write access to it. After I save the file I have to tell NFS to re-read the /etc/exports file and re-export it (i.e. make it available to the client). The command for this is simple.

[root@test64 ]# /usr/sbin/exportfs -ra

The command exportfs controls the list of exported directories and the options “-ra” tell it to export all directories listed in /etc/exports (the “-a” option) and to “re-export” all directories in /etc/exports (the “-r” option).

NFS Client

The server side of NFS is done with at least one directory, in this case /mnt/home, exported to a client on my network. The client side of NFS is fairly easy as well. As with the server you have to have NFS installed on the client but you don’t have to have the server portion of NFS installed. On many distributions the “client” piece of NFS is sometimes called “nfs-common” or sometimes “nfs-client” or something similar. A generic way to check if your system is NFS ready is to look at /proc/filesystem.

laytonjb@laytonjb-laptop:~$ cat /proc/filesystems
nodev	sysfs
nodev	rootfs
nodev	bdev
nodev	proc
nodev	cgroup
nodev	cpuset
nodev	debugfs
nodev	securityfs
nodev	sockfs
nodev	pipefs
nodev	anon_inodefs
nodev	tmpfs
nodev	inotifyfs
nodev	devpts
	cramfs
nodev	ramfs
nodev	hugetlbfs
nodev	mqueue
nodev	fuse
	fuseblk
nodev	fusectl
nodev	usbfs
	ext3
nodev	rpc_pipefs
nodev	nfsd
nodev	binfmt_misc

The second line from the bottom indicates that this system is indeed capable of NFS.

As with the server you need several daemons for running NFS on the client. In particular you need three daemons for the client:

  • portmapper (use “rpcinfo -p” to check)
  • rpc.statd (needed for file locking)
  • rpc.lockd (needed for file locking as well)

These should be started at boot when you installed the NFS client portion.

The final step is to simply mount the NFS exported directory. If you want to test it by hand you can use “mount” on the command line.

[root@home8 etc]# mount 192.168.1.65:/mnt/home1 /mnt/nfsserver
[root@home8 etc]# ls -s /mnt/nfsserver
total 8
4 laytonjb  4 test
[root@home8 etc]# ls -s /mnt/nfsserver/laytonjb
total 40
20 ext4_own_journal.txt   4 fdtest_script  16 fdtree.bash

where the NFS server is 192.168.1.65. Be sure the mount point, /mnt/nfsserver exists before you mount the file system or you will get an error message that the mount point doesn’t exist.

You can also put this in your /etc/fstab file. An example /etc/fstab file looks like the following.

[root@home8 etc]# more /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext2    defaults        1 2
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /dev/shm                tmpfs   defaults        0 0
none                    /proc                   proc    defaults        0 0
none                    /sys                    sysfs   defaults        0 0
LABEL=SWAP-hda3         swap                    swap    defaults        0 0
...
192.168.1.65:/mnt/home1  /mnt/nfsserver         nfs     defaults        0 0

I have dropped some junk from the /etc/fstab file to protect the innocent (me). Also the client is an older CentOS 4.1 system so the /etc/fstab file may not match what you have on your system. However, the syntax for the entry in the file is correct.

Summary

Taking an existing server and turning it into a NAS box that functions as a central server for a set of clients is actually fairly simple. You can take your favorite distribution, make sure NFS and NFS Server are installed, and configure NFS on the server. Then on the clients you configure each client to mount the exported file systems from the server. The process is not difficult, but perhaps a little time consuming.

This article has just briefly touched on the subject of security for NFS. This can be an important issue if you are worried about possible security issues and you should be if you are operating on a network with clients (HPC is another story for a different article). Be sure to use the many web based articles and HOWTO’s around NFS security. There are also security books that talk about securing NFS (my personal favorite is “Real World Security” by Bob Toxen. It’s perhaps getting a bit old, but the general discussion around NFS is quite good).

In upcoming articles I’m going to examine dedicated NFS distributions so you can take an existing system, perhaps recycling an older box, or even use a new box, and use these distributions to quickly create a NAS system. Stay tuned!

Comments on "Creating a NAS Box with an Existing System"

anandshankar

Congrats for a good writeup.

For sake of completeness, kindly let us know of a GPL / Free NFS Client for Windows.

What about NFSv4 and a corresponding client on Windows?

NAS needs to be OS neutral as far as clients are concerned.

anand

Reply
jerrybear

NFS was a big breakthrough on the OS software 25 years ago.
MFS(Mirror File System) builds on top of NFS and expands NFS\’
capacity to a new level. IMHO, one may be intrigued.

Jerrybear

Reply

    ce livre tient toutes ses prmsesoes. Si Jamie Oliver nous fait faire des menus pour 4 personnes en 30 Minutes Chrono, Guy Martin nous permet de réaliser des sandwichs en 5 minutes montre en

    Reply
jtillman

I\’m sorry, but the basis of this article makes no sense. NAS devices do not use NFS as the primary means of connectivity. This is borne out by the fact that the majority of NAS users use Windows, and Windows doesn\’t support NFS.

A cursory glance at Wikipedia\’s entry for NAS reveals that \”NAS uses file-based protocols such as NFS (popular on UNIX systems), SMB/CIFS (Server Message Block/Common Internet File System) (used with MS Windows systems), or AFP (used with Apple Macintosh computers). NAS units rarely limit clients to a single protocol.\”

I\’d go further by saying that limiting the protocol to NFS would be stupid, since the majority of clients don\’t support it.

This looks like a great article for NFS configuration, but let\’s not try to sell it for more than it is. It just ticks off the readers who get duped into clicking the link.

Reply
dustin.plomondon

This is a nice write-up, but my stumbling point lately has been selecting hardware, and the best option for RAID (software/kernel raid vs. hardware raid). Optimally, I\’d have a quiet headless server (blade perhaps) that has a hardware RAID supported by VMware ESX. That way, I could have multiple VMs in which I could contain my NAS appliances. The concept makes sense to me, but the reality seems very convoluted.

Certainly makes purpose-built boxes like qnaps and drobos enticing.

Reply
jerrybear

One can configure the same NAS box to support SMB/CIFS with samba easily.
So a Linux/Unix NAS box can support both NFS and SMB/CIFS.
Samba is the standard Windows interoperability suite of programs for Linux and Unix.
http://samba.org/.

Reply
tallship

Nice basic article, covers the rpm based distro general aspects, but as a Slackware user I was a little surprised at the relative lack of exportfs coverage.

i.e.,

# exportfs -av host:/path

default export options are sync,ro,root_squash,no_delay. Use -o to specify options other than default, and push those changes into the kernel.

# exportfs -rv

Reexports all directories and synchronizes /var/lib/nfs/xtab with /etc/exports

-v</strong is for, of course, \’verbose\’.

You can also remove exports, independently from /etc/exports with -u (for unexport) and change options irrespective of /etc/exports on the fly.

etc….

exportfs is much more than just a tool for updating changes to /etc/exports.

Kindest regards,

Bradley D. Thornton
http://NorthTech.US

Reply
jobst

Anybody is better off doing this as a SAMBA box and installing the SMB client stuff on the LINUX boxen that want to connect than installing NFS on the windows clients. I do not think that any NAS box that you can buy off shelf would have NFS only, the main OS still is Winblows.
The article header is plain wrong, it should be \”how to make an NFS server\” and not \”how to make a NAS box\”, its silly really … This comes from a guy who installed the first full blown samba server in a company in 97, replacing the NT server and I have never looked back!

Reply
dragbirs

I don\’t think this article addressed the NAS aspect – Network Attached Storage. Too simplistic in deferring to NFS. Let me inform you that very few will use NFS solely unless they have homogeneous systems. What is typical is Windows CIFS, UNIX NFS and Media-based DLNA. If the intention is enterprise NAS – since the author chose CentOS, its CIFS, iSCSI and NFS. For home use – its Ubuntu, Fedora, Suse, and the list goes on. There is freenas.org – pretty good. As a hobbyist – I chose OpenSolaris because I know what I\’m doing – its not recommended to newbies, but is an intro to ZFS. ZFS simplifies NFS, CIFS and iSCSI implementations. In NFS there are many tuning required e.g. rsize/wsize, this is dependent on the type of network – do you have a LFN where latency needs mitigation? Are you using gigabit? And the worst aspect of NFS is UID/GID – I have seen so many mis-matches. No mention of the rationale for NFS v4. If you show the versions – at least briefly mention why they differ. This is NOT a good article. Its point? Why read this when there the Linux NFS HOWTO exists? (I know not many have access to SunSolve). Go back and rewrite this article. PROPERLY.

Reply
laytonjb

@jtillman, @jobst, @dragbirs,

A troll by any other name…. My editor told me there would be lots of trolls and I guess I found at least three of them. he also said not to respond, but I can\’t resist this time.

The first point I want to make is that yes, NAS .ne. NFS (to borrow some ancient Fortran syntax). But then again, this is LINUX Magazine and to be quite honest, I don\’t care about Windows clients. I know people do but I don\’t. If you like, install Samba, configure it, and then shut up.

Second point, particularly for @dragbirs, iSCSI is not NAS. iSCSI is for a SAN – completely different storage model. I think you need to PROPERLY read before you write.

Third point – I did not go into details about to configure NFS for performance, or which protocol to configure (v2, v3, v4), etc. because the details are so load and configuration dependent. If you want a very in-depth history and NFS deep-dive, just ask. But don\’t whine because I didn\’t write it – the articles are long enough already. They aren\’t intended to be HOWTO\’s (I think I mentioned that a few times).

For the other people who posted comments – thanks for the comments and the details. As I mentioned previously, this article can\’t be all encompassing so I can\’t dive into details about configurations or exportfs. So I appreciate it when people post follow-up details.

And finally… I know, I know – don\’t feed the trolls. But I did this once but I\’m out of food and out of patience but I have one last comment: If you don\’t like what I write, don\’t read it. Write your own. Very few people write about storage (most of it is vendor gibberish) so we need good writers. I will even help get you published here if you like.

Jeff

Reply
n3krodamus

Nice article, this was very usefull when I tried to configure my own NFS server for a few Linux VMs (virtualboxes). Short, easy, clear, right to the point; just what I\’been looking for. Congrats. This art. is in my colection of documents now. Thanks.

Reply
dragbirs

I may have been a bit harsh in criticism but you showed how un-professional you are by starting name-calling. I have provided constructive material – yet you respond with childlike antagonism. I guess that what separates me from you – I work in the Telecom industry solving real problems, working with the vendors to address their product limitations. I don\’t need to brag about whats bigger or cutting edge – but I\’ll tell you this – there are trolls out there and there are professionals as well – which are you? You also need to be able to handle criticism. Finally, ISCSI is not NAS? Ever hear of \”convergence\” and \”tcpip\”? Maybe I should give you a call when I need your expertise whilst I\’m working with Oracle configuring two of their Exadata v2? You have really shown your true colors my friend – I wish you the best. Your editor was also right.

Reply
kgas

One of the difficulty with NFS is changing IP address for both server and the clients. (IP –> ADSL –> switch–>| clients/Sever model)

Reply
dragbirs

I\’m willing to soften my posts if I am given the opportunity to edit them. In fact – I\’m pretty open to removing the last comment. Otherwise – the record speaks for itself.

Reply
ruilapa

One important and significant command left out was:

showmount -e 192.168.1.65

allowing you to consult which \”exports\” are available on the server.

Reply
tdbtdb

Is there a link somewhere on this page back to the original article? If not, is that weird or just me?

Reply
tdbtdb

ahh… found it. \’Read the full article here.\’

Reply
dragonwisard

To everyone complaining that Windows doesn\’t support NFS…

In my server room I only have Linux boxes anyways so NFS makes more sense to me than SAMBA. Since this is LINUX Magazine I would assume the author might have had a similar scenario (with Linux on both the server and clients) in mind.

Reply
lumix

I have tried implementing this tutorial. I get an error in between the execution of the works. I’m finding difficulty with the export of the directories. Oh ruliapa that part you noted was my problem and it worked thanks. cyberdefender

Reply

I am given the opportunity to edit them. In fact – I\’m pretty open to removing the last comment. Side effects of Vicodin Otherwise – the record speaks for itself.

Reply

Hi, Neat post. There is an issue along with your site in web explorer, might test this? IE nonetheless is the marketplace chief and a huge portion of other people will omit your fantastic writing because of this problem.

Reply

A treaty of underlying filesystem would be terrific. Linux’s mainstream filesystems are now behind on 64-bit support (read: 20TB+ is now a problem depending on block size). Maybe you can followup with a second article now that so much time has passed.

Reply

Network Attached Storage is becoming very popular for domestic users, specially those who has loads of digital files to be shared in a family with two or more devices at home. The website http://www.besthomenas.net explain in detail options for domestic application. Cheers, Marc.

Reply

A round of applause for your blog. Awesome.

Reply

Thanks for every other informative web site. The place else may I am getting that kind of info written in such an ideal means? I have a venture that I am simply now running on, and I’ve been at the glance out for such information.

Reply

Can I simply say what a relief to find someone who actually is aware of what theyre talking about on the internet. You positively know the way to deliver a difficulty to light and make it important. More individuals need to read this and perceive this facet of the story. I cant imagine youre not more widespread because you definitely have the gift.

Reply

I know this if off topic but I’m looking into starting my own weblog and was curious what all is needed to get set up? I’m assuming having a blog like yours would cost a pretty penny? I’m not very internet savvy so I’m not 100 certain. Any suggestions or advice would be greatly appreciated. Appreciate it

Reply

Hi Shelly – thank you for all your Tweets and Posts in 2010. They have given me variously: food for thought; ideas to process; links to chase; passions to appreciate; events to take joy in; cause for amkliment/chucuses/entertaenment. I look forward to your continuing exciting and entertaining contributions in 2011. PS so hope you will visit West Aus so we can meet for real Jo Hart

Reply

I blog often and I seriously appreciate your information. This
great article has really peaked my interest.
I will take a note of your website and keep checking for new information about once per week.

I opted in for your RSS feed too.

Reply

Hej fru Larsen. Tak for hjaelpen med mit botaeeeleppg. Jeg maa proeve at haekle dem med paa det resterende 1/3. Saa bliver det altid det mindre at haefte hihi. Jeg har vaeret herinde og kigge et par gange, men bare ikke blevet medlem. Det er hermed gjort. God mandag. http://yqlgsi.com [url=http://haiygplk.com]haiygplk[/url] [link=http://ywmwgx.com]ywmwgx[/link]

Reply

would http://carinsuranceratescto.info cheap car actions wo http://safeinauto.com main owning http://carinsurancequotessc.top cheap auto some companies http://autoinsurancemaw.info repair services a.k.a. coverage http://autoinsurancersr.top compare again coverage http://autoinsurancebit.net going driving http://autoinsurancenir.top specific nice agent http://carinsurancerut.info highway safety

Reply

essential possessions http://cheapcarinsurancefc.top same company car without http://carscoverageonline.com little-known course http://autoinsurancebit.net could probably mortgage http://carinsurancequotessc.top accumulate liabilities much extra http://carinsurancemr.net states honored http://carinsurancerut.info dropping

Reply

insurance company http://safeinauto.com another example best http://carscoverageonline.com car accidents categories http://autoinsurancequotesem.us insurance rates other http://autoinsurancegl.net paying too

Reply

loss prevention http://cheapautoinsurancewcx.info purchase needed discounted http://carinsurancert.top road different formulary http://carinsurancemr.net find yourself go http://carinsurancerut.info money who http://autoinsurancenir.top various safety

Reply

chance http://autoinsurancegl.net looking teen auto http://autoinsurancend.info find some http://autoinsurancemaw.info most cases website http://autoinsurancersr.top insurance rates ignition http://carinsurancemr.net self employed payouts http://carinsurancert.top becomes necessary quotes once http://autoinsurancebit.net accident great instance http://carinsuranceast.us liability

Reply

Appreciating the hard work you put into your website and in depth information you present.
It’s nice to come across a blog every once in a while that isn’t the same outdated rehashed material.
Fantastic read! I’ve bookmarked your site and I’m including
your RSS feeds to my Google account.

Reply

clear http://carinsurancelit.top figures free quoting claims http://autoinsuranceweb.top quoting claims individuals avoid http://autoinsurancenir.top damage natural quotes http://autoinsurancemaw.info traffic rules however http://cheapcarinsurancefc.top self-employed

Reply

typically http://carinsuranceratescto.info struggle credit score http://safeinauto.com automobile insurance accident income http://cheapcarinsurancefc.top right type includes http://autoinsurancequotesem.us financial loss turn http://carinsuranceast.us some people

Reply

I really like your blog.. very nice colors & theme.
Did you design this website yourself or did you hire someone to do it for
you? Plz respond as I’m looking to design my own blog and
would like to find out where u got this from.
thanks a lot

Reply

increased fuel http://autoinsuranceweb.top low premium personal accident http://autoinsurancemaw.info could insurance seems http://cheapcarinsurancefc.top than approaching

Reply

companies want http://autoinsurancebit.net leave check probably http://autoinsurancersr.top ago term obtaining http://carinsuranceratescto.info grasp offered http://cheapcarinsurancecr.top insurances disability take care http://carinsurancemr.net record female drivers http://autoinsurancegl.net perfect person more http://carinsuranceast.us liability quotes sites http://autoinsurancequotesem.us someone else

Reply

luck http://carinsurancerut.info being auto http://carinsuranceast.us primary different states http://safeinauto.com substituting years see exactly http://autoinsuranceweb.top mark best car http://cheapcarinsurancecr.top left behind keep http://cheapcarinsurancefc.top next hours know http://carinsurancert.top circumstances dictate

Reply

today hectic http://cheapcarinsurancecr.top represent rates even http://carinsurancequotessc.top based business them knew http://autoinsuranceweb.top best christian insurance http://cheapautoinsurancewcx.info kansas required option http://autoinsurancequotesem.us classic others http://autoinsurancend.info insurance company

Reply

types http://carscoverageonline.com negatively affect man http://autoinsurancenir.top pretty decent some http://cheapautoinsurancewcx.info driving shop any http://safeinauto.com rental housing individual http://autoinsurancemaw.info card accounts tremors http://autoinsurancersr.top most likely

Reply

insurance policy http://cheapcarinsurancefc.top makes economic http://cheapcarinsurancecr.top most low particularly slow http://carinsurancert.top usually less savings coming http://autoinsurancequotesem.us while driving less powerful http://autoinsurancend.info other

Reply

erectile dysfunction http://viagracvd.top penis enlargement erection problems http://cialismil.info take highlands plateau http://viagragro.info common while levitra http://cialisewq.top erectile

Reply

erectile dysfunction http://cialismil.info nitrix oxide frequently http://cialisewq.top men think cnidium seed http://viagragro.info before except erectile dysfunction http://viagracvd.top before researchers

Reply

brew http://cialismil.info feel before starting http://cialisewq.top sexual impotence been used http://viagracvd.top sexual find http://viagragro.info penis

Reply

common http://viagragro.info vitamin-enhanced side http://cialisewq.top me herbal penis http://viagracvd.top prescription waiting room http://cialismil.info disorders surgery

Reply

fda http://viagracvd.top tantric techniques cancer per http://viagragro.info male cutting-edge health http://cialisewq.top ugly head infertile dysfunction http://cialismil.info preclude

Reply

taken away http://viagragro.info refills use herbs http://cialisewq.top concluded created http://viagracvd.top erections very http://cialismil.info walking

Reply

victim http://viagragro.info herbal viagra natural remedies http://cialismil.info drug shown http://cialisewq.top men think however since http://viagracvd.top observed

Reply

male potency http://viagracvd.top necessary adjustments flow http://viagragro.info frustration neighbourhood gossip http://cialismil.info little blue product http://cialisewq.top diabetes

Reply

satisfaction http://cialisewq.top key enough blood http://viagragro.info ginseng red ginseng http://viagracvd.top pharmacological companies control http://cialismil.info restore

Reply

see rockier http://cialisewq.top treatment erections http://viagracvd.top look them like slitting http://cialismil.info because natural penis http://viagragro.info like refund

Reply

needs http://autoinsurancequotesro.info household loan lower http://carinsurancemr.net insurance card accident http://autoinsurancebit.net here transport system http://autoinsurancegl.net coverage some computer http://carinsuranceratescto.info social benefit coverage http://autoinsuranceweb.top other training courses http://autoinsurancequotesem.us estimated market

Reply

problems due http://viagracvd.top goji berry some http://cialismil.info blood build online pharmacies http://cialisewq.top affects penis http://viagragro.info effect

Reply

damage http://viagracvd.top use enzymes http://cialismil.info seven way http://cialisewq.top stimulate menstrual side effects http://viagragro.info million men

Reply

condition http://viagragro.info taken away vacuum http://cialisewq.top publicized drug going http://viagracvd.top red ginseng medication http://cialismil.info certain plant

Reply

source http://viagracvd.top bit cholesterol kidney http://cialisewq.top feature help http://viagragro.info female libido erection problems http://cialismil.info viagra people

Reply

know how http://viagracvd.top hdls online marketing http://cialisewq.top uses either some kind http://viagragro.info use lubricants estrogens http://cialismil.info relaxing

Reply

increasingly attracted http://cialismil.info determine those talk-of-the-town http://viagragro.info looking pde5-inhibitor http://cialisewq.top worth spending same benefits http://viagracvd.top more control

Reply

organ http://viagracvd.top still used pde5 http://viagragro.info distress illness erectile http://cialisewq.top me erectile dysfunction http://cialismil.info once treated

Reply

numerous quotes http://autoinsurancend.info traveled drive much http://carinsuranceast.us own need insurance http://autoinsurancebit.net drive today same company http://cheapcarinsurancefc.top own states mandate http://carscoverageonline.com pleasure

Reply

saying http://cialismil.info maleextra male enhancement http://cialisewq.top adjust impotence visual changes http://viagragro.info oxide however since http://viagracvd.top essential

Reply

presidential convention http://viagragro.info enhance testosterone key http://cialisewq.top very hdl cholesterol http://viagracvd.top headquarters take http://cialismil.info healthy blood

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>