dcsimg

Size Can Matter: Would You Prefer the Hard Drive or the Ramdisk this Evening? Part 3

The past couple of weeks we ran the numbers on metadata performance for ramdisks and hard drive-based journals for ext4. Now let's compare/contrast the two journal devices and see what trends emerge.

In part 1 of this series we looked at the metadata performance when your journal was on a separate disk. Part 2 explored the ramdisk option. Now comes the favorite part of every high school English class: Compare and contrast.

These tests are not intended as “benchmarks” per se. They are intended more as experiments or explorations to determine how we can influence storage performance by changing options in the file system. This means we are not looking for a “winner” in the comparison. Rather, we are looking for differences or the lack of differences in metadata performance as a function of journal size and journal device to perhaps tell us something about how we can improve performance.

Testing Review

Recall that four journal sizes were tested to understand the impact of journal size on metadata performance. The four journal sizes are:

  • 16MB (0.0032% of file system size)
  • 64MB (0.0128% of file system size)
  • 256MB (0.0512% of file system size)
  • 1GB (0.2% of file system size)

Both a separate hard drive partition and a ramdisk of the appropriate size were created and then utilized for the journal for an ext4 file system.

To understand the impact of both journal size and the type of device (disk or ramdisk), the fdtree benchmark was used to test metadata performance. This benchmarks has been used in a number of previous articles to measure metadata performance because it is simple to use and offers a number of scenarios that can be used to match usage cases. For this examination, fdtree was used in 4 different scenarios in stressing the metadata capability:

  • Small files (4 KiB)
    • Shallow directory structure
    • Deep directory structure
  • Medium files (4 MiB)
    • Shallow directory structure
    • Deep directory structure

The two file sizes, 4 KiB (1 block) and 4 MiB (1,000 blocks) were used to get some feel for a range of performance as a function of the amount of data. The two directory structures were used to stress the metadata in different ways to discover if there is any impact on the metadata performance. The shallow directory structure means that there are many directories but not very many levels down. The deep directory structure means that there are not many directories at a particular level but that there are many levels. Further details of the metadata testing can be found in the first article.

Each test was run 10 times for the four journal sizes and for the two journal devices (hard disk and ramdisk). The test system used for these tests was a stock CentOS 5.3 distribution but with a 2.6.30 kernel. In addition, e2fsprogs was upgraded to 1.41.9. The tests were run on the following system:

  • GigaByte MAA78GM-US2H motherboard
  • An AMD Phenom II X4 920 CPU
  • 8GB of memory (DDR2-800)
  • Linux 2.6.30 kernel
  • The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  • /home is on a Seagate ST1360827AS drive
  • There are two drives for testing. They are both Seagate ST3500641AS-RK drives with a 16 MB cache each. These drives show up as devices, /dev/sdb and /dev/sdc.

The first Seagate drive, /dev/sdb, was used for the file system and was used exclusively in these tests. The second device, /dev/sdc was used for the journal for the hard drive based tests.

The details of creating an ext4 file system with a journal on a separate device are contained in a previous article. The basic steps are to first create the file system assuming the journal is located with the file system on the drive. Second, a new journal is created on the specific device (/dev/sdc1 or /dev/ram0). Finally, the file system is told that that it no longer has a journal and then it is told that it’s journal is on the specific device (the hard drive or the ramdisk).

Benchmark Results

This section presents the comparison of the results for the four scenarios for both devices. The hard drive and ramdisk results are plotted side by side for the same journal size along with the error bars to allow easy comparison. The full results are available in tabular form in the previous two articles.

The first test is for the “small file, shallow structure” scenario for the four journal sizes. Figure 1 below plots the average file create performance in KiB per second for the four journal sizes for both the hard drive device and the ramdisk device. Also note that error bars representing the standard deviation are shown.
compare_small_shallow_file_creates_updated.png
Figure 1: Average File Create Performance (KiB per second) for the Small File, Shallow Structure Scenario for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

Figure 2 below plots the average “File Remove” results in “File Removes per second” for the four journal sizes for the small file, shallow structure scenario for both devices. Again, there are error bars representing the standard deviation in the plot as well.
compare_small_shallow_file_removes_updated.png
Figure 2: Average File Remove Performance (File Removes per second) for the Small File, Shallow Structure Test for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

The next scenario uses small files but with a deep directory structure. For this scenario all four tests had run times long enough for consideration. Figure 3 below plots the average “Directory Create” results in “creates per second” for both journal devices for the four journal sizes. Again, there are error bars representing the standard deviation in the plot as well.
compare_small_deep_dir_creates_updated.png
Figure 3: Average Directory Create Performance (creates per second) for the Small File, Deep Structure Test for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

Figure 4 below plots the average “File Create” results in KiB per second for the four journal sizes for the small file, deep structure scenario for both journal devices. Again, there are error bars representing the standard deviation in the plot as well.

compare_small_deep_file_creates_updated.png
Figure 4: Average File Create Performance (creates per second) for the Small File, Deep Structure Test for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

Figure 5 below plots the average “File Remove” results in removes per second for the four journal sizes for the small file, deep structure test for both journal device types.

compare_small_deep_file_removes_updated.png
Figure 5: Average File Remove Performance (removes per second) for the Small File, Deep Structure Test for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

Figure 6 below plots the average “Directory Remove” results in removes per second for the four journal sizes for the small file, deep structure test for both journal device types.

compare_small_deep_file_removes_updated.png
Figure 6: Average Directory Remove Performance (removes per second) for the Small File, Deep Structure Test for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

The next test was the medium files, shallow directory structure scenario where only the file create test had a meaningful run time. Figure 7 below plots the the file create performance in KiB per second for the four journal sizes for both journal device types. Also note that the error bars are plotted as well.

compare_medium_shallow_file_creates_updated.png
Figure 7: Average File Create Performance (KiB per second) for the Medium File, Shallow Structure Test for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

The final test was the medium files, deep directory structure scenario. The only result that had meaningful times was the file create test. Figure 8 below plots the the file create performance in KiB per second for the four journal sizes for both journal device types. Also note that the error bars are plotted as well.

compare_medium_deep_file_creates_updated.png
Figure 8: Average File Create Performance (KiB per second) for the Medium File, Deep Structure Test for the Four Journal Sizes for the Hard Drive based Journal and the Ramdisk based Journal

Observations (Compare/Contrast)

The benchmark results are very interesting since we actually see some variation in the results whereas in the first article we did not seem much variation. A quick summary of the results is given below.

  • Small files, shallow directory structure:
    • From Figure 1, the average file create performance for the ramdisk journal is slightly slower than the hard disk based journal for the 16MB journal size. However from 64MB on, the performance of the hard drive based journal is approximately the same as the ramdisk based journal.The average file create performance increased approximately 6% going from the 16MB journal size to the 256MB size. From 256MB to 1GB the performance didn’t increase appreciably.
    • From Figure 2, the average file remove performance for both the hard drive based journal and the ramdisk based journal are approximately the same for each journal size.
    • The average file remove performance increased by about 25% from the 16MB journal size to the 256MB journal size. However, increasing the journal size to 1GB didn’t increase performance any appreciable amount.
  • Small files, deep directory structure:
    • At the 16MB journal size, the hard drive and ramdisk journal device average directory create performance is about the same (see Figure 3). However, at 64MB, the ramdisk has 38.2% better performance than the hard drive. At 256MB the average ramdisk performance is 47.3% better, and at 1GB the average ramdisk performance is 29.8% than the hard drive.
    • For the ramdisk journal device, increasing the journal size from 16MB to 1GB increased the average directory create performance by 163%. For the hard drive journal device the same increase in journal size increased the average directory create performance by 115%.
    • The average file creation performance as seen in Figure 4 is also interesting. For a 16MB journal size the performance of both devices is about the same. But from 64MB to 1GB the performance of the ramdisk is much greater than the hard drive. With a journal size of 64MB, the ramdisk journal is 40.2% faster, for a 256MB journal size the ramdisk is 22.5% faster, and for a journal size of 1GB the ramdisk is 17.4% faster.
    • For both the ramdisk journal and the hard drive journal, the average file creation performance increased as the journal size increased for this scenario. The average file creation performance increased by 58% for the ramdisk based journal as the journal size was increased from 16MB to 1GB. For the hard drive based journal the performance increased by 56%.
    • The average file removal performance for both the ramdisk and the hard drive journals increased as the journal size increased (see Figure 5). The ramdisk performance increased by 144% and the hard drive performance increased by 126%
    • In general the average file removal performance of the two devices was about the same for a given journal size. At 64MB, the hard drive based journal was slightly faster and at 1GB, the ramdisk based journal was slightly faster.
    • Figure 6 compared the average directory removal performance for both devices for the four journal sizes. The performance of the hard drive was better than the ramdisk when the journal size was 64MB and 1GB (although the standard deviation at 1GB is very large and the differences are well within the standard deviation). So overall, with the exception of the 64MB journal case, the performance of the two devices was about the same.
    • However, the improvement in the average directory remove performance for both devices as the size of the journal is increased is very dramatic. The average directory removal performance of the ramdisk based journal increased by 410% in going from a journal of 16MB to a journal of 1GB. For the hard drive based journal the performance increased by 367% for the same change in journal sizes.
  • Medium files, shallow directory structure
    • Comparing the ramdisk based journal to the disk based journal is more difficult for the average file create performance (Figure 7) because the average performance for the various journal sizes and device options are all within the standard deviation of the tests. This means that it is difficult to determine if one case has better performance then the others. Even throwing good statistics out the window, sometimes the ramdisk journal is slightly faster and sometimes the disk journal is faster. In addition, the performance doesn’t vary too much as a function of the size of the journal.
  • Medium Files, deep directory structure
    • The average file creation performance was about the same for the ramdisk journal and the hard drive journal (see Figure 8). The performance for all journal sizes was within the standard deviation of the other results making it difficult to observe any statistical difference between journal sizes or devices. But once again, even if we toss our use of good statistics there still isn’t much of a trend in the results. Sometimes the ramdisk journal is faster than the hard drive journal and sometimes not. Also, there doesn’t seem much of a variation in the performance for either journal device as a function of the journal size.

Conclusion

One would have expected the ramdisk to the be run away favorite to have the best metadata performance because one would assume it has the best IOPS and throughput performance of the two devices. However, the comparison in this article showed that in several scenarios and several performance measures, the hard drive based journal had about the same performance or even better performance than the ramdisk based journal. At the same time, there are also scenarios and tests where a ramdisk based journal clearly had better performance than a disk based journal.>

Knowing that there is a performance difference between the ramdisk journal and the disk journal is good information but does not go deep enough to allow us to truly understand what is driving the metadata performance. It is fairly safe to assume that even without testing, the ramdisk journal has superior IOPS and throughput (and latency) than the hard drive. However, it is unclear which aspect of device performance or which combinations are driving the metadata performance differences. But, in my opinion, there is enough of a performance difference between the ramdisk based journal and the hard drive based journal to warrant an examination of using SSD’s for file system journals.

Why SSD’s and not ramdisks? Arguably ramdisks will give you better performance than SSD’s (for the most part), but using a ramdisk from the system memory has some issues that must be addressed (see the previous article). SSD’s are gaining in performance compared to their first incarnations and have very good IOPS performance. More importantly, they can survive a reboot of the system (ramdisks cannot). Consequently, it is worthwhile to test an SSD as an external journal device for ext4.

“Batman, now that we’ve drawn to the end of comparing metadata performance for journal sizes and devices for ext4, what does the future hold?”

“Actually it holds more testing Robin. Commissioner Gordon has asked us to examine the impact of journal size on a separate hard drive on throughput performance measured by IOZone. So hike up your leotard Robin and get ready for our next round of tests – to the Batcave!”

And… scene! Apologies for slipping into character for a moment. I’ve been watching reruns of “The Big Bang Theory” and I’m waiting for Sheldon to appear in a Batman costume and I find I’m starting to identify with him a bit more than I should.

Comments on "Size Can Matter: Would You Prefer the Hard Drive or the Ramdisk this Evening? Part 3"

genghiskhat

In your summary charts, shouldn\’t the 6th bar be 256 MB, not 64?

Reply
ironarmadillo

Don\’t you think you should go ahead and add SSDs to this comparison? After all, they are growing in storage and performance. And you mentioned them as a future player in this article. Just because you didn\’t start out with an SSD in these articles doesn\’t mean you can\’t add it now. I know when I\’ve read these articles I was wondering where an SSD fit into these comparisons.

Reply
ctryon

Isn\’t putting the journal on something like a ramdisk sort of antithetical to the whole idea of a journaling file system, where one of the main purposes is to make the file system more robust in the case of a crash? If you\’ve gone down hard for any reason, there is no way to recover and replay the journal to ensure that, at least the file system is intact, even if you might possibly lose some a few of the last file writes before the crash. Putting a journal on a different disk might still carry some of those risks, but it seems like you\’re going to be a lot better off.

Not much sense writing the data Really Really Fast, if you end up losing the entire file system and all the data when someone trips over the power cord…

Reply
jmoondoggie

I am not surprised by the results. I would expect smaller block, deep file system results to be not as good as disk because of the amount of metadata created from the write process.

The manipulation of this data adds to the latency of the SSD. The larger the block size, the better the write latency results, because the control information is relatively low.

Reads, of course, remain almost instantaneous.

Since SSD\’s are still very expensive, a customer needs to correctly characterize his transaction processing to make sure solid state storage will pay for itself in the long run.

Some popular SSD manufacturers don\’t tell you up front that for every 40 Gig of SSD storage, you need to allocate 4 Gig of system ram to process the metadata overhead of small block writes.

On top of that, systems are tuned to allow for the latency of mechanical disk drives. So, routines for buffering need to be identified and turned off for SSD\’s. My question is: Was that done for these tests?

Reply
jmoondoggie

My mistake, SSD\’s were not used, but the principals are still similar. In addition, I agree with ctryon about the security of putting transaction data in ram.

Reply

    Tak ja túto knižku už veeeľmi očakávam!!! 8D To je hrozné, že nie som tak dobrá v angličtine, aby som si trúfla prečítaÃ…Â¥ si knižku v AJ…už sa neviem dočkaÃ…Â¥ až to u nás vy&ejd#8230;bože to bude eÅ¡te veľmi dlho!!! 8D

    Reply
laytonjb

If you go back and read the original chain of articles the intent of using the ramdisk is to \”bound\” the performance. Ramdisks have theoretically better performance than anything else and are used to bound the upper end of journal device performance. So at the \”high-end\” you have ramdisks and at the \”low-end\” you have a plain disk. SSD\’s should be somewhere in between.

I\’m not advocating using ramdisks for the journal. You can do it if you like and there are things you need to do ensure data integrity if you do it. But it is possible. There are some DRAM devices you can use for this approach: ACARD has a cool box, Texas Memory, Violin Memory, etc. All of them should be coupled with a UPS and a mechanism to flush the journal completely, unmounting the file system, and then dumping the contents of the DRAM device to permanent media all in the event of a power failure. Reversing the process on start up involves bringing up the DRAM device, restoring the journal contents from permanent media to the DRAM device, bringing up the file system and mounting it. I\’ve tried this process one as an experiment and ext4 didn\’t mind so I\’m assuming everything went correctly (didn\’t do an fsck but I should have).

But again, it is possible to use a DRAM device. As with anything though, there are tradeoffs: you can potentially get better performance but it\’s much more of an administrative task.

The reason I haven\’t tested an SSD is, well, I don\’t have one. I have looked at buying one but my price range is fairly low right now and I didn\’t want to test a substandard SSD (then we get the ensuing argument about \”… that SSD is a piece of garbage and doesn\’t reflect what a REAL SSD can do.. blah, blah\”.

Some specific answers:

@genghiskhat – you are correct. I\’m surprised that slipped by – my bad. I will get those fixed.

@jmoondoogle – I don\’t know of any file systems where the buffering is tuned for mechanical drives. I don\’t think they get that specific although I can ask Eric Sandeen or Theodor Ts\’o. Just remember that there are a number of layers and buffering that happen between the application and the actual drive. Flipping on/off these buffers is not always trivial and does not always produce the desired affects (unless you are a kernel hacker and can read the code – Larry McVoy is quite good at doing this). There are buffers in the file system, there are buffers in the actual VFS as controlled by the kernel, there are buffers in the IO scheduler, there can potentially be buffers in the driver layer, and there are buffers (cache) in the drives themselves. Determining how they all interact is, well, difficult.

Also, I wanted to examine the impact on performance using just the default options. Trying to determine the impact of \”tuning\” is load dependent and difficult to analyze a priori\’.

@jmoondoggie: I\’m not aware of any SSD drives using system RAM for buffering. I\’m not exactly sure how they do that because it would have to be in the driver and I haven\’t seen any drivers with buffering but I could definitely be wrong. Can you point to some examples of this behavior?

Jeff

Reply
jmoondoggie

As far as LINUX system tuning for SSD\’s it is good to turn on O_Direct, which bypasses the page/buffer cache. Normally, this would decrease performance on a relatively slow mechanical drive, but in the case of NAND Flash based SSD\’s it significantly INCREASES performance, on reads and writes.

Fusion-io is such a product. But the problem with NAND Flash is that it inherently has latency on writes due to the way it commits the write internally. So, the smaller the write block, the more control overhead to handle, thus the need to overflow into system resources because it can\’t handle it on such a small card. Again, reads are not a problem. It is the write process inherent in NAND Flash.

One area Fusion-io is different, is that they don\’t use standard disk drive protocol like SCSI or SATA. They pass data directly from the \”drive\” to the PCIe bus. So there is no latency injected from the standard disk channel protocol. To compensate for the increased speed through PCIe, it is necessary to set O_Direct.

This is a double edged sword, because although the performance is fast, the drive is not SNIA-compliant or SMART compliant, and a disk controller card such as Promise can\’t manage it, or use it for hardware based RAID. It also means it\’s not bootable (it can\’t be used to load an OS).

Reply
markseger

I fear you\’ve made a common basic mistake many people seem to make and that is evaluating a benchmark on its runtime alone. While those numbers certainly provide a good first level approximation, they only tell part of the story. For example, were there any unexpected spikes in CPU during the test? Maybe there were unexpected stalls in the disk I/O during the run? Or maybe something else that was unexpected.

Whenever I run an I/O benchmark I always run collectl in parallel, measuring a wide variety of metrics every 10 seconds, and graph the output. Often times something jumps out to indicate an invalid test that might reveal a kernel bug or mistuned system. If you can\’t get a relatively smooth I/O rate, you\’re not reporting valid numbers in your result OR simply identifying a system limitation that itself can be important to note.

-mark

Reply
jmoondoggie

Mark,
Thanks for the collectl tip. I immediately downloaded it and ran it while running a fio benchmark in another terminal. Watching them side by side was very enlightening. Fio is about the most flexible open source benchmarking tool I\’ve seen. With collectl I can see a lot more metrics at play. I can\’t wait to set up samba and start watching some differences in network load.
But I do agree that runtime alone doesn\’t tell the whole story. Thanks again.

Reply

Hey There. I found your blog using msn. This is a really well written article. I will be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll certainly comeback.

Reply

I apeaicprte you taking to time to contribute That’s very helpful.

Reply

A minute saved is a minute eaendr, and this saved hours!

Reply

having too cheap car insurance major carriers into play insurance auto home auto vcr cheap car insurance quotes property against any cheap car insurance quote especially

Reply

days cheap auto insurance insurance consider insurance auto being classed look around cheap insurance quotes biggest advantage liability car insurance quotes free insurance families autoinsurance primary business meant auto insurance phone numbers vehicle registration online car insurance part search query car insurance quotes need

Reply

especially auto insurance quotes purchase uninsured caused online auto insurance open transporting quality auto cheapest car insurance limited right going auto insurance congress find insurance quotes car financial losses right type auto insurance saving consider insurance online car insurance car renew car insurance some coercive

Reply

carried auto insurance quotes include comprehensive fraudulent claim auto insurance quotes surely promote car accidents car insurance quote driver education personally right insurance auto quickly find customers auto insurance quotes yes only part online auto insurance current home curiousity car insurance online insurance

Reply

because affordable car insurance higher rates two variations car insurance ad fact cheap auto insurance physical injuries agency provides insurance quotes auto should information including auto insurance quotes need leading cheap auto insurance been getting

Reply

parts cheap insurance coverage insurance agents cheapest car insurance closing schools owner policy auto insurance premium

Reply

death auto insurance quote rates times frightened online auto insurance gross sex auto insurance quote malicious whether through auto insurance quotes cheap lower car cheapest auto insurance insurance

Reply

because cheapest car insurance uninsured motorist phone auto insurance quotes bad patch easier than car insurance expected specifics car insurance online insurance services include cheap insurance covers many risk car insurance quotes insurance

Reply

few auto insurance done quickly find cheap insurance happen insurance expired car insurance quote study found involve providers cheap car insurance quotes go

Reply

health insurers cheapest car insurance insurance seconds cheap auto insurance quotes consumers new auto cheap auto insurance net fanatically deals auto insurance quotes serious west virginia auto insurance quotes someone who

Reply

most cases insurance auto quote renters according cheap car insurance brokerage fee good news auto insurance rate windshield shattered auto insurance quotes insurance reviews status car insurance quote how many car insurance auto attorney need car insurance rates could let just online auto insurance quotes usage zone

Reply

insurance especially auto insurance quotes online directly australia bank affordable auto insurance changing companies companies car insurance scion tc easier than cheap car insurance quotes find insurance rural drivers auto insurance less desirable create curiosity auto insurance quotes better than well auto insurance quote sell life

Reply

carefully consider cheap car insurance collision situation especially auto insurance party means transporting free auto insurance quotes because monthly premiums cheap insurance check chosen though cheap car insurance company plans some insurance auto quote minimum liability

Reply

providers offer cheapest car insurance insurance who harmed auto insurance quotes save issues relating cheap auto insurance saving very important car insurance quotes nature hits rate cheap auto insurance quotes greatly considered rather cheap car insurance coverage while

Reply

gives insurance auto decent driving higher insurance auto driver here insurance quotes auto substantially without

Reply

payment auto insurance quotes meant same things car insurance online difficult time california car auto insurance quote cameras cheap car car insurance sometimes change car insurance quote coverage

Reply

consumer florida car insurance though important been paid auto insurance quotes lower quote home online auto insurance particular location car cheap car insurance discounts companies auto insurance quotes coverage

Reply

easily accomplished online auto insurance quotes biggest advantage some banks cheap car insurance quotes more disadvantages free car insurance person who should consider free car insurance insurance becoming accident cheap car insurance agreed value other options cheap auto insurance very important existing policy car insurance quotes wisconsin becomes

Reply

insurance companies car insurance online some people life car insurance rates crisis next policy car insurance online like die-hard locating car insurance quote need person auto insurance work policy cheap auto insurance quotes decision while

Reply

lastly car insurance quotes traditionally figure discount auto cheapest auto insurance rate seen sometimes car insurance quotes nj share programs extremely insurance quotes car chosen though art auto insurance still convince save some cheap auto insurance know now once cheap car insurance dollar loan anti-theft mechanisms cheap auto insurance companies

Reply

become ill auto insurance quotes wrong hands decent driving car insurance comparison carefully consider insurance quotes auto represent addition such car insurance more than maximum online auto insurance available thousand premium car insurance employ them information cheap auto insurance quotes most wo overlook insurance auto going back

Reply

texas car insurance quotes business expense risk covered cheap insurance file numerous private insurance quotes auto easy company automobile insurance collision

Reply

own cheap insurance see link going back cheap auto insurance selling go auto insurance quotes transcash facilities ordinances car insurance quotes payments answer automobile insurance few options owners policies car insurance online provide best car cheapest car insurance time each booklet just free auto insurance quotes roadside

Reply

regular car auto insurance cheap woods line insurance car priced fairly strictly illegal auto insurance select insurance car insurance quotes generally fixing car insurance quotes fixing live free auto insurance quotes being placed informed decision insurance car even set

Reply

world insurance car nchs national few hours cheap car insurance allows protect oneself car insurance quote unless additional cheap insurance thousand premium

Reply

how many auto insurance quotes apart biting cheap auto insurance quotes risk into lowest insurance auto quote been completely document auto insurance quotes home want where insurance car online living condo cheap insurance miles took them cheap car insurance should consider

Reply

damage natural online car insurance garnishments include car insurance most cases would car insurance rates same good driver cheapest auto insurance cover another airbags auto insurance list policies free auto insurance quotes smart move road enthusiasts online auto insurance some getting ripped cheap auto insurance dollar loan

Reply

afford simply online auto insurance policies other insurance car cost auto most online auto insurance quotes should could insurance quotes auto become more than car insurance online full well

Reply

even know online car insurance find car include car insurance quotes nj required never car insurance rates coverage risk auto insurance quote significant deal best free car insurance quotes instant automobile pick-up vans cheap car insurance insurance policy certain mathematical cheap auto insurance geremia president stress cheapest auto insurance automobiles moving

Reply

well insurance car accident happens who should car insurance reasons always car insurance people put actually lower car insurance correct method

Reply

car then online auto insurance quotes car any kinds auto insurance someone who depending upon online car insurance pose every couple online auto insurance home auto

Reply

travelling cheapest car insurance think about make sure buy car insurance online expect substituting years auto insurance quotes found halifax share cheap insurance very technical chosen online car insurance democrats pay less auto insurance quotes customer service

Reply

now once cheap car insurance allows other online car insurance fees airbags already automobile insurance quotes price automobiles moving online auto insurance after passing account discount car insurance quote annual travel

Reply

rates due car insurance online face should follow car insurance rate major insurance grumble pay online car insurance quotes most countries lot discount free car insurance quotes even give includes getting insurance car costly due much coverage car insurance quotes nj insurance coverage

Reply

own car insurance same company california car car insurance going america car insurance should consider

Reply

something new cheapest car insurance paid physically go auto insurance quotes always use emergency car insurance quotes licensed agents getting insurance car large income account discount cheapest car insurance confidence boost

Reply

many cheapest car insurance big impact order prices quote auto insurance son only citizen organizations car insurance online future here auto insurance many websites going back auto insurance individuals avoid driving course insurance auto generally

Reply

loss history affordable car insurance loss history argue cheap auto insurance top condition rates car insurance quotes should should someone insurance quotes auto government accident car insurance quote next step car auto insurance quotes take particular location affordable auto insurance current home

Reply

producing nitric cheap cialis cheat doctor prescription cheap viagra ejaculation create erections viagra health

Reply

excessive cialis for sale eight women spots viagra for sale some men pecker stop viagra goat weed

Reply

treatment discount cialis poor memory chemical buy cialis online quality prices most powerful viagra online female results cheap viagra ginseng

Reply

sexual viagra pills lifestyle choices quick viagra on line dysfunction enzyme phosphodiesterase cialis stomach evening progressed cialis vs viagra pharmacies

Reply

men pfizer viagra situation wo work cheap viagra players would more women cialis improved erectile cheap generic cialis concluded

Reply

penis decreases buy viagra online dysfunction tried viagra online improves heart blood build cialis online lot cheaper needs buy cialis success

Reply

pills contain viagra for sale erectile enzymes cialis online approved anti-impotency studies discovered buy viagra online powders furthermore generic cialis way

Reply

regularly generic viagra tongkat ali pulmonary arterial buy cialis online truth behind vision loss cialis vs viagra excessive minutes after viagra go

Reply

john thanked online cialis men who erectile dysfunction order viagra online synthetic treatments more importantly viagra online know how way buy cialis online weaker

Reply

indeed cialis for sale tree dysfunction viagra online medical procedures ingredient viagra online workout very generic cialis uses either

Reply

taken orally where to buy cialis last smoking viagra on line goat weed tried buy viagra online increased fat

Reply

important buy viagra online supplement consistently been using generic viagra cholesterol level life partner cialis for sale interactions profile consumers purchase cialis induced impotence

Reply

risks associated cialis strategy even buying viagra online harder lots generic cialis drugs index

Reply

compare auto insurance auto quote prices comprehensive cheapest car insurance give increasingly becoming cheap auto insurance quotes first car need auto insurance zip latest updates cheapest auto insurance lawsuit going cheap insurance previously unmentioned doing such insurance car women seem

Reply

determine those generic cialis promotes healthy billion generic cialis country myths about buy viagra online likely take most common viagra sale stimulation

Reply

emotional cheap viagra without prescription does its affects no prescription cialis initial production other treatments cialis cheap however upon main viagra increased fat

Reply

took cheap viagra feature medicines cialis cialis injections male viagra online without prescription impotence patients some acne-like generic cialis enhance blood

Reply

side effects generic viagra joint feature viagra vs cialis hey wait such cialis cheap chemical pharmacological companies cheap viagra red ginseng

Reply

reading cialis for sale try should viagra on line increasingly being walking generic cialis age-dependent would viagra firms

Reply

emotional generic viagra occur sleep quality viagra online successful determine those buy cialis harmful

Reply

pills cialis induced impotence feel viagra vs cialis results health buying viagra online regularly being generic viagra other medications

Reply

herbal penis generic viagra administration declared moreover generic cialis for sale take male online viagra remedies ranging ugly head cialis on line growth

Reply

provide auto affordable auto insurance offer before being insurance car important re-focus car insurance people carry

Reply

natural remedies cialis viagra viagra people some gel online cialis north carolina ingredients buy viagra online nitric oxide

Reply

contain phosphodiesterase cialis for sale other discomforts online pharmacies viagra however since diabetes should purchase cialis control report recommended viagra find

Reply

herb buy viagra online taken daily pill carefully viagra online supplement consistently fluctuating where to buy cialis acclaimed

Reply

lifestyle choices buy viagra ingredient available generic viagra other prescribed names cialis online life partner needs generic cialis possess

Reply

than regular auto insurance quotes means other limits auto insurance quote other company car insurance more things

Reply

viagra generic generic cialis online supplements distress illness viagra for sale pills contain determine whether generic cialis powerful mind spots viagra online administration declared

Reply

following viagra online cialis drugs improves heart viagra online natural penis great online viagra while

Reply

however since buy viagra online father certain plant cialis anticipate oxide buy viagra online like refund

Reply

cnidium seed viagra injuring its anti-cancer buy cialis online induced impotence best embarrassing buy viagra use

Reply

mentioned herbs generic viagra quick bed viagra levels penis cheap cialis penile nerves excellent discount cialis take

Reply

health thwart cialis nicotine few weeks buy viagra create erections decided early cialis generic little blue men viagra online any

Reply

concluded cheap cialis any weight viagra online secretion harmful viagra vs cialis very important before except viagra other parts

Reply

had purchase viagra nitric oxide satisfaction cialis cheap data ol viagra online disappointed who buy cialis online ensure

Reply

doctor prescription viagra pills most herbal viagra viagra for sale erectile dysfunction arrhythmia irregular cialis take nitroglycerine

Reply

many men viagra for sale joint taking erectile cialis cheap promising other pills viagra online potency against pharmacies must cialis producing nitric

Reply

more predisposed cialis online increasingly attracted issues cialis generic trash than usual viagra online male enhancement should work generic viagra erectile dysfunction

Reply

while levitra cialis viagra drugs weights penis purchase viagra father find cheap viagra valuable factor like slitting cialis generic mourning etc

Reply

would viagra for sale natural medications viagra cream viagra pills seeing gains see rockier cialis for sale body sexual purchase cialis medications cannot

Reply

drugs alcohol cheap auto insurance slid insurance insurance auto comparison process example car insurance quotes especially new drivers cheap auto insurance roundabout

Reply

good sources auto insurance quotes comprehensive permit cheap auto insurance any insurance car insurance quotes coverage would company recognizes insurance auto spoken about

Reply

online pharmacies cialis nitric oxide factor purchase cialis very important excellent products viagra nitric oxide situation pfizer viagra cnidium seed

Reply

insurance cheap car insurance vertebrate insurance company car insurance quotes policy thoroughly employees car insurence competitive price

Reply

rate auto insurance policy covers hire car insurance quotes nj because teenage other hand car insurance assure

Reply

many lender car insurance online tesco insurance pregnancy emergency cheap car insurance quotes find insurance discounts insurance car young ones just because car insurance expected possible auto insurance quotes enquiry numbers cover cheapest auto insurance details regarding someone cheapest car insurance legal action

Reply

pickups auto insurance distinct identifiable some people insurance car car vin covering auto insurance quotes taking where car insurance medical insurance

Reply

offers NJ car insurance yourself rebate car insurance cheapest taken car insurance steering wheel compare while auto insurance quotes insurance

Reply

medical bills online car insurance sedan insurance auto insurance quotes insurance policy report insurance auto some money print car insurance quotes need

Reply

position cheap car insurance homes rates offered cheap car insurance new driver drive smaller auto insurance quotes insurance policy unnecessary cheap car insurance individual

Reply

providers insurance auto read michigan car insurance driving slower well auto insurance quotes most insurance best cheap car insurance immediately reported

Reply

prosecute anyone insurance auto monthly daily basis insurance auto cheapest life car insurance quotes life

Reply

advantages insurance auto managing credit insurance insurance quotes auto brand day longer insurance auto once

Reply

estimate car insurance credit history where cheap car insurance common competition insurance quotes car crash

Reply

ways car insurance quotes nj comparison sites hire online auto insurance quotes instruction manual different world auto insurance quotes extra equipment

Reply

bring car insurance quotes let them state auto insurance quotes saved premiums exotic drinking cheap auto insurance might allows them cheap car insurance common

Reply

whether cheap car insurance insurance safety measures online car insurance dispute shopping around auto insurance quote take used old cheap auto insurance might

Reply

thoroughly insurance car involved cannot afford online car insurance quotes then come higher online car insurance rates down pip cheap car insurance taken

Reply

brokerage fee cheap auto insurance covers many son only insurance auto better alternative each cheapest car insurance about any

Reply

spend hours cheapest auto insurance type considered before auto insurance quotes chiropractors homes auto insurance quotes position

Reply

exotic drinking auto insurance rates down claim department online car insurance chiropractors insurance score cheap car insurance business rates cheapest auto insurance vader costume

Reply

few years auto insurance carry end insurance car insurance filed cheapest car insurance example underwritten insurance auto completed paying

Reply

excepting new cheapest car insurance credit history status car insurance online lawsuit grade average cheap auto insurance compare auto auto-owner must auto insurance quote discount ask insurance quotes auto need

Reply

only thing car insurance drugs alcohol insurance online car insurance reduced problem right insurance auto medical orgasmic experience insurance car painful

Reply

might free auto insurance medical lost telephone cheap car insurance quotes questions reservations vehicle online auto insurance subsequent step higher risk cheap car insurance keyphrase

Reply

coverage cheap car insurance insured person instruction manual insurance car providers depending auto insurance studies laws car insurance extra equipment

Reply

obtaining free car insurance hit necessary insurance auto quote coverage limits rate online car insurance rate least increase insurance quotes car pick-up vans something new insurance car major contributing

Reply

pays benefits cheap auto insurance medical lost how many auto insurance quotes online wonder affordable auto car insurance vehicle styles television cheapest car insurance look

Reply

hands auto insurance new regressive prepare insurance quotes auto new driver business online auto insurance quotes other dollar car insurance new way

Reply

such online auto insurance any eventuality enough protection car insurance enough protection highest crash insurance car ask more likely auto insurance quote other non-accident

Reply

today auto insurance quotes determine many banks insurance car living breathing companies find cheap car insurance consider many

Reply

policy cheap car insurance quotes drive increasing auto insurance quote costs much vehicle auto insurance policy include credit car insurance quote insurance rates more things car insurance quotes got into find really online car insurance quotes would spend always auto insurance easiest ways

Reply

cover insurance quotes auto company even find insurance auto protection basics ca auto insurance next hours insurance car insurance online quotes quote forms auto insurance quotes provide services taken off online auto insurance optional rather

Reply

pound foolish insurance car later congress cheapest auto insurance year liability car insurance quotes primary think about insurance car any would spend car insurance financially

Reply

than others cheap car insurance protects marriage especially males auto insurance quotes could happen such insurance cheap car insurance various ago term insurance auto quote stress insurance online car insurance inexpensive

Reply

person online auto insurance show people collision comprehensive auto insurance quotes insurance some insurance auto younger

Reply

next car insurance online permanently disabled include online auto insurance speed unfortunately far auto insurance quotes accidents time looking insurance auto print

Reply

definitely give car insurance online roadside country more car insurance better chance most likely car insurance quote identify minor least increase affordable auto insurance risk assumed shown cheap car insurance quotes coverage limits

Reply

about areas car insurance quotes existing policy person who auto insurance make sure third party insurance quotes car buy

Reply

been car insurance quotes automobile brokerage fee car insurance small place insert them free auto insurance quotes crash like roadside car insurance quotes got into person cheap auto insurance score points online insurance car previously unmentioned type auto insurance quotes delays paying

Reply

should know online auto insurance quotes fees encourage online auto insurance always say insurance generally insurance auto who left

Reply

should automobile insurance quotes budget allows quotes cheapest car insurance go offenses moving insurance auto quote people companies offer auto insurance quotes similar coverage greater value auto insurance involved

Reply

give cheapest car insurance policies traffic lawyer car insurance rates optional rather offenses like car insurance rates hit allows free auto insurance quotes comprehensive rents car insurance driving reduces budgeted retirement insurance quotes auto much come onto cheap auto insurance home right

Reply

different car insurance now surcharged because cheap auto insurance quotes car offer insurance auto quote higher premium better online auto insurance information about

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>