In the last article we reviewed the most basic RAID levels. In this article, we take things to the next level and discuss what is called "Nested RAID Levels". These concepts can provide both performance and redundancy for a data-rich world. We then look at RAID-01 and RAID-10; two of the most common Nested RAID configurations.
In our last article, I introduced the standard RAID levels: RAID-0, RAID-1, RAID-2, RAID-3, RAID-4, RAID-5, and RAID-6. While it may have been a review for many, I think it’s always good to go back and brush up on fundamental concepts from time to time. These were “single” RAID levels that have a single number associated with them such as RAID-0, RAID-1, RAID-5, RAID-6, etc. Remember that the intent of RAID is to provide extra performance and/or extra data redundancy so these single RAID levels are defined based these principles.
We found that some of the RAID levels emphasize performance, and some emphasize data redundancy. There are also some that try to provide a balance of the two goals but perhaps emphasized data redundancy more than performance. After a while people recognized that these single RAID level configurations might be limiting what we could achieve with RAID. Perhaps we could use multiple RAID configurations to achieve a better balance of performance and data redundancy.
In this article I want to talk about a further development of RAID that combines various RAID levels to create a “Nested RAID”. Nested RAID is sometimes also called Hybrid RAID with the concept of combining the standard RAID levels to perhaps create new and better RAID configurations.
Fundamentals of Nested RAID
The goals in using Nested RAID are the same as for standard RAID levels – better performance and/or better data redundancy. But typically, Nested RAID configurations look for a balance of performance and data redundancy. They are achieved by combining two or more standard RAID levels to create a “hybrid” RAID configuration. Originally, Nested RAID levels are created to either improve the performance of a standard RAID level that emphasizes redundancy, such as RAID-1, or to improve the redundancy of a standard RAID level that emphasizes performance such as RAID-0.
If you want to get a little geeky about it, you can think of Nested RAID levels like cooking. You sprinkle a little RAID-1 here, a little RAID-5 there, and then add RAID-0 at the end for performance and you have your solution. Now that I’ve finally used a cooking metaphor in a Linux storage column let’s move on to more technical details.
Nested RAID levels are usually labeled with a series of numbers rather than a single number as in the standard RAID levels. The most common have two levels or two numbers. Generically they can be written as RAID X+Y or RAID XY (assuming only two levels). For example, we can write RAID-10 where X=RAID-1 and Y=RAID-0. In the numbering scheme, the first number on the left, which is “X” in the generic scheme, is the lowest level in the nest. So in the case of RAID-10, the Nested RAID starts with RAID-1 at the lowest level.
Then the numbering scheme moves to the right. Using the generic RAID-XY sequence, the next RAID level is “Y”, to add the higher RAID levels. In the example of RAID-10, the lowest level is RAID-1, and then RAID-0 is the highest level. So RAID-0 is layered on top of RAID-1 (performance on top of redundancy).
The order of the RAID levels in the naming sequence is significant. RAID X+Y is not the same as RAID Y+X (sorry – no symmetry). RAID Y+X starts with RAID Y at the lowest level and ends with RAID X on top of it. In the case of RAID 10 (RAID X+Y), X is RAID-1 and is the lowest level, and Y is RAID-0 and is the highest level. For RAID Y+X, the opposite is true. RAID-0 is “Y” and is the lowest level and RAID-1 is “X” and is the highest level.
One might think that RAID-10 (RAID 1+0) and RAID-01 (RAID 0+1) are really the same thing and to some degree they are. They actually have fairly similar performance, capacity, and storage efficiency, but the big difference is in how many disks have to be read from, and written to, during regeneration when a Nested RAID fails. This article will talk more about this in subsequent sections.
One last comment as well. You don’t have to stay with just two levels such as RAID-XY. You can create as many levels as the hardware, or software, will allow. So you can easily create RAID-XYZ if you like (and if the hardware or software will allow it).
With that said, let’s move on to some common Nested RAID Combinations, starting with RAID-01 or RAID 0+1.
RAID 0+1
In this Nested RAID configuration, you start my creating two identical groups of disks and applying RAID-0 across each group. Then you use RAID-1 to combine the two groups. Figure 1 below shows how this is laid out.
Figure 1: RAID-01 layout with Four Drives
In this example, four disks are used. Two groups of two disks each are created and each group is built with RAID-0 (for performance). Then the two groups are combined using RAID-1 (mirroring) for data redundancy.
The data flow in Figure 1 is fairly easy to follow. The data chunk “A” is sent to the RAID group at the highest level (RAID-1). The blocks making up the chunk “A” are mirrored by sending them to the two groups (RAID-1). Then the data is striped across the two disks in each group per the RAID-0. So in Figure 1 you can see where blocks A1, A2, A3, and so on reside on the disks. Be sure to notice how the block mirroring (RAID-1) works so that there are two copies of each block in the RAID configuration (i.e. disks 1 and 3 contain the same blocks).
Since it’s difficult to go into more depth with just four drives, let’s try an example with six drives as shown in Figure 2 below.
Figure 2: RAID-01 layout with Six Drives
Comments on "Intro to Nested-RAID: RAID-01 and RAID-10"
I think you’re missing “n” in some of your total capacity calculations. e.g.
Capacity = 3/5 * capacity of single disk
should read:
Capacity = 3n/5 * capacity of single disk
no?
I think there’s something wrong. In the example of a Linux raid configuration with m=2 and f=2, the diagram shows 4 copies of each piece of data. If this is true, then the capacity would be “(n/4) * capacity of a single disk.”
In this map
Disk 1 Disk 2 Disk 3 Disk 4
—— —— —— ——
A1 A1 A2 A2
A3 A3 A4 A4
A5 A5 A6 A6
A7 A7 A8 A8
.. .. .. ..
A2 A2 A1 A1
A4 A4 A3 A3
A6 A6 A5 A5
A8 A8 A7 A7
.. .. .. ..
We have
Capacity = (num of disks/num of chunks) * capacity of single disk
Capacity = (4/4) * capacity of single disk = capacity of single disk and of course we can lost not only 2 but all 3 disk and not lose any info
May be far-map must be
Disk 1 Disk 2 Disk 3 Disk 4
—— —— —— ——
A1 A2 A3 A4
A5 A6 A7 A8
.. .. .. ..
.. A1 A2 A3
A4 A5 A6 A7
A8 .. .. ..
Then lost any folowing 2 drive – lost some info
Ex: Disk2+Disk3 -> lost A2 and A6
Disk1+Disk3 -> all data OK
PS Last map is 4/5 of 1 disk cap
Regards
Donserg
RAID 0+1 is a sign of incompetence. Look at the data layout for 0+1 and 1+0: it is identical. The only real difference between the two algorithms is that when a drive fails in a RAID 0+1, the RAID engine automatically shoots all the other drives on the same side of the mirror. That’s just plain stupid.
How it _should_ work is: if a user tries to create a RAID 0+1 volume, the RAID engine should make a RAID 1+0 volume instead.
Wasn’t RAID5 supposed to do what RAID10 does? That is, improve both speed and reliability. Hope you compare those in a future article.
RAID 5 will do 4 IO for a write.
RAID 10 will do 2 IO for a write.
RAID 10 cost more in disk space since, space efficiency is 1/n
RAID 5 is cheaper in disk space, space efficiency is 1-1/n
Performance wise, the RAID 10 is the better choice.
But very often money is the first criteria of the management …
Wikipedia has an excellent explanation on subtility between RAID types.
See http://en.wikipedia.org/wiki/RAID
The capacity and redundancy calculations seem to be very mixed-up here.
For simple RAID-10, the article is correct. The capacity of RAID-01 (which no one would ever use) and RAID-10 are both n/2, and can tolerate the loss of any one disk. If the array is made from more than 1 set of RAID-1 pairs, you can loose 1 disk from each pair – but not two disks from the same pair.
But for mdadm RAID10, the article is mostly wrong. It starts off by missing out the “offset” option as an alternative to “near” and “far” – though much less used than “near” or “far”, it is possibly slightly faster for some loads.
The efficiency of single-mirror RAID10 is always 50%, or n/2 – regardless of the number of disks. Think about it – each block of data is written to two disks. So if you have three disks, the capacity is 3/2 – not 2/3 (or 2n/3). Regardless of the number of disks, the RAID10 will tolerate the loss of any one disk. If you have enough disks (4 or more), it /may/ tolerate the loss of other disks, depending on the spread of the data.
With four-way mirroring (near-2 and far-2), the capacity is n/4 (not n/2), and it will tolerate the loss of any /3/ disks (assuming you have at least four disks). This is the point of having multiple-copy mirroring – for each mirror, it costs you space for a duplication of the data, but you get extra redundancy.
It is also possible to use three-way mirroring (such as far-3), for n/3 capacity and two disk redundancy.
The whole last section of the article (from “Linux is special”) needs a re-write, using correct information about sizes, redundancy, and speeds. The maths is so simple for RAID-10, regardless of the number of disks and the types of mirrors, that it is hard to understand how the author got this so muddled. It is good practice to read the relevant wikipedia article before writing your own article!
Your blog is pretty cool to me and your subject matter is very relevant. I was browsing around and came across something you might find interesting. I was guilty of 3 of them with my sites. “99% of blog owners are guilty of these five BIG errors”. http://bit.ly/uEMWS2 You will be suprised how easy they are to fix.
Ongoing RAID10 nightmare.
http://www.dslreports.com/front/shutdown.html
https://docs.google.com/document/d/1kll86bDn_MgWoo6Ja7oHo_yvI0SCqggEvNWwPWIcrHY/edit
Nested raid-0 by partitioning SSD multiple times will increase the number of i/o’s, allowing dispatch from multiple cpu’s and controllers – resulting in a higher queue depth – since benchmarks show higher QD results in best performance the idea would be like an engine to reach peak power (QD) and stay there during the entire process. One might say you don’t have any way to sustain that always? but there is the idea of read ahead which can be as simple as read next block if last 2 blocks are sequential as long as the queue depth is decreasing?
8 drives, 2 controllers, 2 sockets, 4 cores each socket – (or 8 controllers?) would allow 8 cpu execution threads on windows ? assuming you stripe the two ssd raids physical raid-0 into 4 logical drives per physical raid then strip them using windows software raid all back into one?
like a short stroke without the stroke – if an ssd can handle multiple reads at once faster than single – then creating logical volumes of smaller blocks would create artificial higher queue depth than a hard drive which mostly cannot read from two places at once (untrue there is a drive with more than one actuator). nested raid could use variable strip size and a smart i/o drive to create logical volumes of 4,16,64,128 and attempt to send “like” requests and use a simple QOS algorithm like a router.