Nested-RAID: RAID-5 and RAID-6 Based Configurations

A Nested RAID configuration is built on top of a standard single-level RAID configuration in order to address performance and data redundancy limitations. Digging deeper into Nested RAID, we check out RAID-5 and RAID-6, which have some truly amazing features (if you have enough drives).

The minimum number of drives in a RAID-51 is six – the same as RAID-15. But given that RAID-1 typically consists of just two groups, your only flexibility in RAID-51 is the number of drives in the RAID-5 group. Regardless, RAID-51 must also use an even number of drives (much like RAID-15).

The fault tolerance of RAID-51 is three drives. If you lose a drive in a RAID-5 group, that group operates in degraded mode (i.e. one more drive and the group loses data). RAID-1 at the highest level doesn’t know that it lost a drive on one side of the mirror so it continues to use both RAID-5 groups. You can lose a second drive on either side of the RAID-1 without losing data access at this point. If that second lost drive happens on the degraded RAID-5 side of the mirror, then that side is down and RAID-1 operates in degraded mode using the opposite side of the mirror. You could also lose a drive on the unaffected side of the mirror resulting in two RAID-5 groups operating in degraded mode.

After the loss of two drives, losing a third drive would render one side of the RAID-1 as down and the other side would be operating in degraded mode. Losing a fourth drive could result in data loss.

As an example, let’s assume we lose disk 4 (fourth drive from the left). At this point, the RAID-5 group on the left is operating in degraded mode but the right hand side is operating normally. Now let’s assume we lose a second drive, drive 5. Now, both sides of the RAID-1 are operating in degraded mode, but as far as RAID-1 is concerned, both are operating correctly. At this point, let’s assume we lose disk 1. The left hand side of the RAID-1 is now down and RAID-1 uses the right hand side for data access. At this point, losing a fourth drive in the right hand side of the mirror means that the you lose access to the data (i.e. the RAID array is down).

Rebuilding a RAID-51 isn’t too bad. If you have one of the RAID-5 pairs operating in degraded mode (i.e. can’t tolerate the loss of another drive), then putting in a replacement drive and rebuilding only involves the data in that particular RAID-5 group. At worst this means that the number of disks read is

Number of drives read after one lost disk = (Number of disks in RAID-5 - 1)

The number of drives written to is just one (the replacement drive).

In the case of losing one side of the RAID-1, then all of the disks on the other side have to be read. The number of drives read depends upon whether the functioning RAID-5 group is degraded or not. The numbers of drives written to is the following,

Number of drives read after one lost disk = Number of disks in RAID-5

If we compare this to RAID-15, we can see that when one disk is lost, RAID-15 actually uses fewer drives (1) in the rebuild where RAID-51 has to use (Number of disks in RAID-5 – 1) disks.

Table 4 below is a quick summary of RAID-51 with a few highlights.

Table 4 – RAID-51 Highlights

Raid Level Pros Cons Storage Efficiency Minimum Number of disks
RAID-51

  • Excellent read performance because of both the mirroring (RAID-1) and RAID-5.
  • Good write performance because of RAID-5 but RAID-1 reduces the number of drives used in writing relative to RAID-15.
  • Excellent data redundancy with the ability to lose three arbitrary disks without losing data.

  • You have to use at least 6 drives (large number of drives)
  • All disks have to be used in rebuild
  • Storage efficiency is poor – much lower than even RAID-1 but approaches RAID-1 efficiency when the number of drives gets very large.
  • In the case of losing a single drive, RAID-51 uses more drives than RAID-15.
Storage Efficiency = (Number of drives in RAID-5 group – 1 ) / (Number of RAID-1 groups * Number of drives in RAID-5 group) 6 (Note: you have to use an even number of disks)

RAID 0+6 (RAID-06)

RAID-6 was developed for better data redundancy than RAID-5. However, the price for this is that in general, write performance is worse than RAID-5. So to gain back some performance for RAID-6, people combined it with RAID-0. The first Nested RAID level that involves RAID-6 that we’ll examine is RAID-06.

Up to this point, our examination of Nested RAID levels that put the performance RAID level at the lowest level and put the data redundancy RAID level at the highest level has shown us that while the storage efficiency and performance is good, the number of drives used in the event of a lost drive (rebuild) is very high – usually all of the drives in the configuration. So we should expect the same thing for RAID-06.

Figure 5 below presents an example of a RAID-06 configuration using a total of twelve drives with four RAID-6 groups (the minimum number of groups in RAID-6) where each group consists of three drives in RAID-0.

RAID-06_cropped.png
Figure 5: RAID-06 layout with Twelve Drives

In this configuration, the two parity groups for RAID-6 (denoted with a subscript “p” and a subscript “q”) are each striped in lower level RAID-0 groups. Data access is fairly easy to determine with the RAID-6 level first computing the two parity groups and then sending them, and the data, to the RAID-0 groups which then stripe the data onto the disks.

The capacity of the RAID-06 configuration is the following:

Capacity = min(disk sizes) * (Number of disks in RAID-6 group - 2) * (Number of disks in RAID-0 group)

For Figure 5, this means,

Capacity = min(disk size) * (4 - 2) * 3
Capacity = min(disk size) * 6

The resulting storage efficiency is the following:

Storage Efficiency = (Number of RAID-6 groups - 2 ) / (Number of RAID-6 groups)

For Figure 5, the resulting storage efficiency is,

Storage Efficiency = (4 - 2) / (4)
Storage Efficiency = 0.50 (50%)

So the addition of RAID-6 to increase data protection has decreased our storage efficiency compared to RAID-05.

Notice that the minimum number of drives in a RAID-06 set is eight: you need at least four RAID-6 groups and you need at least two drives per RAID-0 (otherwise it is simply RAID-6). However, in some cases there are many ways to take a pool of drives and create a RAID-06 array. For example, if you had 12 drives, you could create four RAID-6 groups that each had three drives in RAID-0 (as in Figure 5). Or you could create six RAID-6 groups that each had two drives in RAID-0. You just need to keep in mind that you need at least eight drives and the number of drives in each RAID-0 group needs to be the same.

The fault tolerance of RAID-06 is two drives – the same as RAID-6. If you lose a drive you lose that RAID-0 group. But the RAID-6 configuration at the highest level allows you to lose up to two RAID-0 groups without losing data. Theoretically, you could lose all of the drives in two RAID-0 groups but that means the drive failure has to follow a specific pattern and that is not under our control.

If you lose a drive in a RAID-06 configuration, then every drive in the configuration must either be read from, or written to. In the example of Figure 5, if we lose disk 4 (fourth drive from the left) and a new drive is inserted into the configuration, then disks 1-3 and disks 7-12 must be read so that the data for the missing RAID-0 group can be reconstructed and written to disks 4-6. As we predicted, putting the most data redundant RAID level at the highest point in the configuration causes all of the drives to be involved in rebuilding the array in the event of a drive failure.

Table 5 below is a quick summary of RAID-06 with a few highlights.

Table 5 – RAID-06 Highlights

  • Excellent read performance because of both the striping (RAID-0) and RAID-6.
  • Good to very good write performance because of the striping (RAID-0). But RAID-6 reduces the performance relative to a RAID-01, RAID-10, RAID-05 or RAID-50 configurations.
  • Very good data redundancy (can tolerate the loss of any two disks).
  • Raid Level Pros Cons Storage Efficiency Minimum Number of disks
    RAID-06

    • You have to use at least 8 drives (large number of drives).
    • All disks have to be used in rebuild.
    • Storage efficiency can be low.
    Storage Efficiency = (Number of RAID-6 groups – 2 ) / (Number of RAID-6 groups) 8

    RAID 6+0 (RAID-60)

    This Nested RAID configuration is the “companion” of RAID-06. However, as with the RAID-05 and RAID-50 comparison, we should see that while RAID-06 and RAID-60 are very likely the same in terms of performance, capacity, storage efficiency, and disk failure tolerance, RAID-60 should come out a little ahead of RAID-60 in terms of the number of drives used in the rebuilding. Figure 6 below is an example of RAID-60 showing the data layout for twelve identical drives.

    RAID-60_cropped.png
    Figure 6: RAID-60 layout with Twelve Drives

    The example configuration consists of three groups for RAID-0 at the highest level, and each group consists of a four drive RAID-6 group. The two parity blocks for each RAID-6 are denoted with the subscripts “p” and “q”. The data flow is fairly easy to follow with incoming data being striped first, followed by RAID-6 on each incoming chunk of data (including the computation of the two parity chunks).

    The capacity of the RAID-60 configuration is the following:

    Capacity = min(disk sizes) * (Number of disks in RAID-6 group - 2) * (Number of RAID-0 groups)
    

    For Figure 6, this means,

    Capacity = min(disk size) * (4 - 2) * 3
    Capacity = min(disk size) * 6
    

    The resulting storage efficiency is also fairly easy to compute:

    Storage Efficiency = (Number of drives in RAID-6 group - 2 ) / (Number of drives in RAID-6 group)
    

    For Figure 6, the resulting storage efficiency is,

    Storage Efficiency = (4 - 2) / (4)
    Storage Efficiency = 0.50 (50%)
    

    As with RAID-06, the addition of RAID to increase data protection (RAID-6) has decreased our storage efficiency.

    Notice that the minimum number of drives in a RAID-60 set is eight: you need at least two RAID-0 groups and you need at least four drives per RAID-6. However, in some cases there are many ways to take a pool of drives and create a RAID-60 array but this is typically for larger number of total drives.

    The fault tolerance of RAID-60 is two arbitrary drives – the same as RAID-06. If you lose an arbitrary drive, then the particular RAID-6 group can still operate. You can even lose a second drive in that group without losing data. But if you lose a third drive in that group, then the group fails and subsequently, RAID-0 fails. Theoretically, you could lose two drives in each RAID-6 group but that means the drive failure has to follow a specific pattern and that is not under our control.

    If you lose a drive in a RAID-60 configuration, then only the drives in the specific RAID-6 group are involved in the rebuilding. So the total number of drives read from is (Number of drives in RAID-6 group – 1) and the total number of drives written to is one (this is in the event of the loss of a single drive). In the case of the loss of two drives, then the number of drives that are read is (Number of drives in RAID-6 group – 2 ) and the number of drives written to is two.

    Using Figure 6 as an example, if we lose disk 4 (fourth drive from the left) and a new drive is inserted into the configuration, then disks 1-3 must be read from to reconstruct the data for the missing RAID-0 group. The replacement for disk 4 is written to while rebuilding the array. As we predicted putting the most data redundant RAID level at the lowest point in the configuration results in fewer drives involved in rebuilding the array in the event of a drive failure.

    Table 6 below is a quick summary of RAID-60 with a few highlights.

    Table 6 – RAID-60 Highlights

    Raid Level Pros Cons Storage Efficiency Minimum Number of disks
    RAID-60

    • Excellent read performance because of both the striping (RAID-0) and RAID-6 (no parity is used during reading).
    • Good to very good write performance because of the striping (RAID-0). But RAID-6 reduces the performance relative to a RAID-01, RAID-10, RAID-05 or RAID-50 configuration. Moreover, the write performance should be about the same as RAID-06 but could be a little less depending upon the configuration.
    • Very good data redundancy (can tolerate the loss of any two disks)
    • In the event of a drive failure, only the number of drives in a RAID-6 group are involved in rebuilding.

    • You have to use at least 8 drives (large number of drives)
    • Storage efficiency can be low.
    Storage Efficiency = (Number of drives in RAID-6 group – 2 ) / (Number of drives in RAID-6 group) 8

    RAID 1+6 (RAID-16)

    We characterized RAID-15 and RAID-51 as wearing a belt and a pair of suspenders but RAID-16 and RAID-61 go a bit beyond that. They use titanium reinforced suspenders and a belt with a Kevlar core – just to make sure. These two RAID configurations are really not for performance, but more for data protection but RAID-6 does provide better performance relative to RAID-1. Let’s examine RAID-16 first.

    Figure 7 below is an example showing the Nested RAID-16 data layout for eight identical drives.

    RAID-16_cropped.png
    Figure 7: RAID-16 layout with Eight Drives

    There are four groups in the RAID-6 level (highest RAID level) that are each constructed from RAID-1 pairs. When data is written to the RAID-16 group it first hits RAID-6. The two parity chunks are computed (subscripts “p” and “q”) and they are sent, along with the data chunks, to the various RAID-1 pairs for writing.

    The capacity of the RAID-16 configuration is fairly easy to compute:

    Capacity = min(disk sizes) * (Number of RAID-1 groups - 2)
    

    For Figure 7, this means,

    Capacity = min(disk size) * (4 - 2)
    Capacity = min(disk size) * 2
    

    The resulting storage efficiency is also fairly easy to compute:

    Storage Efficiency = (Number of RAID-1 groups - 2) / ( (Number of RAID-1 groups) * (Number of drives in RAID-1) )
    

    For Figure 7, the resulting storage efficiency is,

    Storage Efficiency = (4 - 2) / (4 * 2)
    Storage Efficiency = 0.25 (25%)
    

    The combination of RAID-6 and RAID-1 in this case really reduces the storage efficiency. All of that data protection has an impact on the useful space in the array.

    Notice that the minimum number of drives in a RAID-16 set is eight: you need at least four groups at the RAID-6 level (highest level) and you need at least two drives per RAID-1 group (lowest level). However, in some cases there are many ways to take a pool of drives and create a RAID-16 array using more than two drives per RAID-1 (3+ drives).

    The fault tolerance of RAID-16 is amazing. You can lose up to five arbitrary drives without losing access to data. This allows for the worst case of completely losing two RAID-1 pairs (I’m assuming RAID-1 only has 2 drives) and one drive on one of the remaining RAID-1 mirrors.

    In the event of a single drive failure, just the mirror drive is used in the rebuild. In the event that a complete mirror pair fails (two drives), then the remaining RAID-1 pairs will be used to rebuild the failed RAID-1 pair. The same is true when four drives are lost (the remaining drives are all used for rebuilding).

    Table 7 below is a quick summary of RAID-16 with a few highlights.

    Table 7 – RAID-16 Highlights

    Raid Level Pros Cons Storage Efficiency Minimum Number of disks
    RAID-16

    • Excellent read performance because of both the mirroring (RAID-1) and RAID-6 (no parity is used during reading).
    • Outstanding data redundancy (can tolerate the loss of any five disks).
    • In the event of a single drive failure, only the mirrored drive is involved in the rebuild.

    • You have to use at least 8 drives (large number of drives).
    • Storage efficiency can be very low (lower than RAID-1).
    • Reasonable write performance because of RAID-6 and RAID-1. However, the combination uses two of the slowest write RAID levels: RAID-1 and RAID-6.
    Storage Efficiency = (Number of RAID-1 groups – 2) / ( (Number of RAID-1 groups) * (Number of drives in RAID-1) ) 8

    RAID 6+1 (RAID-61)

    RAID-61 is the companion to RAID-16 but uses RAID-6 at the lowest level and RAID-1 at the highest level. Figure 8 below is an example showing the data layout for eight identical drives in a RAID-61 configuration.

    RAID-61_cropped.png
    Figure 8: RAID-16 layout with Eight Drives

    This layout uses the minimum number of drives for RAID-61 (eight). There are four drives in each RAID-6 group and two RAID-1 groups for a total of eight drives. When writing to a RAID-61, the data comes into the RAID-1 layer first. Then the data is mirrored and sent to each mirrored pair of RAID-6 drives. Then RAID-6 computes the two parity chunks and along with the data, writes them to the drives.

    The capacity of the RAID-61 configuration is fairly easy to compute:

    Capacity = min(disk sizes) * (Number of drives in RAID-6 group - 2)
    

    For Figure 8, this means,

    Capacity = min(disk size) * (4 - 2)
    Capacity = min(disk size) * 2
    

    The resulting storage efficiency is also fairly easy to compute:

    Storage Efficiency = (Number of drives in RAID-6 group - 2) / ( (Number of RAID-1 groups) * (Number of drives in RAID-6) )
    

    For Figure 8, the resulting storage efficiency is,

    Storage Efficiency = (4 - 2) / (2 * 4)
    Storage Efficiency = 0.25 (25%)
    

    As with RAID-16, the combination of RAID-6 and RAID-1 in this case really reduces the storage efficiency. All of that data protection has an impact on the useable space in the array.

    Notice that the minimum number of drives in a RAID-61 set is eight: you need at least four drives in a RAID-6 group and you need two RAID-1 groups (at least). However, in some cases there are ways to take a pool of drives and create a RAID-61 array using more than two drives per RAID-1 (3+ drives) if your RAID controller allows this.

    The fault tolerance of RAID-61 is outstanding. You can lose up to five arbitrary drives without losing access to data. This allows for the worst case of losing three drives on one of the RAID-6 groups as well as two drives in other mirrored RAID-6 group.

    In the event of one or two drive failures within the same RAID-6 group, the rebuild uses all of the drives in that particular RAID-6 group. If one RAID-6 pair loses three or more drives, then all of the drives in the mirrored group are used for reading, and all of the drives in the first RAID-6 group are used for writing.

    Table 8 below is a quick summary of RAID-61 with a few highlights.

    Table 8 – RAID-61 Highlights

    Raid Level Pros Cons Storage Efficiency Minimum Number of disks
    RAID-61

    • Excellent read performance because of both the mirroring (RAID-1) and RAID-6 (no parity is used during reading).
    • Outstanding data redundancy (can tolerate the loss of any five disks)

    • You have to use at least 8 drives (large number of drives)
    • Storage efficiency can be very low (lower than RAID-1).
    • Reasonable write performance because of RAID-6 and RAID-1. However, the combination uses two of the slowest write RAID levels: RAID-1 and RAID-6.
    • In the event of a drive failure, the drives in the RAID-6 group with the failed drive are involved in rebuilding.
    Storage Efficiency = (Number of drives in RAID-6 group – 2) / ( (Number of RAID-1 groups) * (Number of drives in RAID-6) ) 8

    Summary

    This is the third article in the series on RAID and has shown that Nested RAID levels that include RAID-5 and RAID-6 can be very interesting. In general, RAID-5 and RAID-6 are considered more about data redundancy than performance, although the performance of these two standard RAID levels is better than RAID-1.

    We first examined the combination of RAID-5 and RAID-6 with RAID-0 to add more performance back to the data redundancy RAID levels. We also examined the “companion” Nested RAID configurations for any differences. Specifically we examined RAID-05 versus RAID-50 and RAID-06 versus RAID-60. But I use the term “companion” very loosely since these comparison pairs are really different in how they handle data. However, from a capacity, storage efficiency, lost drive tolerance, and performance perspective, each pair has about the same performance. So RAID-05 is very similar to RAID-50 and RAID-06 is very similar to RAID-60. The biggest difference between them is that when the lowest RAID level is RAID-0, more drives are involved in the rebuild in the event a drive fails. This can lead to longer rebuild times and can come close to reaching the URE (Unrecoverable Read Error) limits for large number of drives and large capacity drives. So in general, it is recommend that you use RAID-50 over RAID-05 and RAID-60 over RAID-06.

    Then we examined combining RAID-5 and RAID-6 which use parity for data protection with RAID-1 that uses mirroring for data protection. These combinations, RAID-15, RAID-51, RAID-16, and RAID-61, are for the truly data paranoid (which isn’t always a bad thing). For these Nested RAID levels “pairs” (RAID-15 and RAID-51, and RAID-16 and RAID-61), the capacity, storage efficiency, data redundancy, and performance aspects are virtually the same. As with the the “RAID-0″ Nested RAID levels, the differences boil down to how many drives are used in a rebuild in the event of a drive failure.

    The RAID-15 and RAID-51 combination have the ability to tolerate the loss of three drives before losing data. However, the most common failure scenario is the loss of a single drive. In this case, RAID-15 offers advantages over RAID-51 in that only the mirrored drive is used during the rebuild. In the event of two drives failing, the worst case means that for both RAID-15 and RAID-51, all of the drives in the array are used for for the rebuild (the same is true for losing three drives as well). So there is little difference between the two at that point. However, RAID-15 is recommended over RAID-51 because of the single drive failure scenario.

    The same is true for RAID-16 and RAID-61 – RAID-16 is recommended over RAID-61. However, the differences between the two are small and really only manifest themselves when a single drive is lost. But the really cool thing about either RAID-16 or RAID-61 is that they can tolerate the loss of five drives without losing data! That is a huge number!.

    Overall you can see that Nested RAID configurations using RAID-5 and RAID-6 really allow you to develop some performance oriented configurations that have some data redundancy or to develop some really paranoid configurations that can lose a large number of drives without losing data. Lots of flexibility.

    Fatal error: Call to undefined function aa_author_bios() in /opt/apache/dms/b2b/linux-mag.com/site/www/htdocs/wp-content/themes/linuxmag/single.php on line 62