The SuperComputing Conference/Exhibition is always a great conference for learning about storage trends in the HPC world. This year the alert attendee could spot two emerging trends: smaller companies developing innovative storage solutions and the rise of flash storage units.
High-Density Flash Storage
There was a past article about really fast storage (Ramdisks – Now We are Talking Hyperspace!) but flash based storage units are becoming more popular. For dense flash based storage units, the performance density (IOPS/U or Throughput/U, where “U” is the common rack unit measure – 1.75″) can be better than DRAM based storage units, especially when price is considered. This is particularly true for IOPS driven applications.
There are several vendors who offer Flash Storage units – Texas Memory, Violin Memory, and now Sun are offering flash based storage units with performance that is amazing.
The basic concept is that you take a standard rack unit, stuff them full of flash units (drives or flash based DIMMs), possibly combine them with RAID controller(s), and put FC, or IB connections, or direct connect via a PCIe cable to a host node, and you have a storage unit with very high performance. It sounds simple but it is actually more difficult than you think.
Texas Memory has been making very high-speed storage units for a number of years. Originally they focused more on DRAM based solutions but in the last few years they have been offering flash based storage devices as well. The latest unit is the RamSan-620.
The RamSan-620 is a 2U unit that has a capacity of 1-5 TB of SLC Flash (the good but more expensive kind of flash) and uses only 230W of power. It has two 4 Gb FC connections in the back of the box with InfiniBand connections and 10GigE connections coming soon. It uses super capacitors in case of power loss so the DRAM on the flash chips can be flushed to the flash storage. It does have some really good management features including the ability to use 512 byte blocks rather than the standard 4KB blocks (if operating systems and file systems can make use of it). It also allows you to carve the storage into 1 to 1,024 LUNs with variable capacity in each LUN and the ability to assign LUNs to specific ports.
However, the really cool aspect of the unit is the performance. It is rated at up to 3 GB/s throughput and 250,000 random IOPS. Given that a hard drive can perform between 100-200 IOPS, this kind of performance is short of remarkable. An equivalent number of hard drives to match the IOPS capability is in the range of 1,250-2,500.
Violin Memory is selling a high performance storage unit, the Violin 1010, that can use either DRAM or Flash based storage units. The basic 2U unit can accommodate FC and Ethernet connections via a “network head” or it can be attached directly to a node via a PCIe interface (dual x4 and x8 interfaces). The Flash based version of the 1010 has a raw capacity of up to 4 TB of SLC flash units. The 1010 uses 62 Violin Intelligent Memory Modules (VIMMs) that come in either 32GB or 64GB capacities.
Like the Texas Memory unit, the Violin Memory 1010 has fantastic performance. For a fully loaded unit with 64GB VIMM’s, the 1010 has a performance of 345,000 4K Read IOPS and 219,000 4K Write IOPS. These numbers are assuming a x8 PCIe connector. A x4 PCIe connection will have a performance of 215,000 4K Read IOPS and 145,000 4K Write IOPS. The sustained read throughput performance is just a little above 1.4 GB/s, a peak write of about 1 GB/s, and a sustained random write performance of about 850 MB/s.
Recently, Sun introduced a totally flash based storage unit, the Sun F5100. It is a 1U box that has 20, 40, or 80 SO-DIMM based SLC Flash Modules (FMOD’s). Each FMOD currently has 24GB of usable capacity but is really a 36GB flash module (the extra space is used for over provisioning). Figure 2 below, courtesy of Robin Harris at StorageMojo, shows the SO-DIMM based flash unit.
Figure 2: Sun F5100 SO-DIMM FMOD
The image was taken from a Youtube video by Andy Bechtolsheim from Sun. Each SO-DIMM is capable of about 200 MB/s throughput and 30,000 read IOPS and 10,000 write IOPS.
The SO-DIMMs are put int a 1U box as shown in Figure 3, again courtesy of Robin at StorageMojo.
Figure 3: Sun F5100 Unit
Internally there are 64 SAS channels to connect the SO-DIMMs and you can break them into 4 domains. You can connect each domain via a SAS connector to a host system or connect all 4 external SAS lines to a single host (you still need 4 SAS ports on the host though).
The performance of the F5100 is the amazing part of the unit. Table 1 below lists the basic performance characteristics of the unit (take from the datasheet on Sun’s site) for the 20 FMOD, 40 FMOD, and 80 FMOD configurations.
Table 1 – F5100 Capacity/Performance Specifications.
||20 Flash Modules
||40 Flash Modules
||80 Flash Modules
|Random Read IOPS (4K)
|Random Write IOPS (4K)
|Sequential Read Rate
|Sequential Write Rate
Notice the IOPS numbers – in the 80 FMOD configuration the performance is well over 1,000,000 IOPS for both reads and writes. For applications that are very IOPS driven, as many HPC applications can be, this kind of performance for an application is nothing short of amazing.
The FMODs have a small amount of DRAM on-board, so the F5100 has up to four very large capacitors called Energy Storage Modules (they look like the drives in the front of the unit). These capacitors have enough power to allow the DRAM’s to flush their data to the flash storage.
Each of the FMOD’s shows up as a block device to the OS. For ZFS, this isn’t such a big deal since it can use the drives individually and manage them as part of the overall storage pool. For something like Linux you would have to use md to manage all of the FMOD’s to create a RAID group.
The price for the entry level unit is also very attractive. According to several articles the price point for 20 FMODs (480 GB raw capacity) is $45,995 (397,000 IOPS read and 304,000 IOPS write). Moreover, the entire unit only uses about 300W of power (less than your desktop system). For that level of IOPS performance and power consumption, the price is very, very attractive.
While there were a number of other announcements at SC09, these two trends were very noticeable. The first trend, smaller companies doing more innovation/development of storage solutions than larger companies, is not just about technology. It is also about integrating everything together to make the whole work better than just the sum of the components. Coupling this with a definite focus on the business aspects results in some of the most interesting storage solutions. Scalable Informatics is a very good example of the type of company who is doing more innovation that others.
The second trend, flash storage units, has been developing for the last few years. Texas Memory and Sun both recently announced flash storage units and were showing them at SC09. The level of performance and the improving density and price of flash storage units makes them very attractive. When you can get over 1 million IOPS from a 1U device, the options for addressing new problems or rethinking your approach to problem solving suddenly improve.
Jeff Layton is an Enterprise Technologist for HPC at Dell. He can be found lounging around at a nearby Frys enjoying the coffee and waiting for sales (but never during working hours).