In Praise of Openness
In my experience, the open approach, promotes open discussion, open experimentation, open solutions, and rapid growth. If you were looking for a market where the concept of "give a little and get a lot" works, you have arrived.
A common mistake people make is to assume open software is “free as in beer.” It is not, and in a future column, I will explain why. For now, however, I would like to talk about “free as in speech.” Recently I read about the High Performance SSH/SCP (HPN-SSH) project at the Pittsburgh Supercomputing Center (PSC). The HPN-SSH project is an enhancement of the ubiquitous ssh protocol. As a basis the project uses the OpenSSH. In case you tuned in late the game, “Open” usually means open source code, which means, depending on the license, you are allowed to modify and re-distribute the source code and binary data.
In essence, ssh is an encrypted version of the telnet (or rsh) command. Back in the day, when we all trusted each other, telnet was the way one logged into machines over the network and even the Internet. Conversations with the other machines including your passwords went flying around the Internet in plain text. The ssh protocol solves this problem by encrypting the entire session — from login to logout. If you need to copy files there was also scp that could be used in place of things like rcp and ftp. Score one for the good guys.
For most users, ssh and scp work just fine, but HPC is different. We often need to run things at full speed. The trouble with ssh/scp is that the underlying SSH2 protocol is limited by statically defined internal flow control buffers. These buffers often end up acting as a network bottleneck. Chris Rapier (PSC), Michael Stevens (Carnegie Mellon), and Benjamin Bennett (PSC) decided to improve things a bit. They have modified the SSH2 protocol so that the buffers can be defined at run time. In addition, they multi-threaded the encryption portions of the code so that the computationally heavy encryption can now run on a second core. As they state, “The amount of improvement any specific user will see is dependent on a number of issues. Transfer rates cannot exceed the capacity of the network nor the throughput of the I/O subsystem including the disk and memory speed. The improvement will also be highly influenced by the capacity of the processor to perform the encryption and decryption. Less computational expensive ciphers will often provide better throughput than more complex ciphers.” These changes can make a big difference network performance. Take a look at the benchmark data on the project site.
While the HPN-SSH project is interesting in it’s own right, there is yet again, a higher principle at work here. I touched on this when I answered the question: Why Linux On Clusters?. In my opinion, the ability of the HPC community to modify software to suit their needs, is a fundamental driving force behind the rapid market growth of Linux clusters. There is no marketing department tollbooth to pass through. In a traditional business approach, marketing needs to determine what features the customers want, do a cost/profit analysis for adding these features, possibly implement them, and charge the customer for an upgrade. Both closed-source and open-source companies/groups may do a similar analysis, but in the case of the closed-source approach, if your feature does not make it through the marketing tollbooth you are SOL. (And, that would be — Sorry. Options Locked. What were you thinking?)
In my experience, the open approach, promotes open discussion, open experimentation, open solutions, and rapid growth. If you were looking for a market where the concept of “give a little and get a lot” works, you have arrived. Still a little doubtful? Consider the high performance file system options for Linux. Enough said. Unfortunately, not every one gets the open thing. Some people find the free beer part hard to swallow. Giving away software and/or IP, that is insane. But that happens everyday in the “closed source” world. I’ll explain that next time.
Douglas Eadline is the Senior HPC Editor for Linux Magazine.