The Undeniable Power of the Source

People generally consider the code that powers the Linux kernel and other open source packages to be both superlative and secure because “many eyes make all bugs shallow.” But even the most reliable open source packages are still burdened with security flaws. Why? To find the answer, you have to look at the source.

August 2006

The Undeniable Power of the Source

Improving the security of software starts with a whole new vernacular

Compile Time

People generally consider the code that powers the Linux kernel and other open source packages to be both superlative and secure because “many eyes make all bugs shallow.” But even the most reliable open source packages are still burdened with security flaws. Why? To find the answer, you have to look at the source.

Ben Chelf

Before the introduction of the personal computer, computer security was largely the purview of computer operations centers that powered behemoth businesses, such as banks, insurance companies, airlines, and government. Machine rooms were off-limits, access to computing resources was strictly controlled and policed, and each system was essentially its own island, surrounded by a great sea of anonymity. Later, when the personal computer arrived, security remained an issue, but if you diligently scanned incoming media (namely, floppy disks), you could avoid most “viruses.”
Today, following the remarkable and pervasive adoption of the personal computer and the Internet, computer security has (unfortunately) become a popular topic. Viruses, worms, and other forms of exploits are a new fact of life — like death and taxes — and the hint of a new, significant threat quickly makes the headlines of the evening news. And it’s no wonder: virtually every computer can connect to another, granting malware a ready opportunity and avenue to propagate and wreak havoc. Computers users, rightly so, have learned to be paranoid.
To be sure, both proprietary software and open source software can be exploited, and both must often be patched to prevent repeated or further damage. But where proprietary software is a big black box, open source provides an opportunity to examine, qualify, and classify what went wrong. One of the enduring benefits of open source is you can look at the software and learn from others’ mistakes. Indeed, using the source is the only hope of improving the overall quality of software.

When a proprietary software vendor discloses a security vulnerability, the vendor doesn’t parade its code for all to see. Instead, the vendor discusses the effects of the vulnerability and provides information about how to avoid or repair the affected system, usually in the form of a patch, or more euphemistically, “an update.”
A patch is sufficient for an end-user, who downloads the patch, applies it, and breathes a sigh of relief, knowing that his or her system is (a bit) more secure. However, such secretive fixes do little to advance the science and practice of making software more secure.
By comparison, it’s typical for an open source software vendor to not only publish a patch for its end-users (for example, consider the invaluable apt-get upgrade), but to also publish a patch to the source code itself. Hence, if you, the end-user, want to see exactly why your system could have been compromised by some malicious teenager a thousand miles away, you can simply examine the patch.
For example, you might see that errant code…
strcpy(small_dest, large_src);
… blows away the stack, allowing miscreants to execute arbitrary code on your system.

When Is a Bug a Bug?

Arbitrary code execution. Privilege escalation. Denial of service. SQL injection. Lord Voldemort. We’ve learned (from experience, sadly) to fear these words and shiver at their mere mention. But what exactly does it say about the source code if you have a privilege escalation? Or a denial of service?
Let’s look at this line of code again:
strcpy(small_dest, large_src); 
Agreed, the previous line of code “is bad,” as security experts might say. But what about this next line of code?
strcpy(dest, src); 
Is there a bug? (Or is that just a rhetorical question? Or a trick question? Or is the answer just plain simple?) The right answer is, “It depends.” Whether or not the line of code portends trouble depends highly on the context in which this line of code appears. In fact, this line of code might not be a problem at all!
In general, it is difficult to classify a defect as a particular vulnerability, even if you’re able to examine the code in context. Many kinds of programming errors can yield the same vulnerability. Furthermore, most of the time, the eventual effect of a vulnerability also depends on what the malicious user chooses to do with it. For example, imagine that an attacker uses a “buffer overflow” (shiver) to insert “arbitrary code to be executed” (shudder) and the code shuts down the box, causing a “denial of service” (gasp!). That’s three security vulnerability buzzwords, but only one real source code defect.

Evil Tetris

Open source is revealing a better way to talk about software vulnerabilities– a way to move from paranoia to action, educate developers, and ultimately, improve source code of the future.
While open source software benefits from the “many eyes” that weed out defects early in the development process, a security vulnerability occasionally slips in. For instance, one of the nineteen Linux high-severity vulnerabilities listed in the National Vulnerability Database (NVD, http://nvd.nist.gov/) was in Tetris, a popular game included in many Linux distributions. As described by the NVD:
Multiple buffer overflows in the checkscores() function in scores.c in tetris-bsd in bsd-games before 2.17-r1 in Gentoo Linux might allow local users with games group membership to gain privileges by modifying tetris-bsd.scores to contain crafted executable content, which is executed when another user launches tetris-bsd.
Or, in other words, you’re endangering your system by playing Tetris.
In the spirit of getting to the root cause of the vulnerability, let’s take a look at the code that leads to this problem:
* Set times to 0 except for high score on each level.
for (i = MINLEVEL; i < NLEVELS; i++)
levelfound[i] = 0;
for (i = 0, sp = scores; i < nscores; i++, sp++) {
if (levelfound[sp->hs_level])
sp->hs_time = 0;
else {
sp->hs_time = 1;
levelfound[sp->hs_level] = 1;
This particular snippet of code shows the current high scores read from a file. For completeness, the scores variable is declared in the global scope, alongside nscores:
static int nscores;
static struct highscore scores[NUMSPOTS];
The data is read via an initialization routine:
nscores = fread(scores, sizeof(scores[0]),
So, the scores variable contains the data from the high scores file. Notice, however, that in the for loop above, the sp variable is assigned these scores, and the hs_level field is read from the structure without a bounds check. So, if the file has a bogus value for that field, a malicious user can corrupt the stack in the assignment to the levelfound[] array.
The fix is to simply perform a bounds check on the potentially dangerous field:
for (i = MINLEVEL; i < NLEVELS; i++)
levelfound[i] = 0;
for (i = 0, sp = scores; i < nscores; i++, sp++) {
if (sp->hs_level < NLEVELS && sp->hs_level >= 0) {
if (levelfound[sp->hs_level])
sp->hs_time = 0;
else {
sp->hs_time = 1;
levelfound[sp->hs_level] = 1;
By adding a simple check, the code avoids an exploit capable of usurping control of the system. (And you can go back to worrying about beating your previous high score!)
Returning to the NVD description, notice that the categorization of this defect is “Arbitrary code execution.” Yes, that’s the result, but it isn’t a good description of the defect itself. In particular, it doesn’t provide any information about the cause of the defect or how to fix it. It would be better to classify this vulnerability as “failure to provide a bounds check for tainted data” (where “tainted data” is a term for any data that must be vetted before it is used in the program). When speaking about the source code, this type of categorization is much more useful in understanding what went wrong.
If you want programmers to write better code, you have to give them good information about the problems you want them to avoid. Arbitrary code execution may help scare an end-user into patching a system, but it doesn’t help the developer learn to avoid the same mistake in the future.

Being Careful With Input Data Isn’t Enough

That same list of nineteen high severity security vulnerabilities discovered in or around Linux includes thirteen that fail to properly validate potentially tainted data. Of the remaining six, two of the issues were design flaws, and the rest were other types of coding defects, including null pointer dereferences, using memory after it was freed, and the following defect found in recent (March 2006) X.org code (if you are using Linux, you are very likely using the X Window System):
1682     if (getuid() != 0 && geteuid == 0) {
1683 ErrorF("The ’-configure’ option can only be used by root.\n");
1684 exit(1);
1685 }
Can you spot the problem? Many malicious users can. In fact, an exploit was published a mere five hours after this vulnerability was disclosed. This defect is described by NVD as:
X.Org server(xorg-server) 1.0.0 and later, X11R6.9.0, and X11R7.0 inadvertently treats the address of the geteuid function as if it is the return value of a call to geteuid(), which allows local users to bypass intended restrictions and(1)execute arbitrary code via the -modulepath command line option, or(2)overwrite arbitrary files via -logfile.
One contributor to X.org described this vulnerability as one of X.org’s “worst case scenarios” and something only found “every 3 to 5 years”. And what type of vulnerability is this? “Execute arbitrary code,” just like the Tetris example above. But in terms of the source code responsible, the two bugs are much different problems!
How can such a problem be better categorized? Maybe bad pointer comparison? Or unintended comparison? Or “treating a function pointer as a pointer value in comparison instead of actually calling the function”? Or maybe just “missing parentheses”? Any of those categorizations provide a lot more information about what went wrong in the code than describing the problem’s effects. And more information is good, because developers can use that to write better code.
The previous section of code highlights another problem in categorizing defects and security vulnerabilities. Namely, if you’re looking at the source code, how do you know that one defect is a security vulnerability and another one isn’t? Certainly, it’s a defect in the code to forget parentheses when you mean to call a function, but not all defects like this are security vulnerabilities. Even if you restrict the definition to “forgetting parentheses in calls to geteuid() “, that doesn’t guarantee that a code defect in this category has security ramifications.
In this particular case, the call to the geteuid function was instrumental in performing the right privilege check. But there are other cases where you may choose to call this function and doing it wrong doesn’t lead to a “privilege escalation” problem. Perhaps you are checking for a non- root user for another reason.
Just looking at source code doesn’t always tell you how “exploitable” a defect in the code is. For some defects, a tremendous amount of experience and brain power is required to figure out the potential run-time implications. That’s just another fact of life.

When Many Eyes Are Reduced To a Few

Keeping up with vulnerabilities is a lot like painting the Golden Gate Bridge.
Keeping the Golden Gate Bridge “golden” red (actually, “orange vermillion”) is quite a chore. In fact, the task is so large that aesthetic maintenance is a full-time (indefinite) job for 38 people, who spend their lives patching up the paint to retain the bridge’s beauty.
Now, Paul[ a friend of the author] is a very smart security guy who works for a company that makes a tremendous amount of software. (You’ve probably used the company’s services, especially around December 21st, if you’re a procrastinator.) A few years ago, Paul had about six weeks of dead time at work between projects, so he shut himself in a closet and spent six full weeks performing a manual audit of the company’s production source code. He said he fixed hundreds of problems, some that would lead to security issues, others that wouldn’t.
However, after six weeks, he had combed through only eight percent of all the code. Again, amazingly, one of the smartest security guys around, who’s spent years and years thinking about these problems, vetted only a twelfth of his company’s code in six weeks. Furthermore, if Paul continued his audit all the way to the end of the code, by the time he was done, it would have all churned to the point where he’d have to do it all again.
You can’t help but think about the dedicated folks who paint the Golden Gate continually.

No Silver Bullet

The point here may be obvious, but it’s worthwhile to summarize: The task of scouring proprietary source code for serious defects is more than even the most dedicated software team can manage. And while open source benefits from the “many eyes” approach, that doesn’t lead to perfect code bases either. Security vulnerabilities are a fact of life. To prevent exploits, code must be vetted, even attacked as early as possible though education, good software process, and sophisticated technology.
For example, software from Coverity[ the author’s employer] helps find defects in source code. In fact, the X.org bug discussed earlier was found automatically with Coverity’s technology as part of a Department of Homeland Security contract awarded by the Science and Technology Directorate titled “Vulnerability Discovery and Remediation, Open Source Hardening Project.”
But the key notion here is “help”. No one can sell you security anymore than they can sell you quality– security is an ideal, a goal, not something you grab off the shelf. Security has to be built using a comprehensive strategy and the right frame of mind. You must build secure code from the beginning and be ever vigilant.
There’s no silver bullet. Linux, and the open source movement, has shed light on the problem and taught a valuable lesson: To tackle security problems most effectively, developers must look beyond the scary consequences of a security failure and focus on its origins.
Building security into software means going straight to the source.

Ben Chelf is Coverity’s Chief Technology Officer. Mr. Chelf was a founding member of the Stanford Computer Science Laboratory team that architected and developed Coverity’s technology. He is one of the world’s leading experts on the commercial use of static source code analysis. In his role at Coverity, Mr. Chelf works with organizations such as the US Department of Homeland Security, Cisco, Symantec, and IBM to improve the security and quality of software around the world. He holds an M.S. and B.S. in computer science from Stanford University. Mr. Chelf often provides expert insight into software security and quality for the press, public audiences, and in published writings.

Comments are closed.