Software Development the Apache Way

May 2005

Software Development the Apache Way

An inside look at how Apache Software Foundation projects produce code

In the second in his series, Apache Software Foundation (ASF) co-founder Ken Coar describes the rules that all ASF projects must abide by — rules that are fundamental to the “Apache Way.”

Ken Coar

As the Apache Software Foundation (ASF) likes to say, “Community is more important than code.” Indeed, the role of the ASF is to provide a framework that facilitates communication and mutual respect among developers. The creation of code simply takes place within that framework. It’s the “Apache Way.”

There is a definite attitude and mindset surrounding activities at the Apache Software Foundation. Although individual projects can have their own specific rules and guidelines, there are some things that are part of the Apache Way that are essentially required of all ASF projects. These requirements exist because they provide some sort of uniformity across the entire Foundation. Some affect the interface between the ASF and the outside world, and some have to do with the internal operations of the ASF and its projects.

One License for All

One of the mandated outward-looking requirements is that all code developed by an ASF project must be licensed and distributed under the Apache Software License (available online at http://www.apache.org/licenses/LICENSE-2.0.html). A single, mandated license ensures that all pieces of Apache code, from whatever packages, are distributed under the same rules.

Another requirement is that all committers (those developers who have access to the master repositories of sources) must execute a Contributor License Agreement (CLA, http://www.apache.org/licenses/icla.txt) and file it with the ASF secretary. This is intended to guarantee that all code changes committed are appropriately licensed to the Apache organization for inclusion, modification, and redistribution.

As a matter of fact, access to the repositories is largely granted by the ASF’s Infrastructure Group, which is composed of the Apache contributors who have volunteered to keep the systems running. The infrastructure folks won’t grant access until they see confirmation that the CLA form has been received and filed.

Decisions, Decisions

One of the inward-facing requirements has to do with how decisions are made. This is actually at the very heart of the Apache Way, and if you ask someone, “What is the Apache Way?” there’s an excellent chance they might respond with a description of the decision-making mechanism.

Almost all decisions are reached by voting in email. This might sound like a hopelessly heavyweight and cumbersome process, but in fact, it generally moves very quickly and smoothly. (Of course, there are the occasions in which someone puts a monolithic or extremely controversial matter up for a vote, but those tend to be uncommon.)

In a fashion only geeks may appreciate, all decision makers express their vote in numeric form:

*+ 1 means, “I’m in favour of this.”

*–1 is either a veto or a vote against the proposal.

*Anything else is essentially an abstention with an opinion attached.

Because of the distributed and volunteer nature of the community, votes typically have a minimum duration that allows everyone in every timezone a chance to review the issues. How the results are determined depends on the kind of decision being made. These fall into two categories: technical and non-technical.

Non-technical votes are decided by simple majority. Typical majority votes include such things as deciding when to release a new version, granting someone commit access, adding a new subproject, or some other non-technical matter.

Technical decisions have to do with such things as whether changes to the code (or to documentation) are acceptable or not. A technical vote has special requirements: to pass, there must be at least three+ 1 votes and no –1 votes. In this case, a –1 vote is referred to as a veto and requires technical justification. In Apache terms, this is called a “consensus vote.”

The technical voting mechanism is the part that is unusual to the Apache organization. If a matter can’t garner at least three people in favor of adopting it, the consensus is that it’s probably a waste of time.

For example, a typical email thread might look something like this:

From: Julius
Subject: [VOTE] krantz-fripping patch

Rasputin’s patch for fripping the krantz looks pretty good.
I’ve tested it and it applies cleanly and works as described.
I’m +1 on committing this.

From: Nikolai
Subject: Re: [VOTE] krantz-fripping patch

Looks good to me. +1

From: Victoria
Subject: Re: [VOTE] krantz-fripping patch

I thought the krantz was supposed to be unfripped by design?
-0 if that’s right, or +1 if I got that wrong and it’s
*supposed* to be fripped.

From: Charlemagne
Subject: Re: [VOTE] krantz-fripping patch

+1

According to the voting rules, the patch would be accepted and incorporated if those were the only votes cast, because at least three favored it (“plus-one’d”) and no-one vetoed it.

A veto is a very powerful tool, because it is unappealable. There are only three ways to deal with a veto: abandon the issue and let the veto stand; convince the vetoer to withdraw it; or address the goal through some other mechanism that won’t incur a veto.

Because of its power, a veto is not supposed to be exercised frivolously. Every veto needs to be accompanied by a technical justification explaining what’s wrong with the item and why the veto is appropriate.

Technical justification can be a fairly subjective thing: what a person exercising a veto thinks represents justification might not seem a strong enough argument to the rest of the community. Even in cases such as that, however, the power of the veto is not overruled. Instead, social and peer factors come into play, with each side trying to convince the other.

In pathological situations, developers may try to ignore vetos or veto each other’s alternatives “just because,” but the Apache culture comes down pretty hard on the participants if such a scenario develops. “Hard” can mean anything from public censure or reprimand, to actual revocation of commit access.

None of this happens in a vacuum. Sometimes these discussions are resolved amicably, and sometimes they result in considerable acrimony and lasting bad feelings. Taking the bad with the good is an accepted part of the process, and maintaining the feeling of camaraderie and teamwork in the face of arguments and “flame wars” is one of the challenges that the Apache perspective is dedicated to meeting.

Casting a Vote, Lobbying for a Change

The question of who is eliglble to vote is a critical piece of this. Who may vote is different for each group, and sometimes for different activities within a group.

For instance, votes on whether someone should be granted commit access might be limited to the project’s management committee (for a description of the structure of projects within ASF, see the first article in this series at http://www.linuxmagazine.com/2005-01/), or perhaps to the full set of existing committers for the project. The choice of a logo or mascot for the project may be something on which everyone, developer or user, is welcome to express an opinion.

Unfortunately, the “who can vote” criteria are not always clearly described. There have been situations in which votes were cast by people who weren’t entitled to do so, and it may or may not have been explained to them after the fact — after their votes were ignored. Chalk it up to the stereotypical social clumsiness of the the culture, if you like.

In the original days of the Apache HTTP Server project, back when it was called the “Apache Group,” every patch was submitted to the mailing list for discussion and voting. Most submitters found that had to become their own advocates, often reminding people to vote on their changes. A+ 1 meant, “I have applied this patch and tested it, and found it ‘good.’” A list of outstanding issues and patches was maintained by one of the developers and he’d update it with any votes or new patches and send it out to the list about once a week.

This method of applying patches became known as “Review-Then-Commit,” or RTC (arr-tee-see). Its strength is that no changes make it into the code without having been tested. Its weakness is that it tends to slow down progress, since each change needs to be approved.

The flip side to RTC is “Commit-Then-Review” (CTR), which involves what you might expect: changes can be committed to the sources at any time and stay there by default unless someone asserts a retroactive veto. The strength of CTR is that development can move quite rapidly, pretty much as fast as the developers can churn out code. Its weakness is that inadequately tested code can be committed.

Both RTC and CTR have social ramifications as well. Review-Then-Commit can strengthen the feeling of community, since people need to work together in a quid pro quo arrangement to commit changes. Everyone needs to at least look at each submission to have one’s own reviewed (and potentially approved) in turn.

The very deliberation it imposes can cause friction, however, with some people feeling they’re being held back. Whether that’s true or not, the perception can be formed if one’s submissions don’t get voted on. It also can be problematic in small communities, where its requirements can cripple development if one or two people are unavailable and the quorum of three can’t be achieved without them.

The CTR model, on the other hand, can be seen as sacrificing community involvement for the sake of rewards on a more personal and individual level. Use of the CTR model indicates a high degree of trust among the peer participants; one of the “individual rewards” is the trust implicit in being allowed to contribute without prior review.

It is possible for both models to be used simultaneously in the same project. For instance, the HTTP Server Project has occasionally adopted guidelines under which it is appropriate to submit bug fixes and small changes under the CTR model, while more sweeping and potentially controversial changes require prior examination and therefore follow the RTC method, with each being adopted only after discussion and at least three people approving the change — and no one vetoing it.

The Review-Then-Commit model ensures that there is ongoing discussion, since it requires interaction for progress to be made. It might be viewed as being better suited for areas in which a lot of people are working together on a specific concept and potentially overlapping each other. If every developer is working on something distinct from every other developer, having to examine code that’s completely unrelated to your own work might become a distraction or an irritant. The CTR approach may be better suited in such cases.

Producing Releases

One thing that may appear odd — and has actually caused confusion — is that releases are not managed as technical issues. In other words, a pending release cannot be stopped in its tracks by a veto. In fact, in most cases it can’t even be stopped by a consensus vote. This is because the release process is under the absolute control of the release manager, who is solely responsible for setting the freeze and release dates and saying whether the release will actually happen at all.

Obviously the release manager must be sensitive to the will of the developers, but he or she has the final say. It can be a pretty stressful operation, so a particular individual may only fill the role once in a while. Sometimes, someone has a particular skill for it, and acts as release manager for more than one release. On the other hand, many people never do it at all. It can be an extremely harrowing role.

1. Making the Most of Commit Hooks with Subversion
2. Using Perl and LDAP
3. Praise for the Humble gedit
4. Looking After Your Bugs with Request Tracker
5. strace: The Friend You Never Knew You Had

Comments via RSS

Sorry, the comment form is closed at this time.