The Apache Incubator

Apache Software Foundation packages don’t materialize out of thin air. Like its very management, projects must follow prescribed rights of passage to earn its membership. The Apache Incubator guides the acolytes.

The “Apache Way” is not a simple concept. Indeed, given the evolutionary and organic growth of the Apache Software Foundation, it’s taken some time for even Foundation insiders to fully grasp its tenets. In fact, it took a global confab (started by Ryan Bloom) to determine just what the ASF’s goals were originally and to codify what the ASF should strive to do in the future.

As an example, according to the spiritual charter of ASF, an individual should progress from a user, to a contributor, to a committer, and finally to an ASF member. However, to the surprise and dismay of a lot of the long-time participants, an alarming number of people saw no value in becoming a member of the Foundation — or worse, considered it a step backward, into bureaucracy and politics instead of coding. Other deviations from the original vision turned up, as well, some concerning fundamental aspects of the Apache way of doing things, such as voting and vetos.

There’s no fault or blame to assign. “Straying” in certain projects was simply a matter of miscommunication, unspoken assumptions, and mismatched expectations. However, the lack of a common vision needed to be resolved– and since the Apache Way is one of the fundamental principles of the ASF, the solution required clarification to bring everything into line.

Things don’t happen quickly in the world of Apache development– at least not procedural things. It was nearly a year after the Great Big Reorganization Discussion (the Discussion) before the means to solve the problem was put in place. The instrument? The Apache Incubator.

Birthing the Incubator

The notion of an ASF project incubator had been raised for several years, but there was no obvious, compelling need for it — until the Discussion revealed the inadvertent schism. What’s odd is that an idea for something like the Incubator had been discussed before, intended to to represent, document, and teach (to new participants) the Apache Way.

At its inception, Apache’s board of directors mandated that all external software projects joining the ASF must come through the incubator. Of course, an obvious question comes immediately to mind: “What about packages started by people who are already involved in Apache?” Unfortunately, that’s a question that’s still somewhat unresolved (at least as this issue of Linux Magazine goes to press.

If the new package is a descendant of, or is logically related to, an existing project, there’s nothing special to be done. The project just starts another codebase and work commences.

If, on the other hand, the package is sufficiently different from any existing project that it deserves to be its own self-contained top-level project (TLP), that’s where things get a little weird. The specific case hasn’t arisen yet; all to-be-TLP packages have either originated outside the ASF or began life under an existing project.

According to the rationale that led to the incubator’s creation, such a package really ought to start life under the wing of the Incubator. After all, one of the Incubator’s tasks is to test and verify the viability of a project’s community, which means that it has both enough people involved to be able to survive the departure of the original team and sufficient diversity that it isn’t a cliquish, closed shop.

To date, the typical case is that of a external package being brought to Apache, along with its pre-existing development communiy. Some packages — such as Nutch and SpamAssassin — were aready open software projects; others, such as Derby[ see the feature story on developing applications with Derby on page XX] were previously proprietary. Given that “adoption” has been the most visible aspect of the Incubator, it makes sense to take a look at how one would actually bring an external package to Apache through the Incubator.

Package Delivery

The first thing to do, of course, is to have offline discussions with Apache staff and measure the merits of the project. It’s entirely reasonable to actually start working on a proposal and hammering out the details through such discourse.

Once it’s been determined — or guessed — that the ASF would be receptive to the package, the next step is to decide where the project would ultimately fit within the ASF (after graduation).

1.The package would become the nucleus of a brand-new top-level Apache project; or

2.The package would become part of an existing TLP.

In the former case, the Incubator (or less commonly, the ASF board of directors) must sponsor the project directly. The alternative is to have an existing project commit to accepting and finding a place for the package after graduation.

Since finding a sponsor means convincing a group of people at Apache that your package fits, the second case — folding a package into an existing TLP — tends to be easier. It’s simpler to convince people who are already familiar with at least the basic technology that yours would make a fine addition.

Technically, the Incubator isn’t supposed to refuse any submissions, but in practice,ones that don’t appear to fit or that have insurmountable issues are strongly discouraged. The alternative is to accept them despite known fatal flaws and have them die in the incubation process — but that does no one any good, and is bad for the reputations and motivation of all parties.

Now, assuming that informal discussions have yielded a place for your package in the Apache milieu and you’ve found a sponsor, the next step is to write a proposal and submit it to the Incubator. The proposal must address a lot of questions, so a sort of “template” has evolved to make the process easier.

The proposal’s whole purpose is to describe what the package is and why it belongs at Apache. Sometimes, proposals are submitted in plain text via email, and sometimes they’re built as pages in the Incubator’s wiki. (The sidebar “Previous Incubator Proposals” has links to five previous proposals that you can crib from.)