With dozens of software projects involving hundreds of developers, keeping data flowing smoothly is an involved process for the Apache Software Foundation (ASF). With tens of machines distributed worldwide, gigabytes of daily downloads, and fifty hits per second on the Apache home page, system maintenance requires the varied skills of a small legion of volunteers. In the fourth in an ongoing, exclusive series, ASF co-founder Ken Coar pulls back the curtain to reveal how it all works.
The Apache Software Foundation, or ASF, is a non-profit organization chartered with creating freely-available software for the public good. But the ASF represents much more than that, though. The group is also an embodiment of a philosophy of collaboration, often called the “Apache Way.”
With dozens of software projects involving hundreds of developers, keeping data flowing smoothly is an involved process. The ASF looks both inward and outward, but focuses on different things according to the direction. When looking outward, the emphasis is on how the Foundation interacts with the rest of the world, safeguarding the Apache brand, participating in various standards committees, and so on. Looking inward, the focus is on helping all the participants do what they do — write code.
As the ASF has grown and its needs become apparent, individuals have stepped-up to meet the challenges and eventually coalesced into teams or groups devoted to a purpose. For instance, the Public Relations Committee is concerned with things like interactions with the press, fielding requests to use the Apache marks, and so on. There’s also the Infrastructure group, a team of overworked volunteers who manage the systems, the network, the software repositories, and all of the Web sites and other bits that allow the Foundation to get its work done.
There are actually two kinds of web sites associated with the Foundation. There’s the kind tied to a specific project, such as http://httpd.apache.org/ (for the Apache HTTPD Server) or http://tomcat.apache.org/ (for Tomcat), and there’s the kind which has no such specificity– like the main Web site at http://apache.org/ or the package download mirror system. The infrastructure team is responsible for seeing that all of the sites are accessible, but isn’t responsible for the content of those sites.
The infrastructure team is responsible for all of the sites, to various degrees. For each project-specific site, the infrastructure staff makes sure that the site is functional and can be updated by the members of the project involved. For the non-project sites, the infrastructure group has primary responsibility for both content and operation.
Tending the Code
Probably one of the most important responsibilities of the infrastructure team is keeping the source management systems in operation. Since the primary “product” of the ASF is software, source code, and documentation, it stands to reason that the continuity and integrity of the sources is of utmost concern. For years the ASF used the Concurrent Versioning System (CVS), but that package has some known deficiencies, but recently switched to Subversion (SVN, http://subversion.tigris.org/). (A number of people from ASF became involved with SVN while it was being developed, and provided valuable input to make Subversion better than CVS. This has proven quite beneficial: as the ASF moved from CVS to SVN, it had experts in the new software available “in house.”)
In the beginning, the infrastructure needs of the Apache development organization were quite modest: a web site, a mail server, and a software repository. Originally, all of these services lived on a single computer, sharing the system with a number of unrelated domains and handled by one of the Apache developers. As time passed and the various service loads increased, the Apache domain moved to its own dedicated hardware, and then to multiple machines, and most recently to systems that are geographically distributed. At the time I’m writing this, there are more than a dozen sizeable systems devoted solely to the Foundation.
The need for this much horsepower becomes understandable when you realize that the Apache infrastructure supports sixteen server systems; more than three hundred mailing lists; over one million email messages per day; over two dozen top-level projects; dozens of ASF project web sites; more than a thousand developers working on dozens of packages; fifty hits on the web site per second (more than four million per day); over one hundred gigabytes downloaded from the Web site per day; repositories containing over fifteen gigabytes of code and documentation; and hundreds of thousands of revision records. (This doesn’t even touch on the world-wide mirroring system. More on that in a bit.)
You can see the size and activity of some of these operations http://people.apache.org/~coar/mlists displays information about the Foundation’s public mailing lists, the number of people subscribed, and graphs of activity. http://people.apache.org/~henkp/ shows various collections of information about the relationships and activities within the Foundation. http://monitoring.apache.org/status/ is the Nagios page used by the infrastructure team to track system status. This system also automatically notifies the team in real time of incidents such as “system down.” http://svn.apache.org/viewcvs/ provides a view into the living repositories on which those hundreds of developers are busily working.
Graphs and charts are great for showing trends at a glance. For example, Figure One shows the count of committers over time, and Figure Two pictures ASF membership over time.