| The User-Developer Convergence: Innovation and Software Systems Development in the Apache Project | ||
|---|---|---|
| Prev | Next | |
1995: The Internet is still predominantly a command line-based medium. While the World Wide Web has been around for the better part of a year, its adoption is still fairly limited. On-line communities almost exclusively use e-mail, Usenet and Internet Relay Chat to communicate. File sharing is predominantly FTP-based. Predating the Web is Gopher. Gopher is a file sharing service in between FTP and the Web. It employs clickable links to download files. In 20/20 hindsight, almost a decade down the line, it may seem obvious the World Wide Web will succeed. In 1995, though, its success is not that obvious. In fact, due to its widespread use, Gopher seems better positioned to become the next generation Internet information system.
The Apache team is distributed across the United States and Europe. Its members use the Internet to communicate. Their primary communications channel is the new-httpd mailing list. They use the mailing list for announcements and discussions about the Web server's development. As e-mail is an asynchronous mode of communication, there are always a number of discussions running in parallel on the mailing list. Threads of discussion overlap in time and topic. At times several threads merge, but more often a single thread forks into several separate sub-threads. In turn, these sub-threads run in parallel, overlapping each other in time and topic, and at times even merging again.
Browsing the mailing list archives may prove confusing. There are discussions within discussions. Threads are dropped and left unfinished, to be picked up the next day, several months later, or never again. While the archives store e-mails on a monthly basis, and Hypermail giving the possibility of sorting messages by either author, subject or date, the sheer wealth of information present makes it difficult to extract interesting data. To isolate the data foundations for this thesis, I choose to present vignettes from the new-httpd mailing list. Each vignette is a limited selection of e-mails bound together by topic and/or time. To preserve the flow of events and illustrate the complexity of issues the developers have to work with, I first present the vignettes with minimal explanation recognizing that many details may be unfamiliar to the reader. Each vignette is then followed by a commentary that provides some background information about the topics discussed and the vignette itself.
We enter the story upon the last day of February 1995, and the new-httpd mailing list participants are in the process of setting up the project that will be known as Apache a few months later. Until now the mailing list participants have merely been a group of disgruntled users of the NCSA Web server. Early morning local time February 28 Robert Thau posts a declaration of independence to new-httpd [THAUMAR95A](Thau, posted on the new-httpd mailing list February 28 1995). It is becoming apparent that the NCSA team is not going to update their Web server, a situation that calls for action. If the NCSA team isn't going to continue developing their server, someone else has to take over.
Thau has only recently gotten involved with the new-httpd crowd. With a Harvard degree in Biology from the mid-eighties, he is at the time a graduate student by MIT's Department of Brain and Cognitive services. He is working on robotics and various sorts of neural network modeling. Thau will in some ways become the Apache group's mad scientist, constantly experimenting with new features and playing with different approaches to solving old problems, exploring the emerging Web technology. For now, though, he is concerned with announcing the new-httpd mailing list's autonomy.
Even though Thau's declaration of independence is the first e-mail archived in the new-httpd mailing list archives, it is obvious from the subject field and its contents that Thau's message is the conclusion of a longer discussion. The e-mail is a summary of the new-httpd crowd's collective sentiments. It is a mix of a declaration of independence and an action plan. At the crux of the matter is the synchronization problem that stems from having no central authority to organize, synchronize and distribute the plethora of patches available for the NCSA Web server release 1.3.
As the NCSA developers seem uninterested in synchronizing the existing patches for 1.3, the people on the new-httpd list will take on the task themselves to create a revised version of the software with all popular patches included. Things have gotten out of control, and the new-httpd participants are taking matters in their own hands. In fact, Thau says that the biggest problem facing the Apache group will be the announced beta of the NCSA Web server version 1.4. Will the new-httpd group have the right for life, a justification for their dissident activities, if the NCSA group finally gears up and releases a new version of their software?However, 1.3 is not a perfect product, and in the absence of further visible development from NCSA, a lot of people have found themselves fixing or extending 1.3 to meet their needs --- in the process fixing the same bugs and deficiencies over and over, at different sites. [THAUMAR95A](Thau, posted on the new-httpd mailing list February 28 1995)
At this stage, though, the project is very much a group of individuals sharing their individual work. People are not sitting around waiting for someone else to take the initiative. The whole effort is, from a software managerial point of view, a complete chaos where anyone is more or less free to do whatever pleases them. Things are so uncoordinated that even active participants on the mailing list are having problems keeping track of the activity. There is no control over what is being worked on, and who is doing what. There are even several servers being used as patch repository independent of each other. Robert Thau's work is based around his own site where he collects patches, while Cliff Skolnick has announced the hyperreal.com FTP site as central repository. Despite this, which has its obvious problems, the project seems to be progressing nicely.
The amount of work organizing the patches is quite massive. Often several patches solve the same problem in different ways and depending of different combinations of preceding patches. This makes interdependency between patches at best unclear, at worst mutually exclusive. As new patches rely on the combination of patches applied to the code, the situation is becoming increasingly complicated by the day. After a fortnight Cliff Skolnick posts to new-httpd:
Skolnick is a former Sun Microsystems employee and long-time low-level Unix/C programmer. Together with Brian Behlendorf he founded Organic Online, a San Francisco Web design consultancy, the year before. They had found the only real Web server alternative, the NCSA server, lacking in both functionality and stability. Trying to improve on this, they became the main driving force behind setting up the new-httpd mailing list. Robert Thau proposed a loose process in his declaration of independence, but now Skolnick finds the team in the same bind as previous to setting up the mailing list, a situation he wants to change.OK. I think we are getting ourselves into a bind here and loosing track of which patch does what, which is the latest, etc. I think we needs some formal process (ick) for voting patches in or out and stuff. More importantly we need to link the patch reports and bug reports to the discussions somehow, which will not be an easy trick. [SKOLNICKMAR95A](Skolnick, posted on the new-httpd mailing list March 15 1995)
In his e-mail, Skolnick proposes a three phased process to deal with the problem. The first phase consists of creating a patch. All new patches are assigned a unique identification number. Once the patch has been submitted, it enters the second phase which consists of peer review and bug fixing. Editing a patch is handled centrally, and every day a change log is distributed to show the past 24 hours' activity. Bug reports are posted to the mailing list. When posting a bug report the affected patch's identification number must be included for reference, and only one patch is allowed pr. bug report. This is done for easier retrieval of patch information from the mailing list archives. Each week a round of votes will be made to determine which of the finished patches shall go into the main code base. Once a patch passes the vote, it enters the third phase: integration. It is now integrated with the main code base. Even though not explicit in Skolnick's e-mail, it is implied that integration is handled by a designated individual who cleans up the code and does a proper integration job. Skolnick's central argument for such a process is that it is "a big part of software engineering as opposed to hacking" [SKOLNICKMAR95A](Skolnick, posted on the new-httpd mailing list March 15 1995).
Robert Thau doesn't completely agree. While he agrees with a minimum of traceability in patches and that peer review is important, he is worried that the project will loose momentum. Based on his experience with the NCSA developers, Thau argues the results of too slow a development organization. Having learnt the importance of rapid updates from his own situation with the NCSA group, he does not want the Apache to get into a situation where every minor bug fix has to go through the review process, impeding the update frequency. He is willing to forsake some of the advantages of an administrative system for the sake of speed "… many of the items on the functional enhancements list will take some discussion to decide if we want them, and in what form, and I don't think those discussions should be allowed to hold up the project" [THAUMAR95B](Thau, posted on the new-httpd mailing list March 9 1995). Rob Hartill supports Thau's sentiments: "Our goal as I see it at the moment is to maintain some momentum in the process of discussing ideas and adding them to what was NCSA 1.3" [HARTILLMAR95A](Hartill, posted on the new-httpd mailing list March 21 1995).
The actual development organization to emerge from these initial exchanges of opinions is somewhat of a compromise. Rob Hartill, proponent of the need for speed, is actually the first to make use of the voting system [HARTILLMAR95B](Hartill, posted on the new-httpd mailing list March 15 1995), and even Robert Thau agrees to the need for patch identification [THAUMAR95C](Thau, posted on the new-httpd mailing list March 15 1995). The rest of Skolnick's proposal is never found necessary, and therefore never effectuated. The project does not seem to suffer in momentum from this introduction of a more rigid process, though. Within less than two weeks of the declaration of independence Cliff Skolnick announces his plans for releasing a first alpha version of the Apache Web server [SKOLNICKMAR95B](Skolnick, posted on the new-httpd mailing list March 12 1995). A week after that Rob Hartill has the first pre-release of Apache ready for the group [HARTILLMAR95C](Hartill, posted on the new-httpd mailing list March 19 1995).
The patch is central to this vignette. A patch or a patch file is the difference listing produced by the diff program. diff compares the contents of two files. One file is the base which changes to the second file are found. The output of diff can be stored in a file. This file is called a patch, so called because its contents can be used to update a copy of the base file with the tool named patch. A patch is relative to the contents of the base file. Herein lies the problem facing the Apache developers. The situation is better explained using an example.
Say we have a file, F. Brian finds the bug B1 in F, and fixes it. This revised file is called F1. Using diff he creates a patch P1 that he shares with fellow developer Cliff. Robert finds another bug B2 in F, and fixes it. Let's denote Robert's revision F2. Using diff he produces the patch P2. Cliff has applied patch P1 to F. Working with F1 he realizes there is yet another bug in the the code. Cliff fixes the bug resulting in a new revision F3. Using diff, Cliff produces patch P3 that he shares with Brian and Robert.
There are now three revisions of the original file. F3 relies on the revision F1, while both F1 and F3 stem directly from the original file F.
The problem arises when Robert wants to apply the changes contained within P3 to his F2. He first has to apply P1. With trivial code, this is really not a problem in itself. The problem arises when P1 and P2 have changes in a common segment of lines. These conflicts will have to be resolved by merging the contents of the two patches. This usually has to be done by hand. Another problem would be if Robert wants the changes from P3 but not P1. Such a situation may prove a bit more tricky, and made even worse if the code is complex or difficult to understand. The task is at best time-consuming, usually fraught with pit-falls, and in worst case impossible to follow through as P3 relies too deeply on P1.
The Apache team experienced as situation where numerous patches had inter-relationships that were difficult to chart. For them it was becoming next to impossible to share the many patches to solve the original NCSA code's many bugs.