Crisis, chrysalis, Shambhala

Late April, the Apache project is scant two months old, but cracks are starting to appear. "You can add honey to shit, but you only get sweet tasting shit", Cliff Skolnick vents his frustration after a long night of hacking [SKOLNICKAPR95C](Skolnick, posted on the new-httpd mailing list April 26 1995). It is a few minutes past 5 in the morning when Skolnick sends his e-mail. "I'm getting more and more frustrated trying to wedge stuff into a large program that was not well architected … I am not proud of the patches I need to do, but there is no clean easy was of integrating the junk". He is not the only one frustrated with the turn of events: "Some time soon we'll need to take a look at redesigning Apache completely", Andrew Wilson adds [WILSONAPR95A](Wilson, posted on the new-httpd mailing list April 26 1995). It is apparent that new features are becoming increasingly harder, to the point of impossible, to integrate with the existing code base. "It's [the Web server] been growing organically and while individual new components are well designed they *are* bolted on to a rather mixed up program that could benefit from a comb and cut" [WILSONAPR95A](Wilson, posted on the new-httpd mailing list April 26 1995).

Since release 0.5 on April 8, getting sufficient votes is becoming increasingly harder for each release. Comes May, and the voting system starts breaking down as there is simply not sufficient activity for anything substantial to be voted on. Hartill is able to gather sufficient votes for release 0.6.3, but upon starting work on release 0.6.4 in mid-May it is becoming increasingly apparent that he is fighting an uphill battle. Hartill is determined to follow through with new releases of the Web server. By now there are simply no new patches to vote on. Brian Behlendorf later describes Hartill's fight as "shoving an elephant down the eye of a needle" [BEHLENDORFJUN95A](Behlendorf, posted on the new-httpd mailing list June 30 1995). The lack of patches doesn't reflect the Web server's state. The server has serious technical problems.

Early May sees the release of Apache 0.6.3, but the release is later skipped because an incompatibility problem with W3C's Arena browser. The browser is not able to properly content negotiate with Apache 0.6.3 [THAUMAY95A](Thau, posted on the new-httpd mailing list May 5 1995). Then a problem is discovered in with handling of CGI scripts in an earlier release, 0.6.2. The issue sorely needs fixing, but nobody rises to the occasion [HARTILLMAY95A](Hartill, posted on the new-httpd mailing list May 11 1995). 0.6.2 is in turn discarded in mid-May due to this unresolved issues. A new bug-fix release, 0.6.4, is released in mid-May. It is meant to resolve the situation. Yet, the problems are not over. In quick succession bug fix versions 0.6.4b through d are released over the next few days. Amidst all this, there are still unresolved issues like the XBITHACK, unresolved since early April[BEHLENDORFAPR95C](Behlendorf, posted on the new-httpd mailing list April 7 1995), and no decision has been made as to Apache's non-forking behavior.

A statistic analysis of new-httpd speaks clearly of the problems. The volume of traffic on the mailing list is halved between April and May. By June the traffic is only one fourth that of April. Activity is at an all time low, and traffic on the new-httpd mailing list will never again drop this far. By mid-May development activity seems to have ceased almost completely. Hartill is practically keeping the project alive single-handedly. He alone is responsible for 26% of the mailing list's total traffic, having started 44% of all threads.

Throughout May and June Rob Hartill is trying to sort out the problems with the release 0.6.x series on his own, while also working on a new release, Apache release 0.7. The release cycle slows down as people stop voting. It is obvious that the Apache development effort is quickly going stale. Within less than 6 months after breaking free from the NCSA group, Apache seems fast heading into the same bog of problems with infrequent updates as the NCSA team suffered under. When Apache 0.7 is released [HARTILLMAY95B](Hartill, posted on the new-httpd mailing list May 29 1995), it is the result of Rob Hartill's hard work. The development effort is no longer a group effort. Looking back Randy Terbush recalls that "Rob Hartill has … done a great job of biting the bullet and pulling 0.7 into a publicly consumable form." [TERBUSHJUN95A](Terbush; posted on the new-httpd mailing list June 30 1995). Yet, the 0.7 release cycle is not without its own share of problems [BEHLENDORFJUN95B] [COXJUN95A] [WATKINSJUN95A](Behlendorf, posted on the new-httpd mailing list June 6 1995; Cox, posted on the new-httpd mailing list June 7 1995; Watkins, posted on the new-httpd mailing list June 8 1995).

Mid-June, and Robert Thau shares his garage project with the rest of the team:

Over the past few weeks, I've been off on a garage project to explore some possible new directions I thought *might* be useful for the group to pursue. … What I have right now is a server I'm calling Shambhala. … The basic idea of Shambhala is to make a modular "tinkertoy" server, to which people can easily add code which is valuable to them (even if it isn't universally useful) without hairing up a monolithic server. [THAUJUN95A](Hartill, posted on the new-httpd mailing list June 13 1995)

Not only is Thau trying to modularize the Web server, his rewrite even takes measure to remove some of the old error sources. Thau's code is still missing a great deal of Apache's extra functionality, and it is fairly unstable. A week later, and the Shambhala code is both more stable and has some more functionality added.

Working in parallel with Thau's Shambhala project, is an effort to make the existing code more stable. 0.7.2 is released first week of June, only to suffer the same fate as 0.6.4. A series of rapid bug fix releases a through k are unleashed over the next two weeks. Lead by Hartill, work on the original code is facing serious problems: "I just grabbed a copy of 0.7.2d, compiled it under Solaris 2.4 w/ gcc and it still seems to have a problem I've seen before - random links, usually images, hang, and the error log shows an socket error and then dead children", Ryan Watkins reports [WATKINSJUN95A](Watkins, posted on the new-httpd mailing list June 8 1995). The server's memory footprint is too large [HARTILLJUN95A](Hartill, posted on the new-httpd mailing list June 16 1995). Trying to fix it a new problem arises: "well the suckers are down to 260k each at startup (was 548k), so that's promising.. trouble is that a SIGHUP (restart) causes the new children to die when servicing any request" [HARTILLJUN95B](Hartill, posted on the new-httpd mailing list June 17 1995).

After two weeks of frequent bug fixes to the 0.7.2 release, 0.7.2k is abandoned as it still does not handle redirection correctly. The next day Rob Hartill releases 0.7.3, which fixes the redirection problem. Later the same day release 0.7.3 is already at bug fix release c. There seems to be no end to the problems. "Apparently there is an fd leak happening somewhere... and killing the children doesn't seem to free them, only killing the parent", Brian Behlendorf reports [BEHLENDORFJUN95B](Behlendorf, posted on the new-httpd mailing list June 20 1995). "I've had a quick look at apache 0.7, and it sees to suffer from a major design flaw, namely that it always starts up a fixed number of processes." David Robinson reports [ROBINSONJUN95A](Robinson, posted on the new-httpd mailing list June 26 1995).

Interest in Apache is gone: "Feature development on Apache 0.7 has been static … I don't remember anyone reporting that they were working on any ideas/code.", Rob Hartill sighs [HARTILLJUN95C](Hartill, posted on the new-httpd mailing list June 30 1995). Despite its unfinished state, an increasing number of people are moving from Apache to Shambhala. The excitement is returning:

Has anyone else in the group taken a look at RST's Shambhala? The code is ultra clean. Truely non-forking. Predictable. Fast! [TERBUSHJUN95B](Terbush, posted on the new-httpd mailing list June 29 1995)

Suddenly the mailing list bursts into activity. Rob Hartill is wondering if this means the demise of the Apache project [HARTILLJUN95C](Hartill, posted on the new-httpd mailing list June 30 1995). At the very least he wants to know how Shambhala relates to the Apache project [HARTILLJUN95D](Hartill, posted on the new-httpd mailing list June 30 1995). Over the next few weeks the original code base that Hartill has put an enormous amount of effort into, is scrapped and the Shambhala code replaces it as the official Apache code from release 0.8. For a while Thau replaces Hartill as the project's central authority. Unlike Hartill, Thau doesn't have to pull the project by himself. Even though Thau decides what goes into the Shambhala code and what doesn't, a great number of developers contribute with bug fixes and several developers are porting their features to this new modular architecture.