The Empire (No Longer) Strikes Back

One of the most worrying moments in recent open source history was when it became clear that Microsoft was determined to wrench away Apache's crown as top Web server. This began in early 2006, and was soon showing dramatic results, as the April 2006 Netcraft survey commented:

This month's survey brings one of the largest one-month swings in the history of the web server market, as Microsoft gains 4.7 percent share while Apache loses 5.9 percent. The shift is driven by changes at domain registrar Go Daddy, which has just migrated more than 3.5 million hostnames from Linux to Windows. Go Daddy, which had been the world's largest Linux host, is now the world's largest Windows Server 2003 host, as measured by hostnames. The company said it will shift a total of 4.4 million hostnames to Windows Server 2003.

This was a staggering shift, and I feared it might presage a real effort by Microsoft to achieve a major PR win. Things reached their nadir in September 2007:

Apache gains over 3 million hostnames, and around 0.9 million active sites this month. But this is not enough to prevent its market share declining closer to the 50% mark, as Microsoft also gained over 3 million hostnames (a large part of which come from MySpace and Live Spaces, both of which use its Internet Information Server.

At that time, the gap between Apache and Microsoft's IIS was just 15%, down, from 50% just a couple of years earlier.

But since then, Apache has gradually pulled ahead; today the gap is around 18% - still far smaller than it once was, but increasing. I feel that the danger has passed, not least because Microsoft has realised that it was fighting yesterday's battles.

Tomorrow's fight will be about owning the cloud, and the main threat there is not so much Apache, as customised versions of open source software, of the kind employed by Google for its vast server farms: in the latest Netscape survey, Google has around 5% of the Web server market. It's still open vs. closed, but not as we know it.

The crucial point is that Microsoft failed to displace Apache, despite its almost limitless resources. This is the crucial lesson for the future, more important than any particular percentage market share: that Microsoft's attacks can - and have been - beaten off.

Open Cloud Conundrum, Open Cloud Consortium

One of the hot areas in 2008 was cloud computing, and 2009 looks likely to be a year that is equally occupied with the subject. But cloud computing represents something of a conundrum for the open source world.

On Open Enterprise blog.

The Real Story Behind GNU/Linux

If the prospect of another week stretching out before you is getting you down, I've got good news. There's a post about GNU/Linux that is guaranteed to bring a smile to your face. It's a real stonker - try this for a start....

On Open Enterprise blog.

What Comes After “Embrace, Extend”?

Here are two small, but significant moves by Microsoft....

On Open Enterprise blog.

Openness is the Solution to the (Double) Subprime Crisis

As I listen to all this talk of lack of trust in the banking system, of inflated values ungrounded in any reality, of “opacity”, and of “contaminated” financial instruments, I realise I have heard all this before. In the world of software, as in the world of finance, there is contamination by overvalued, ungrounded offerings that have led to systemic mistrust, sapped the ability of the computer industry to create real value, and led it to squander vast amounts of time and money on the pursuit of the illusory, insubstantial wealth that is known as “intellectual property”....

On Linux Journal.

The Olympian Apache

Here's a nice reminder that open source - in the form of Apache - has been head of the field for more than 12 years, despite what certain companies would have us believe:

In 1996 the World Wide Web was truly in its very early stages. The Olympics took place less than a year after Netscape went public, which many consider the key event marking the transition of the Internet from a research network used primarily by the technical community to the commercial behemoth that it went on to become.

The new World Wide Web had the feeling of magic, but, in 1996, it was pretty primitive magic. To begin with, the vast majority of people accessing the Web at the time were doing so over slow dial-up modems with bandwidths of 56 kilobits per second or less. Only at work, if you were lucky, did you have access to faster broadband speeds. It wasn't until years later that broadband usage in the home became commonplace.

As we were planning the IT infrastructure for the Olympics website, hardware was not an issue. We used IBM's SP family of parallel supercomputers which we were confident would provide us with all the computing power we could want.

But the software for web servers was quite immature. Netscape's web software was the most widely used in those days, and while it was adequate for small workloads, its scalability was suspect. We could not use it. Instead, we used the open source Apache Server as the basic web server, and custom built the extensions needed to support its content, applications and other capabilities.

We were pretty sure that the Atlanta Olympics website was the largest such web project anyone had undertaken so far. Because it was all so new, we did not know how many people would come to our website and what features they would use once they got there. We were well aware of the considerable risks inherent in doing such a complex, new project on such a global stage. We knew, for example, that beyond a certain number of users, the response time would start to degrade, and if sufficiently stressed beyond its capabilities, the system could become unstable and crash.

...

Our Olympics website worked quite well, except for some unduly slow response times when traffic got very heavy. Overall, the site handled 187 million hits – that is, individual pieces of information served to users. We learned a lot about the requirements for building and operating large, complex websites. All in all, it was a very successful experiment.

Big Blue is Back

On Open Enterprise blog.

Should We Fear the (Microsoft) Geeks, Bearing Gifts?

On Open Enterprise blog.

Is Apache the Greatest Open Source Project?

On Open Enterprise blog.

Apache on the Up

Not much, mind you, but given that I commented when it was on the way down, it's only fair to point out moves in the other direction:

Apache continues its recovery after steep falls in share over the last eighteen months and is back over 50%. Its share had been negatively affected over that period by the increasing number of blog sites in the survey on large providers like Microsoft and Google, using their own server software. But it is also benefiting from growth at other blog providers like multiply.

Google's Secret Weapon

In Redmond Magazine (no, really.)

Open Source Unoriginal? - How Unoriginal

Here's a tired old meme that I've dealt with before, but, zombie-like, it keeps on coming back:

The open-source software community is simply too turbulent to focus its tests and maintain its criteria over an extended duration, and that is a prerequisite to evolving highly original things. There is only one iPhone, but there are hundreds of Linux releases. A closed-software team is a human construction that can tie down enough variables so that software becomes just a little more like a hardware chip—and note that chips, the most encapsulated objects made by humans, get better and better following an exponential pattern of improvement known as Moore’s law.

So let's just look at those statements for a start, shall we?

There is only one iPhone, but there are hundreds of Linux releases.

There's only one iPhone because the business of negotiating with the oligopolistic wireless companies is something that requires huge resources and deep, feral cunning possessed only by unpleasantly aggressive business executives. It has nothing to do with being closed. There are hundreds of GNU/Linux distributions because there are even more different kinds of individuals, who want to do things their way, not Steve's way. But the main, highly-focussed development takes place in the one kernel, with two desktop environments - the rest is just presentation, and has nothing to do with dissipation of effort, as implied by the above juxtaposition.

chips, the most encapsulated objects made by humans, get better and better following an exponential pattern of improvement known as Moore’s law

Chips do not get better because they are closed, they get better because the basic manufacturing processes get better, and those could just as easily be applied to open source chips - the design is irrelevant.

The iPhone is just one of three exhibits that are meant to demonstrate the clear superiority of the closed-source approach. Another is Adobe Flash - no, seriously: what most sensible people would regard as a virus is cited as one of "the more sophisticated examples of code". And what does Flash do for us? Correct: it destroys the very fabric of the Web by turning everything into opaque, URL-less streams of pixels.

The other example is "the page-rank algorithms in the top search engines", which presumably means Google, since it now has nearly two-thirds of the search market, and the page-rank algorithms of Microsoft's search engine are hardly being praised to the sky.

But what do we notice about Google? That it is built almost entirely on the foundation of open source; that its business model - its innovative business model - would not work without open source; that it simply would not exist without open source. And yes, Yahoo also uses huge amounts of open source. No, Microsoft doesn't, but maybe it's not exactly disinterested in its choice of software infrastructure.

Moreover, practically every single, innovative, Web 2.0-y start-up depends on open source. Open source - the LAMP stack, principally - is innovating by virtue of its economics, which make all these new applications possible.

And even if you argue that this is not "real" innovation - whatever that means - could I direct your attention to a certain technology known colloquially as the Internet? The basic TCP/IP protocols? All open. The Web's HTTP and HTML? All open. BIND? Open source. Sendmail? Open source. Apache? Open source. Firefox, initiated in part because Microsoft had not done anything innovative with Internet Explorer 6 for half a decade? Open source.

But there again, for some people maybe the Internet isn't innovative enough compared to Adobe's Flash technology.

A Bit of a Shindig

One of the great things about open standards is that anyone can implement them - including those in the free software world. An obvious candidate for this treatment is the new OpenSocial set of APIs from Google, and here's an Apache project doing just that:

Shindig will provide implementations of an emerging set of APIs for client-side composited web applications. The Apache Software Foundation has proven to have developed a strong system and set of mores for building community-centric, open standards based systems with a wide variety of participants.

A robust, community-developed implementation of these APIs will encourage compatibility between service providers, ensure an excellent implementation is available to everyone, and enable faster and easier application development for users.

The Apache Software Foundation has proven it is the best place for this type of open development.

The Shindig OpenSocial implementation will be able to serve as a reference implementation of the standard.

What Does This Mean for NetBeans?

NetBeans has always been something of a mystery to me. I'd always regarded it as the runner-up IDE for Java, after Eclipse. But it's clear that I'm behind the times:

Netbeans 6.1 will have plugin support for creating, editing, deploying to Apache HTTPD, running and even debugging PHP projects.

And according to the NetBeans site:

You get all the tools you need to create professional desktop, enterprise, web and mobile applications, in Java, C/C++ and even Ruby.

Add in PHP, and that's increasingly impressive, but it does beg the question: Do we really need another all-purpose IDE alongside Eclipse? Doesn't that just dissipate the effort? Answers on the back of a postcard. (Via Tim Bray.)

Yahoo! Goes Whoop! About Hadoop! (and Pig!)

Now why on earth would Yahoo be doing this?

Yahoo! Inc., a leading global Internet company, today announced that it will be the first in the industry to launch an open source program aimed at advancing the research and development of systems software for distributed computing. Yahoo!'s program is intended to leverage its leadership in Hadoop, an open source distributed computing sub-project of the Apache Software Foundation, to enable researchers to modify and evaluate the systems software running on a 4,000 processor supercomputer provided by Yahoo!. Unlike other companies and traditional supercomputing centers, which focus on providing users with computers for running applications and for coursework, Yahoo!'s program focuses on pushing the boundaries of large-scale systems software research.

Currently, academic researchers lack the hardware and software infrastructure to support Internet-scale systems software research. To date, Yahoo! has been the primary contributor to Hadoop, an open source distributed file system and parallel execution environment that enables its users to process massive amounts of data. Hadoop has been adopted by many groups and is the software of choice for supporting university coursework in Internet-scale computing. Researchers have been eager to collaborate with Yahoo! and tap the company's technical leadership in Hadoop-related systems software research and development.

As a key part of the program, Yahoo! intends to make Hadoop available in a supercomputing-class data center to the academic community for systems software research. Called the M45, Yahoo!'s supercomputing cluster, named after one of the best known open star clusters, has approximately 4,000 processors, three terabytes of memory, 1.5 petabytes of disks, and a peak performance of more than 27 trillion calculations per second (27 teraflops), placing it among the top 50 fastest supercomputers in the world.

M45 is expected to run the latest version of Hadoop and other state-of-the-art, Yahoo!-supported, open-source distributed computing software such as the Pig parallel programming language developed by Yahoo! Research, the central advanced research organization of Yahoo! Inc.

It's cool that Yahoo's backing the open source Hadoop, and doubly cool that one of the projects is called Pig. But it's also shrewd. It's becoming abundantly clear that open beats closed; Google, for all its use of open source software, is remarkably closed at its core. Enter Hadoop, running on a 4,000 processor supercomputer provided by Yahoo, with the real possibility of spawning a truly open rival to Google.... (Via Matt Asay.)

Apache and the Art of the Press Release

Here's some interesting commentary on my recent post about Apache's declining market share in the Netcraft survey:

The Netcraft numbers are changing for one reason and one reason only: because a very large and powerful entity is doing whatever they can to change those numbers, even if it means creating millions of bogus sites. Even if it means paying registrars large sums of cash to move their parked domains over to IIS. Anything. It is of prime importance for them to be able to "beat" Apache, and we are seeing the result. People aren't switching to IIS. Companies aren't switching to IIS. Hosts aren't switching to IIS. At least not for technical reasons. MS needs this marketing success. It needs to "prove" that IIS is beating Apache and, by logical conclusion, MS is better than Open Source. Can't we all already predict what the press release will say? So with something so important on the line, and with a survey that can now be easily fudged, the battle call is "Change Netcraft!"

Of course, the graph itself makes it clear that on a certain day Microsoft decided: we will overtake Apache, cost what it may. But my point stands: whatever dirty tricks Microsoft may use to achieve that goal, it doesn't matter - it's too late.

The same post also links to this alternative web survey, by Security Space, where Apache still dominates utterly.

Is Apache About to Get Scalped?

Certainly looks like it:

Apache loses 2.8% share this month, partly through the strong growth at the major blogging systems, and partly due to 2.5 million domains on Apache expiring at trouble-free.net. Apache has around a 10% market share advantage over IIS now, which is the smallest gap between the two since IIS was launched in 1996.

The only consolation is that Apache's job is done: it has the shown the way. Today, if necessary, we can live without Apache as an example of how Microsoft can be beaten in a market, because the total open source story is now so strong.

The Penguin Goes to Redmond

Well, to Redmond Magazine, that is....

After Netcraft

For over a decade, it has been a point of faith that Apache is not only a better Web server than Microsoft's IIS, but that this is demonstrable: the Netcraft survey of public Web servers shows that Apache has been consistently ahead.

Alas, for a variety of reasons - not least Microsoft's determination to reduce the gap, whatever the cost - Apache's lead is falling. So it's good to have this new survey that re-asserts Apache's superiority, and adds a nice extra twist:

Linux websites have better uptime and load faster than Windows-based websites. Research by WatchMouse, a website monitoring company, also shows that web server platform Apache outperforms the Microsoft IIS platform. Therefore, having a Linux website and an Apache webserver platform offers the best choice for professional web pages.

I feel another Microsoft-funded piece of research on its way....

Google Points Finger at Microsoft IIS

Interesting bit of shin-kicking here:

Web sites running Microsoft Corp.'s Web server software are twice as likely to be hosting malicious code as other Web sites, according to research from Google Inc.

Last month, Google's Anti-Malware team looked at 70,000 domains that were either distributing malware or hosting attack code. "Compared to our sample of servers across the Internet, Microsoft IIS features twice as often as a malware-distributing server," wrote Google's Nagendra Modadugu, in a Tuesday blog posting.

Together, IIS (Internet Information Services) and Apache servers host about 89 percent of all Web sites, but collectively they're responsible for 98 percent of all Web-based malware. Google actually found an equal number of Apache and IIS Web sites hosting malicious software, but because there are so many more sites hosted by Apache servers (66 percent versus Microsoft's 23 percent) malicious sites make up a much larger percentage of all IIS servers.