04 August 2006

Linus Does Not Scale

One of the darkest moments in the history of free software occurred in September 1998. For perhaps the first - and one hopes the last - time the Linux kernel came perilously close to forking.

The problem was simple: Linus had become a victim of Linux's success. He was unable to cope with the volume of patches that were being sent to him. In the memorable words of Larry McVoy at the time, "Linus does not scale."

That scaling problem was solved by working on a better version control system (what became BitMover, later replaced by the memorably-named Git), as wll as handing off some of Linus's work to others. In the case of the kernel, this could be achieved by mutual agreement, but more generally it is hard to divide up a task among many contributors.

There are now several sites that have sprung up to address this problem. One of them is Amazon's Mechanical Turk, which I wrote about some time back, although I rather missed the key point, which is the use of distributed human intelligence to carry out those kind of tasks that computers presently struggle with. A more recent entrant is Mycroft, discussed in this C|net piece.

Also worth noting is the Crowdsourcing blog, which is a follow-up to the Wired article on the same (and doubtless a feeder to the inevitable book on the subject).

What's interesting about the crowdsourcing idea is that it represents a kind of open source without the openness: that is, participants are essentially computing drones with no way of knowing what the bigger picture is, unlike open source programmers, who can always look at the code. In a sense, then, crowdsourcing is a dilution of the idea at the heart of all the opens, but it's also a broadening in that it enfranchises more or less anybody with basic human processing abilities.

Update: And here's another crowdsourcing blog, called, aptly enough, Crowdsource.

2 comments:

Jonathan Rasmusson said...

Hiya Glyn,

Great post on Linus and the challenges that come with scaling open source projects (like Linux).

I found your comment on crowdsourcing interesting to. I view Open Source as a form of crowdsourcing. There is nothing in crowdsourcing that prevents developers from seeing code.

I agree with your through that crowdsourcing is so broad, that different projects will execute differently.

If you ever write more on crowdsourcing, I would love to hear your thoughts on our company:

www.cambrianhouse.com

We crowd source software ideas at fiece speeds.

I also have a blog at:

www.cambrianhouse.com/blog

All the best - JR

glyn moody said...

Thanks for the information about Cambrian House: what you're doing sounds very interesting.

I suppose it's partly a matter of semantics whether you call it open source or crowdsourcing. Or maybe it's one of scale: I tend to think of crowdsourcing involving many people doing small tasks, whereas typically open source is a few key coders doing most of the heavy lifting, with plenty of crowd-sourcing type support.

I will watch your company's work with interest.