Open Source's (Not-so-)Secret Sauce: Modularity

Why does open source succeed? Apart, that is, from the fact that it is created by huge numbers of amazingly clever and generous people? Or, put another way, what is a key ingredient that must be present for the open source methodology to be applicable to other spheres?

Modularity.

If the stuff to hand isn't modular, you can't really share, because your stuff isn't compatible with other people's stuff. If it isn't modular, you can't share out tasks and scale. If you can't share out tasks, you can't have people working independently, at their own pace and in their own way, which means the project isn't really open. If it isn't modular, you can't swap in some new elements while leaving everything else untouched, which means no "release early, release often", no experimentation, no rapid evolution. Modularity is indispensable.

I think that's why open source hardware has singularly failed to take off. It's difficult to make bunches of atoms modular in the way that bunches of bits are (at least until we have general 3D printers, in which case we're done...)

But could there be a way of introducing that modularity at a higher level so as to enjoy the benefits outlined above? I do believe there is, and with hindsight, it was pretty obvious (er, so why didn't I think of it?). It's called OpenStructures:

The OS (OpenStructures) project explores the possibility of a modular construction model where everyone designs for everyone on the basis of one shared geometrical grid. It initiates a kind of collaborative Meccano to which everybody can contribute parts, components and structures.

As you can see, the clever people behind this project have the magic word "modular" in there. Specifically, they have devised a very simple grid system that ensures that things fit together, even when they're made by different people at different times and for different purposes. Significantly, the grid is based on binary multiples and subdivisions:

If you choose to apply the OS grid for the dimensions of a part, at least one of the measurements of this part (length, wideness and thickness or height) should correspond to either 0,125cm / 0,25cm / 0,5cm / 1cm / 2cm and multiples of 2cm in order to be compatible with other parts. (see part examples)

What's really impressive about this project is not just this insight into the modularity of elements, but the completeness of the vision that results. For example, there is an explicit hierarchy of elements, starting from OS Parts, which combine to form OS Components, from which are made OS Structures, and finally OS Superstructures.

It's an amazing vision, and I think it could have a major impact on the world of open source hardware, at least of this particular construction-set type. If you want to see some of the exciting objects that have already been created, don't miss the fab photos on the project's blog. (Via @opensourcerer.)

Follow me @glynmoody on Twitter or identi.ca.

Open Knowledge, Open Greenery and Modularity

On Saturday I attended the Open Knowledge 1.0 meeting, which was highly enjoyable from many points of view. The location was atmospheric: next to Hawksmoor's amazing St Anne's church, which somehow manages the trick of looking bigger than its physical size, inside the old Limehouse Town Hall.

The latter had a wonderfully run-down, almost Dickensian feel to it; it seemed rather appropriate as a gathering place for a ragtag bunch of ne'er-do-wells: geeks, wonks, journos, activists and academics, all with dangerously powerful ideas on their minds, and all more dangerously powerful for coming together in this way.

The organiser, Rufus Pollock, rightly placed open source squarely at the heart of all this, and pretty much rehearsed all the standard stuff this blog has been wittering on about for ages: the importance of Darwinian processes acting on modular elements (although he called the latter atomisation, which seems less precise, since atoms, by definition, cannot be broken up, but modules can, and often need to be for the sake of increased efficiency.)

One of the highlights of the day for me was a talk by Tim Hubbard, leader of the Human Genome Analysis Group at the Sanger Institute. I'd read a lot of his papers when writing Digital Code of Life, and it was good to hear him run through pretty much the same parallels between open genomics and the other opens that I've made and make. But he added a nice twist towards the end of his presentation, where he suggested that things like the doomed NHS IT programme might be saved by the use of Darwinian competition between rival approaches, each created by local NHS groups.

The importance of the ability to plug into Darwinian dynamics also struck me when I read this piece by Jamais Cascio about carbon labelling:

In order for any carbon labeling endeavor to work -- in order for it to usefully make the invisible visible -- it needs to offer a way for people to understand the impact of their choices. This could be as simple as a "recommended daily allowance" of food-related carbon, a target amount that a good green consumer should try to treat as a ceiling. This daily allowance doesn't need to be a mandatory quota, just a point of comparison, making individual food choices more meaningful.

...

This is a pattern we're likely to see again and again as we move into the new world of carbon footprint awareness. We'll need to know the granular results of actions, in as immediate a form as possible, as well as our own broader, longer-term targets and averages.

Another way of putting this is that for these kind of ecological projects to work, there needs to be a feedback mechanism so that people can see the results of their actions, and then change their behaviour as a result. This is exactly like open source: the reason the open methodology works so well is that a Darwinian winnowing can be applied to select the best code/content/ideas/whatever. But that is only possible when there are appropriate metrics that allow you to judge which actions are better, a reference point of the kind Cascio is writing about.

By analogy, we might call this particular kind of environmental action open greenery. It's interesting to see that here, too, the basic requirement of modularity turns out to be crucially important. In this case, the modularity is at the level of the individual's actions. This means that we can learn from other people's individual success, and improve the overall efficacy of the actions we undertake.

Without that modularity - call its closed-source greenery - everything is imposed from above, without explanation or the possibility of local, personal, incremental improvement. That may have worked in the 20th century, but given the lessons we have learned from open source, it's clearly not the best way.

The Marvels of Modularity

One word that has cropped up time and again on this blog is "modularity". It's one of the prime characteristics of the open source way - and one of its greatest strengths. Now wonder, then, that Microsoft has finalled cottoned on - helped, no doubt, by the abject failure of its Vista monster:

When Windows 7 launches sometime after the start of 2010, the desktop OS will be Microsoft's most "modular" yet. Having never really been comfortable with the idea of a single, monolithic desktop OS offering, Microsoft has offered multiple desktop OSes in the marketplace ever since the days of Windows NT 3.1, with completely different code bases until they were unified in Windows 2000. Unification isn't necessarily a good thing, however; Windows Vista is a sprawling, complex OS.

A singular yet highly modular OS could give Microsoft the best of all possible worlds: OSes that can be highly customized for deployment but developed monolithically. One modular OS to rule them all, let's say.

Modularity has another huge benefit for Microsoft: it will allow it to address the nascent ultraportable market, something that it finds hard to do with its current operating systems.

Needless to say, though, even in making this sensible move, Microsoft manages to add a touch of absurdity:

Unsurprisingly, Microsoft already has a patent on a "modular operating system" concept.

A *patent* on modularity? Give me a break....

A Theory of Modularity

I've mentioned a few times how important modularity is to the efficiency of openness. This seems pretty obvious, intuitively, but it's nice to know that some academics have produced a rather nice, rigorous demonstration of why this should be the case for software:

Important software modularity principles, such as the information hiding criterion, have remained informal. DSM modeling and Baldwin and Clark’s design rule theory have the potential to formally account for how design rules create options in the form of independent modules and enable independent substitution.

This paper evaluated the applicability of the model and theory to real-world large-scale software designs by studying the evolution of two complex software platforms through the lens of DSMs and design rule theory. The results showed that (1) DSM models can precisely capture key characteristics of software architecture by revealing independent modules, design rules, and the parts of a system that are not well modularized; (2) design rule theory can formally explain why some software systems are more adaptable, and how a modularization activity, such as refactoring, conveys strategic advantages to a company.

Er, quite. (Via Michael Tiemann.)

Microsoft Starts to Get the Modularity Bug

First, this incredible opening par:

Some of the changes in the upcoming release of Windows Server 2008 are a response to features and performance advantages that have made Linux an attractive option to Microsoft customers.

Er, say that again? Windows Server 2008 is explicitly responding to GNU/Linux?

Then, this little nugget:

"Having less surface area does reduce the servicing and the amount of code you have running and exposed, so we have done a lot of work in 2008 to make the system more modular. You have the server manager; every role is optional, and there are more than 30 components not installed by default, which is a huge change," Laing said.

Ah, yes, modularity....

In Praise of Modularity (Again)

News that Firefox users tend to be more up-to-date with their security patches is interesting, especially for on account of the suggested explanation:

Much of this patching success has to be credited to Firefox's automatic update mechanism, which debuted in version 1.5 but was improved in version 2.0. The browser checks to see if a new version is available and notifies the user when it finds one. The security updates tend to be small (around 200KB to 700KB), which also makes the updating process less painful.

Internet Explorer, in contrast, is typically updated along with the rest of the system with Windows Update. Regular users of Windows Update automatically got upgraded from IE 6 to IE 7, so it is not surprising that people still stuck on IE 6 are not updating as much as IE 7. It's possible to assume that many of the people who aren't using Windows Update are avoiding it because the Windows Update web site checks (using WGA) to see if the user has a legitimate copy of the operating system, but as critical updates for IE 6 are still automatically downloaded by Windows even if WGA fails, it seems more likely that the numbers include legitimate users who have turned automatic updates off.

Once again, the virtues of modularity become clear - and turn out to have very clear real-world benefits too, in this case.

Open Science and Modularity

As the open meme sweeps through field after field, there is a tendency to assume that openness on its own is enough. But as this wise post by Pedro Beltrão about open science explains, there's something else you need if you are to get the full benefits of opening up: modularity.

Open source thrives because major tasks are split up into smaller ones, joined by clean interfaces. This enables tasks to be distributed, and sometimes performed in parallel. Competition operates at the level of the small tasks - the best solutions are chosen - rather than at the top level, which is how proprietary software typically works.

But as Beltrão points out, science is still encouraging competition at the topmost level - at the point when results are published - which leads to teams being scooped and work wasted. Far more sensible if the whole were split up into smaller tasks where competition can operate more fruitfully, and he has some practical suggestions about how that might be achieved.

We Are All Modular Now

One of the central theses of this blog is that for things like software, modularity produces more and better code, because it allows a kind of Darwinian selection to kick in on an atomistic basis.

But wait: isn't another of my theses that openness is appropriate across a whole range of activities - notably content production? And so...that would suggest that content should become more modular too, allowing a similar kind of winnowing process to take place.

Eek!

Substitutability and Modularity

Wise words from Bob Sutor on substitutability:

the ability to take one software application from one provider and put in its place another application from a possibly different provider. Open standards enable interoperability and hence substitutability.

Open standards may enable such substitutability, but it is open source that lives and breathes the principle, thanks to its higly modular structure. This means substitutability can be applied at the level of the sub-routine, and not just for entire apps.

In Praise of Modularity

On Open Enterprise blog.

Modularity Gets Down to Business

On Open Enterprise blog.

Open Source Hardware: A Meme That Won't Die

Open source hardware is nice in theory, but currently self-contradictory in practice. The key thing about open source is that it's generated by code, and the code can be hacked. The same is true of open content, open data, open genomics and the rest. Until they come out with better fabbers whose underlying generative code is both available and hackable, we're doomed to pale imitations of true open source hardware.

In the meantime, Bug Labs has come up with a fun waystation on the road to that end goal:

BUG is a collection of easy-to-use, open source hardware modules, each capable of producing one or more Web services. These modules snap together physically and the services connect together logically to enable users to easily build, program and share innovative devices and applications. With BUG, we don't define the final products - you do.

Note that one key open source feature that you can reproduce in hardware is modularity, and indeed it's key to Bug's approach. And in a real sense, Bug has its heart in the right place:

BUGbase is the foundation of your BUG device. It's a fully programmable and "hackable" Linux computer, equipped with a fast CPU, 128MB RAM, built-in WiFi, rechargeable battery, USB, Ethernet, and a small LCD with button controls.

Spicing Up Thunderbird

As I've noted before, one of the key features of free software is its modularity. From this and the underlying licence flows the ability to mix and match different elements to produce new applications.

Here's a good example:

Synovel, a startup based on Hyderabad, India founded by a group of International Institute of Information Technology (IIIT) graduates, has released a preview of Spicebird, a Mozilla-based collaboration suite.

Spicebird is built on Thunderbird and Lightning, the powerful extension that adds calendaring functions to Thunderbird. Additionally it seems to integrate SamePlace, a Firefox extension that provides instant messaging capabilities based on the Jabber protocol.

Interesting to see that this is coming out of India - not currently a hotbed of such open source startups, but an area I'm sure we'll be hearing more from in the future.

And as David Ascher, who heads up a new company that aims to build software based on Thunderbird, points out:

There are lots of young companies in the same space, each promoting their own angle on solving the problem that they’ve identified. There are companies playing within the Outlook/Exchange framework. There are companies coming at it with Exchange replacements. There are companies focusing on collaboration rather than communication. There are companies with a web focus, others with a mobile focus, others with a social network focus.

But as he also notes, there are very particular advantages to working in the open source space:

From the project health point of view, I think it’s good to have various companies building products off of the Mozilla codebase in general. At the very least, it means that the platform won’t get too tied to any one product’s requirements. I don’t think there’s a huge risk of that happening, because Mozilla already supports several active products (Firefox, Thunderbird, Seamonkey, Komodo, Songbird, Miro, Joost, etc.). But having more people care about the mail/news bits should at least help with the engineering work we need to do there which is product-independent. There are long-standing architectural problems with the system which haven’t been fixed because of a lack of resources. With several companies betting on this platform, as long as the discussions happen in public and in good faith, we should be able to work together to improve things for all.

Of Bazaars and Dangerous Co-location

I often bang on about modularity in this blog, and its critical importance to creating and running open projects. Here are some more thoughts on the subject, along with many interesting ruminations on creating a Raymondian bazaar, and the state of open source companies today. It concludes by answering a key question it posed itself:

Why do so many open-source projects not have the active community of external contributors they are hoping for? Because they have been largely developed by co-located teams of hired software engineers, 100% dedicated to the project, managed and organized like any traditional software development effort. This seems to be especially true for the new crop of ‘custom build’ open-source companies, which would like to take advantage of the open-source business model. They might hope to also enjoy the advantages of the open-source development model one day, but achieving that requires a conscious effort.

Good stuff.

Widgetification = Modularisation

An interesting piece by Om Malik (not on GigaOM) about the rise of the widget. But no surprise here really: the atomisation of programs is just another reflection of the tidal wave that is open source currently sweeping over programming in general. As I've written several times, modularity is key to free software's success: widgetification is simply the same idea applied to Web services.