Why We Need Open Data

Despite the good-natured ding-dong he and I are currently engaged in on another matter, Peter Murray-Rust is without doubt one of the key individuals in the open world. He's pretty much the godfather of the term "open data", as he writes:

Open Data has come a long way in the last 2-3 years. In 2006 the term was rarely used - I badgered SPARC and they generously created a set up a mailing list. I also started a page on Wikipedia in 2006 so it’s 2-and-a-half years old.

The same post gives perhaps the best explanation of why open data is important; it's nominally about open data in science, but its points are valide elsewhere too:

* Science rests on data. Without complete data, science is flawed.

* Many of todays global challenges require scientific data. Climate, Health, Agriculture…

* Scientists are funded to do research and to make the results available to everyone. This includes the data. Funders expect this. So does the world.

* The means of dissemination of data are cheap and universal. There is no technical reason why all the data in all the chemistry research in the world should not be published into the cloud. It’s small compared with movies…

* Data needs cleaning, flitering, repurposing, re-using. The more people who have access to this, the better the data and the better the science.

Open data is still something of a Cinderella in the open world, but as Peter's comments make clear, that's likely to change as more people realise its centrality to the entire open endeavour.

Labour's Open Hypocrisy

The "O" word has been much on the lips of the UK government recently, what with all the nice things it's been saying about open source, and now this:

The independent Power of Information Task Force published its report on 2 March. The report contained 25 challenging recommendations to government aimed at improving the use of information in this new world. The Task Force's work has been recognised internationally as providing a cutting-edge vision, with examples of what modern public service delivery might be.

The Government welcomes the task force’s vision, accepts its overall messages and will be responding on the detailed recommendations shortly. We are already taking steps to implement this vision and in 2009 we will seek to deliver the following:

Open information. To have an effective voice, people need to be able to understand what is going on in their public services. Government will publish information about public services in ways that are easy to find,easy to use, and easy to re-use, and will unlock data, where appropriate, through the work of the Office of Public Sector Information.

Open innovation. We will promote innovation in online public services to respond to changing expectations. The Government will seek to build on the early success of innovate.direct.gov.uk by building such innovation into the culture of public services and public sector websites.

Open discussion. We will promote greater engagement with the public through more interactive online consultation and collaboration. We will also empower professionals to be active on online peer-support networks in their area of work.

Open feedback. Most importantly, the public should be able to have a fair say about their services. The Government will publish best practice in engaging with the public in large numbers online, drawing on the experience of the www.showusabetterway.com competition and the www.londonsummit.gov.uk, as well as leading private sector examples like www.ideastorm.com.

Open information, open innovation, open discussion, open feedback: well, that's just super-duper and fab and all that, but why not allow a little openness about what the UK government is doing? How about getting rid of the absurd Official Secrets Act, the very antithesis of openness? How about putting the teeth back in the Freedom of Information Act? How about not refusing to publish documents about the Iraqi war? How about letting us see details of MPs' expenses? How about letting us know where our MPs live? How about letting the public openly rate the government itself - the one group that seems excluded from the wonderful plans to "ebay-ise" UK public life?

Because, strange as it may seem, openness does not have hard lines: if you're going to be open, you're going to be *really* open, everywhere. Otherwise, it just further debases an increasingly fashionable concept, takes our cynicism up a notch or three, and alienates those of us fighting for *real* openness.

Do Open Source Eyeballs Really Work?

One of the most contentious areas in computing is whether open source is more or less secure than closed source systems. Open source is open for everyone – including the black hats – to poke around and find the bugs, but it's also open for anyone skilled enough to fix them. Closed source is (theoretically) harder to peek into, but (practically) impossible to fix unless you work for the company that wrote it.

Here's some nice empirical evidence that many eyeballs looking at open source code *do* make a difference...

On Open Enterprie blog.

He Wants a Million Quid? Don't Give it to Him

Tom Steinberg has a simple request: he wants a million quid.

If you want to know how I think mySociety could change the world, this is your answer. I don’t want a million quid because I want some sort of open source empire: I want a million quid because we can’t cross this chasm with any less.

I don't think we should give him a million quid; I think we should give him a *ten* million - OK, make it a hundred million - plus the job of overseeing the opening up government in this country to such an extent that its buttons start popping.

Oh, and as another precondition for the dosh, could he kindly respond to my Twitter request to follow him (not that I'm bitter).

After all, it's not a question of how much that would cost, but how much it is costing us - economically, politically and socially - by *not* doing it.

The True Begetter of Innovation is Openness

One of the persistent myths peddled by lovers of intellectual monopolies is that you need things like patents to promote innovation. The idea is that patents encourage new research, which then feeds into more research, and the world is a better place.

Not so, according to some rigorous new research into the effects of intellectual monopolies on science:

Scientific freedom and openness are hallmarks of academia: relative to their counterparts in industry, academics maintain discretion over their research agenda and allow others to build on their discoveries. This paper examines the relationship between openness and freedom, building on recent models emphasizing that, from an economic perspective, freedom is the granting of control rights to researchers. Within this framework, openness of upstream research does not simply encourage higher levels of downstream exploitation. It also raises the incentives for additional upstream research by encouraging the establishment of entirely new research directions. In other words, within academia, restrictions on scientific openness (such as those created by formal intellectual property (IP)) may limit the diversity and experimentation of basic research itself. We test this hypothesis by examining a “natural experiment” in openness within the academic community: NIH agreements during the late 1990s that circumscribed IP restrictions for academics regarding certain genetically engineered mice. Using a sample of engineered mice that are linked to specific scientific papers (some affected by the NIH agreements and some not), we implement a differences-in-differences estimator to evaluate how the level and type of follow-on research using these mice changes after the NIH-induced increase in openness. We find a significant increase in the level of follow-on research. Moreover, this increase is driven by a substantial increase in the rate of exploration of more diverse research paths. Overall, our findings highlight a neglected cost of IP: reductions in the diversity of experimentation that follows from a single idea.

This work basically shows that recent attempts to introduce intellectual monopolies into science in order to "promote innovation" have actually been counter-productive.

our results offer direct evidence that scientific openness seems to be associated with the establishment of entirely new research lines: more specifically, increased openness leads to a significant increase in the diversity of the journals in which mouse-articles in the treatment group are cited, and, perhaps even more strikingly, a very significant increase in the number of previously unused “keywords” describing the underlying research contributions of the citing articles.

In this context at least, it's openness that leads to more innovation, not its polar opposite. (Via Open Access News.)

Follow me on Twitter @glynmoody

How Openness Can Regulate the Real World

Yes, even the really messy bits:

Participatory regulation is arguably the best way to surface and defeat corruption in government and industry. I’ve highlighted a range of impressive efforts below. They range from Transparency International’s more top-down survey and index approach to the bottom-up Wikileaks site where anybody can post documents that uncover instances of corruption.

The post explores several examples: Transparency International’s Corruption Perceptions Index; The Kimberley Process (KP) - a joint government-industry-civil society initiative to stem the flow of conflict diamonds; and the Extractive Industries Transparency Initiative (EITI), which is "similar in intent to TI’s bribe payer’s index — it also aims to strengthen governance by improving transparency and accountability in the extractives sector" (apparently the "extractive industries" refer to mining, oil, gas and similar companies).

What's really noteworthy here is that openness is being used to make a difference not in airy-fairy realms of genteel, abstract concerns, but in some of the most brutal, real-world contexts imaginable. Who knows, it might even work for something as corrupt as the British political system.

Update: Simon Phipps has pointed out the new Stimulus Watch, which works on similar principles.

A Tale of Two Consultations

One of the running themes on this blog is the importance of engaging with the powers that be, specifically through responding to government and European requests for comments on proposals.

You would think that those putting together such requests would do everything in their power to maximise the feedback they receive, but alas that's not always the case....

On Open Enterprise blog.

Change in How Change Happens

Another cracking post from Kevin Kelly:

Of all the tricks that evolution came up for increasing its evolvability none compare to minds. Minds – and not just human minds – bestow on life a greatly accelerated way to learn and adapt. This should not be surprising because minds are built to find answers, and one of the key things to answer might be how to learn better, quicker. If what minds are good for is learning and adaptation, then learning how to learn will accelerate your learning. Even though most of the learning a mind does is not transferred directly into biological evolution, there are several ways in which minds accelerate evolution (see the Baldwin Effect), even in the lower animal kingdom. So the presence of minds in life has increased its evolvability; the discovery of mindness has driven evolution in many new directions while also creating a new territory to explore – the territory of possible minds.

The most recent extension of this expansion is technology. Technology is how human minds explore the space of possibilities. We power our minds via science and technology to make possible things real. More so technology is how our society learns and introduces change. It is almost a cliché to point out that technology has brought as much change on this planet in the last 100 years as life has in the last billion years.

Ray Kurzweil can provide you with dozens of graphs charting the accelerating change brought about by technology in the last 100 years or so. From the speed of computers, the bandwidth of communications, the power of engines, the yield of crops – all are accelerating in performance. Change is this century's middle name.

But meta-change is not about acceleration itself; it is not about faster change. Rather, the acceleration of evolution or increased evolvability is about the change in the nature of change. The basic mechanism by which our collective minds – as expressed by technology – adapt and produce change is undergoing a shift. In fact the most important change at work in our world right now is "the change in how change happens."

"Change in how change happens": that's a pretty good description of what openness is doing. It has changed *how* we change. It's also what we need to achieve on a *planetary* scale if was are going to save much of the world as we know it. It doesn't get much more profound than that.

Openness is Good for Everyone – Even MPs

It will come as no surprise to readers of this blog that I believe that openness is a pretty good thing, pretty much everywhere. Strangely, the UK Government doesn't agree with me: it seems to think telling members of the public how MPs spend the public's money is a Really Bad Thing. As mySociety explains:

Ministers are about to conceal MPs’ expenses, even though the public has just paid £1m to get them all ready for publication, and even though the tax man expects citizens to do what MPs don’t have to. They buried the news on the day of the Heathrow runway announcement. This is heading in the diametric wrong direction from government openness.

Yes indeedy. MySociety also has some helpful suggestions on what you might want to do about it:

1. Please write to your MP about this www.WriteToThem.com - ask them to lobby against this concealment, and tell them that TheyWorkForYou will be permanently and prominently noting those MPs who took the opportunity to fight against this regressive move. The millions of constituents who will check this site before the next election will doutbtless be interested.

2. Join this facebook group and invite all your least political friends (plus your most political too). Send them personal mails, phone or text them. Encourage them to write to their politicians too.

3.Write to your local paper to tell them you’re angry, and ask them to ask their readers to do the above. mySociety’s never-finished site http://news.mysociety.org might be able to help you here.

I've already done the first two (not quite sure about the third), and I urge you to do the same. Remember: it's not about the money, it's about the openness.

For those who are interested, here's my letter:

I am writing to you to express my profound disquiet that the Government is about the go back on its decision to make detailed information about MPs expenses available to the public.

As the Government likes to remind us, those who have nothing to fear have nothing to hide, and the request that the British electorate – the people that ultimately foot the bill of MPs' expenses – should be allowed to see the costs claimed by MPs is simply a question of justice.

It is also a question of fairness: at a time when ordinary citizens are being asked to give up more and more information about themselves to the Government, it is only right that politicians should do the same if they are not to be branded as hypocrites.

Moreover, it is a question of good sense: much time and money has already been spent preparing this information. To throw it away now, at a time when many families are struggling to make ends meet, would be a real slap in the face for the general public, and a clear sign that the Government is contemptuous of their everyday problems.

I know you as an MP who has always conducted a laudably frank and open dialogue with your constituents, and so I hope that you will agree that making politics as transparent as possible can only strengthen our democracy, while creating exceptions for MPs will only increase the public's cynicism and lead to an ever-great alienation from the political system.

For these reasons, I urge you to vote against this measure to conceal MPs' expenses.

Extreme Openness: the Rise of Wikileaks

There is a long journalistic tradition of looking back at the end of the year over the major events of the preceding 12 months - one that I have no intention of following. But I would like to point out an important development in the world of openness that has occurred over that time-span: the rise and rise of Wikileaks....

On Open Enterprise blog.

Will 2009 Be Open or Closed?

As the end of 2008 approaches, people's thoughts naturally turn to 2009, and what it might hold. The dire economic situation means that many will be wondering what the year will bring in terms of employment and their financial situation. This is not the place to ponder such things, nor am I qualified to do so. Instead, I'd like to discuss a matter that is related to these larger questions, but which focusses on issues particularly germane to Linux Journal: will 2009 be a year in which openness thrives, or one in which closed thinking re-asserts itself?

On Linux Journal.

BBC, Meet Plughole....

We are grateful to Andrew Pierce for his informative article about how the Foreign Office minister misled parliament with regard to the advertising of the post of Director of the World Service.

...

To maintain the BBC World Service's reputation and credibility, the new Managing Director must be chosen through a fully open selection process, with full consideration of the availability and qualification of external candidates. In addition, a new managing director must be authoritative in news and current affairs, have wide international perspectives, must be capable of resisting pressure both from the UK government and from other governments and should not believe that the World Service can be founded on the perceived importance of marketing. To impose a closing date for applications of January 4, 2009 is to foreclose all these options.

Read it, and weep.

Open Me Kangaroo Up, Sport

Having had their plan to combine their broadband TV services kyboshed by the Competition Commmission, the BBC and ITV today said they plan to do it anyway - but this time to open up the infrastructure to all comers.

The two broadcasters, along with BT, said they want to foster a "common industry approach" that's "open for all public service broadcasters, device developers and other ISPs". All this will be founded upon "a standards based open environment".

Yes, but *how* open?

Principles for an Open Transition

Talking of openness and Obama:

President-elect Obama has made a clear commitment to changing the way government relates to the People. His campaign was a demonstration of the value in such change, and a glimpse of its potential. His transition team has now taken a crucial step in making the work of the transition legally shareable, demonstrating that the values Obama spoke of are values that will guide his administration.

To further support this commitment to change, and to help make it tangible, we offer three “open transition principles” to guide the transition in its use of the Internet to produce the very best in open government.

That openness meme is certainly getting popular.

Openness We Can Believe In

Of course, no danger of any of this dangerous "21st century" openness cropping up here in the UK:

President-elect Obama has championed the creation of a more open, transparent, and participatory government. To that end, Change.gov adopted a new copyright policy this weekend. In an effort to create a vibrant and open public conversation about the Obama-Biden Transition Project, all website content now falls under the Creative Commons Attribution 3.0 License

(With thanks to Alan Lord for reminding me this deserves to be highlighted.)

The Fear of Openness...

...strikes again:

A political row erupted last night after counter-terrorism police arrested the shadow Home Office minister, Damian Green, after he published leaked documents allegedly sent to the Tories by a government whistleblower.

...

George Osborne, the shadow chancellor, told the BBC: "I think it is extraordinary that the police have taken that decision. It has long been the case in our democracy that MPs have received information from civil servants. To hide information from the public is wrong."

This is getting serious.

Der Doppelgänger

Here's a typical Moody text I never wrote:

A brief explanation of what the free culture movement is and the various factors that led to its fighting to preserve the commons, including corporations and special interests trying to restrict the commons to protect their interests, the development of the open source community, technological developments, such as the Internet and digital copying of media, the developmentof web 2.0 and its philosophies, current state of copyright law and youth culture.

It's by one "David W. Moody, California State University, San Jose School of Library and Information Science." Sad, then, that he makes no mention of Rebel Code in his bibliography about openness, since it pre-dates by far other sources that he does mention. But I'm not bitter.

Much.

Du bleicher Geselle!

Andrew Keen is Right...

...when he writes:

The hungry and cold unemployed masses aren’t going to continue giving away their intellectual labor on the Internet in the speculative hope that they might get some "back end" revenue.

People aren't going to give away intellectual labour in that hope because...that's not why people contribute to Wikipedia, or Linux or any of the thousand other endeavours built around sharing, collaborating and giving.

As studies have shown, if you start paying people to do something that they are doing for the sheer pleasure of doing it, they suddenly lose much of the satisfaction they hitherto derived: people don't *want* to be paid for doing it - but they will want to be paid for doing something that do in order to get paid (aka "work").

And Mr Keen is absolutely spot-on again when he adds:

"Free" doesn’t fill anyone’s belly; it doesn’t warm anyone up.

Indeed not; but it does fill the *heart*, which has its own imperatives quite separate from the undeniable ones of the belly.... (Via Slashdot.)

Investing Out in the Open

As the recent financial fun has shown, investing can soon turn into an ungrounded exercise in fantasy wealth creation based on trickery, deceit and general exploitation of ignorance. Part of the problem is the lack of openness.

So here's an interesting idea from a company called Covestor: investing out in the open.

Covestor is not a bulletin board or fantasy trading game, it's all about actions. Covestor is about real-trades, real people and real results - where you can both build your credibility and see what other real people are doing to achieve their goals. Secondly, it's about helping people make more money by leveraging the hard work that is already being done. Of course, discussion is part of the investment process.

Many of our members also have their own stock blogs and are active on discussion sites. Our role is not to replace that, but to help add trust to what they are saying elsewhere.

Ah yes, trust: that's the glue that holds the opens together; it's also the stuff that, in the financial world, was melted down and sold off like lead from a church roof. Let's hope that Covestor can get its idea to, er, stick. (Via Mark Taylor.)

Opening Up ISO's Can of Worms

Nothing shows better what is wrong with the ISO, and why we need to replace it with a new global standards organisation, than the following post....

On Open Enterprise blog.