A Victory (of Sorts) for e-Petitions

It's easy to be cynical about 10 Downing Street's e-petitions (I should know). But here's a case where it might even have done some good.

Thank you for your e-petition, which asks that The National Archives convert its electronic records to Open Document Format rather than Microsoft Open XML Format, in order to make them accessible to users.

The National Archives is committed to preserving electronic records that are both authentic, and easily accessible by users. Wherever possible, records are made available online in a format which can be accessed using any standard web browser. Electronic records transferred to The National Archives are always preserved in their native format; if the native format is not suitable for online access then a separate ‘presentation’ version is created. No single format can address the diversity of electronic records held by The National Archives. At present, documents transferred in Microsoft Office formats are converted to Portable Document Format for online access. PDF is an international standard (ISO 32000-1: 2008) and is supported by all major browsers, either natively or via freely available plug-ins. The National Archives does not currently plan to convert any records to Microsoft Office Open XML format.

It's the last bit that's important: there were rumours circulating that some dark deal was being done to lock up the Archives in OOXML. For the moment we seem safe....

The Next Great Microsoft Lock-in

I've written a lot here and elsewhere about Microsoft's faux-open file format OOXML. I've also noted that there is an unhealthily close relationship between the BBC and Microsoft over the former's iPlayer and its chosen file formats. Now it seems that this kind of chummy lock-in is happening elsewhere, at the UK's National Archives and beyond.

The National Archive story is not new, but dates back to July of this year, when I first noted it. Here's what a Microsoftie said:

The announcement we just made with the National Archives is trying to address the issue of digital conservation head-on. With billions of documents in the world wrapped up in proprietary document formats (from Microsoft and many many other vendors) we felt it was important to focus on how we can help the body in the UK which has the biggest headache and do what we can to assist them in:

* Migrating documents to the latest Office format (Open XML) via our document conversion tools to ensure they can be accessed by the public in the future

Since then, people have started taking notice, to the extent that there is now an e-petition all us Brits can sign asking that nice Mr Brown to use ODF instead of OOXML for the National Archives (but don't hold your breath.)

However, I've just noticed that the Microsoftie quoted above mentions this little factette:

Well, we've actually been working with The British Library and The National Archive for about 18 months now on digital preservation with some other European organisations as members of an EU project called Planets.

And as far as the latter is concerned:

The Planets consortium brings together the unique experience required to research, develop, deliver and productise practical digital preservation solutions. Coordinated by the British Library

The British Library, you may recall, is also in cahoots with Microsoft when it comes to locking up our digital heritage. Some now we have the prospect of the OOXML cancer spreading to other institutions, and large chunks of European culture being locked up in proprietary formats.

This is getting serious. It's obviously time to call in the heavy mob: the Open Source Consortium....

Selling (Digital) Brooklyn Bridges

In a move that seems like a great model of public and private cooperation, the National Archives and Amazon.com have reached a pact under which Amazon will sell films and video footage gathering dust in the archives’ vaults. These videos and films, which capture some of our most intriguing and important moments in history, are already available at no charge to folks who want to visit the archives’ facilities in College Park, MD, but now they’ll become available to anybody via the Internet.

So let me get this straight. Amazon makes money selling digital copies of archive material, which is freely available, back to the people who own that material: and that's a great model? When will Amazon starting selling digital Brooklyn Bridges, I wonder....

Decoupling Software and Standards

As you may have noticed, there is a big bust-up over office file formats going on at the moment. On the one hand, we have ODF, which is a completely open, vendor-independent standard that is supported by multiple programs, and on the other, we have Microsoft's OOXML, which is a vendor-dependent standard of sorts, unlikely to be fully implemented by anyone other than Microsoft.

The only reason this debate is taking place is because of the huge installed base of Microsoft Office, which is naturally biased towards OOXML. But with the release of Sun's ODF Plug in 1.0 for Microsoft Office, things have changed:

The Sun ODF Plug in for Microsoft Office gives users of Microsoft Word, Excel and Powerpoint the ability to read, edit and save to the ISO-standard Open Document Format. The ODF Plug in is available as a free download from the Sun Download Center (SDLC). Download the ODF Plug in.

The Plug in is easy to setup and use, the conversion happens transparently and the additional memory footprint is minimal. Microsoft Office users now can have seamless two-way conversion of Microsoft Office documents to and from Open Document. The ODF Plug in runs on Microsoft Windows and is available in English. More language support will be available in later releases.

This is important, because it decouples the file format from the program. Now anyone - including Microsoft Office users - can opt for a truly open format, not one that aspires to this condition.

We can only hope that the UK's National Archives, making an extraordinary amount of noise about solving a problem largely of Microsoft's making, will use the new plug-in to convert files stored in proprietary formats into the safest long-term solution - ODF.

Open Public Data: Halfway There

Well, now, here's some progress:

The OFT's market study into the commercial use of public information has found that more competition in public sector information could benefit the UK economy by around £1billion a year.

Download Commercial use of public information (pdf 707 kb).

Examples of public sector information include weather observations collected by the Met Office, records held by The National Archives used by the public to trace their family history, and mapping data collated by Ordnance Survey. The underlying raw information is vital for businesses wanting to make value-added products and services such as in-car satellite navigation systems.

Public sector information holders (PSIHs) are usually the only source for much of this raw data, and although some make this available to businesses for free, others charge. A number of PSIHs also compete with businesses in turning the raw information into value-added products and services. This means PSIHs may have reason to restrict access to information provided solely by themselves.

The study found that raw information is not as easily available as it should be, licensing arrangements are restrictive, prices are not always linked to costs and PSIHs may be charging higher prices to competing businesses and giving them less attractive terms than their own value-added operations.

It's good news that the Office of Fair Trading has grasped that the public sector trough-scoffers cost taxpayers serious money through their greed; however, realising that making the information freely available - not just for commercial use - would generate far more dosh still will take a while, I fear. (Via Open Knowledge Foundation.)

Murdering Memory

This press release from the US National Archives raises a key issue for the digital age: the need for archives to act in a completely transparent fashion. If, as has been happening, archives can be silently "disappeared" by security forces, history - built on sand at the best of times - becomes even more unstable.

The words of the grandly-named Archivist of the United States should be framed on the walls of everyone working in the world of digital memories:

There can never be a classified aspect to our mission. Classified agreements are the antithesis of our reason for being.

Imagine, for example, if the great and wonderful Internet Archive were forced to delete materials, without even leaving a notice to that effect. Perhaps they already have.

Google's Creeping Cultural Imperialism

Another day, another Google launch.

As the official Google blog announced, the company is launching a pilot programme to digitise national archive content "and offer it to everyone in the world for free."

And what national archives might these be? Well, not just any old common-or-garden national archives, but "the National Archives", which as Google's blog says:

was founded with the express purpose of ... serving America by documenting our government and our nation.

Right, so these documents are fundamentally "serving America". A quick look at what's on offer reveals the United Motion Newsreel Pictures, a series which, according to the accompanying text, "was produced by the Office of War Information and financed by the U. S. government", and was "[d]esigned as a counter-propaganda medium."

So there we have it: this is (literally) vintage propaganda. And nothing wrong with that: everybody did it, and it's useful to be able to view how they did it. But as with the Google Print/Books project, there is a slight problem here.

When Google first started, it did not set out to become a search engine for US Web content: it wanted it all - and went a long way to achieving that, which is part of its power. But when it comes to books, and even more where films are concerned, there is just too much to hope to encompass; of necessity, you have to choose where to start, and where to concentrate your efforts.

Google, quite sensibly, has started with those nearest home, the US National Archives. But I doubt somehow that it will be rushing to add to other nations' archives. Of course, those nations could digitise and index their own archives - but it wouldn't be part of the Google collection, which would always have primacy, even if the indexed content were submitted to them.

It's a bit like Microsoft's applications: however much governments tell the company to erect Chinese walls between the programmers working on Windows and those working on applications, there is bound to be some leakiness. As a result, Windows programs from Microsoft have always had an advantage over those from other companies. The same will happen with Google's content: anything it produces will inevitably be more tightly integrated into their search engine.

And so, wittingly or not, Google becomes an instrument of cultural imperialism, just like that nice Mr Chirac warned. The problem is that there is nothing so terribly wrong with what Google is doing, or even the way that it is doing it; but it is important to recognise that these little projects that it sporadically announces are not neutral contributions to the sum of the world's open knowledge, but come with very particular biases and knock-on effects.