Open Source and Open Research Computation

Free software was inspired in part by the scientific method, but it is only now that science is starting to apply free software's key insights. For example, opening up the source code would imply that scientific papers should be made freely available for anyone to read and use. And yet it is only in the last few years that this open access approach, as it is called, has made significant headway against the prevailing proprietary system, which says that you have to pay - often handsomely - if you want to read a paper.

On Open Enterprise blog.

Abundance Obsoletes Peer Review, so Drop It

Recently, I had the pleasure of finally meeting Cameron Neylon, probably the leading - and certainly most articulate - exponent of open science. Talking with him about the formal peer review process typically employed by academic journals helped crystallise something that I have been trying to articulate: why peer review should go.

A recent blog post has drawn some attention to the cost - to academics - of running the peer review process:

So that's over £200million a year that academics are donating of their time to the peer review process. This isn't a large sum when set against things like the budget deficit, but it's not inconsiderable. And it's fine if one views it as generating public good - this is what researchers need to do in order to conduct proper research. But an alternative view is that academics (and ultimately taxpayers) are subsidising the academic publishing to the tune of £200 million a year. That's a lot of unpaid labour.

Indeed, an earlier estimate put the figure even higher:

a new report has attempted to quantify in cash terms exactly what peer reviewers are missing out on. It puts the worldwide unpaid cost of peer review at £1.9 billion a year, and estimates that the UK is among the most altruistic of nations, racking up the equivalent in unpaid time of £165 million a year.

Whatever the figure, it is significant, which brings us on to the inevitable questions: why are researchers making this donation to publishers, and do they need to?

The thought I had listening to Neylon talk about peer review is that it is yet another case of a system that was originally founded to cope with scarcity - in this case of outlets for academic papers. Peer review was worth the cost of people's time because opportunities to publish were rare and valuable and needed husbanding carefully.

Today, of course, that's not the case. There is little danger that important papers won't see the light of day: the nearly costless publishing medium of the Internet has seen to that. Now the problem is dealing with the fruits of that publishing abundance - making such that people can find the really important and interesting results among the many out there.

But that doesn't require peer review of the kind currently employed: there are all kinds of systems that allow any scientist - or even the general public - to rate content and to vote it up towards a wider audience. It's not perfect, but by and large it works - and spreads the cost widely to the point of being negligible for individual contributors.

For me what's particularly interesting is the fact that peer review is unnecessary for the same reason that copyright and patents are unnecessary nowadays: because the Internet liberates creativity massively and provides a means for bringing that flood to a wider audience without the need for official gatekeepers to bless and control it.

Follow me @glynmoody on Twitter or identi.ca.

Beyond Open Access: Open Publishing

Another splendid piece from Cameron Neylon calling into question the value of traditional peer review:

Whatever value it might have we largely throw away. Few journals make referee’s reports available, virtually none track the changes made in response to referee’s comments enabling a reader to make their own judgement as to whether a paper was improved or made worse. Referees get no public credit for good work, and no public opprobrium for poor or even malicious work. And in most cases a paper rejected from one journal starts completely afresh when submitted to a new journal, the work of the previous referees simply thrown out of the window.

Much of the commentary around the open letter has suggested that the peer review process should be made public. But only for published papers. This goes nowhere near far enough. One of the key points where we lose value is in the transfer from one journal to another. The authors lose out because they’ve lost their priority date (in the worse case giving the malicious referees the chance to get their paper in first). The referees miss out because their work is rendered worthless. Even the journals are losing an opportunity to demonstrate the high standards they apply in terms of quality and rigor – and indeed the high expectations they have of their referees.

What Neylon has exposed here is that scientific publishing - even the kind that wears its open access badge with pride - simply isn't open in any deep way. We need to be able to see the whole process, for the reasons he mentions. Open access isn't enough, not even with open data: we need *open publishing*.

And yes, that's going to be a huge shift, and painful for many. But if that's the price of producing better scientific papers - and hence better science - surely it's a price worth paying. (Via Nat Torkington.)

Follow me @glynmoody on Twitter or identi.ca.

Good Thoughts on Naughty Lord Mandelson

One of the heartening things about the disheartening three-strikes saga currently playing out in the UK is the quality of the opposition that has provoked. I've already tweeted today about Cameron Neylon's splendid polemic - written with a rigour that only a scientist can provide - which I strongly urge you to read.

Meanwhile, here's an important point made by Monica Horten:

What he [Lord M.] doesn't get is that the Internet is not an entertainment system. It is a public communications network. The powers that he could acquire have serious implications for civil liberties, in particular for freedom of speech. Under the UK's own Human Rights Act, freedom of speech may only be restricted where there is a genuine public interest objective, and any measures must be specific and limited.

I think this goes to the heart of the problem with Lord Mandelson's intervention: he thinks the Internet is like radio or television, and does not appreciate how much bigger it is than that. As Horten points out, the UK's Human Rights Act may well provide the coup de grâce to his plans.

Follow me @glynmoody on Twitter or identi.ca.

Another Reason for Open Access

Yet again, Cameron Neylon is daring to ask the unasked questions that *should* be asked:

Many of us have one or two papers in journals that are essentially inaccessible, local society journals or just journals that were never online, and never widely enough distributed for anyone to find. I have a paper in Complex Systems (volume 17, issue 4 since you ask) that is not indexed in Pubmed, only available in a preprint archive and has effectively no citations. Probably because it isn’t in an index and no-one has ever found it. But it describes a nice piece of work that we went through hell to publish because we hoped someone might find it useful.

Now everyone agreed, and this is what the PLoS ONE submission policy says quite clearly, that such a paper cannot be submitted for publication. This is essentially a restatement of the Ingelfinger Rule. But being the contrary person I am I started wondering why. For a commercial publisher with a subscripton business model it is clear that you don’t want to take on content that you can’t exert a copyright over, but for a non-profit with a mission to bring science to wider audience does this really make sense? If the science is currently unaccessible and is of appropriate quality for a given journal and the authors are willing to pay the costs to bring it to a wider public, why is this not allowed?

Why not, indeed? For as Neylon points out:

If an author feels strongly enough that a paper will get to a wider audience in a new journal, if they feel strongly enough that it will benefit from that journal’s peer review process, and they are prepared to pay a fee for that publication, why should they be prevented from doing so? If that publication does bring that science to a wider audience, is not a public service publisher discharging their mission through that publication?

Which is only possible, of course, in open access journals adopting a funder pays approach, since traditional publishers need to be able to point to the uniqueness of their content if they are trying to sell it - after all, why would you want to buy it twice? Open access journals have no such imperative, since they are giving it away, so readers have no expectations that the stuff is unique and never seen before.

Follow me @glynmoody on Twitter or identi.ca.

Who Can Put the "Open" in Open Science?

One of the great pleasures of blogging is that your mediocre post tossed off in a couple of minutes can provoke a rather fine one that obviously took some time to craft. Here's a case in point.

The other day I wrote "Open Science Requires Open Source". This drew an interesting comment from Stevan Harnad, pretty much the Richard Stallman of open access, as well as some tweets from Cameron Neylon, one of the leading thinkers on and practitioners of open science. He also wrote a long and thoughtful reply to my post (including links to all our tweets, rigorous chap that he is). Most of it was devoted to pondering the extent to which scientists should be using open source:

It is easy to lose sight of the fact that for most researchers software is a means to an end. For the Open Researcher what is important is the ability to reproduce results, to criticize and to examine. Ideally this would include every step of the process, including the software. But for most issues you don’t need, or even want, to be replicating the work right down to the metal. You wouldn’t after all expect a researcher to be forced to run their software on an open source computer, with an open source chipset. You aren’t necessarily worried what operating system they are running. What you are worried about is whether it is possible read their data files and reproduce their analysis. If I take this just one step further, it doesn’t matter if the analysis is done in MatLab or Excel, as long as the files are readable in Open Office and the analysis is described in sufficient detail that it can be reproduced or re-implemented.

...

Open Data is crucial to Open Research. If we don’t have the data we have nothing to discuss. Open Process is crucial to Open Research. If we don’t understand how something has been produced, or we can’t reproduce it, then it is worthless. Open Source is not necessary, but, if it is done properly, it can come close to being sufficient to satisfy the other two requirements. However it can’t do that without Open Standards supporting it for documenting both file types and the software that uses them.

The point that came out of the conversation with Glyn Moody for me was that it may be more productive to focus on our ability to re-implement rather than to simply replicate. Re-implementability, while an awful word, is closer to what we mean by replication in the experimental world anyway. Open Source is probably the best way to do this in the long term, and in a perfect world the software and support would be there to make this possible, but until we get there, for many researchers, it is a better use of their time, and the taxpayer’s money that pays for that time, to do that line fitting in Excel. And the damage is minimal as long as source data and parameters for the fit are made public. If we push forward on all three fronts, Open Data, Open Process, and Open Source then I think we will get there eventually because it is a more effective way of doing research, but in the meantime, sometimes, in the bigger picture, I think a shortcut should be acceptable.

I think these are fair points. Science needs reproduceability in terms of the results, but that doesn't imply that the protocols must be copied exactly. As Neylon says, the key is "re-implementability" - the fact that you *can* reproduce the results with the given information. Using Excel instead of OpenOffice.org Calc is not a big problem provided enough details are provided.

However, it's easy to think of circumstances where *new* code is being written to run on proprietary engines where it is simply not possible to check the logic hidden in the black boxes. In these circumstances, it is critical that open source be used at all levels so that others can see what was done and how.

But another interesting point emerged from this anecdote from the same post:

Sometimes the problems are imposed from outside. I spent a good part of yesterday battling with an appalling, password protected, macroed-to-the-eyeballs Excel document that was the required format for me to fill in a form for an application. The file crashed Open Office and only barely functioned in Mac Excel at all. Yet it was required, in that format, before I could complete the application.

Now, this is a social issue: the fact that scientists are being forced by institutions to use proprietary software in order to apply for grants or whatever. Again, it might be unreasonable to expect young scientists to sacrifice their careers for the sake of principle (although Richard Stallman would disagree). But this is not a new situation. It's exactly the problem that open access faced in the early days, when scientists just starting out in their career were understandably reluctant to jeopardise it by publishing in new, untested journals with low impact factors.

The solution in that case was for established scientists to take the lead by moving their work across to open access journals, allowing the latter to gain in prestige until they reached the point where younger colleagues could take the plunge too.

So, I'd like to suggest something similar for the use of open source in science. When established scientists with some clout come across unreasonable requirements - like the need to use Excel - they should refuse. If enough of them put their foot down, the organisations that lazily adopt these practices will be forced to change. It might require a certain courage to begin with, but so did open access; and look where *that* is now...

Follow me on Twitter @glynmoody

Peer-Reviewed Open Journal of Science Online

One of the most eloquent proponents of the idea of open science is Cameron Neylon. Here's an interesting post about bringing peer review to online material:

many of the seminal works for the Open Science community are not peer reviewed papers. Bill Hooker’s three parter [1, 2, 3] at Three Quarks Daily comes to mind, as does Jean-Claude’s presentation on Nature Precedings on Open Notebook Science, Michael Nielsen’s essay The Future of Science, and Shirley Wu’s Envisioning the scientific community as One Big Lab (along with many others). It seems to me that these ought to have the status of peer reviewed papers which raises the question. We are a community of peers, we can referee, we can adopt some sort of standard of signficance and decide to apply that selectively to specific works online. So why can’t we make them peer reviewed?