Abundance Obsoletes Peer Review, so Drop It

Recently, I had the pleasure of finally meeting Cameron Neylon, probably the leading - and certainly most articulate - exponent of open science. Talking with him about the formal peer review process typically employed by academic journals helped crystallise something that I have been trying to articulate: why peer review should go.

A recent blog post has drawn some attention to the cost - to academics - of running the peer review process:

So that's over £200million a year that academics are donating of their time to the peer review process. This isn't a large sum when set against things like the budget deficit, but it's not inconsiderable. And it's fine if one views it as generating public good - this is what researchers need to do in order to conduct proper research. But an alternative view is that academics (and ultimately taxpayers) are subsidising the academic publishing to the tune of £200 million a year. That's a lot of unpaid labour.

Indeed, an earlier estimate put the figure even higher:

a new report has attempted to quantify in cash terms exactly what peer reviewers are missing out on. It puts the worldwide unpaid cost of peer review at £1.9 billion a year, and estimates that the UK is among the most altruistic of nations, racking up the equivalent in unpaid time of £165 million a year.

Whatever the figure, it is significant, which brings us on to the inevitable questions: why are researchers making this donation to publishers, and do they need to?

The thought I had listening to Neylon talk about peer review is that it is yet another case of a system that was originally founded to cope with scarcity - in this case of outlets for academic papers. Peer review was worth the cost of people's time because opportunities to publish were rare and valuable and needed husbanding carefully.

Today, of course, that's not the case. There is little danger that important papers won't see the light of day: the nearly costless publishing medium of the Internet has seen to that. Now the problem is dealing with the fruits of that publishing abundance - making such that people can find the really important and interesting results among the many out there.

But that doesn't require peer review of the kind currently employed: there are all kinds of systems that allow any scientist - or even the general public - to rate content and to vote it up towards a wider audience. It's not perfect, but by and large it works - and spreads the cost widely to the point of being negligible for individual contributors.

For me what's particularly interesting is the fact that peer review is unnecessary for the same reason that copyright and patents are unnecessary nowadays: because the Internet liberates creativity massively and provides a means for bringing that flood to a wider audience without the need for official gatekeepers to bless and control it.

Follow me @glynmoody on Twitter or identi.ca.

Beyond Open Access: Open Publishing

Another splendid piece from Cameron Neylon calling into question the value of traditional peer review:

Whatever value it might have we largely throw away. Few journals make referee’s reports available, virtually none track the changes made in response to referee’s comments enabling a reader to make their own judgement as to whether a paper was improved or made worse. Referees get no public credit for good work, and no public opprobrium for poor or even malicious work. And in most cases a paper rejected from one journal starts completely afresh when submitted to a new journal, the work of the previous referees simply thrown out of the window.

Much of the commentary around the open letter has suggested that the peer review process should be made public. But only for published papers. This goes nowhere near far enough. One of the key points where we lose value is in the transfer from one journal to another. The authors lose out because they’ve lost their priority date (in the worse case giving the malicious referees the chance to get their paper in first). The referees miss out because their work is rendered worthless. Even the journals are losing an opportunity to demonstrate the high standards they apply in terms of quality and rigor – and indeed the high expectations they have of their referees.

What Neylon has exposed here is that scientific publishing - even the kind that wears its open access badge with pride - simply isn't open in any deep way. We need to be able to see the whole process, for the reasons he mentions. Open access isn't enough, not even with open data: we need *open publishing*.

And yes, that's going to be a huge shift, and painful for many. But if that's the price of producing better scientific papers - and hence better science - surely it's a price worth paying. (Via Nat Torkington.)

Follow me @glynmoody on Twitter or identi.ca.

Why not Participatory Medicine?

If this participation thing is so great, why don't we apply it to something really important, like medicine? Why not, indeed?

Welcome to JoPM, a New Peer-Reviewed, Open Access Journal

Our mission is to transform the culture of medicine to be more participatory. This special introductory issue is a collection of essays that will serve as the 'launch pad' from which the journal will grow. We invite you to participate as we create a robust journal to empower and connect patients, caregivers, and health professionals.

More specifically:

Because the Journal of Participatory Medicine is a new journal publishing multidisciplinary articles on topics within a new, not yet defined field, we have established draft parameters that define the journal’s range of interest. We anticipate that these parameters will change somewhat as the field develops. In the meantime, the following characterize the field of participatory medicine.

I particularly liked the following section, with its emphasis on openness:

New Knowledge Creation stems from the collaboration of researchers and patients, as individuals and as groups.

1. Health professionals and patients sharing in the discussion of scientific methods, including open discussion about the level of evidence of the research

2. Open, transparent process that demonstrates collaboration and participation in research

3. Patients with significant interest in a topic joining together to create repositories for research, including (but not limited to) registries, tissue banks and genetic databases; demonstrating mutual respect for the contributions of the data owners and health research professionals with the tools to gain insight from those repositories. Interpretation of results and conclusions including involvement of all stakeholders.

Important stuff, worth a read.

Follow me @glynmoody on Twitter or identi.ca.

Another Reason We Need Open Access

One of the more laughable reasons that traditional science publishers cite in their attempts to rubbish open access is that it's somehow not so rigorous as "their" kind of publishing. There's usually a hint that standards might be dropped, and that open access journals aren't, well, you know, quite proper.

And then this comes along:

The Scientist has reported that, yes, it's true, Merck cooked up a phony, but real sounding, peer reviewed journal and published favorably looking data for its products in them. Merck paid Elsevier to publish such a tome, which neither appears in MEDLINE or has a website, according to The Scientist.

Now, open access in itself isn't going to stop this kind of thing, but it seems highly unlikely that anyone would try it, given that the results would be freely available for any Thomas, Richard or Harold to peruse.

One reason why Elseview probably thought they could pull it off was that they knew few people would look at this stuff - which is why it's not in Medline, and why it doesn't have a website. Given enough eyes, all bugs are shallow and all that.

So, next time high-falutin' publishers look down on open access journals - especially if it's Elsevier - just remind them about the Australasian Journal of Bone and Joint Medicine episode....

Peer-Reviewed Open Journal of Science Online

One of the most eloquent proponents of the idea of open science is Cameron Neylon. Here's an interesting post about bringing peer review to online material:

many of the seminal works for the Open Science community are not peer reviewed papers. Bill Hooker’s three parter [1, 2, 3] at Three Quarks Daily comes to mind, as does Jean-Claude’s presentation on Nature Precedings on Open Notebook Science, Michael Nielsen’s essay The Future of Science, and Shirley Wu’s Envisioning the scientific community as One Big Lab (along with many others). It seems to me that these ought to have the status of peer reviewed papers which raises the question. We are a community of peers, we can referee, we can adopt some sort of standard of signficance and decide to apply that selectively to specific works online. So why can’t we make them peer reviewed?

Rice's Digital University Press

I've written before about Rice University's Connexions platform and programme, which aims to make courseware freely available for all kinds of interesting re-use. But there's another side to Rice's re-invention of academic publishing:

Rice University has re-launched its university press as an all-digital operation. Using the open-source e-publishing platform Connexions, Rice University Press is returning from a decade-long hiatus to explore models of peer-reviewed scholarship for the 21st century. The technology offers authors a way to use multimedia — audio files, live hyperlinks or moving images — to craft dynamic scholarly arguments, and to publish on-demand original works in fields of study that are increasingly constrained by print publishing.

Rice's digital press operates just as a traditional press, up to a point. Manuscripts will be solicited, reviewed, edited and resubmitted for final approval by an editorial board of prominent scholars. But rather than waiting for months for a printer to make a bound book, Rice University Press's digital files will instead be run through Connexions for automatic formatting, indexing and population with high-resolution images, audio and video and Web links.

Users of Rice University Press titles are able to view the content online for free or, thanks to Connexions' partnership with on-demand printer QOOP, order printed books in every style from softbound black-and-white on inexpensive paper to leather-bound, full-color hardbacks on high-gloss paper.

Perhaps the best place to find out about why and how Rice is doing this is an interview with Charles Henry, publisher at Rice University Press (you might need to register to access this article, but it's free). (Via Open Access News.)

The Open Medicine Paradigm

Here's a paradigmatic tale:

The editors who were fired or resigned over the editorial-independence controversy at the Canadian Medical Association Journal have reunited to start their own free, online medical journal.

Open Medicine will be a peer-reviewed, independent open-access journal that does not accept advertising from pharmaceutical or medical-device companies.

Until now, the big publishing houses have held all the cards: do it our way, or you don't do it. No longer. If you don't like it, leave and start your own.

The issues at stake are important:

As a medical librarian, I believe that information (in all its forms, good and bad) is central to human health.

It is also essential to the health of democracies. Without free, open access to information - particularly from a global perspective - our freedoms are limited, and more specifically physicians are unable to practice evidence-based medicine.

Open Peer Review: Not in Their Nature

One door opens, another door closes: Nature has decided to bin its open peer review experiment:

Despite the significant interest in the trial, only a small proportion of authors opted to participate. There was a significant level of expressed interest in open peer review among those authors who opted to post their manuscripts openly and who responded after the event, in contrast to the views of the editors. A small majority of those authors who did participate received comments, but typically very few, despite significant web traffic. Most comments were not technically substantive. Feedback suggests that there is a marked reluctance among researchers to offer open comments.

Nature and its publishers will continue to explore participative uses of the web. But for now at least, we will not implement open peer review.

I suspect that Nature was probably the worst possible place to try this experiment. Nature is simply the top spot for scientific publishing: getting a paper published there can make somebody's career. So the last thing most people want is anything that might increase the risk of rejection. Public discussion of submitted papers certainly falls into that category, both for the commenter and commented (think scientific mafia).

In a way, this is what makes PLoS ONE so important: it's a tabula rasa for this kind of thing, and can redefine what scientific publishing is about. Nature and its contributors are hardly likely to want to do the same. Kudos to the title for trying, but I bet they're relieved it flopped. (Via Techdirt.)

True Open Access

One of the things that continues to amaze me about blogs is the quality of some of the writing. A case in point is this fantastic essay by Richard Poynder. It's an extremely thorough consideration of whether open access means that peer review is on the way out.

Here are a couple of ideas that were new to me:

In September, for instance, a group of UK academics keen to improve the way in which scientific research is evaluated launched a new OA journal called Philica.

Unlike both Nature and PLoS ONE, Philica has no editors, and papers are published immediately on submission — without even a cursory review process. Instead, the entire evaluation process takes place after publication, with reviews displayed at the end of each paper.

and

Philica is not the only new initiative to push the envelope that bit further. Another approach similar in spirit is that adopted by Naboj, which utilises what it calls a dynamical peer review system.

Modelled on the review system of Amazon, Naboj allows users to evaluate both the articles themselves, and the reviews of those articles. The theory is that with a sufficient number of users and reviewers, a convergence process will occur in which a better quality review system emerges.

And you're getting it all for free: true open access. I just hope you are grateful.

Is Helium Enough of a Gas to Lift Off?

User-generated sites are becoming increasingly accepted as a viable way of creating information and even money. But there's a huge problem faced by any new entrant in this sector. For a site to draw in new contributors, it needs lots of readers; but to gain those readers, you need good content, which means lots of good contributors.

A good example of this Catch-22 situation in practice is the new Helium: there's nothing wrong with the site, but equally there's nothing special about it either (though the use of peer review is interesting), so it's hard to see it gaining the momentum needed for it to succeed. (Via ContentBlogger.)

Now There's an Idea: Peer Review of Patents

I almost had to pinch myself for this one: the US Patent and Trademark Office has apparently

created a partnership with academia and the private sector to launch an online, peer review pilot project that seeks to ensure that patent examiners will have improved access to all available prior art during the patent examination process.

(Via Peer to Patent and Boing Boing.)

But wait: they can't possibly do this. I mean, it's so obviously sensible, and the right first step in fixing a manifestly broken system, there must be a catch. Maybe not: the full, wikified details of this potential wonder sound strangely plausible....

Wikipedia, Science and Peer Review

Last week, Wendy Grossman wrote a wise article about how all those making a fuss over Wikipedia's inaccuracies and lack of accountability forget that precisely the same charges were levelled against the Web when it rose to prominence around ten years ago. (Parenthetically, it's interesting to note more generally how Wikipedia is in some sense a Web 2.0 recapitulation of those early Web 1.0 days - a heady if untidy attempt to encompass great swathes of human knowledge).

It seems to me that another story, on BBC News, which considered the fate of the peer review process in the wake of Dr Hwang Woo-suk's faked stem cell papers, is relevant here.

Many of Wikipedia's critics claim that everything would be fine if only it added some rigorous peer review to the process, just like science does when papers are published. But as the BBC item explains, the peer review process employed by Science magazine (where most of Hwang's apparently ground-breaking papers appeared) signally failed to spot that Hwang's papers were frauds (it also explains why, in any case, this is probably asking too much of peer review).

These recent problems with scientific peer review re-inforce Grossman's point about Wikipedia: that we should stop worrying about the inaccuracies and instead learn how to live with them, just as we do for the Web. Far better to develop a critical sense that submits all results - wherever they come from - to a minimum level of scrutiny.

Open Access - Get the Facts

A piece that writes very positively about open access's future quotes a survey from the Centre for Information Behaviour and the Evaluation of Research (CIBER) that examined academics' attitude to different forms of publishing. According to figures given in a story referred to by the first article, some 96.2% of those surveyed support peer review - the standard academic process whereby a paper is sent to referees for comments on its accuracy. So far, so good.

Except that the headline given on the second site is "Academic authors favour peer review over open access" - as if the two were in opposition. In fact, most open access titles employ peer review, so the 96.2% in favour of it were not expressing any opinion about open access, just about peer review.

However, the second article does quote two other figures: that "nearly half" of the academics surveyed thought that open access would undermine the current system (which requires academic institutions to take out often hugely-expensive subscriptions to journals), and that 41% thought that this was a good thing.

To find out whether this 41% refers to the entire sample, or only to those who thought open access would undermine the old system, I naturally went to the CIBER site in order to find out what the real figures were. It turns out that the 41% refers to the whole sample, not just those who viewed the rise of open access as likely. Among the latter group, more than half were in favour.

The Publishers Association and the International Association of STM Publishers, which sponsored the report, must be pretty gutted by the results that a significant proportion of academics rather like the idea of open access destroying the current system. But not peer review. As Microsoft likes to say, in a rather different context, and with a rather different effect, Get The Facts.