A recent blog post has drawn some attention to the cost - to academics - of running the peer review process:
So that's over £200million a year that academics are donating of their time to the peer review process. This isn't a large sum when set against things like the budget deficit, but it's not inconsiderable. And it's fine if one views it as generating public good - this is what researchers need to do in order to conduct proper research. But an alternative view is that academics (and ultimately taxpayers) are subsidising the academic publishing to the tune of £200 million a year. That's a lot of unpaid labour.
Indeed, an earlier estimate put the figure even higher:
a new report has attempted to quantify in cash terms exactly what peer reviewers are missing out on. It puts the worldwide unpaid cost of peer review at £1.9 billion a year, and estimates that the UK is among the most altruistic of nations, racking up the equivalent in unpaid time of £165 million a year.
Whatever the figure, it is significant, which brings us on to the inevitable questions: why are researchers making this donation to publishers, and do they need to?
The thought I had listening to Neylon talk about peer review is that it is yet another case of a system that was originally founded to cope with scarcity - in this case of outlets for academic papers. Peer review was worth the cost of people's time because opportunities to publish were rare and valuable and needed husbanding carefully.
Today, of course, that's not the case. There is little danger that important papers won't see the light of day: the nearly costless publishing medium of the Internet has seen to that. Now the problem is dealing with the fruits of that publishing abundance - making such that people can find the really important and interesting results among the many out there.
But that doesn't require peer review of the kind currently employed: there are all kinds of systems that allow any scientist - or even the general public - to rate content and to vote it up towards a wider audience. It's not perfect, but by and large it works - and spreads the cost widely to the point of being negligible for individual contributors.
For me what's particularly interesting is the fact that peer review is unnecessary for the same reason that copyright and patents are unnecessary nowadays: because the Internet liberates creativity massively and provides a means for bringing that flood to a wider audience without the need for official gatekeepers to bless and control it.
Follow me @glynmoody on Twitter or identi.ca.
Hard to word this... replacement is perhaps the key. If a reviewer will spend more time combing through non-reviewed papers when actively researching than they spend in the review process then...
ReplyDelete@zotz: but there's no reason that should happen. If the Net's natural filtering process is allowed to kick in, people will get to hear about the good papers - and avoid the bad ones.
ReplyDeleteAdditionally, there is, believe it or not, "office politics" at play in the larger academic community. Such "low-stakes treachery" may be remedied if the peer-review process were dismantled.
ReplyDelete@TheImplementor: yes, that's a good point - transparency would bring all sorts of benefits.
ReplyDeleteThis sparked a good FF discussion: http://friendfeed.com/cameronneylon/d78e49d3/abundance-obsoletes-peer-review-so-drop-it
ReplyDelete@Bill - thanks for letting me know.
ReplyDeleteThis is *exactly* what I hoped would happen: that people who actually knew what they were talking about in this field (unlike me) would explore some of the ideas I was throwing out.
And you are right, I am talking about abundance in publishing (I'll try to add something to make that clear).
"But that doesn't require peer review of the kind currently employed: there are all kinds of systems that allow any scientist - or even the general public - to rate content and to vote it up towards a wider audience."
ReplyDeleteAs flawed as the peer review process is, I'm not sure this works either. Isn't the suggested model fairly close to what PLoS One is currently doing? I've found in my field, at least, very few papers have any comments, let alone ratings, on the PLoS One site. Why not?
I've also found that the "voting it up" tends not to work when it comes to controversial data. The forum tends to become an echo chamber where dissenting opinions are consistently voted down.
Opening it up to the general public will result in only the fringe, conspiratorial, or "sexy" topics being "voted up." Have you seen general public's comments and votes at YouTube? Do you really want to see scientific manuscripts treated that way?
@Carl: you're right, this is a non-trivial exercise, and PLoS One hasn't exactly been a roaring success.
ReplyDeleteBut I think that's partly because people haven't really tried to do this much: more experiments are needed to handle the problems you mention.
The point of the peer review process is not only to determine what research is interesting, but what research is scientifically valid. Peer review is carried out by experts in the article's field of study who evaluate whether the best possible methods have been used, whether the data and statistical analysis are of high quality, whether the proper controls have been used, and whether the conclusions are valid. This isn't something that can be replaced by a 'Like' button open to everyone in the world. I'm happy to see that even the editors of journals don't claim that they're knowledgeable enough to make these decisions. Science is too diverse and too dynamic for all of us to be able to discern whether research coming from a different subfield is of sufficient quality for us to take it seriously.
ReplyDeleteI would argue that the abundance of literature and the ease with which it can be published makes peer review even MORE necessary - without it, agenda-pushing and biased studies would become even more prevalent than they already are. I'm not arguing that the peer review process is perfect - even now there is plenty of poorly done research that gets published. But if you take peer review away, then there will be no critical analyses of research provided by the scientific community save on a voluntary basis, which would therefore be both biased and rare (as observed in PLoS One) - even with peer review being the norm, journals have a hard time finding referees. Without peer review, lower quality research would get published and we (neither scientists nor the general public) won't know what to believe. These are not problems you can fix by providing an option for upvotes and YouTube-style comments.
@Raeka: peer review isn't perfect, as you say. So maybe there are other ways of doing it - not just through simple voting up or down, but parallel efforts that seek to check validity and provide the rigour you rightly seek.
ReplyDeleteThe trouble is, if we don't try out new ways, we will be stuck with the imperfections of peer review - which may well grow as the volume of papers increases.
I certainly agree that new strategies should be tried, and indeed journals are already thinking of ways to improve the peer review process. These include making it more transparent by publishing reviewers' comments along with an article, which would discourage referees from being nasty and unconstructive, and providing incentives or a merit system for scientists who review lots of papers, which would hopefully ensure that all the papers out there do get reviewed and in a timely fashion.
ReplyDeleteBut I stand by the opinion that pre-publication peer review is necessary. The publication of research is a sign of its quality, reliability, and novelty. The Net's natural filtering process cannot guarantee any of these things. Adopting the approach of "publish-first-judge-later" will, quite frankly, result in more crap being published, which doesn't help anyone.
@Raeka: yes, maybe what we need is a new kind of peer review - more open, as you say - to avoid abuses, and to allow more input from a wider circle of people.
ReplyDelete