banner
You are not using a standards compliant browser. Because of this you may notice minor glitches in the rendering of this page. Please upgrade to a compliant browser for optimal viewing:
Firefox
Internet Explorer 7
Safari (Mac and PC)
Post Archive
2014 (0)2011 (1)
January (1)

Et tu Odysseyus?
Tuesday, January 11, 2011
2010 (37)
December (5)

First most influential paper
Tuesday, December 14, 2010

Twelve (minus one) Months of pondering Blather
Tuesday, December 7, 2010

I'll save Tideliar the trouble
Tuesday, December 7, 2010

Thinking differently
Friday, December 3, 2010

Music Warz! - The Maccabees
Thursday, December 2, 2010
November (6)

Standing out in a crowd: An addendum
Monday, November 8, 2010

Standing out in a crowd
Friday, November 5, 2010

Ripping your arms off
Tuesday, November 2, 2010

The Tea Party explained
Tuesday, November 2, 2010

Words of wisdom for prospective graduate students
Monday, November 1, 2010

Grant advice
Monday, November 1, 2010
October (10)

Planet of the Apes
Wednesday, October 27, 2010

Too many postdocs?
Tuesday, October 26, 2010

Oh! Rats! [UPDATED]
Thursday, October 21, 2010

Rethinking Education
Monday, October 18, 2010

Elephant man, rabies and leprosy
Wednesday, October 13, 2010

Over-priced mochas and syphilis
Sunday, October 10, 2010

DonorsChoose - give early and give often. [UPDATED]
Friday, October 8, 2010

Question
Wednesday, October 6, 2010

What I would be doing if I weren't doing science
Tuesday, October 5, 2010

Time spent reviewing
Monday, October 4, 2010
September (6)

Dear PI's who wrote the NSF proposals I am now reviewing...
Wednesday, September 29, 2010

Funding Illusions
Tuesday, September 28, 2010

FIve years ago today
Wednesday, September 22, 2010

A (temporary) cure for vortices of suckitude
Wednesday, September 15, 2010

Things that make Odyssey grumpy
Tuesday, September 14, 2010

What I wish I knew before starting my faculty position
Tuesday, September 7, 2010
August (10)

Flying 101
Tuesday, August 24, 2010

Don't panic!
Monday, August 23, 2010

One to Rule Them All
Friday, August 20, 2010

The NSF review panel process
Thursday, August 19, 2010

Peer review, schmeer review
Friday, August 13, 2010

Hypotheses: The most disposable of lab supplies
Thursday, August 12, 2010

How much do you need to want it?
Monday, August 9, 2010

Bugger
Sunday, August 8, 2010

REPOST: How Many Papers for Tenure?
Thursday, August 5, 2010

Checking it out
Thursday, August 5, 2010
Rate This Post
Total votes: 2
Blogger Profile

Odyssey

I'm a molecular biophysicist in a biochemistry department. In a college of medicine. And I'm funded by the NSF. Not too sure my dean likes that... I'm here to blather on about things that interest me and to raise the average age of the bloggers here by at least 1.2567 years. And I'm Australian.

My posts are presented as opinion and commentary and do not represent the views of LabSpaces Productions, LLC, my employer, or my educational institution.

Blog RSS Feed
RSS Add to My Yahoo Add to Google
Recent Comments

Haha the comment above is funny! Research before hypothesis! . . .Read More
Aug 09, 2012, 1:48pm
Comment by Tim in Peer review, schmeer review

Good article! "Do proponents of this system have any idea just how hard it is to find reviewers under the current system?" I've just stumbled across this artcile, so my comments are. . .Read More
Oct 13, 2011, 5:20pm
Comment by Tideliar in Standing out in a crowd

Rules of the Internets, #3"Don't necro dead threads n00b"Corollary Rule 3.1"If the best you can add is content-lite, don't fucking necro dead threads n00b"Corollary Rul. . .Read More
Jul 06, 2011, 5:48pm

anyone interested in helping my kindergarteners? Check out http://www.donorschoos. . .Read More
Jan 25, 2011, 10:47am
Comment by Anonymous in Standing out in a crowd

Thanks for the post! Very helpful. - Reshmi . . .Read More
Jan 13, 2011, 12:44am
Awesome Stuff
Wikio RSS:

http://www.wikio.com

ResearchBlogging:

researchblogging.org

Site Meter

Locations of visitors to this page

Blogroll
Jan 26, 2014, 10:58am
Jan 02, 2013, 8:53pm
Feb 29, 2012, 10:31am
Nov 19, 2011, 4:38pm
Feb 17, 2011, 9:40am
Feb 11, 2011, 8:27pm
Nov 07, 2010, 4:50pm
Friday, August 13, 2010

There is a recent piece in the The Scientist titled "I Hate Your Paper" that outlines the problems with the current peer review system and describes some of the alternatives that have been suggested. This has been discussed by Orac, which was followed up on by DrugMonkey.

There is no doubt that the current peer review system has some major problems. I won't defend that. However, some of the so-called "alternatives" seem to me to be prone to the same, if not worse, issues. Let us not forget,

Peer review in any shape or form is done by people.



Yes, reviewers can be snarky, unreasonable, stupid, vindictive and/or lazy.
And editors can be snarky, unreasonable, stupid, vindictive and/or lazy.

So can authors.


I've certainly been on the receiving end of what I consider to be completely unreasonable reviews.
As a reviewer I've probably written reviews that the authors have considered completely unreasonable.*
And as an editor I've probably made decisions that authors have considered completely unreasonable.*

Peer review in any shape or form is done by people.





So let's consider some of the suggested alternatives (not all, just some).

Anonymous Authors
This system would work just the way current peer review does, except that the names and affiliations of the authors are not revealed to the reviewers. The idea is that the biases of the reviewers against competitors, old enemies, humanity in general, will be minimized.

Nice idea, but it won't work. Most publications are chapters in an ongoing story being told by a lab (or labs). The authors cite the work done leading up to the current chapter, usually with emphasis on the preceding work done in the lab(s) of the authors. It is just too easy to figure out who the senior authors are.

Non-Anonymous Reviewers
In this model, apparently already adopted by some publications, the names of the reviewers are revealed at the time of publication. The thought here is that reviewers will take their jobs much more seriously if they know their name will be associated with the final product.

Riiiiggggghhhhhtttt.

Do proponents of this system have any idea just how hard it is to find reviewers under the current system? Any reviewers, not just good ones? Working in an editorial capacity I routinely have to ask 6-8 people before I can get two to agree to review (and here we're assuming those two will actually complete their reviews and in a timely manner).** And those lists of potential reviewers authors have to provide when submitting a manuscript? Often useless. Once I cull out cronies, former (and sometimes present!) collaborators and people with absolutely no expertise in the field, there's often no one left.

If I now have to tell potential reviewers their names will be revealed to the authors, what do you think is going to happen? If I ask someone to review a manuscript by Professor Standing-Member-Of-The-Study-Section-Your-Grant-Goes-To, what do you think they're going to do? What about a manuscript by Professor Senior-Dood-Who-Has-The-Most-Influence-In-Your-Field? Or Professor I-Have-A-Nobel-Prize? Most will find some excuse not to accept the invitation to review (especially young faculty if they have any sense). Those that do may not be as unbiased as one would hope. Are they really going to trash a manuscript (that deserves it) by someone who has some direct control over their future, given that, should the manuscript actually be published the names of the reviewers will be revealed? Maybe, maybe not. At least reviewer anonymity provides some semblance of protection against retribution.***

Under this system people will be very, very circumspect about which manuscripts they agree to review. Making it harder to find reviewers and making the whole process even longer than the current one.

Post-Publication Review
The idea here is that authors can go ahead and publish without review, likely on a web site somewhere, and the scientific community can decide whether or not the work holds up. There would be some sort of ranking system, perhaps comments associated with the manuscript in the same manner comments are linked to blog posts, or enumeration of how many times a manuscript is incorporated into someone's personal on-line library. This is kind of like a mega-peer review system.

I like this idea in principle. But...

Reviewing via comments? Despite valiant efforts by organizations such as PLoS, this just hasn't worked. I'm not sure why.

Ranking by counting how often an unreviewed manuscript is included in someone's personal on-line library? Seriously? Who has time to read all the relevant papers in their field that have been through the current peer-review system? Not me. And at least 50% (I suspect more like 70%) of all manuscripts never make it into print. So we're going to critically read twice as many, if not three times as many, manuscripts in order to figure out which are reasonable? No. What I suspect will happen is the known authors in a given field will have their manuscripts included in libraries, and the rest will be largely ignored...


I don't know what the answer is. As stated above, there is no doubt that the current peer review system has some major problems. But when trying to come up with an alternative, let's all keep in mind that

Peer review in any shape or form is done by people.









* The authors may or may not be right.
** I'm not talking just about over the summer when it's notoriously difficult to find people to review. I'm talking about year round, over at least the last 4-5 years.
*** And before you say "But I can always figure out who reviewed my manuscript!", let me tell you no, you can't. A very senior editor once told me that he often gets people coming up to him to tell him they know exactly who reviewed their manuscript. According to him, in 99% of cases they're wrong.

This post has been viewed: 1143 time(s)

Blog Comments
Simon Higgins

Guest Comment
I'm surprised by your editorial experience being that it is so difficult to get reviews. The only time I refuse to review a MS is when I feel it's outside my area of competence, and I nearly always review on time. Talking to colleagues here, it seems they largely operate the same way. Maybe it's field-dependent (I'm a chemist)? However, I have to admit that reviews I complete for non-learned-society, specialist commercial journals may be done with somewhat less thoroughness.

Odyssey
Rate Post:

Like 0 Dislike
@Simon:
It may well be field dependent. I have colleagues on various editorial boards that complain of the same issues I run into. Of course you shouldn't admit publicly to being such a good reviewer - you may be inundated with requests to review... :-)

Dr Becca, Ph.D.
Rate Post:

Like 0 Dislike
Great analysis, Odyssey--couldn't agree more. As much as we'd like there to be an objective "right" or "wrong" in science, it's just not always the case. Think about all the flaws we find in papers that do make it through peer review, even in really well-respected journals! Peer review is and always will be done by people, and with that, I leave you this:


Odyssey
Rate Post:

Like 0 Dislike
@Dr. Becca:
:-)

Daniel Mietchen
Rate Post:

Like 0 Dislike
I think you left out the most interesting of the variants being discussed - publishing the review reports (and the authors\' replies) along with the manuscript. This way, reviewers can still be as anonymous and blunt as they wish, but they are coerced into bringing up real arguments, and anyone in the field can flag reviews for being unfair either way. This kind of quality control is not possible in classical behind-closed-doors peer review. Example: http://www.biogeosciences.net/

Odyssey
Rate Post:

Like 0 Dislike
@Daniel:
I left that out mostly because I'm still thinking through the pros and cons of that approach (and partly because I got tired of typing). Thanks for the link - it makes for interesting reading.

Brian Krueger, PhD
Duke University
Rate Post:

Like 0 Dislike
Daniel is all over this open access peer review stuff, he's a great resource :)

biochem belle
Rate Post:

Like 0 Dislike
Re: the authors' list of suggested reviewers, I've found it an odd request and wondered just how useful they actually are.

I very much agree with the points you made about anonymous authors and non-anonymous reviewers. For the latter, the only argument you need is your main pont: Peer review is done by people. And people can be vindictive when they don't like what you've got to say. And some people have long memories.

On the last part, it's interesting to note that Nature experimented with an open peer-review system a few years back. In this case, it was pre-publication-manuscripts were posted for open review (where anyone could leave comments) and sent out for the standard anonymous review. In the end, it was essentially a failure. Participation on both sides was low, and the open reviews were no better than the closed ones. (I will hunt down a link to the experiment and post it here later.)
Kay

Guest Comment
I fully agree with this post. In my opinion, the biggest problem with current peer review is the editorial culture of dealing with rebuttal leters. A while ago, I have written about this in my blog, see here.
In brief, if you get a bad (unfair/incompetent/lazy) review for your manuscript and write a rebuttal to the editor, most likely he/she will immediately take side with the reviewer - irrespective how compelling you argue that the reviewer is wrong. I think this automatism has to be abolished - editors should understand enough of the subject matter to enable them to decide if the reviewer or the author is right. Most likely, good editors already do it like this, but there are plenty out there who either have no time to actually read the reviews and the rebuttal, or are not able to.

Odyssey
Rate Post:

Like 0 Dislike
@Kay:
Most likely, good editors already do it like this, but there are plenty out there who either have no time to actually read the reviews and the rebuttal, or are not able to.

The good ones not only take the time to read the reviews and rebuttal, but also the original submission. The really good ones filter out the unfair/incompetent/lazy reviews in the first place. One caveat though - there isn't always an editor with the relevant expertise available to handle a given manuscript. In that case the editor is very reliant on the reviewers.

The editors who "have no time" should step down from their editorships. If you accept the role, you also accept the responsibility.

One should be very careful how a rebuttal is written. If you call a reviewer unfair/incompetent/lazy the editor is likely to be automatically defensive - they were after all responsible for choosing the reviewer. Some tact/diplomacy can go a long way.

Odyssey
Rate Post:

Like 0 Dislike
@Biochem Belle:

Thanks for the Nature links!
DrugMonkey

Guest Comment

GUEST COMMENT said:

I have not found this to be the case at all, in fact I find that a reasoned, point by point response that takes the criticisms seriously gains serious credit with editors. I'd have to say that my experience is that editors will side with the authors most of the time in a situation of split review and a reasonable response to the bad reviewer
Kay

Guest Comment
Drug monkey said:
"I have not found this to be the case at all, in fact I find that a reasoned, point by point response that takes the criticisms seriously gains serious credit with editors. I'd have to say that my experience is that editors will side with the authors most of the time in a situation of split review and a reasonable response to the bad reviewer."

Hmm, it seems like you are dealing with other editors than I have to. My experience (and that of several friends and colleagues) is that a split review essentially means that the paper is dead. My interpretation is that those journals don't want to risk publishing something that turns out to be wrong later (with a reviewer standing up, yelling "see, I told you so")

And as we all know, the best method (for a competing reviewer) to kill a manuscript is not to write "this is all rubbish", but rather to say "this is potentially a really interesting story - the only thing that needs to be done for publication is this tiny but highly important control (which doesn't really add anything, but will keep the authors busy for another 6 months, giving me sufficient time to scoop them)"

Finally, I don't want to leave the impression that I am a conspiracy theorist who doesn't get his papers published. This is not the case, and I am really a big fan of peer review as it exists today. Just try to convince an editor that a control experiment requested by a rogue reviewer isn't really necessary (and is next to impossible to do)
Tim

Guest Comment

Good article!

"Do proponents of this system have any idea just how hard it is to find reviewers under the current system?"

I've just stumbled across this artcile, so my comments are a bit late. But given the question above, I have two comments.

First, I'd happily review a paper if the reviewer's knew my identity. I have rejected papers before and given the authors my email address if they didn't understand my comments. I find that they are always polite when they contact me. I suspect it is because I make a great effort not to be dismissive and over critical, and I also either make sure that I'm right, or I openly admit my doubt if I'm not sure. Nothing pisses me off more than a dismissive reviewer comment that I'm quite convinced they wouldn't make if they had to put their name to it.

Second, one possible solution to your problem could be used if most journals agree to implement it: to submit a paper to a journal, you have to have reviewed X papers for that journal first (where X is probably 2-3). This wouldn't work for new journals, which would have to use a policy that to get back a review, you would first have to review X papers. This latter  works for special issues and book chapters that I have submitted to -- I see no reason why it wouldn't help for journals in general.

Cheers,

Tim

Add Comment?
Comments are closed 2 weeks after initial post.
Friends