Peer review

Recently, I have submitted four papers to a conference with different co-authors. After the peer review process, one was accepted for a talk, the three others for a poster. I do not copy all the reports here because it would be boring, but just the marks we received to the question “is this worth a talk”, ranging from +3 to -3. You will see a pattern emerge.

Paper 1 (the one that was accepted) had three reviewers: marks 3, 2 and 1

Paper 2 had three reviewers: 2, 0, -2

Paper 3 had two reviewers: 1, -2

Paper 4 had two reviewers: -3, 2 (the first reviewer, having noticed a few typos, mentioned “poor right up” [sic] as one of the reasons not to consider our submission).

Do you see the pattern? No? Look more closely… YES, you have got it: peer reviewing is random number generation 😉

Technical corollaries:

(1) With little post-processing, any correlation with the content of the paper can be removed for papers 2-4.

(2) Paper 1 is special, not because there is no spread, but because the average is not centered around 0. This bias is robust and can be eliminated only by suppressing buzzwords.

Advertisements

About valerio

Principal investigator at Centre for Quantum Technologies and professor at National University of Singapore

Posted on March 6, 2012, in Academic advices. Bookmark the permalink. 1 Comment.

  1. Peer review is the worst form of deciding what to publish, except for all the other forms that have been tried from time to time…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: