Category Archives: Peer reviews
In the remote preparation for my Coursera on randomness, I read Nate Silver‘s The signal and the noise. I am not sure how much of it will enter my course, since I don’t plan to enter into the topics he deals with (politics, the stock market, climate change, prevention of terrorism, baseball and poker). But the conclusion struck a cord.
The author lists seven approximations to describe the “efficient market hypothesis”, which run: 1. No investor can beat the stock market, 2. No investor can beat the stock market over the long run, and so on until approximation 7 which a is five lines long sentence. Then he adds (emphasis is mine):
“The first approximation — the unqualified statement that no investor can beat the stock market — seems to be extremely powerful. By the time we get to the last one, which is full of expressions of uncertainty, we have nothing that would fit on a bumper sticker. But it is also a more complete description of the objective world.”
Sounds familiar? Let’s give it a try:
- Bumper sticker: No extension of quantum theory can have improved predictive power
- Expression full of uncertainty: the authors work under the assumption of no-signaling (so, if you are Bohmian, don’t worry, our result does not concern you). Then they assume a lot of quantum physics, but not all of it, otherwise the claim would be tautological. Beyond the case of the maximally entangled state, which had been settled in a previous paper, they prove something that I honestly have not fully understood. Indeed, so many other colleagues have misunderstood this work, that the authors prepared a page of FAQs (extremely rare for a scientific paper) and a later, clearer version.
- Comment: the statement “Colbeck and Renner have proved that quantum theory cannot be extended” is amazingly frequent in papers, referee reports and discussions. Often, it comes in the version: “why are people still working on [whatever], since Colbeck and Renner have conclusively proved…?” It is pretty obvious however that many colleagues making that statement are not aware of the “details” of what Colbeck and Renner have proved: they have simply memorized the bumper sticker statement. I really don’t have a problem with Colbeck and Renner summarizing their work in a catchy title; what is worrisome is other experts repeat the catchy title and base decisions solely on it.
- Bumper sticker: The quantum state cannot be interpreted statistically [Yes, I know that the title of the final version is different, but this is the title that sparked the curiosity of the media]
- Expression full of uncertainty: the authors work with a formalization of the notions of “ontic” and “epistemic” that is accepted by many people, though not by Chris Fuchs and some of his friends. They add a couple of other reasonable assumptions, where by “reasonable” I mean that I would probably have used them in a first attempt to construct an epistemic model. Then they prove that such an epistemic model is inconsistent.
- Comment: too many people have commented on this paper. The latest contrary claim has been posted online today, I have not read it because I am really not following the debate, but for those who are interested, here it is.
- Bumper sticker: either our world is fully deterministic or there exist in nature events that are fully random [the use of “either-or” makes it too elaborated for a real bumper sticker, but for someone who browses these papers, the sentence is basic enough]
- Expression full of uncertainty: the authors consider a very weak source of randomness, something like a very biased coin; in fact, it can be more perverse than that, because it can have correlation over various tosses. But it cannot be completely perverse: the authors make an assumption about its structure (technically known as “Santha-Vazirani” by the names of the first two persons who proposed it). Then they prove that, if this source is used as seed for a specific quantum experiment, the outcomes of the experiment are guaranteed to be much more random. In the limiting case of an experiment lasting infinitely long time, and whose results do not deviate by any amount from the optimal result allowed by quantum physics, the source can contain almost no randomness, while the final list will be almost fully random.
- Comment: in a paper just published, we studied what happens if we remove the Santha-Vazirani assumption, so that the source can be as perverse as you wish. Not surprisingly, the conclusions become more pessimistic: now, one would need a fair amount of initial randomness in order for the quantum step to produce further randomness. Nothing wrong at all: some guys get a good result with an assumption, others test the limit of the assumption, this is the normal course of science. But read again the bumper-sticker statement: taken in itself, out of the paper where it belongs, that statement has not been “scientifically proved” — it even sounds closer to being impossible to prove, without the crucial assumption
Recently, I have been very lucky with the referees of my papers. It is a good time to recall the occasions where I had been not so lucky in the past, without the danger of venting unresolved personal angers; and give some advice to young referees.
So, imagine you receive an article to referee from a prestigious journal; you don’t really understand what it is about, but your gut feeling is that it should be rejected. There are good and bad ways of doing that.
Here are some hateful strategies which border the unethical:
- The paper presents a new idea and studies only the simplest case: dismiss it with “There is no much interest in the topic” or “The idea is interesting but is not applied to a realistic scenario”.
- The paper provides the solution to an open problem: dismiss it with “The idea is not new” or “The result is not surprising”.
- The ever-successful “it is not of sufficient broad interest to justify publication in this journal” can be applied at any time.
Notice how 1 and 2 can be used in sequence to block a full series of papers: one can first reject the initial idea by saying that the authors should work more, then reject the extensions by stressing that the idea has already been presented before.
How do you see that these strategies are nasty? Basically, they use a very generic statement, against which the authors have no possibility of scientific reply. If the authors are told “not broad interest”, their only hope is to argue “yes, look better, it is interesting”, not very convincing; if the authors are told “not surprising”, they can only try and convince the editor that “surprise” is maybe not a criterion, but editors get very nervous if you try to teach them criteria for acceptance…
So, what are better ways of rejecting? It depends on each paper, but when you write a report, you can check some items:
- If your criticism can be applied to a paper you admire, it is wrong. For instance, with strategy 1 above, you could have rejected the original quantum teleportation paper: bad idea, for sure. As for criterion 2, I am sure you have written yourself some paper with very good, but “not surprising” results. Never use an argument that can be used against yourself!
- If your report is topic-independent, it is wrong. Always base your rejection on the scientific content of the paper: “This is a marginal improvement over [work that you consider similar]”, “This is not new since it was already mentioned in [review paper]”…
- As a referee you must feel moderately competent in the field, otherwise it is better to decline (journals know that I decline a lot). Assuming you think you are competent, the burden of clarity lies on the authors, not on you. So it is fine to write something like “It is not clear to me what these guys are trying to prove, they say it is very different from [reference] but it looks the same to me. Unless they clarify their contribution, I cannot accept”.
P.S. as written in the first paragraph, this post refers to the case where the referee does not have strong objective arguments for rejection, but just a gut feeling (which may be perfectly right and licit) that a paper is of low quality. If the referee has objective arguments, of course he/she should use them!