A peer review of peer review

I had a paper published last week which was notable for two reasons: first, it was the culmination of about three years work on a project to ensure consistent application of a European Directive across the EU, and second because it was the first time that I had complained to an editor about the quality of a peer review.   The paper was entitled Comparing aspirations: intercalibration of ecological status concepts across European lakes for benthic diatoms and had a total of twenty co-authors representing twelve different countries. I do not pretend that it was a perfect manuscript and, indeed, hammering out a consensus with so many points of view on offer inevitably leads to loose ends that are hard to tuck out of sight. But one of the two referees appeared to me to have an agenda that went beyond an impartial evaluation of the paper.

Now I have recently had the task of finding peer reviewers for a special issue of Freshwater Science that I co-edited and I know that it is a thankless task.   People who I thought were the best suited to evaluate papers turned out to be busy, ill or otherwise unavailable. I had to cast the net wider to find individuals with enough specialist knowledge to assess the work and, in many cases, send out reminder emails to make sure that they completed their reviews within a reasonable period.   My own experience of having papers peer reviewed is that the process nearly always results in a better paper, simply by having two or three dispassionate readers commenting on it, but it is not perfect. I also read enough papers that seem to have slipped through the peer review process with serious flaws to make me wonder if peer-review really is giving us the gold standard of papers that is often claimed.

One big conceptual flaw lies at the heart of peer review: Thomas Kuhn outlined it in The Structure of Scientific Revolutions in 1962 when he argued that science advances by episodic leaps that he described as “paradigm shifts” rather than by the gradual accumulation of facts and evidence. He used the replacement of Ptolemaic astronomy by the new ideas of Copernicus and others as a case study, noting how the flaws in the old theory of the sun and planets orbiting the earth had been accommodated by making the calculations of planetary motion more and more complicated and it needed the radical thinking of Copernicus to provide the paradigm shift which, coincidentally, resulted in a much more straightforward set of calculations.

My concern is that, for a very conservative discipline such as that within which I work, the likelihood of getting a referee, or even two, steeped in “Ptolemaic” thinking is greater than getting a Copernicus.   Indeed, most of the mid-career onwards scientists who form the backbone of the peer reviewer community will have made their careers within the “Ptolemaic” paradigm. Of course, for every Copernicus there were probably a dozen wannabe paradigm busters whose work deserved to be rejected and it is probably better that peer reviews default to the conservative option. But, at the same time, peer review can also serve to reinforce the conservative position.

In my case, the editor reached her own opinion which was favourable to our paper.   The rogue reviewer’s comments were rambling and poorly reasoned and, indeed, many of his (or her) criticisms were, in fact, already answered elsewhere in the paper. So I guess that the system worked in the end. In any case, for all its faults, I cannot think of a workable alternative to peer review so I guess that we are stuck with it for the moment. I do sometimes wonder, however, how Copernicus would have fared if the publication of his treatise depended on a favorable judgement by two or three reviewers steeped in Ptolemaic thinking.


One thought on “A peer review of peer review

  1. As a non-professional reading journals out of personal curiosity more than anything else I’ve often wondered about peer review; particularly that the “peer” portion should represent a tiny portion of the specialized subset of a scientific community, rather than a broad section of scientists in associated fields. Then I remember that fans of NASCAR disputing the superiority of a particular driver could pass for peer review and die a little inside that the judgment of a minority should constitute a test of merit.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s