Peer Review: Reform and Renewal in Scientific Publishing

Challenges of Peer Review

Peer review has become ubiquitous in scholarly journals, being seen as the hallmark of a journal’s credibility and what qualifies research as science. Yet, as we have seen, peer review did not originate as the guarantor of the validity of research, and it is clear that it has not always performed that function effectively. For example, a study was conducted at the British Medical Journal (BMJ), taking a paper that was about to be published and introducing eight errors. The paper was then sent out to 420 potential reviewers, 221 of whom responded. None of the respondents spotted more than five of the errors; the average respondent identified only two errors; and 16 percent didn’t identify any errors at all (Godlee, Gale, and Martyn). However, it is unclear how many of the reviewers continued to review the paper after spotting the first two or three errors.

A similar conclusion is demonstrated looking at published papers. García-­Berthou and Alcaraz (2004) found statistical inconsistencies in 38 percent of papers in Nature and 25 percent in the BMJ. The number of papers retracted every year for problems in methodology, for breaches of research ethics, or for fabrication of data is indicative of the fact that peer review does not, and cannot, be a truth sieve through which only valid science can pass (see http://​www​.retractionwatch​.com/). John Ioannidis (2005) presented a probability algorithm to demonstrate the likelihood that most published studies are false. As well as methodological problems, factors such as conflicts of interests, prejudice, and perceptions concerning what are the “hotter” topics have served as ineffective filters for journals. These are problems of editorial peer review that, among other things, increase the likelihood that a journal will publish articles with false conclusions. If we accept that the majority of published work is incorrect and that the primary purpose of peer review is to prevent the publication of false and invalid research, then it is clear why many now question the system of traditional peer review.

One can say that peer review does serve a useful function in the filtering of submissions, citing research into the fate of rejected articles. Fifty to sixty-­five percent of articles rejected from the Annals of Internal Medicine were published elsewhere, indicating that as many as 50 percent of submissions were simply unpublishable and rightly kept out of the scientific literature (Williamson).

If we were to set the standard lower and say that we expect peer review to improve the quality of papers, then the results are less clear. A study published in the Journal of the American Medical Association (JAMA) would seem to bear this out, concluding that “editorial peer review, although widely used, is largely untested and its effects are uncertain” (Jefferson et al., “Effects”). Similarly, a Cochrane review (2003, updated in 2007) of studies into peer review found “little empirical evidence to support the use of editorial peer review as a mechanism to ensure quality of biomedical research, despite its widespread use and costs” (also see Hopewell et al.).

However, the absence of study data about the effects of peer review is not the same as evidence that peer review has a null effect. Surveys of researchers consistently find that peer review is perceived as improving the quality of papers. A survey of 4,000 people conducted by Sense about Science found that 9 out of 10 authors believe that peer review improved their last published paper (Mulligan, Hall, and Raphael). This is the same proportion reported by a similar survey of 3,040 academics conducted by the Publishing Research Consortium (Ware; also see Lu). In the perception of authors and reviewers, the question is not whether peer review improves the quality of papers but to what extent and in what areas. In descending order, Ware (2008) found that authors perceived some improvement in presentation (94 percent), language or readability (86 percent), missing or inaccurate references (78 percent), scientific errors (64 percent), and statistical errors (55 percent). These opinions indicate that peer review does improve (or at least is perceived to improve) the quality of papers but is less significant in those areas that might be considered most integral—­that is, questions of validity.

The effectiveness of peer review is also challenged by the fact that reviewers are not often unanimous in their recommendations. Rothwell (2000) found that the probability of two reviewers agreeing were only slightly better than chance. While using two to three independent reviewers is usually considered the standard for peer review, Rothwell concluded that you would need six reviewers per paper to produce a reliable result. Those journals that operate with only a single reviewer per paper—­and there are plenty of such journals—­are particularly vulnerable to the inconsistency in peer judgment. This discrepancy occurs not only in the general recommendation but in the specific requests for revisions. Those authors unfortunate enough to have new reviewers look at a manuscript they have revised can sometimes find themselves being requested to add in elements that the previous reviewers asked them to remove (or vice versa).

In addition to inconsistency, there are other perceived problems with peer review, including the speed of the process, financial cost, potential bias, instances of abuse, and difficulty of detecting fraud or misconduct (Williamson).

The speed of peer review, and thus the speed of publication, is often a surprise for the uninitiated. The fact is that peer review is conducted unpaid by academics or practitioners who have other, more rewarding, calls on their time: “A single peer review takes about four hours, but organizing two or three reviews takes on average four months or more” (Johnston). In the fast-­paced area of medicine, for instance, waiting a quarter of a year or more for a first decision delays dissemination of significant, sometimes life-­changing, results. While there are tools and techniques for reducing unnecessary delays, improving the speed of peer review will be difficult while it depends on reviewers donating their spare time.

The cost of peer review is a significant concern, especially for publishers in an age of limited growth in subscriptions and disruption to the market from open access and other forms of digital dissemination. The cost of the editorial process is between $90 and $600 per article, far more costly than copyediting or printing or other publishing services (Williamson). This cost is incurred even though reviewers are not paid; it includes editor honoraria, fees for electronic systems, and the salaries of editorial office staff, among other things. Were reviewers to be paid, it is unlikely that they would ever be paid equivalent to the time they devote to peer review, and in most cases, such reviewer payments would prove prohibitive to continued publication. Those journals that do pay reviewers often cover the costs by charging a submission fee.

Bias in peer review is more difficult to evaluate. Williamson (2003), for example, asserts that there is evidence of prestige bias, geographical bias, and gender bias. Other common concerns about peer review are the danger of confirmation bias (the tendency to read evidence as confirming your own preconceptions) and the conservatism of peer review, favoring consensus over proposals from outside the current paradigms. However, in a wide-­ranging review examining such studies, Lee et al. (2012) questioned Williamson’s methodological assumptions and found little empirical evidence for these concerns about bias. Anecdotal evidence does indicate that bias and prejudice occurs, though it is difficult to determine how widespread it is and what impact it has on publication.

It is well known that peer review can be open to abuse. One common problem is that the line between authors and reviewers can actually become blurred. Some have encountered cases where a supervisor of the lead author was initially recommended as reviewer of a paper only to later claim to also be an author on the same paper. There have been other cases where reviewers (or even editors) have attempted to gain authorship credit on a paper by offering to collaborate with authors for a revised version. This problem of “gift authorship” can be preempted by adopting clear policies about what qualifies one as an author, but the incentive of an easy publication means that this form of abuse will always be a danger.

More recently, so-­called peer review rings have emerged, where authors have created fabricated reviewer accounts that allowed them to subvert and manipulate the peer-­review process. Hundreds of published articles were retracted after it became apparent that authors reviewed their own work or had a friendly third-­party review done on their behalf (Oransky).

Other recorded forms of abuse would leave the casual observer incredulous, ranging from attempts by reviewers to plagiarize studies they review, “scooping” the original authors by rapidly publishing or releasing their data in a public forum such as an academic conference, and even giving authors the option to recommend reviewers (many journals struggle to find reviewers, which is why they sometimes ask authors for help—­the potential for abuse is obvious, though). There are now agencies that offer to create fake e-mail accounts for unscrupulous authors to use for their recommended reviewers so that, if selected, these accounts can provide favorable reviews and increase the chances of publication. This problem highlights an overdependence on recommended reviewers by editors, who often struggle to get reviewers to agree, as well as an overdependence by editors on the recommendations of reviewers in preference to their own assessments of a paper.

In noting that reviewers are “almost useless” at spotting fraud or misconduct, Williamson is really only highlighting the fact that reviewers are limited by circumstance. Reviewers were not present when a research study was conducted, when the results were analyzed, or when the conclusions were authored. Often the only information the reviewer will have is the submitted paper. If the authors have misreported their methodology, omitted reference to their unethical conduct, or fabricated their data, how would the reviewer be able to tell? Sometimes a reviewer might be able to identify that the reported results look “fishy” or too good to be true, but accusations of misconduct would be hugely detrimental to a researcher’s career, and reviewers are unlikely to bandy such accusations lightly. To address this concern, efforts such as the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network have emerged. The EQUATOR Network is an international initiative that promotes transparent and accurate reporting and wider use of robust reporting guidelines. Currently they provide information on more than 300 guidelines (http://​www​.equator​-network​.org).

A reviewer might spot other types of fabrication, such as image manipulation. But most lack the training and expertise to identify all possible forms of manipulation. Similarly, a reviewer can only spot plagiarized material if he or she happens to have read and remembered the prior publication. While reviewers can assist in spotting cases of misconduct, it is simply unrealistic to expect reviewers to identify such misconduct consistently (let alone infallibly).

Another threat to the peer-­review system is the advent of “predatory” open-­access publishers. These are sham journals that exist only to take advantage of the author-­pays business model by publishing almost anything for a price. Most of these fake publications claim to conduct peer review but in fact do not. Young researchers from developing countries are particularly at risk of becoming the victim of predatory publications (Xia). Awareness of predatory open-­access publishers has grown due the work of Jeffrey Beall, a librarian at the University of Colorado Denver, as well as some high-­profile exposés. In 2010, Beall coined the term “predatory publisher” and created what has come to be known as “Beall’s List of potential, possible, or probable predatory scholarly open-­access publishers” (Butler). In 2009, Phil Davis, a Cornell University doctoral student interested in investigating the increasing prevalence of scam journals, submitted a manuscript composed of computer-­generated nonsense to a suspected predatory journal (Basken). The paper was accepted and the ruse exposed on the popular industry blog the Scholarly Kitchen. More recently, John Bohannon, a staff writer for Science magazine, targeted the open-­access system in 2013 by submitting an intentionally deeply flawed paper to more than 300 open-­access journals. Approximately 60 percent of those journals accepted the fake medical paper (Bohannon).

While some have tried to use the practices of predatory publishers to paint all open-­access journals and all forms of peer review as illegitimate, it is important to note that most open-­access journals are authentic and do carry out peer review in an effort to publish material that contributes to the body of scientific knowledge.