The Peer-Review Crisis

Gale Sinatra, professor of psychology and the Stephen H. Crocker Professor of Education at the University of Southern California, is stepping down as associate editor of the American Psychological Association’s Journal of Experimental Psychology: Applied. Officially, it’s because she’s becoming an associate dean for research and won’t have as much time to devote to her editorship. But the new job is only part of it: like so many other journal editors, Sinatra is facing a serious shortage of available scholars to review submitted articles, and it’s a problem she can’t solve on her own.

Worse Than Ever

This issue isn’t new: academic publishing has long been a delicate system that operates—tenuously—on goodwill, in the form of comprehensive, unpaid article analyses from expert volunteers. But the pandemic has pushed this system to breaking, or close to it. With academics’ professional and personal lives disrupted in so many ways for years now, this kind of labor is increasingly harder to source: journal editors across fields say scholars are significantly less likely to accept article-review requests, if they respond at all, and (to a lesser degree but concerningly nonetheless) they are more likely to return reviews that are late or even rushed.

At the same time, journals’ overall submission numbers haven’t decreased to the extent many anticipated during COVID-19, and they have actually increased in many fields, especially those in which researchers were studying the pandemic in some way.

“I’ve had a good run. I’ve done three journals, [and] I’ve enjoyed all three of these experiences. But I’ve peaked out because it’s just become too difficult,” Sinatra said of trying to find reviewers, chasing down late reviews and, worst of all, apologizing to the scholars who understandably want to know if and when their delayed articles will be published. These are often scholars who are looking for jobs, going up for tenure or facing other high-stakes decisions that turn on their publication records. (This is not to mention the fundamental problem of delays in the timely publications of valuable research.)

Using very rough estimates, Sinatra said an article that may have taken three months to get reviewed before the pandemic—a turnaround time for which many journals strive—could take six months now. And in some cases, Sinatra hasn’t been able to find a third reviewer at all and had to offer more extensive editorial comments to make up for having only two.

“I just don’t know what we’re going to do. The model has to change,” Sinatra said.

Ken Hanson, an assistant professor of sociology at the University of Wyoming and managing editor of Socius: Sociological Research for a Dynamic World, an open-access journal from the American Sociological Association, said it used to take three or four tries to find a willing reviewer for an article. Now it takes around eight, he said. At the same time, the journal is busier than ever in terms of submissions.

Consistent with other reports, Hanson said that the real downturn in reviewer availability happened not at the beginning of the pandemic but this past academic year. (One study of ecology journals actually found no drop-off in reviewer participation in the first six months of the pandemic, and that reviewers actually replied more quickly to invites and agreed to return their reviews more quickly then.)

In 2020, Hanson, said, “I think that people were excited to review COVID papers, and there was a sense of, ‘This is important. This is going to help us get out of this and better understand the pandemic.’ So there wasn’t that reviewer lag. Plus, I think people were more actually locked down. So people that weren’t burdened with having children or having the disease—whatever the situation was—were like, ‘I can review a paper because I’m just sitting at home.’”

Yet as things have “gotten closer to quote-unquote normal,” Hanson continued, “the burnout has kind of caught up to people. The 2021–2022 academic year has been particularly bad. And I would say since March it’s been severe.”

Tanya Joosten, co-director of the National Research Center for Distance Education and Technological Advancements at the University of Wisconsin at Milwaukee, said she recently served as a special editor for a project and asked “everyone in the U.S. that was doing research in this area” to review a particular article on digital learning, to no avail.

“Every single person said, ‘No, we would love to help out in this sort of thing, but we just don’t have the time,’” Joosten said, noting that the digital learning field has been especially flooded with submissions since and about COVID-19. Joosten said one of her own articles was stalled at a journal, as well: after she inquired about it multiple times, an editor told her they’d only been able to secure one reviewer. She’s since pulled the article and seen it accepted elsewhere, but she said the experience delayed her study results getting out by a year.

“It’s delaying the dissemination of scientific knowledge across the board,” she added. “It’s like our knowledge creation as a society is slowing down and speeding up at the same time.”

In Joosten’s experience, junior faculty members are often motivated to review because they’re enthusiastic about being at that stage in their careers, she said, and she’s personally motivated to review because she’s trying to build the “rigor” of the digital learning field. But for many others in academe, she asked, “What’s the motivation to review?”

Paid Reviews

One possible answer? Paying reviewers for their work.

Roland Hatzenpichler, assistant professor of environmental microbiology at Montana State University, said it’s been about a year since he started refusing to review for for-profit journals for free. In response to one journal’s recent request that he review an article, for instance, Hatzenpichler thanked the editor for the invite but said that because the publication is owned by a major for-profit company with high profit margins, “my consulting fee of $200 per hour applies. Please let me know if these terms are acceptable and I will consider whether I can accept the invitation and/or suggest alternative reviewers. Please note that I will charge a one-time fee of $50 for the latter because I would be effectively doing the work you are being paid for free otherwise.”

No journal has taken Hatzenpichler upon his offer thus far. He doesn’t necessarily expect any to do so. He won’t stop asking, however.

“I’m not saying I have the solution to the problem, I’m just saying that I cannot offer my services to a company that gives nothing back to the community,” he said. (Hatzenpichler said his lab publishes almost exclusively in open-access journals and that he remains happy to review for publications run by nonprofit organizations such as scholarly societies. Although he started demanding payment from for-profit journals during COVID-19, he said the timing of his decision was much less about the pandemic than about his personal maturation as a scientist.)

Just as problems with the peer-review system aren’t new, only worse, paying reviewers for their work is an old idea that’s gotten a lot more attention during the pandemic.

One example: scientist James Heathers wrote in a widely read late 2020 essay—with some irony and perhaps more foresight—that it was only after leaving precarious academic employment for a secure, well-paying job at a technology start-up that he realized it makes sense to pay reviewers for their time and expertise.

In “our bold new astonishingly tenuous academic hellscape, this is a straightforward matter of commerce. Fiscal reality,” he wrote, warning that “the plague” would make it harder than ever for many to “afford” being an academic.

Heathers explained that his own corporate consulting rate was $250 per hour, and that his academic consulting rate is far lower or even free, even though “journal groups have more money than God.” Considering the taxes that consulting work is subject to, Heathers estimated that his peer-reviewing fee would be about $50 to $150 per hour. He spends anywhere from three to nine hours reviewing a paper from start to finish, he concluded, so “if we put the low with the high, and the high with the low we get … $450, actually, both ways. Give me $450” per article.

Heathers’s “450 Movement” was debated at last year’s Researcher to Reader conference on scholarly communication, where Heathers had a perhaps-unexpected ally: Brad Fenwick, senior vice president at Taylor & Francis, one of a handful of major for-profit publishers.

“Some editors are well compensated for their efforts. So why would the same approach not be applied to peer reviewers?” Fenwick said during the debate, according to Science. “Universities provide faculty with the freedom to supplement their income as paid consultants and/or by being involved in for-profit businesses. There’s no reason that their contribution to the publishing industry should be treated in a lesser fashion.”

On the other side of the debate, Alison Mudditt, CEO of PLOS, the nonprofit publisher of open-access articles, said at the event, “There is no practical way to pay reviewers without wrecking peer review. Reviews vary wildly in length, quality and complexity. Where would we start with assessing an appropriate fee? Why pick $450? There are some articles that are so intricate that perhaps only a handful of experts on earth can review them. One large society publisher tells me that [a fee of only $350] would wipe out the surplus they returned to the society—no more investment in the society’s research and researchers.”

Mudditt also cited a 2018 survey by Publons, a website where academics can track their editorial contributions, which found that cash payment is not a significant incentive to review, ranking No. 6 in a list of initiatives that would make researchers more likely to review (the survey specifically addressed reviewing grant proposals, but this in many ways parallels peer reviewing publications). More professional recognition for this work came in at No. 1. The same survey found that the reviewing workload is not evenly distributed, with just 4 percent of reviewers doing 25 percent of reviews.

A separate Publons survey on global peer review found that 85 percent of respondents think institutions should more explicitly require and recognize peer review. This survey also found that article submissions grew 6 percent between 2013 and 2017, and article publication volumes grew by about 3 percent over the same period.

Sinatra, of USC, also disagreed with the idea of paying reviewers, saying she worried it would introduce perverse incentives into the work and result in poorer-quality reviews. Instead, Sinatra proposed that that academics with a paper under review at particular journal should be required to complete one review for that same journal, in a kind one-to-one exchange of labor: “If you’re under review, you should be expected to complete a review. I think it’s only fair and that would help alleviate the backlog.”

To increase the pool of reviewers, Sinatra said she’s also been on Twitter encouraging journal scholars to join editorial review boards and to write to editors in their areas of expertise saying, “‘I’m available,’ because it is sometimes an issue of not knowing who’s out there.”

Sara Shulist, an associate professor of linguistic anthropology at Queen’s University in Canada, said her own philosophy on peer-review requests is that “I try to accept more often than not,” but she didn’t like the idea of reviewing requirements based on submissions.

“I don’t love a hard rule,” in general, Shulist said, “and I also think it depends on the people submitting. I think there’s a big difference between tenured people and securely employed people and contingently employed people. Contingent-employed people may want to submit but shouldn’t be asked to do reviews. I think that should be distributed among senior scholars more.”

A corollary of this is whether the work is paid or not, Shulist said—most urgently for non-tenure-track scholars who are compensated for teaching only, not for the research and service tenured jobs are understood to involve.

“I can reasonably argue that service of this type is covered under the broad category of my tenured job, whereas other people who don’t have such a job are doing this completely for free.”

Regarding Fenwick’s participation in the $450 debate, Taylor & Francis Group said at the time that it was an intellectual exercise, not company policy, and that there were no plans to broaden a company practice of paying for accelerated reviews for a small share of journals focused on pharmaceutical development.

Hatzenpichler, at Montana State, said that while he understood some academics’ concerns about payment resulting in sloppier or biased reviews, he argued that it’s important to remain at least “agnostic” about the issue until it’s been studied extensively.

Other Solutions—and a Bigger Problem

Ken Kolb, chair of sociology at Furman University, said he got some pushback this month for tweeting that “if you are tenured, you’ve gotta do some peer reviews, like pretty regularly,” as “junior folks’ lives are on hold, waiting on reviewers.” He remained unapologetic, saying it’s increasingly challenging to help junior scholars in his department map out their paths to finding jobs or getting tenure when the timelines for their career-making publications are in flux because journals can’t find reviewers.

“It’s almost like we need an Ice Bucket Challenge of ‘do one review a month, you know, for 12 months,’ just to clear the backlog—whatever it could be,” Kolb said, referring to the successful 2014 fundraising campaign for amyotrophic lateral sclerosis research and care. “I know that’s not the most popular thing, because it’s work. It’s extra work. I’m salaried, so I feel like everything that I do is paid for in such a way, but the number of things that I’ve been asked to do over the course of my career has gotten more and more and more, in terms of service to the discipline, service to my institution—all this other stuff besides my core tasks of teaching and research. But I just feel for human beings who are up against the clock, and they don’t really have time to wait for long-term reforms.”

That said, Kolb told Inside Higher Ed that he’s willing simultaneously to entertain longer-term solutions to the problem. This includes curbing the expansion of—and possibly eliminating—revise-and-resubmits, he said, citing a new paper from sociologist Christine L. Williams called “Abolish the R&R.”

Of revise-and-resubmits, in which journal editors offer extensive feedback on papers but don’t accept them, Williams wrote, “The goal of this labyrinthine and lengthy process is to publish scientifically sound sociology, but it has significant downsides. Many scholars can tell horror stories about revising a paper for years only to have it rejected after multiple rounds of reviews. What can be done to improve the publication process?”

The first step, Williams continued, “is to eliminate the R&R. After peer review, editors should have two options: reject or conditionally accept. A conditional acceptance means that the editor or their deputy commits to working with the author until the paper is published. There could never be a second round of anonymous reviews, which only encourage reviewers to dig in their hills, pick at nits, become combative, etc. There may be occasions when the editor needs to consult with a specific reviewer a second time to verify that the author has correctly answered the questions raised by the review. However, in this case, the editor has already become a negotiator and advocate for the author, not a disinterested arbiter.”

Kolb said other long-term concerns include “the proliferation of journals,” which results in more editors asking for reviews, and the “diminishing amount of tenured faculty to be able to review.”

This last point is the most intractable, given that the vast majority of the academic workforce is now employed off the tenure track. And it’s at the very root of the peer-review crisis, to the extent there is one: when academe’s shrinking share of securely employed professors compensated for work across teaching, research and service becomes suddenly more stressed, forcing them to triage their many duties more than they already do, there is no slack in the system to absorb the work that goes undone. And yet this undone work remains vital, especially for those waiting on their work to be reviewed.

As Ryan Cordell, associate professor of information sciences and English at the University of Illinois at Urbana-Champaign, said on Twitter last week, “The peer review crisis in higher ed may seem narrow, but it’s an example of how COVID exposed the precariousness of broader labor structures—peer review runs almost exclusively on the civic spirit of field professionals—so the systems only work while there isn’t *any* disruption.”

It’s “hard to see a way out because *everyone* is so overextended & burnt out—there’s no secret, massive reserve of refreshed & energized peer reviewers who could sweep in & fix things—it’s burnt out folks all the way down,” he added. “Which means the real answers are, like they are across labor sectors in 2022: hire more people, give them fairer contracts, reduce exploitative workloads. [In other words] real solutions would require labor solidarity across academic tracks & ranks because everything else is a bandaid.”