Everyone gets rejected. And it never stops being painful not matter how successful or how long you have been in the business. Some of this is inevitable; not everyone is above average. But some of it isn't. I thought that I would offer some dos and don’ts for reviewers out there to improve the process and save some hurt feelings, when possible. Some are drawn from personal experience; others, more vicariously. I have done some of the "don’ts" myself, but I feel bad about it. Learn from my mistakes.
(Author's note: I'd like people to focus on the ideas in this piece, not the strong language, so I've substituted a new version with all the same points, but a few different words.)
First, and I can’t stress this enough, READ THE PAPER. It is considered impolite by authors to reject a paper by falsely accusing it of doing THE EXACT OPPOSITE of what it does. Granted, some people have less of a way with words than others and are not exactly clear in their argumentation. But if you are illiterate, you owe it to the author to tell the editors when they solicit your review. It is O.K. – there are very successful remedial programs they can recommend. Don’t be ashamed.
Second, and related to the first, remember the stakes for the author. Let us consider this hypothetical scenario. In a safe estimate, an article in a really top journal will probably merit a 2-3 percent raise for the author. Say that is somewhere around $2,000. Given that salaries (except in the University of California System) tend to either stay the same or increase, for an author who has, say, 20 years left in his/her career, getting that article accepted is worth about $40,000 dollars. And that is conservative. So you owe it more than a quick scan while you are on the can. It might not be good, but make sure. Do your job or don’t accept the assignment in the first place. (Sorry, I don’t usually like scatological humor but I think this is literally the case sometimes.)
Third, the author gets to choose what he/she writes about. Not you. He/she is a big boy/girl. Do not reject papers because they should have been on a different topic, in your estimation. Find fault with the the paper actually under review to justify your rejection.
Fourth, don’t be petty and whiny. Articles should be rejected based on faulty theory or fatally flawed empirics, not a collection of little cuts. Bitchy grounds include but are not limited to – not citing you, using methods you do not understand but do not bother to learn, lack of generalizability when theory and empirics are otherwise sound. The bitchiness of reviews should be inversely related to the audacity and originality of the manuscript. People trying to do big, new things should be given more leeway to make their case than those reinventing the wheel.
Fifth, don’t be a jerk. Keep your sarcasm to yourself. Someone worked very hard on this paper, even if he/she might not be very bright. Writing “What a surprise!”, facetiously, is not a cool move. Rejections are painful enough. You don’t have to pour salt on the wound. Show some respect.
Sixth, remember that to say anything remotely interesting in 12,000 words is ALMOST IMPOSSIBLE. Therefore the reviewer needs to be sympathetic that the author might be able to fix certain problems when he/she is given more space to do so. Not including a counterargument from your 1986 journal article might not be a fatal oversight; it might have just been an economic decision. If you have other things that you would need to see to accept an otherwise interesting paper, the proper decision is an R&R, not a reject. Save these complaints for your reviews of full-length book manuscripts where they are more justifiable.
Seventh, you are not a film critic. Rejections must be accompanied by something with more intellectual merit than "the paper did not grab me" or "I do not consider this to be of sufficient importance to merit publication in a journal of this quality." This must be JUSTIFIED. You should explain your judgment, even if it is something to the effect of, "Micronesia is an extremely small place and its military reforms are not of much consequence to the fate of world politics." Even if it is that obvious, and it never is, you owe an explanation.
Brian C. Rathbun is associate professor in the School of International Relations at the University of Southern California. This essay is adapted from a blog post by Rathbun at The Duck of Minerva.
I am only in my early 50s, yet I have lost count of the number of grant and contract review committees on which I have served. I wish I could say it is because I am some sort of academic superstar. In fact, it is just the combination of an odd specialization that crosses technical boundaries and a substantially cooperative nature. I always agree when asked by panel officers from the National Institutes of Health for two reasons: I have been on the government side of the aisle, and I desperately need people to review my stuff. The upside of my unfettered willingness to do what is essentially continuous community service is that I hear about interesting research and novel techniques that would have never crossed my desk of their own accord. The downside is that I spend a lot of time sitting in windowless hotel conference rooms in and around Washington, trying to stay engaged in conversations at 3 o’clock on Friday afternoons. I have developed waiting habits not dissimilar to those I use for long train rides and delayed flights. I watch and write narrative in my head.
I am happy to say that the story of review panels is a happy one. Despite a well-developed penchant for cynicism, I have found grant review panels actually accomplish their tasks with almost the idealistic aplomb one might hope for. This is by no means a Nicholas Sparks novel with clean moral progression and sentimental tidy outcomes, but rather more a Kazuo Ishiguro story with enough complex ambiguity to render it believable. How can 10 to 25 people of disparate background and expertise, with only limited preparation and exact knowledge, decide on the future of scientific research project funding in a 20-minute discussion? It is done the same way that a memorable high school prom is planned and executed: the power of small group process.
The first challenge of grant review is the scoring algorithms. The NIH has changed this quite recently from a 1-5 scale with decimals to a 1-9 scale with no decimals. Not surprisingly, changing the scale has not done a great deal for how people map their evaluations to the metric. Some struggle mightily to develop their own algorithms to allocate points and retain measurement fidelity across all applications. Others lead with their gut and are quite idiosyncratic from application to application. The adjudication of these mapping processes happens in those airless hotel conference rooms when the applications are discussed and scored. People engage in brinkmanship, acquiescence, passionate articulate speechifying, and occasionally embarrassing backtracking. The consequence, however, is a scoring mechanism for any given application is constructed through a rough form of consensus building. Obviously most of the debate occurs in slicing up those on the margins. Everyone in these rooms recognizes the Elvis on Velvet and the Picasso of applications. Everything else is much harder.
The second challenge is maintaining the democratic process in light of the many and varied egos and rivalries. Small group process is often derailed by a single strong personality. Grant review panels are no different, but the NIH has embedded some mechanisms to shape the process and the panels themselves enforce others. First, there is always a big alpha dog who acts as chair. The panel officers at NIH know their dogs well and usually choose those who understand the value of careful, graceful, and humorous guidance rather than bullying. I have fallen in love with many a chair over a two-day period as he or she gently guides the group to its destination. Second, for the unsolicited grant proposals NIH receives continuously, there are standing panels that have members who stay on in rotating cohorts for two to four years. The combination of familiarity and new blood expedites the process dramatically. It means that the first half-morning is not spent in the painful process of taking everyone’s measure. By the third or fourth meeting, the habits, humor, and disposition of your fellow cohort members are well-known to you and everyone else, so you spend less time thinking about process and much more time thinking about science. And because there are overlapping cohorts, the old guys train and discipline the young ones. It is usually gently done particularly for those who have not served on a panel before and are over-eager or simply nervous.
The famously contentious nature of academics is only rarely on display in these meetings because it is tough to remain in the room once you have been openly ugly and particularly hard if you have to return to a similar room four months later. Finally, the groups themselves retain an almost puritanical sense of justice and fairness that is quite surprising given that these panels usually consist of people with some mileage on them. I have seen panels collectively and clearly redirect any member who tries to subvert due process. In a panel not so long ago, a young gentleman who had been in a lifelong headlock with another researcher tried to scuttle an application simply because of this ongoing animosity. The outrage in the room was both palpable and sharp. He was, indeed, redirected.
These scientific grant panels are not some gloriously functional happy family by any means, but the careful structuring and dynamics of well-intentioned human beings in groups actually has the intended outcome. The decisions are on average correct and mostly just. There are, of course, built-in biases toward larger universities and established researchers, which some misguided champion of the underdog like me will point out with passion on a regular basis. Mistakes are occasionally made because of oversight, hubris, or a lack of expertise. But surprisingly and delightfully, it usually works. Prom night does finally come magically lit and complete with the best band ever, and most people go away happy.
Felicia B. LeClere is a principal research scientist in the Public Health Department of NORC at the University of Chicago, where she works as research coordinator on multiple projects including the National Immunization Survey and the National Children's Study. She has 20 years of experience in survey design and practice, with particular interest in data dissemination and the support of scientific research through the development of scientific infrastructure.
In today’s Academic Minute, Antonio Sagona of the University of Melbourne discusses his archaeological excavation of one of the First World War’s most famous battlefields. Learn more about the Academic Minute here.
Faculty leaders at the University of Illinois are circulating a petition calling for the removal of Michael Hogan as president of the university system, The News-Gazette reported. Faculty critics cite Hogan's push to centralize enrollment management decisions, and his "extraordinary bullying" of Phyllis Wise, chancellor of the Urbana-Champaign campus, whose e-mail messages reveal that Hogan did not think she was pushing her faculty members hard enough to back his views on enrollment management. The letter in circulation says of Hogan: "In our view he lacks the values, commitments, management style, ethics, and even manners, needed to lead this university, and his presidency should be ended at the earliest opportunity." A spokesman for the university system said that Hogan was not resigning and had "unwavering support" from the board.
The American Educational Research Association announced Friday that, in response to recent Georgia laws viewed as hostile to immigrants, the association will move its 2013 annual meeting from Atlanta to San Francisco. "As a matter of policy, AERA has an affirmative obligation to operate its own functions and monitor its own behavior in accordance with the research policies it supports, its code of ethics, and a commitment as a democratic organization to the values of equity, equality, and transparency. The relocation from Georgia helps to ensure that AERA members and other annual meeting participants have equal access to engage in AERA activities free of constraint, distraction, and intimidation that could occur under this law," said the association's statement.
The issue of when disciplinary associations should relocate annual meetings -- the locations for which are typically selected years in advance -- has created controversies in numerous fields. Currently, historians are sponsoring an online discussion on the topic.
Faculty members at Coppin State University have voted -- overwhelmingly -- that they lack confidence in President Reginald Avery, The Baltimore Sun reported. "[Avery] has brought neither a clear vision of mission to CSU, nor established a coherent or viable strategic plan, nor wisely allocated resources," wrote Nicholas Eugene, the leader of the Faculty Senate, in a letter to William E. Kirwan, chancellor of Maryland's university system. "We feel that despite the efforts of faculty, Dr. Avery's leadership has resulted in a dilution of the academic quality at CSU." Avery told the newspaper that he would work to improve communication with professors, and he vowed to continue his work at the university.