Faculty

Everyone gets rejected. And it never stops being painful not matter how successful or how long you have been in the business. Some of this is inevitable; not everyone is above average. But some of it isn't. I thought that I would offer some dos and don’ts for reviewers out there to improve the process and save some hurt feelings, when possible. Some are drawn from personal experience; others, more vicariously. I have done some of the "don’ts" myself, but I feel bad about it. Learn from my mistakes.

(Author's note: I'd like people to focus on the ideas in this piece, not the strong language, so I've substituted a new version with all the same points, but a few different words.)

First, and I can’t stress this enough, READ THE PAPER. It is considered impolite by authors to reject a paper by falsely accusing it of doing THE EXACT OPPOSITE of what it does. Granted, some people have less of a way with words than others and are not exactly clear in their argumentation. But if you are illiterate, you owe it to the author to tell the editors when they solicit your review. It is O.K. – there are very successful remedial programs they can recommend. Don’t be ashamed.

Second, and related to the first, remember the stakes for the author. Let us consider this hypothetical scenario. In a safe estimate, an article in a really top journal will probably merit a 2-3 percent raise for the author. Say that is somewhere around $2,000. Given that salaries (except in the University of California System) tend to either stay the same or increase, for an author who has, say, 20 years left in his/her career, getting that article accepted is worth about $40,000 dollars. And that is conservative. So you owe it more than a quick scan while you are on the can. It might not be good, but make sure. Do your job or don’t accept the assignment in the first place. (Sorry, I don’t usually like scatological humor but I think this is literally the case sometimes.)

Third, the author gets to choose what he/she writes about. Not you. He/she is a big boy/girl. Do not reject papers because they should have been on a different topic, in your estimation. Find fault with the the paper actually under review to justify your rejection.

Fourth, don’t be petty and whiny. Articles should be rejected based on faulty theory or fatally flawed empirics, not a collection of little cuts. Bitchy grounds include but are not limited to – not citing you, using methods you do not understand but do not bother to learn, lack of generalizability when theory and empirics are otherwise sound. The bitchiness of reviews should be inversely related to the audacity and originality of the manuscript. People trying to do big, new things should be given more leeway to make their case than those reinventing the wheel.

Fifth, don’t be a jerk. Keep your sarcasm to yourself. Someone worked very hard on this paper, even if he/she might not be very bright. Writing “What a surprise!”, facetiously, is not a cool move. Rejections are painful enough. You don’t have to pour salt on the wound. Show some respect.

Sixth, remember that to say anything remotely interesting in 12,000 words is ALMOST IMPOSSIBLE. Therefore the reviewer needs to be sympathetic that the author might be able to fix certain problems when he/she is given more space to do so. Not including a counterargument from your 1986 journal article might not be a fatal oversight; it might have just been an economic decision. If you have other things that you would need to see to accept an otherwise interesting paper, the proper decision is an R&R, not a reject. Save these complaints for your reviews of full-length book manuscripts where they are more justifiable.

Seventh, you are not a film critic. Rejections must be accompanied by something with more intellectual merit than "the paper did not grab me" or "I do not consider this to be of sufficient importance to merit publication in a journal of this quality." This must be JUSTIFIED. You should explain your judgment, even if it is something to the effect of, "Micronesia is an extremely small place and its military reforms are not of much consequence to the fate of world politics." Even if it is that obvious, and it never is, you owe an explanation.

I am only in my early 50s, yet I have lost count of the number of grant and contract review committees on which I have served. I wish I could say it is because I am some sort of academic superstar. In fact, it is just the combination of an odd specialization that crosses technical boundaries and a substantially cooperative nature. I always agree when asked by panel officers from the National Institutes of Health for two reasons: I have been on the government side of the aisle, and I desperately need people to review my stuff. The upside of my unfettered willingness to do what is essentially continuous community service is that I hear about interesting research and novel techniques that would have never crossed my desk of their own accord. The downside is that I spend a lot of time sitting in windowless hotel conference rooms in and around Washington, trying to stay engaged in conversations at 3 o’clock on Friday afternoons. I have developed waiting habits not dissimilar to those I use for long train rides and delayed flights. I watch and write narrative in my head.

I am happy to say that the story of review panels is a happy one. Despite a well-developed penchant for cynicism, I have found grant review panels actually accomplish their tasks with almost the idealistic aplomb one might hope for. This is by no means a Nicholas Sparks novel with clean moral progression and sentimental tidy outcomes, but rather more a Kazuo Ishiguro story with enough complex ambiguity to render it believable. How can 10 to 25 people of disparate background and expertise, with only limited preparation and exact knowledge, decide on the future of scientific research project funding in a 20-minute discussion? It is done the same way that a memorable high school prom is planned and executed: the power of small group process.

The first challenge of grant review is the scoring algorithms. The NIH has changed this quite recently from a 1-5 scale with decimals to a 1-9 scale with no decimals. Not surprisingly, changing the scale has not done a great deal for how people map their evaluations to the metric. Some struggle mightily to develop their own algorithms to allocate points and retain measurement fidelity across all applications. Others lead with their gut and are quite idiosyncratic from application to application. The adjudication of these mapping processes happens in those airless hotel conference rooms when the applications are discussed and scored. People engage in brinkmanship, acquiescence, passionate articulate speechifying, and occasionally embarrassing backtracking. The consequence, however, is a scoring mechanism for any given application is constructed through a rough form of consensus building. Obviously most of the debate occurs in slicing up those on the margins. Everyone in these rooms recognizes the Elvis on Velvet and the Picasso of applications. Everything else is much harder.

The second challenge is maintaining the democratic process in light of the many and varied egos and rivalries. Small group process is often derailed by a single strong personality. Grant review panels are no different, but the NIH has embedded some mechanisms to shape the process and the panels themselves enforce others. First, there is always a big alpha dog who acts as chair. The panel officers at NIH know their dogs well and usually choose those who understand the value of careful, graceful, and humorous guidance rather than bullying. I have fallen in love with many a chair over a two-day period as he or she gently guides the group to its destination. Second, for the unsolicited grant proposals NIH receives continuously, there are standing panels that have members who stay on in rotating cohorts for two to four years. The combination of familiarity and new blood expedites the process dramatically. It means that the first half-morning is not spent in the painful process of taking everyone’s measure. By the third or fourth meeting, the habits, humor, and disposition of your fellow cohort members are well-known to you and everyone else, so you spend less time thinking about process and much more time thinking about science. And because there are overlapping cohorts, the old guys train and discipline the young ones.  It is usually gently done particularly for those who have not served on a panel before and are over-eager or simply nervous. 

The famously contentious nature of academics is only rarely on display in these meetings because it is tough to remain in the room once you have been openly ugly and particularly hard if you have to return to a similar room four months later. Finally, the groups themselves retain an almost puritanical sense of justice and fairness that is quite surprising given that these panels usually consist of people with some mileage on them. I have seen panels collectively and clearly redirect any member who tries to subvert due process. In a panel not so long ago, a young gentleman who had been in a lifelong headlock with another researcher tried to scuttle an application simply because of this ongoing animosity. The outrage in the room was both palpable and sharp. He was, indeed, redirected.

These scientific grant panels are not some gloriously functional happy family by any means, but the careful structuring and dynamics of well-intentioned human beings in groups actually has the intended outcome.  The decisions are on average correct and mostly just. There are, of course, built-in biases toward larger universities and established researchers, which some misguided champion of the underdog like me will point out with passion on a regular basis. Mistakes are occasionally made because of oversight, hubris, or a lack of expertise. But surprisingly and delightfully, it usually works. Prom night does finally come magically lit and complete with the best band ever, and most people go away happy.