'What Just Ain't So'

Mark Twain once observed that “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” Twain may not have been referring to what scholars assume humanities publishers want out of the peer-review process, but he probably would have appreciated the application.

According to the troubling “Report of the MLA Task Force on Evaluating Scholarship for Tenure and Promotion” (2007), monograph publishing in the arts and humanities remains the key to the city of Academe. Moreover, in spite of plunging sales and soaring costs, monographs remain central to the mission of university presses and vital to the careers of scholars. In the monograph publishing equation, scholars play dual, crucial roles: they both write manuscripts and judge the value of others’ manuscripts. Reviewing manuscripts can benefit scholars and bring them into a respected circle. It can also inadvertently become detrimental to one’s career. Both roles demand academic rigor and a commitment to objectivity.

While the demonstration of academic rigor and objectivity is demanded of the author, it unfortunately at times collapses on the side of the reviewer. I’m not sure why this happens, but I know that it does. And when it does, the process gets dicey for publisher, author, and reviewer alike. Interestingly, while authors and publishers likely see the problem especially clearly, reviewers often seem alarmingly unaware of the stakes. In their role as peer reviewers, scholars might profit from knowing the publisher’s unspoken expectations and objectives; such knowledge might help them avoid the potential fallout from missing those expectations.

Keeping in mind that university presses publish roughly 10,000 new books annually, and assuming that they publish only 1 out of every 10 manuscripts, that means university presses are filtering 100,000 manuscripts per year. Of those, probably 15,000–20,000 get sent out for peer review. Granted, reviewing a manuscript for a university press is tantamount to taking a vow of poverty, but it will be part of your career either as an author or a reviewer. Odds are pretty good that you will, at some point, be asked to review a manuscript. Odds are even better that you will not tackle the assignment in the same way as others, with the same skills, or with the same assumptions about what the publisher wants.

Most publishers do provide — or should provide — readers with guidelines for evaluating a manuscript. Generally publishers want feedback about originality, documentation, how the work interacts with scholarship, suitability for classroom use, etc. This article will not revisit those issues. Here I’d like to dig a little deeper to reveal what I look for in the review process that scholars may think they know, but which “ain’t so.” By becoming aware of what publishers want (and don’t want), you might improve the value of your evaluation for all parties. Ultimately, a well-executed review should fulfill the professed scholarly allegiance to objectivity and academic rigor. Equally important, an ineffective review not only fails to give me what I want and need, it could jeopardize your future professional relationships with both publishers and other scholars.

Reviewers, even those who follow our guidelines, may think that they’re giving me what I want, but they often wind up submitting something else. Here are a few of the more egregious disconnects.

The New York Times book review. The request to peer review a manuscript is not the same as the request to review a book. Evaluating a manuscript’s merits and recommending or not recommending publication may seem like a book review, but it’s not. A scholarly book review unpacks a book’s argument, sets it in the context of similar works, and assesses the book’s value in that scholarly discussion. The book reviewer thus must assay a fixed object and place it in the scholarly grid of information. While the evaluation of a manuscript may (perhaps even should) include some of these elements, reviewing an unpublished manuscript is a fundamentally different process. Most obviously, an unpublished manuscript is still pliable: a case can be argued more strongly, a thesis written more clearly, and conclusions can sharpened to further advance understanding. The genre of the manuscript in hand should also be weighed: If the volume is designed as a course text, it should not be evaluated in the same way as a revised dissertation. Less obviously, reviewers often forget –– or take perverse pleasure in knowing –– that the career and future of a scholar hinge on the successful publication of a book manuscript. The published monograph can be the signal event in a professor’s career. Don’t get me wrong: the responsible reviewer should not take this last circumstance too lightly in either direction. This poses no ethical dilemma but a heuristic decision. You must be both responsible and decisive in either recommending or not recommending publication, and you must be able to articulate the “why or why not” behind either. In other words, as a first step in evaluating a manuscript, you should apply the same academic rigor and objectivity to assessing a manuscript as you would in writing your own book. Then look over your shoulder and remember that the review –– for good or ill –– may have consequences for your own career.

The "why-didn’t-you-write-a-different-book?" review. Really. It may not happen regularly but it occurs enough to be a vexing problem: A reviewer evaluates a manuscript as if it were written on a topic other than what the author has claimed. Many an editor has secured two reviewers who seem to be evaluating two different manuscripts. What is happening? Often one reviewer is evaluating an imaginary document other than the one that was written. For example, consider a manuscript examining the sociopolitical motives behind the burning of witches in 15th-century Europe. The author might have given only a passing nod to the economic and political history of firewood and forestry in 15th-century Europe. As one might imagine, a reviewer with expertise in late medieval firewood and forestry might perceive the lack of a discussion of the latter unforgivable. Piqued by the oversight the reviewer contends that the author has missed the whole point and takes the author to task for having discussed neither the history of the woodcutters’ guild nor the origins of German hardwoods. And while that may be a fine book, it’s not the manuscript the reviewer was asked to review and clearly not the book the author wished to write. Admittedly other causes can stand behind such a misreading: The author may have written unclearly in the first place. If that’s the case, then the book’s lack of an identifiable or a convincing thesis should be noted as a legitimate shortcoming. You should posit why the thesis fails or is indiscernible. But that’s not the same as peremptorily deciding that the manuscript should be about another subject and reviewing it accordingly.

A second, familiar permutation of misreading a manuscript takes place when the reviewer puts the manuscript back in its manila folder, pours a glass of Merlot, and discusses instead the reviewer’s own work, which by the way, should have been cited in the manuscript. This might be precipitated by ego, or it might be triggered by the failure to have actually read the manuscript. Lack of expertise might figure here as well. The outcome, however, is the same: I am left holding a feckless review that says more about you than the manuscript. As you submit the report, you may be pondering how good it feels to have corrected a major shortcoming of the manuscript, but I’m thinking it “ain’t so.” A review that so wildly misses the point forces me into an untenable position. What do I do with a vacuous review? The review has become part of the record. Sharing the review with the author risks a confrontation if not a lawsuit. Not submitting the review risks alienating you or violating procedures for handling manuscripts. Then there’s the question, should I recruit a new reviewer or just scratch the project? The next step toward writing a successful review is reviewing the manuscript sent.

The slashing review. And then there’s the slashing review. Clearly, more than a few peer reviewers regard the evaluation as an opportunity, to paraphrase Mencken, “to raise the black flag and begin slitting throats.” Regrettably, such actions come off more like Freddie Krueger than H.L. Mencken. Some seem to assault a manuscript gleefully: Brandishing their own cleverness and wit, they slash through the pages, intent not on evaluating the manuscript but on demonstrating their prowess. While everyone loves a black flag –– and Mencken –– the purpose of a peer review is neither to slit the throat of the manuscript’s author nor parade the reviewer’s rapier wit.

First, that kind of criticism proves helpful neither to the author nor ultimately to you, and, again, such actions push me into a dark corner. How do I return a manuscript stained with the author’s own blood? How do I say “thanks”? The slashing technique is misguided, but its creator clearly possesses an eye for weaknesses in the manuscript that might have be turned toward a more constructive reading. That would have actually helped. But the damage is done, and not just to the author and publisher. The slasher’s review –– and any inappropriate review –– will probably receive its own asterisk in the file: “Don’t ask Professor X to review manuscripts.” When paired with the “why-didn’t-you-write-a-different-book? review, the slasher’s review is a show-stopper. Then there’s the question, even in a double-blind arrangement, will I be able to protect your identity? Remember: in Academe only two degrees, not six, separate the author from the reviewer. Overspecialization means everyone knows everyone else. The short-term delight in savaging a manuscript will be hard to remember when the only person qualified to review your next book on woodcutter guilds in late medieval Germany is the individual whose manuscript you just shredded.

The intellectual comb over. Perhaps the most frustrating review and most impervious to fixing is the non-review, the intellectual comb over. This ploy simultaneously fails to review the manuscript and tries to cover up the failure by posing as an astute and perceptive scholarly assessment. While one might guess that this conceit occurs most frequently in negative evaluations, that is not the case. Reviews become even more problematic when they recommend publication without having exercised the kind of scholarly judgment and objectivity demanded by the evaluation process. The comb over is characterized by vague, unsubstantiated recommendations: tell-tale phrases like “needs to tighten up argument,” “could use some reorganizing,” “should consider restructuring argument,” “needs to add signposts,” or “needs to flesh-out this point” appear throughout. Add a long strand of so-called essential bibliographic references and a few names in the field, and voilà: A bogus review thinly conceals what might be (mis)interpreted as either laziness or incompetence. While all of these recommendations may indirectly address valid complaints, they are so vague as to be useless. Publishers — and authors –– need concrete, specific advice and the rationale behind it. I desire — indeed require — in the peer-review process the same academic objectivity and scrutiny that is demanded of scholarly writing itself.

I’ve read too many unhelpful reviews, plenty of valuable reviews, and a few stellar ones. The stellar ones remind me that the art of peer reviewing a manuscript remains one of the hallmarks of scholarship. Academics, especially humanists, often speak of themselves as being in “the guild.” If ever there was a time for a member to mentor a fellow-guild member, it’s in the peer-review evaluation. Here the craftsman or grandmaster can instruct the apprentice in the fine art of scholarship. And though the instruction may or may not result in publication, the report should emulate the twin standards of the guild: academic rigor and objectivity. Applying these standards to the peer-review process not only ensures a quality review; it keeps you in good stead with your peers. I also won’t file your name under “Don’t ask Professor X to review manuscripts,” which, on the one hand, may sound like a relief, but on the other, it places you and maybe your career on the periphery of a vital scholarly circle.