Are editors manipulating citation scores in order to inflate the status of their publications? Are they corrupting the rankings of scholarly journals?
While any allegations about cheating or other academic chicanery are cause for concern, journal rankings to date continue to offer one rough but useful source of information to a wide variety of audiences.
Journal rankings help authors to answer the omnipresent question “Where to publish?” Tenure review committees also use rankings as evidence for visibility, recognition and even quality in the academic review process, especially for junior candidates. For them, journal ranking becomes a proxy when other, more direct measures of recognition and quality are not available. Given that many candidates for tenure have recent publications, journal rankings become a surrogate measure for the eventual visibility of that research.
Yet it is easy to rely unduly on quantitative rating scores. The trouble arises when journal rankings becomes a stand-in for the quality of the research. In many fields, research quality is a multifaceted concept that is not reducible to a single quantitative metric. For example, imposing a single rule -- for example, that top-quartile journals count as “high-quality” journals while others do not -- assigns more weight to journal rankings than they deserve and generates the temptation to inflate journals’ scores.
In an editorial in the journal Research Policy, editor Ben R. Martin voiced his concern that the manipulation of journal impact factors undermines the validity of Thompson/Reuters Journal Citation Reports (JCR). He concludes that “… in light of the ever more devious ruses of editors, the JIF [journal impact factor] indicator has lost most of its credibility.” A journal’s impact factor represents the average number of citations per article. The standard, one-year impact factor is calculated by summing up citations to articles published in a journal within the last year, divided by the number of articles published.
I share the suspicion and unease that many academics feel about excessive reliance on journal impact scores for the purposes of academic evaluation and tenure decisions. Yet, while I am not a fan of impact scores calculated over a one-year period, my research on journal rankings leads me to conclude that Martin’s concerns are overstated.
The two main sources of manipulation that Martin discusses are coercive citations (whereby editors require authors to add citations to the journal in question) and creating a queue of online articles, which artificially inflates the number of citations per published article. While any intentional manipulation of journal rankings is reprehensible, to date the overall effect of this type of behavior in practice is quite limited. I arrive at this sanguine conclusion after exploring a variety of indexes and data sources in a forthcoming assessment of journals in my own field of sociology.
A clear hierarchy of journals in sociology is evident no matter what data source (Web of Science or Google Scholar) one uses. There is a great degree of commonality across measures in describing this gradient, even though many low-ranked journals are bunched together with quite similar scores. Manipulation of one-year data has not altered the overall picture a great deal (at least not yet) because five-year measures yield very similar rankings. And even to manipulate the one-year impact factor, editors would have to insist that new authors cite the most recently published articles in that journal.
Substantively, I doubt that much manipulation in sociology journals occurs because, first, the raw scores have not inflated over time and second, the relative ranking of more than 100 journals has been quite steady. Individual journals here and there have moved up and down slightly, but these changes are much more readily attributable to changes in the level of scholarly interest in particular subfields and editorial choices than to any individual editor’s efforts to game the system.
The main reason I discount concerns about manipulation is that different approaches to journal rankings produce a broadly similar picture of inequality. In my study, I use Google Scholar data to calculate the h-index for journals. This measure focuses on the top-cited articles over an extended time period rather than the average citation in a short time frame. It would not be easy for journal editors to manipulate this measure, even if they were aware of it its use.
Let’s take citations to Martin’s own journal, Research Policy, as an example. I obtain an h of 246 over the period from 2000 to 2015. That means that 246 articles cited at least 246 times have been published in this journal during this time frame. That is an impressive score, exceeding the visibility of the American Economic Review (h=227 over the same time period) and the American Sociological Review (h=162). (I calculated all figures with A. W. Harzing’s 2015 Publish or Perish software using Google Scholar data.)
The h statistics just cited reflect the remarkable visibility of these leading journals. It would be quite difficult to develop strategies to artificially generate enough citations to significantly alter those scores. I prefer the use of h as a measure because it attempts to capture the skewed nature of scientific scholarship. Yet the fact remains that the overall hierarchy of journals is broadly similar whether the h-index or the conventional impact factor is used.
In their 2012 study, Allen W. Wilhite and Eric A. Fong present data of concern regarding the prevalence of coercive citations. The pattern of coercive citation was particularly pronounced in lower-tier journals, and especially in the field of business and management. Yet again, I doubt that the overall journal regime is appreciably altered by dubious editorial gaming stratagems. Wilhite and Fong identify eight journals in which this practice might be common enough to matter (more than 10 reports of coercive rankings), but none of those journals has made its way into the top tier in the field (as measured in the JCR standings). In other words, by dint of a relentless and long-term commitment to manipulation, some third-quartile journals might be able to inch their way into the second quartile by manipulating scores, but that is unlikely to alter the overall contours of the field.
If a significant group of low-visibility journals undertook a major effort to increase their citations, that would make them as a group harder to distinguish from the top journals. In the field of sociology, there is no indication that middle- and lower-tier journals are narrowing the distance from the most frequently cited journals. Indeed, this enduring gap is itself interesting, in that it suggests that search engines are not increasing the visibility of journals to which few individuals subscribe.
At the same time, we need to remember that journal rankings serve as only a rough proxy for visibility and recognition of individual papers. In other words, articles published within the same journal will vary in how often they are cited. In my analysis of 140 sociology journals over the period from 2010 to 2014, most of the 10 most frequently cited papers were not published in the top-ranked journals. Thus, substantial variability in visibility (citations) within journals coexists with broadly stable patterns of inequality between journals.
In addition, the list of top-cited articles is largely impervious to self-citation. It is simply too difficult to cite oneself enough to vault one’s research into this echelon on visibility. For example, the top 10 cited journal articles in sociology from 2010 to 2014 had 400 or more citations. To catapult one’s own paper into this citation stratosphere would require publishing hundreds of papers in just a few years. No one could possibly publish frequently enough and cite themselves regularly enough to affect inclusion in the list of top-cited papers. And anyone prolific enough to implement such a strategy would not need to game the system.
Authors have a natural desire to seek outlets that will enhance the visibility of their research. In the field of sociology, that involves a choice of pursuing the most selective generalist journals, the top journals in each specialty area within the field, the second-tier generalist journals and then other remaining specialty outlets and interdisciplinary journals. The use of journal ranking data may be marginally useful in informing such choices. Other important factors include each journal’s particular focus, its selectivity, turnaround time, policies regarding second and third rounds of revisions, and so on.
Journal rankings are likely to remain with us because such rankings are of interest to so many parties, as research by Wendy Nelson Espeland and Michael Sauder suggests, even while their value is likely to remain contested. Perhaps a clearer recognition of the imprecision inherent in journal rankings will mean that they will be used judiciously, as a complement rather than a substitute for important and difficult academic evaluations. And perhaps the use of a variety of different journal indexes will reduce the temptation to game the system and redirect efforts back toward selecting high-quality research for consideration by the scholarly community.
Jerry A. Jacobs is professor of sociology at the University of Pennsylvania and former editor of the American Sociological Review.
The story is told of how, during an interview at a film festival in the 1960s, someone asked the avant-garde director Jean-Luc Godard, “But you must at least admit that a film has to have a beginning, a middle and an end?” To which Godard replied, “Yes, but not necessarily in that order.”
Touché! Creative tampering with established patterns of storytelling (or with audience expectations, which is roughly the same thing) is among the basic prerogatives of artistic expression -- one to be exercised at whatever risk of ticket buyers demanding their money back. Most of the examples of such tampering that Robert L. Belknap considers in Plots (Columbia University Press) are drawn from literary works now at least a century old. That we still read them suggests their narrative innovations worked -- so well, in fact, that they may go unnoticed now, taken as given. And the measure of Belknap’s excellence as a critic is how rewarding his close attention to them proves.
The late author, a professor of Slavic languages at Columbia University, delivered the three lectures making up Plots in 2011. Belknap’s preface to the book indicates that he considered the manuscript ready for publication at the time of his death in 2014. Plots has an adamantine quality, as if decades of thought and teaching were being crystallized and enormously compressed. Yet it is difficult to read the final paragraphs as anything but the author’s promise to say a great deal more.
Whether the lectures were offered as the overture to Belknap’s magnum opus or in lieu of one, Plots shuttles between narrative theory (from Aristotle to the Russian formalists) and narrative practice (Shakespeare and Dostoevsky, primarily) at terrific speed and with a necessary minimum of jargon. Because the jargon contains an irreducible core of the argument, we might as well start (even though Belknap does not) with the Russian formalists’ contrast between fabula and siuzhet.
Each can be translated as “plot.” The more or less standard sense of fabula, at least as I learned it in ancient times, is the series of events or actions as they might be laid out on a timeline. The author tweaks this a little by defining fabula as “the relationship among the incidents in the world the characters inhabit,” especially cause-and-effect relationships. By contrast, siuzhet is how events unfold within the literary narrative or, as Belknap puts it, “the relationship among the same incidents in the world of the text.”
To frame the contrast another way, siuzhet is how the story is told, while fabula is what “really” happened. The scare quotes are necessary because the distinction applies to fiction and drama as well as, say, memoir and documentary film. “In small forms, like fairy tales,” Belknap notes, fabula and siuzhet “tend to track one another rather closely, but in larger forms, like epics or novels, they often diverge.” (Side note: A good deal of short fiction is also marked by that divergence. An example that comes to mind is “The Tell-Tale Heart” by Edgar Allan Poe, where the siuzhet of the narrator’s account of what happened and why is decidedly different from the fabula to be worked out by the police appearing at the end of the story.)
Belknap returns to Aristotle for the original effort to understand the emotional impact of a certain kind of siuzhet: the ancient tragedies. An effective drama, by the philosopher’s lights, depicted the events of a single day, in a single place, through a sequence of actions so well integrated that no element could be omitted without the whole narrative coming apart. “This discipline in handling the causal relationship between incidents,” says Belknap, “produces the sense of inevitability that characterizes the strongest tragedies.” The taut siuzhet chronicling a straightforward fabula reconciled audiences to the workings of destiny.
Turning Aristotle’s analysis into a rule book, as happened in later centuries, was like forcing playwrights to wear too-small shoes. The fashion could not last. In the second lecture, Belknap turns to Shakespeare, who found another way to work:
“He sacrificed the causal tightness that had served classic drama so well in order to build thematic tightness around parallel plots. Usually the parallel plots involve different social levels -- masters and servants, kings and courtiers, supernatural beings and humans -- and usually the plots are not too parallel to intersect occasionally and interact causally at some level, though never enough to satisfy Aristotle’s criterion that if any incident be removed, the whole plot of the play should cease to make sense …. Similarity in plots can be represented as the overlap between two areas, and those areas may be broken down into individual points of similarity, dissimilarity, contrast, etc. Without knowing it, a Shakespearean audience is making such analyses all the time it watches a play, and the points of overlap and contrast enter their awareness.”
It’s not clear whether Belknap means to include the modern Shakespearean audience -- possibly not, since contemporary productions tend to trim down the secondary plots, if not eliminate them. But the Bard had other devices in hand for complicating fabula-siuzhetarrangements -- including what Belknap identifies as “a little-discussed peculiarity of Shakespearean plotting, the use of lies.” In both classical and Shakespearean drama, there are crucial scenes in which a character’s identity or situation is revealed to others whose confusion or deception has been important for the plot. But whereas mistakes and lies “are about equally prevalent” in the ancient plays, Shakespeare has a clear preference: “virtually every recognition scene is generated primarily out of a lie, not an error.”
In a striking elaboration of that point, Belknap treats the lie as a kind of theatrical performance -- “a little drama, with at least the rudiments of a plot” -- that often “express[es] facts about the liar, the person lied to or the person lied about.” The lie is a manipulative play within a play in miniature. And in Hamlet, at least, the (literal) play within a play is the prince’s means of trying to force his uncle to tell the truth.
Now, such intricate developments at the level of form also involve changes in how the writer and the audience understand the world (and, presumably, themselves). The Shakespearean cosmos gets messier than that of classical drama, but loosening the chains of cause and effect does not create absolute chaos. The motives and consequences of the characters’ actions make manifest their otherwise hidden inner lives. To put it another way, mutations in siuzhet (how the story is told) reflect changes in fabula (what really happens in the world) and vice versa. Belknap suggests -- tongue perhaps not entirely in cheek -- that Shakespeare was on the verge of inventing the modern psychological novel and might have, had he lived a few more years.
By the final lecture, on Dostoyevsky’s Crime and Punishment, Belknap has come home to his area of deepest professional interest. (He wrote two well-regarded monographs on The Brothers Karamazov.) Moving beyond his analysis of parallel plots in Shakespeare, he goes deep into the webs of allusion and cross-referencing among Russian authors of the 19th century to make the case that Crime and Punishment contains a much more deliberate narrative architecture than it is credited with having. (Henry James’s characterization of Russian novels as “fluid puddings” undoubtedly applies.)
He even makes a bid for the novel epilogue as being aesthetically and thematically integral to the book as a whole. Other readers may find that argument plausible. I’ll just say that Plots reveals that with Belknap’s death, we lost a critic and literary historian of great power and considerable ingenuity.
Everyone knows how hard it is for new Ph.D.s, especially for new Ph.D.s in the humanities, to find tenure-track jobs, and everyone knows about the outrageous working conditions faced by contingent faculty. The problem with these problems, however, is that they are one problem we have come to think of as two. It is a labor problem of epic proportions -- at a time when labor is having its own problems.
If I had a dollar for every time I’ve heard someone proclaim that there are “no jobs” in the humanities, I’d be able to buy every graduate student in my department a hot meal. Just this month, the American Academy of Arts and Sciences published the latest addition to the literature on the decline in jobs for faculty in the humanities. Yet the truth is that there are jobs: lots and lots of jobs. The prevalence of MOOCs notwithstanding, classrooms across the nation still need to be staffed by instructors. The problem is that most of these are per-course assignments that pay horribly and offer few benefits, if any. Demand for instructors remains high. It is the egregious working conditions and compensation for those who instruct the majority of our students at the majority of our universities and colleges that are at the heart of the matter.
The consequences of thinking about the plights of graduate students and adjuncts as if they were separate problems are many and insidious. Because there are “no jobs,” many people advise their brightest undergraduates not to enter academe. It is actually considered irresponsible not to try to dissuade the young and talented from following in our footsteps. Some people applaud the shutdown of graduate programs in the humanities at institutions of lesser prestige and the downsizing of elite programs because there are “too many Ph.D.s.”
While I do understand the desire to protect students from the anguish of not finding tenure-track jobs, culling the competition is a mistaken and destructive approach to the problem of perceived scarcity. Instead, we need to turn our attention to the ways our institutions hire, compensate and retain educators. This is a labor problem that can be resolved -- but it will not be resolved by thinning our own ranks.
The consequences of our bifurcated thinking about graduate students and contingent faculty can be subtle, as long as you are not an adjunct instructor, in which case they are toxic. Adjunct faculty members, for example, are generally hired in a completely different manner than the full-time faculty members whom we groom our graduate students to aspire to be. Whereas the hiring process for tenure-stream faculty most commonly calls for the input of several or all members of a department as well as the approval of umpteen levels of an administration, it is customary for department heads alone to hire adjunct instructors, sometimes sight unseen. By conducting our adjunct hiring sub rosa, as it were, we reinforce the split between the “real” jobs we prepare our students to compete for and the invisible (but all-too-real) jobs of people who are often teaching alongside them.
The message these vastly divergent hiring practices send to graduate students and contingent faculty alike is that much of the teaching performed in their midst is somehow something to be ashamed of (and that teaching doesn’t really matter). Everything about the manner in which we select adjunct faculty suggests that they occupy the status of a necessary evil -- the less spoken of, the better.
The Normalization of a Two-Tiered System
This rotten system arose on our watch. How did we let it happen? Speaking for myself, I was so busy trying to find a job after completing my doctorate in 1997 that I didn’t pay much attention to the bigger picture. All I could think about was my own situation. Even though I understood that the odds of getting a tenure-track position were against me, I spent my time trying everything I could think of to improve my chances. Getting a job was up to me, I told myself. Oblivious to the highly individualistic ethos implanted in me in graduate school, I figured that if I was good enough, I would succeed. I did not think of the many other graduates who were also desperate to find tenure-track jobs -- except for when I wanted to make myself feel better about the jobs I didn’t get.
I found a job -- a three-year term position that turned into a six-year term position -- whereupon I devoted myself to becoming even more irresistible as a job candidate the next time I had to go on the market. When I finally got an assistant professorship at the institution that employs me today, my thoughts turned to getting tenure.
It’s embarrassing to admit this, but even though I disapproved of the treatment of contingent faculty, I just wasn’t paying attention to the way the naturalization of their exploitation was taking place concurrently with my own professionalization. I never thought of myself as having any say in the matter: without a stable position from which to voice my opposition, I just looked on as administrations chipped and hacked away at humanities programs across the country, cutting costs by depleting programs of their tenure lines and replacing them with adjunct slots. Like most people I knew in the humanities, I felt helpless to do anything about the seemingly irreversible decline of the profession.
No one asked me or my fellow graduates how we felt about the adjunctification of the professoriate as we were trying to claw our way into permanent positions. But no one ever does ask you about such things. The normalization of the two-tiered system just manages to steal up on you, and suddenly, you’ve got tenure in an industry dominated by massive exploitation, denial, resurgent elitism and magical thinking. I am pretty sure that most tenured and tenure-track faculty do not approve of the current condition of adjunct workers, but I am absolutely certain that even those who don’t think about adjunctification do feel terrible about the lack of decent employment prospects for their own graduate students. Relinking these problems will allow us to see clearly this dilemma for what it is: a labor problem.
Of late, adjunct faculty members have bravely come together in unions and affinity groups that are creative and ambitious in character. Such organizations as the New Faculty Majority have formed nonprofit and lobbying entities to unite adjuncts and broadcast their plight to the public at large. Graduate students have united to form unions and political action groups of their own. And yet tenure-track, and more significantly, tenured faculty have yet to form a national organization dedicated to resolving this problem. Why is that the case? Why is it that those of us who occupy relatively privileged positions are the readiest to accept that this is just the way things are?
Calling on Tenured Faculty
The American Association of University Professors has focused valiantly on these issues for a long time, but the AAUP cannot do it alone. It’s time for those of us who have relatively secure positions to speak out in our communities and on a national level. Tenured professors have considerably more leverage than graduate students or adjunct instructors in our institutions; it’s up to us to come together to put pressure on our administrations to make the many invisible positions we fill under the table into “real” jobs. We need to do it for all of our students, present and future, undergraduate and graduate, academe bound and otherwise. If many of us are already working under austerity conditions at our institutions and feel our own jobs imperiled, so much the more reason to act now to secure a living wage for all who teach at the university level. It is in the interest of all faculty members to band together to demand a future for higher education.
So how can we begin to change this situation? Unionization is an obvious answer and an important part of the solution, but when the interests of the full-time faculty are pitted against those of the part-time faculty, with each group represented by a different union, it is hard to reach any sort of common ground. When the full-time and adjunct faculty are represented by the same bargaining unit, adjunct faculty do not have an easy time getting support from their tenure-stream colleagues, in large part because of the great difference in the modes of hiring and reviewing that I discussed above. As long as we continue to accept this two-tiered labor model, there is only so much that unions can do. (A rare exception was the strike two years ago at the University of Illinois at Chicago, where full-time faculty walked out in solidarity with their adjunct colleagues.)
The labor movement we will need to assure the future of higher education requires action on the local and national fronts. Tenured faculty must get involved, and vocally involved, at every level of governance at our institutions. We must vigilantly attend and participate in every kind of meeting open to us that bears any relation whatsoever to mission, staffing, funding and budgeting priorities -- and demand representation when such meetings are closed to us. We must insist, loudly and incessantly, on full-time positions in our departments. We might want to experiment with the establishment of a more rigorous screening procedure for hiring contingent faculty so as to draw attention to the need for higher pay to draw and keep the most qualified applicants. We need to write letters, articles, emails, blogs -- anything we can -- about this problem. We need to get creative.
And we need to lobby our elected representatives at the local, state and national levels to address this systemic labor problem. Tuition rates have been skyrocketing across the country at the same time as full-time positions have been slashed; wherever this money is going, it is not being spent on full-time positions. Like increased tuition and student debt, this two-tiered system plays a role in the dynamics of income inequality. Legislators need to reverse the decades-long retreat of state and federal governments from the funding of higher education. Colleges and universities are not going to change their practices unless we put pressure on them from without and from within.
Some people will say it’s too late to do anything about this trend. But do we really believe so little in ourselves and in what we do as educators to give up on the future so easily? As long as we keep thinking about the fate of our graduate students as if it were separate from the condition of adjunct labor, we will continue to be complicit in the dismantling of higher education. And this would be tantamount to collective professional suicide. Have we really given this battle our all?
Carolyn Betensky is an associate professor of English at the University of Rhode Island.