Defying a Gendered ‘Narrative’

Are biased letters of recommendation one reason that women remain seriously underrepresented in experimental particle physics, at just about 15 percent of the field’s faculty? A new study suggests not.

The new paper, published in open-access format, includes some other interesting findings that challenge the existing literature on gendered language in letters of recommendation.

“When we looked at female-authored versus male-authored letters written about the exact same female candidates, we did not find evidence of bias against women,” said co-author Wendy M. Williams, professor of psychology at Cornell University. “Remarkably, we found that women were actually described as ‘brilliant’ three times as often as men were.”

In contrast to these findings, numerous earlier studies have found that women in academic science are half as likely as men to receive excellent versus good letters of recommendation, and that recommenders use significantly more “agentic” (powerful) and “standout” (exceptional) adjectives to describe men, whereas women are more often described in “communal” (community-oriented) and “grindstone” (hardworking) terms.

For this new study, researchers analyzed more than 2,200 recommendation letters for assistant professor job candidates in experimental particle physics, developmental psychology and sociology. Experimental particle physics is an especially male-dominated corner of the traditionally male-dominated discipline of physics. Developmental psychology and sociology have much higher shares of female faculty members. Williams and her co-authors were thus able to analyze letters of recommendation in physics by candidate and recommender gender and compare them to letters from different fields. (The physics data set included 963 letters for 206 male candidates and 198 letters for 39 female candidates submitted to the Fermi National Accelerator Laboratory in Illinois between 2011 and 2017. The social sciences data set included 40 letters for 163 men and 605 letters for 222 women submitted to Cornell.)

The letters did differ in some gendered ways but, for the most part, not in patterns that were predictable or punitive to women. For instance, male physicists were indeed found to use more grindstone words when writing about female candidates, but—regardless of discipline or gender of writer—men were not more commonly depicted as agentic or as standouts, nor were women depicted as more communal.

In the social sciences, female letter writers used communal words more frequently than male writers, but they did this equally for male and female candidates. Physicists used more grindstone, communal and agentic terms than did the developmental social scientists, but female and male writers in physics were equally likely to use these terms.

Regarding letter length, the researchers did find a large, significant effect: letters for female candidates in social science were 65 words longer than letters for men over all—about 6 percent longer, somewhere between several sentences and a paragraph. In physics, however, letter length did not significantly vary by candidate gender.

Across all disciplines, women recommenders wrote longer letters than did men, a difference of 63 words.

Letters for men in social science were 19 percent more likely to be male-authored. This was not the case in physics.

Seeing Women as Scientists, Too

The researchers also found some letter features that defied standard analyses. In one case, a female writer compared a male candidate to Eeyore from Winnie-the-Pooh, apparently as a compliment. Female writers also described some male candidates as “lovable,” “charming,” “delightful” or “adorable.”

One female writer said that a male candidate had “two adorable children,” although it had nothing to do with the position, and a male writer said that a female candidate “demonstrated her ability to multitask through her raising of children while pursuing an academic career.”

As Williams noted, letters for female candidates were about three times more likely to include the term “brilliant,” something that is certainly more “standout” than “grindstone.”

Despite the overall findings, the authors warn against concluding that recommendation letters “are free of gender bias.” Standard lexical measures rely on pre-existing word lists that may include terms used more for women and other terms used more for men, “with offsetting differences that could obscure the use of gendered language,” the paper says: women might have more descriptions saying they are creative but fewer saying they are innovative, with the aggregate score showing little or no gender difference, for example.

The study’s open-ended analysis attempts to capture some of this nuance. While women in physics were indeed described as brilliant more often than men, men in the field were also significantly more likely than women to be described as talented, intellectual, innovative and creative.

Interestingly, and perhaps very significantly, two of the largest gender disparities were for “physicist” in the physics letters and “scientist” in the social science letters, with both used to describe men more often than women.

“Although other explanations are possible, future research should investigate the possibility that recommendations can reflect unconscious gender stereotypes in which physicists and scientists are imagined as men,” the study says.

The paper also acknowledges a major limitation, even though it attempted in various ways to control for candidate qualities: it “cannot rule out the possibility that female candidates were superior to male candidates and deserved stronger letters than those they received.”

Lead author Robert Bernstein, a senior scientist at the Fermi lab, said it remains unclear just why the letters in his study defied prior findings on biased language in recommendations.

“The standard narrative before our study was that [letters of recommendation] depict men as agentic standouts and women as communal grindstones, and that was claimed in studies for periods overlapping our letters,” he said. “There’s too much going on to conclude that specific interventions like training make a difference without a controlled study on the same sort of long, carefully written letters we have.”

Bernstein said he’d attribute any reduction in biased language use “more to the ongoing discussions we have in [experimental particle physics] among ourselves than formal training, but you have to be aware of the problem, and that’s where training comes in.”

In any case, he added, “because of this paper, I make sure that I refer to women as scientists when I write letters.”

Williams has previously published work that challenges the notion that academic science is biased against women. One 2015 study found that women were actually preferred over men in hiring for tenure-track jobs in STEM fields, for instance (that study was widely cited but encountered criticism, namely that it paid insufficient attention to the kind of bias that women can encounter in STEM once hired). Even with this background, Williams said that in approaching this new study, “based on the literature and many personal testimonials, we were expecting to find that women candidates were downgraded when their letters of recommendation were written by men as compared to women.”

Ultimately, Williams said, these findings “accord well with our past and current work showing that women in academic science are treated fairly in many domains, while still experiencing bias in some areas.”

Colleges are starting to rethink requiring letters of recommendation for admissions, due to their potential to introduce bias into the process, among other concerns about equity. Asked if letters of recommendation remain a valuable part of the very different job application process in STEM fields, Bernstein said yes—enthusiastically.

“Letters of recommendation are an essential element of the process for these positions. All these candidates are incredibly talented, and the more information the better,” he said. “CVs and standardized forms don’t provide the richness and detail we get in letters. Our letters are about 1,000 words, or two or three pages long—the writers put a lot of thought and effort into their letters and give us insights we can’t get in any other way.”

Letters of recommendation aren’t perfect, he said, “and they can introduce bias, but writing down a detailed assessment with your name on it puts the writer’s reputation on the line, too, adding accountability.”