Bias Seen in Bias Studies

Professors are all Democrats, except those who are communists. Professors all hate Bush. Professors favor like-minded students and love converting those who love God, country and the president. You've read all the claims and more, in right-leaning blogs and columns. Frequently, these claims are based on studies -- many have been released in the last two years -- of professors. Party registration is documented, or professors respond to surveys, or syllabus content is rated.

A new study being released today aims to debunk all of those studies. "The 'Faculty Bias' Studies: Science or Propaganda," takes eight of the recent studies on faculty politics and judges them by five general tests of social science research. Today's study finds that the eight all come up short in adhering to research standards. The new study was sponsored by the American Federation of Teachers and the work was conducted by John B. Lee, an education researcher and consultant who said that once the AFT commissioned the work, it did not restrict his approach or findings in any way.

The various studies analyzed are by no means identical, but they tend to have two major themes (although some stress just one of the themes): that faculty members are liberal and that their liberal inclinations are significant in considering their performance.

Lee's analysis finds some support for the first theme. "Taken together, these studies at best suggest that college faculty members are more likely to be Democrats than Republicans," he writes. However, even on this theme, he notes that the studies tend to exclude community college faculty members and to focus on faculty at elite institutions -- probably skewing the results.

The second theme takes a more thorough beating in the study. "Among the most serious claims the authors make is that this liberal dominance results in systematic exclusion of conservative ideas, limited promotion opportunities for conservative faculty, and expression in the classroom of liberal perspectives that damage student leaning," Lee writes. "These claims, however, are not supported by the research. Basic methodological flaws keep a critical reader from accepting the conclusions suggested by the authors."

The flaw Lee identifies most frequently with this theme is one in which researchers note a correlation and -- in Lee's opinion -- then see a causal relationship without sufficient evidence that one exists.

AFT officials said that they commissioned their study out of concern that the drumbeat of reports on political bias were suggesting to the public and politicians that faculty members are unprofessionally injecting politics into the curriculum, hiring and grading. Some of those whose work is criticized in the AFT's report, however, said that it was the faculty group's report that was guilty of bias, and they questioned the legitimacy of the new study, which they termed propaganda.

The New Research

The new AFT study looks at eight studies, including some that have attracted substantial attention (both praise and criticism), such as work published in 2005 in The Forum that analyzed faculty attitudes at four-year institutions and concluded that conservatives, practicing Christians and women are less likely than others to get faculty jobs at top colleges. That study was based on a survey of 1,643 faculty members. Other studies looked at faculty attitudes in certain disciplines or at certain institutions.

Some of the studies were prompted by specific events, such as the American Council of Trustees and Alumni's "How Many Ward Churchills?," which analyzed class materials online at top institutions and found that the controversial Colorado professor's ideas -- which have been in the news while his university has considered whether to fire him -- are shared by many professors. Some of the reports are by social scientists, published in peer-reviewed journals. Others were issued by associations that are players in the culture wars of academe.

Lee said that to test the validity of the studies, he wanted standards that could not be considered partisan, so he used a 2006 statement by the White House Office of Management and Budget about objectivity in research. Based on that statement, he asked five questions about each of the faculty bias studies:

• Can another researcher with a different perspective replicate the results using the information provided by the author?
• Are the definitions used in the studies clear enough?
• Does the research eliminate alternative explanations for the results?
• Do the conclusions follow logically from the evidence?
• Has the author guarded against assumptions that could introduce systematic bias into the study?

Using this framework, Lee gives the studies failing grades. Four studies had data that could be replicated, and he gave three studies acceptable reviews on clarity of terms, but it was downhill from there, and he argues that none of the reports can truly back up their contentions.

Besides offering that general rubric, Lee goes through each study, summarizing problems he found with it. For example, "How Many Ward Churchills?" was based on a review of online materials at various colleges. Lee notes that the researchers for the study appeared to focus on syllabuses or courses that had certain key words: activism, discrimination, gay issues, Marxism, oppression, pornography, radical, women's studies, among others. Lee writes that selecting 65 courses at 48 colleges "does not allow for the sweeping generalizations the authors make."

Even for those courses, he notes, the authors of "How Many Ward Churchills?" didn't actually observe the courses, so while they may know that certain topics or perspectives are covered, they have no way of judging the intellectual character of a classroom. While that report's authors wrote that Americans should be "outraged by the one-sided doctrinaire perspective" of their courses, Lee writes that they had no evidence to assert much of anything about the courses.

In several of the studies, Lee notes that relatively small subgroups of college faculty were surveyed, generally professors at elite and/or four-year institutions. Because community college professors, on average, are more centrist and more religious than colleagues at four-year institution, Lee questions whether their exclusion limits the ability of the studies' authors to make statements about academe as a whole.

Another theme he returns to over and over again is one of demonstrating (or not) causal relationships. He notes that there are many explanations for political trends and demographics among the professoriate, so it is unfair to assume that a liberal tilt (assuming one exists) reflects bias. He notes, for example, that the studies do not explore whether there could be non-political explanations.

Many have questioned, for example, the lack of data on applicant pools for faculty positions, and compared the disparity in political inclinations to that of Wall Street, where there are not suggestions that any Republican tilt is the result of bias or results in any discrimination against Democratic investors. Lee also compares the military, where recent polls have found a Republican tilt in opinions, but no evidence that soldiers service to their country is affected by whether they are seeking to protect members of one party or another.

While Lee finds flaws in all of the studies, he says that they have had influence, and notes that the studies have been widely cited by conservative pundits. Looking at the studies together, he says that it is clear that the authors "have a clear agenda" of charging professors with unprofessional conduct, and yet lack the evidence to make their case. Not a single study, he says, shows political bias in the classroom or hiring decisions.

"Until credible studies are conducted to provide a more grounded and systematic approach to understanding the subtle relationship between political beliefs and professional responsibilities, it is irresponsible to suggest that the conclusions reached in these reports represent a scientifically derived set of facts. They do not," Lee writes. "Passing off personal opinions as facts is not science; it is the antithesis of what serious researchers try to do, regardless of whether they are conservative or liberal."

Two of the authors whose work is criticized by the AFT took issue with the conclusions, and questioned whether the organization could fairly look at these issues.

"Critical commentary is always socially useful, and this new report is no exception. Even just a cursory reading will teach us much about the moral and intellectual character of its sponsors -- the AFT, the AFL-CIO,  and Free Exchange on Campus," said Daniel Klein, a professor of economics at George Mason University and the co-author of two of the studies reviewed in the report.

Anne Neal, president of the American Council of Trustees and Alumni (which issued two of the reports reviewed), criticized the AFT for commissioning the study. Via e-mail, she said: "Faced with mountains of evidence from ACTA and others documenting a troubling lack of professionalism in the academy, AFT chooses, instead, to shoot the messenger. In doing so, far from undermining ACTA, it discredits itself. AFT's study is severely flawed. It is filled with inaccurate and tendentious interpretations -- for instance, framing the debate in terms of politics rather than professional standards outlined by ACTA; applying irrelevant 'scientific' standards to textual analysis; and offering such shoddy research that the sections on ACTA totally confuse and conflate two different reports, rendering the critique invalid, even laughable. "

She added: "In the face of troubling evidence of a politicized classroom, has AFT conducted any studies of its own to see if there is problem? Taken concrete steps to explore the atmosphere in the classroom? The answer, of course, is no. AFT's report is not science -- it's propaganda."

With more studies of the sort the AFT criticized in the works now, stay tuned for more debate.