News
New Test for Bias in Peer Review
Peer review is supposed to assure fair consideration of scholars' work for placement in journals, the awarding of grants and so forth. But many have their doubts and believe that fairness is much more theory than practice. Many scientists say in fact that incompetence and bias hinder the peer review process.
Andrew J. Oswald, an economist at the University of Warwick, has proposed a new way to test for bias. His system and some tests of it are outlined in a paper he wrote for the Cornell Higher Education Research Institute. While the system proposed is primarily a tool for looking at how journals rank articles, and he did the testing with his own field, economics, Oswald argues that it could be applied to other disciplines and avoids some of the pitfalls of other systems for detecting possible bias.
Oswald's system is simple: He looks at journal articles and the order in which they appear in a given publication, with the assumption that journals put the work that editors believe will have the most impact at the top of the table of contents. Then Oswald examines citation records for articles that are adjacent. The expectation, he writes, is that the top article should be cited more. If a journal is biased for or against a particular group, Oswald writes, this should show up. If, for example, a journal had a bias in favor of Harvard University scholars, they would not perform as well on citations as articles beneath theirs.
Many people who believe bias is a problem with journal selection cite the relative proportions at which articles from different groups are accepted or rejected by a journal, Oswald notes. Such a system assumes relative equity in quality among articles from the various groups. Oswald's system tests journals on what they actually publish, with the expectation that citations would match the journal's judgment on an article's quality.
The article includes demonstrations of how his system works. And although Oswald notes that the approach could be used to study bias for or against members of racial or gender groups, he looked at university affiliations. He finds some evidence of bias -- but with a suggestion that should reassure those who worry about cronyism. Oswald tested whether the Journal of Political Economy, published by the University of Chicago Press, favors Chicago authors. Using his system, he found the opposite: If there is bias, it is against Chicago authors, as they appear to be held to a higher standard than others, he writes.
