New Questions on Test Bias

For many years, critics of the SAT have cited a verbal question involving the word "regatta" as an example of how the test may favor wealthier test-takers, who also are more likely to be white. It's been a long time since the regatta question was used -- and the College Board now has in place a detailed process for testing all questions and potential questions, designed to weed out questions that may favor one group of students over another.

But a major new research project -- led by a scholar who favors standardized testing -- has just concluded that the methods used by the College Board (and just about every other testing entity for either admissions or employment testing) are seriously flawed. While the new research doesn't conclude that the tests are biased, it says that they could be -- and that the existing methods of detection wouldn't reveal that.

"I'm not saying that they are being dishonest or hiding something, but if there is bias today, you are not going to detect it," said Herman Aguinis, professor of organizational behavior and human resources at the Kelley School of Business of Indiana University at Bloomington. The research -- co-written with Steven A. Culpepper of the University of Colorado at Denver and Charles A. Pierce of the University of Memphis -- has just been published in the Journal of Applied Psychology.

Aguinis said that the question of bias remains crucial to the fairness of admissions tests. Since, on average, white and Asian students earn higher scores than do black or Latino students, colleges can be justified in using standardized tests only if they are certain that the score gaps are not related to bias.

The College Board, ACT and just about every other major testing outfit (not to mention colleges) typically say that score gaps are caused by the inequities in American society, in which many black and Latino students are less likely to benefit from the educational and economic advantages of other students. And the testing services all say that they use sophisticated testing techniques -- plus much more sensitivity than was the case in the regatta question era -- to prevent biased questions from ever being used.

In the common approach, individual questions are analyzed. What the new paper suggests is another way to look for bias. The scholars created a database with literally trillions of questions and scores on a range of tests, including all the major standardized tests used in college admissions. And this database featured trillions of questions that had been determined to have bias. But when samples were pulled out for analysis of a given question on a given test, the results came back negative for bias.

The conclusion, Aguinis said, is that question-by-question analysis doesn't detect bias.

"Given our research, the conclusion that tests are unbiased should be revisited," he said. "We need a much bigger question."

He also noted that much of the research that has been done to show no bias in tests has been supported by the testing industry. Aguinis noted that the research he and his colleagues conducted did not receive any outside support. And he stressed that he would like to see a new discussion so that bias could be detected and eliminated, not so testing should be stopped.

"I think our research does not negate the importance of testing," he said. "Our research does not say that testing is not useful. On the contrary, I believe that merit-based testing and testing that truly identified the people with the greatest potential is needed, What we need to do is strive to develop fair and valid tests."

The new study on bias techniques is the second major research effort this summer to question whether the SAT is detecting and preventing racial bias in questions. The other study, which found some bias against black students in some kinds of questions, was published in the Harvard Educational Review and has been criticized by the College Board.

A spokeswoman for the College Board said that officials there did not have time to study and respond to the new research on Friday.

Robert Schaeffer, public education director of the National Center for Fair and Open Testing, said that the new findings "challenge a fundamental paradigm" of the testing industry. "Given the increasingly high stakes attached to standardized exams, failure to follow the recommendation for additional investigation into potential bias would be irresponsible," he said.