New Criticisms of GRE

A new study questions whether doctoral programs in science and technology should be using the Graduate Record Examination in admissions.

The study, published in October in the journal PLOS ONE, was based on an analysis of the academic performance of 1,805 students who constituted a representative sample of those enrolled at four flagship universities.

Men had significantly higher quantitative scores than women at all four universities. But there were no gender differences in completion rates or time to degree.

The completion rates for women were around 60 percent -- regardless of the quartile ranking of their scores. In a finding that the researchers found particularly surprising, men with the bottom quartile of GRE quantitative scores had a higher completion rate (74 percent) than those in all other quartiles, including those in the highest quartile, 56 percent of whom completed their degrees. This pattern was true even in engineering, which the researchers thought might be a field where the quantitative scores would have more value in predicting ultimate success in the doctoral programs.

The paper suggests that these findings should raise serious questions about using GRE scores in STEM doctoral admissions.

Sandra L. Petersen, the lead researcher on the project, is professor of molecular neuroendocrinology of reproduction and director of the STEM Diversity Institute at the University of Massachusetts at Amherst.

She noted in a statement that although the Educational Testing Service explicitly discourages the use of cutoff scores on any part of the GRE, "this is a common practice." She added that "our study shows convincingly that the scores are not useful for identifying students most likely to finish STEM doctoral programs. In fact, the scores are negative predictors of completion rates for men."

Further, she said that at a time when many universities are trying to diversify their STEM doctoral student population, the study raises questions about why scores may matter more for some than for others.

“One reason given for using GRE scores to compare students is to ‘level the playing field’ for students coming from undergraduate institutions differing in prestige,” Petersen said. “It has been suggested that without these scores, admissions committees may show implicit bias that could hurt the chances of admitting students from underrepresented groups who often come from lesser-known institutions. However, for undetermined reasons women and non-Asian minorities continue to score less well than white males and Asian Americans and, therefore, the pool of ‘acceptable’ women and minority candidates is reduced substantially.”

Also working on the research was Evelyn S. Erenrich, associate graduate dean at Rutgers University; Dovev L. Levine, assistant graduate dean at the University of New Hampshire; Jim Vigoreaux, associate provost for faculty affairs at the University of Vermont; and Krista Gile, associate professor of mathematics and statistics at UMass.

The study arrives as some graduate programs are questioning the use of the GRE in admissions. The philosophy program at the University of Pennsylvania this year dropped the GRE as a doctoral admissions requirement, setting off a wide discussion in that field.

Critics of the way graduate programs use the GRE have launched a campaign with the tagline GREXIT to encourage departments to rethink how they use testing in admissions.

ETS itself has acknowledged that some departments may have relied too much on test scores, or used cutoffs, and the organization has publicized ways to use holistic admissions, in which a variety of factors are considered and cutoffs are avoided.

Via email, David Payne, vice president and COO of ETS, said that the findings are not relevant because they are based on claims that ETS does not make.

“The GRE test does not predict graduate or doctoral completion rates. It was never intended to do this,” Payne said. “Rather, the test provides a measure of graduate school readiness by assessing skills that are necessary to handle graduate-level work: verbal and quantitative reasoning, critical thinking and analytical writing. Investing more money in research that proves the same point over and over is wasteful when there is so much need in the graduate community for research that: identifies what characteristics are correlated with completion; develops more inclusive admissions processes that will help to identify which applicants have the academic skills as well as the personal attributes to be successful; and creates programs that will support students in their chosen programs.”

As to the findings about completion, Payne said, “This research attempts to blame the GRE test for why students aren’t completing doctoral degrees. This is not a fault of the test. The top reasons that students drop out of graduate programs, according to a study from the National Center of Education Statistics, were: changes in family status (30 percent), job/military conflict (17 percent), dissatisfaction with the program (16 percent), the need to work (14 percent), personal problems (13 percent), and other financial reasons (12 percent). These are not situations that an academic test can predict. Furthermore, in that same study, only 1 percent of interviewees said they dropped out due to academic problems. One of the reasons this number is so low is in part due to the role that standardized testing plays in graduate admissions, particularly its ability to predict outcomes related to academic performance.”