The Nudges That Didn't Work

A 2015 study gave great hope to those worried about "undermatching," the term scholars gave to the patterns in which most high-achieving, low-income high school seniors don't apply to a single competitive college.

The 2015 study -- by two respected researchers, Caroline Hoxby and Sarah Turner, who were involved in the original undermatching work -- found that the right interventions (largely providing certain types of information) can in fact alter the choices of these undermatched students. These interventions could send a significant number of low-income, high-ability students to the nation's top colleges, they found. The study was based on a national sample of 18,000 high school seniors who had scores in the top decile on the SAT or ACT, had family income in the bottom third for families with high school seniors, and did not enroll at "feeder" high schools (such as magnets) that regularly send many students to top colleges. To many who were encouraged by the findings, the next step was to conduct research on a much larger scale.

That happened, and the results are now out -- and they are disappointing to those who hoped the interventions might work on a large scale. Hardly any changes were found in the college application process of students. The results were published as a working paper by the Annenberg Institute for School Reform at Brown University.

The project was financed by the College Board and reached 785,000 low- and middle-income students in the top 50 percent of the PSAT and SAT score distributions.

The students in the project were sent information about academically strong colleges, the application process (for admission and financial aid) and other materials. Information came in multiple forms, including mail, email and text messages. The College Board also provided application fee waivers and let participants in the program send SAT scores without fees to an unlimited number of colleges.

All of these tools were in theory designed to encourage students to apply to and enroll at more competitive colleges. The original study on undermatching noted that the most competitive colleges in the country -- public and private -- tend to have higher graduation rates and more resources to help students than do other colleges. So if talented, low-income students are not enrolling at these institutions, they are missing a chance to thrive at institutions that are for many stepping-stones to successful careers.

But the study found "no changes in college enrollment patterns," with the exception of a very modest increase in the academic quality of colleges considered by some African American and Latinx students. But even those gains were "extremely small" and not consistent, the report said. Students appeared to look at the materials sent, and to send SAT scores to more colleges than was typically the case for similar students, but the way they went about considering and applying to colleges did not actually change.

Hoxby, a professor of economics at Stanford University, and Turner, a professor of education and economics at the University of Virginia, did not respond to a request for comment on the findings.

But the researchers who did the study -- all from the College Board except for one from the University of Missouri at Columbia -- offered a number of ideas about why the project did not yield the results many educators hoped to see.

The researchers noted differences (beyond size) in the sample used for the new study. The students included, for example, were a broader cross-section than those in the top decile, students from some "feeder" high schools were included, and email was a primary tool for communication with students. This may have "diluted impacts due to distaste of electronic correspondence," the study said. Still, "our best attempt at mimicking their sample still produces no statistically significant effect," the researchers said.

So why might the informational approach not work?

"Our evidence suggests that one key issue is students received the information but did not use it to consistently apply to colleges of higher quality," the report said. "Data on SAT score sends suggests that students became interested in both higher and lower quality institutions, though even these changes were of a relatively small magnitude and unlikely to result in large changes to observed enrollment. Thus it appears that efforts to shift college enrollment were thwarted at the application stage. Given the influence of neighborhood, family and peers in the college selection process, the type of information we provided may not have been sufficiently novel or compelling to change student behavior."

In addition, it may be the case, the researchers write, that prior discussion of undermatching has encouraged more highly selective colleges to themselves reach out to the same groups of students that were the targets of this intervention.

But generally, the researchers wrote, their findings may point to the need for more labor-intensive (and more costly) efforts to reach low-income students.

"College outreach or direct service programs, [which] provide a more intensive but human touch working directly with students, may be more efficacious than information-based initiatives in substantially altering college application behaviors," the study said.

Education policy experts sharing the results on Friday suggested that the effort was admirable, but that this may be evidence that inexpensive approaches are unlikely to work.

The authors of the new study are Oded Gurantz from the University of Missouri and from the College Board, Jessica Howell, Michael Hurwitz, Cassandra Larson, Matea Pender and Brooke White.

What to do about undermatching continues to be a challenge for many scholars of higher education. Hoxby and Turner in January published research finding that some selective colleges have responded to pressure to hit national benchmarks for enrolling low-income students in ways that hurt other needy students who are equally academically worthy. The research specifically criticizes the "shaming" approach -- in which colleges are compared on their enrollment of low-income students -- for leading some colleges to add more such students without developing real campaigns to promote the enrollment of talented undergraduates from all socioeconomic groups.