Higher Ed Scholars' Voucher War

From the high-profile perch of The Wall Street Journal's op-ed page last month, Matthew M. Chingos and Paul E. Peterson trumpeted the findings of their new study: that African-American students who used New York City vouchers to attend private schools in a 1990s program were 24 percent more likely to enroll in college than were black students who didn't win a voucher lottery. While the Journal's opinion editors may have been impressed by Chingos's and Peterson's scholarly skills, the findings probably earned a spot on the coveted op-ed page because of the researchers' embedded critique of President Obama's continued opposition to school vouchers.

The president "seems to have nothing against private school per se, as he has long sent his own daughters to private schools," wrote Chingos, a fellow in governance studies at the Brookings Institution's Brown Center on Education Policy, and Peterson, a Harvard University government professor. "Yet -- apparently thanks to opposition to vouchers from powerful teachers unions -- the president still hasn't taken the next step and helped open private-school doors for low-income children as well."

Now another higher education researcher is questioning the methodology and findings of the Chingos-Peterson study. In a paper published today by the National Education Policy Center, Sara Goldrick-Rab asserts that Chingos and Peterson do not make the case that "the statistically significant result for African Americans that is set forth in the report is truly statistically significant or different from the non-statistically significant result for Hispanics."

Further, argues Goldrick-Rab, associate professor of educational policy studies and sociology at the University of Wisconsin at Madison, the two scholars play down what she argues is the study's true main finding: the absolute dearth of "evidence that the vouchers were effective in advancing the participation of students in higher education."

"Policy makers and practitioners interested in the effectiveness of school voucher programs should indeed attend to the results of this study, which -- contrary to the interpretation of the authors -- convincingly demonstrates that in New York City a private voucher program failed to increase the college enrollment rates of students from low-income families," she writes.

Normally a K-12 Issue

School vouchers are among the most politically volatile issues in all of education policy, with many (mostly) Republicans arguing that government-funded payments directly to families expand the choices of families who might otherwise be relegated to poor-quality public schools, and many Democrats asserting that they divert essential resources from public schools to private and religious schools. But the subject rarely touches on matters directly related to higher education, and therefore rarely appears in these pages.

Last month's study by Chingos and Peterson, the Henry Lee Shattuck Professor of Government and director of Harvard's Program on Education Policy and Governance, however, sought to make a direct link between school vouchers and college enrollment. The study, published by Brookings and Harvard's Kennedy School, examined the privately funded New York School Choice Scholarships Foundation Program, which offered three-year scholarships (chosen by the luck of the draw) to up to 1,000 low-income families with young children. Recipients could attend secular or religious private schools in New York City.

Tracking participants in the program through the years based on enrollment information from the National Student Clearinghouse, Chingos and Peterson found "no significant effects of the offer of a school voucher on college enrollment." But they conclude that the offer of a voucher increased the overall college enrollment rate of African Americans by 7.1 percentage points, and that the college enrollment of African-American students who used the vouchers to attend private schools rose by 8.7 percentage points, or 24 percent, over those who did not.

The impact was even stronger, the researchers said, when it came to enrollment of black youth in selective colleges, more than doubling the likelihood for those who received the offer of a voucher.

The researchers said they found a small -- yet statistically insignificant -- impact on college enrollment for Hispanic students, and attributed the differences between the two groups to "educational and religious reasons."

"African-American students were especially at risk of not going on to college, and families sought a private school -- even one outside their religious tradition -- that would help their child overcome that disadvantage," Chingos and Peterson wrote. "Hispanic students were less at risk of not enrolling in college and likely sought a voucher for some combination of religious and educational benefits."

A Challenge From Madison

Goldrick-Rab cites a series of perceived flaws (some of which only a social scientist could love) in the study's methods and conclusions. She summarizes them this way in a news release:

• There are no statistically significant differences in the estimated impact for African Americans as compared to other students.
• There is important but unmentioned measurement error in the dependent variables (college attendance outcomes) affecting the precision of those estimates and likely moving at least some of them out of the realm of statistical significance.
• The authors fail to demonstrate any estimated negative effects that could help explain the average null results.
• There are previously existing differences between the African American treatment and control groups on factors known to matter for college attendance (e.g., parental education).

Asked for an explanation that a layman journalist might grasp, she offered this: "Every impact estimate is that -- an estimate. The modeling is an attempt to find a 'signal' in a room, and to be certain you're hearing things correctly and aren't just hearing noise. A very loud signal is easy to hear, and others are less so.  Every signal is surrounded by noise that can make it very hard to hear -- call that the 'error' around your estimated 'signal.' You have to prove the signal exists and isn't actually a false alarm -- that means you have a statistically significant impact.  

"In this case they don't prove that, on average. They say, 'We hear a signal for blacks and it's different from Hispanics.' Except it's not -- both of those signals are quite noisy and the result is that they are statistically indistinguishable from each other. The authors fail to emphasize that, and it's crucial -- they can't establish what they hear isn't just noise over all, or noise just for Hispanics, or that the difference in the signal for either blacks or Hispanics isn't really just noise either. They have nothing but noise -- and still claim to hear a signal."

In a statement e-mailed to Inside Higher Ed, Chingos and Peterson strongly defend their study. On the apparent lack of difference between the impact of black and Hispanic students, they write: "The review says that an interpretation of the African American results is not appropriate because they do not differ significantly from those observed for Hispanic students. As stated in our report, it is true that the effects for African Americans and Hispanics are both positive and do not differ from one another by an amount that is statistically significant. But we can confidently say that the effect for African-American students is positive (i.e., greater than zero), whereas we cannot say the same for Hispanic students. "

And they describe Goldrick-Rab's assertion that the study proves that the New York City program did not improve outcomes for low-income families generally as false. "The overall impact estimate is not estimated with enough precision to conclude that the voucher intervention had no effect. The overall impact is not statistically significant from zero, but it is also not statistically significant from a negative impact of 3 percentage points or a positive impact of 4 percentage points."

They conclude: "The one result that can be reached with confidence is that the impact of vouchers for African Americans was positive.  None of the issues raised in this commentary compromise that conclusion."