The New Orleans Miracle: A Continuous Sale of Snake Oil

To the Editor:

Scott Jaschik cited a recent study by Glenn and Harris (2020) reporting the academic improvements in New Orleans K-12 schools that occurred a few years after Hurricane Katrina have been sustained. Studies that have showed miraculously improved student outcomes as a result of the post Katrina school “reforms” have been mainly authored by Harris and his colleagues (e.g., Harris, 2015; Harris & Larsen, 2018). The reforms involved turning public schools into charter schools and introducing accountability contracts for administrators and teachers. Given such incredulous success, perhaps all U.S. schools should follow the lead of New Orleans. If you believe that, then we’ve got some oceanfront property in the Midwest we would like to sell you.

Although several closer looks at the data have debunked the New Orleans miracle (e.g., Gabor, 2015; Hatfield, 2014; Kimmett, 2015; Robinson & Bligh, 2019), it remains yet another zombie concept that refuses to die (Sinatra & Jacobson, 2019). Selling it as an example of charter school and accountability success is pure snake oil. There is but one cause of the immediate success of New Orleans schools after Katrina. Prior to the hurricane, there were 65,000 students enrolled in the public school system. Post-Katrina, in 2013, there remained only about 45,000 students who were then enrolled in the charter schools. What happened the 20,000, or 31% of the students, who disappeared? Sadly, they were forced to leave the city after flooding destroyed their homes. Similarly, by 2016, almost one third of the black residents who were forced to flee had not returned to the city (Rivlin, 2016). The students who left were likely among the poorest and, therefore, on average, likely ranked among the lowest students in terms of achievement. The average achievement test scores of the students who remained greatly increased immediately, regardless of any so-called reforms. This is an example of a threat to internal validity called history. It happens when something else that occurred during the treatment might also explain the change in the dependent variable. If an alternative explanation accounts for the results, then the treatment effect becomes bogus.

Imagine if we claimed the Kansas City Chiefs winning the Super Bowl in February 2020 was the reason why a dramatic increase in the numbers of deaths worldwide due to respiratory failure happened in the following year. We could simply compare the numbers of deaths in the year prior to the Super Bowl and in the year after. Have we made a convincing case? Should the Chiefs be banned from the National Football League?

In the Super Bowl example, the bogus treatment is the Kansas City Chiefs winning, the dependent variable is number of deaths due to respiratory failure, and the “something else” that also occurred would be the Covid-19 pandemic. No reasonable person would blame the deaths on the Kansas City Chiefs. For the New Orleans example, the bogus treatment is the switch from public to charter schools, the dependent variable is student achievement, and the “something else” is Hurricane Katrina. Just because two things are related does not mean one caused the other. Correlation is not causation.

Comparing data from pre-Katrina K-12 students to post-Katrina K-12 students is a fool’s errand. If the Omaha, Nebraska school district were to experience the loss of its 31% lowest-performing students, its reading proficiency rate would instantly jump from the current 63% to 91%, leaving the state average of 77% in the dust. This “difference” would happen without any increase in academic performance by the remaining students. Average academic “improvement” following the removal of the poorest of the poor is certainly not a successful  “reform” claimed by the charlatans. Unfortunately, such unsupported claims are often repeated in later publications that cite the original research (Shaw et al., 2010). This particular New Orleans Miracle snake oil has somehow continued to be peddled (e.g., Chait, 2014; Langhorne, 2018; Leonhardt, 2018; Nowakowski, 2015). Please put this zombie out of its misery.

Robert M. Bligh

Daniel H. Robinson