A reader, Enrique Diaz-Alvarez, offered the following critique of the new national study of charter schools by CREDO. The “improvement” seems to be a result of closing low-performing charter schools. Is this like kicking out the low-performing students and declaring that your test scores are great?

He writes:

Diane, you *must* read the scenario analysis that starts in page 89.

They actually think the survivorship bias is a *feature*, not a bug! They run a bunch of scenarios detailing the dramatic improvement that you get in results if you eliminate from the study different kinds of underperforming charters (in addition to the ones that were closed down, of course), and theyconclude:

“The purpose of these simulations is not to advocate for any particular approach. Rather, the different scenarios make obvious the fact that the impact on quality that accompanies closure is more dramatic and enduring than efforts to improve the current stock of schools. The glimpse of what the future holds provided by these scenarios should quicken the collective resolve to use closure policies where charter schools are clearly underperforming. If the commitment to quality is to be fully realized, everyone
needs to put the interest of students first and use all the resources at their disposal to ensure the best possible student outcomes.”

Hey! Why not close *every* charter school except the top 10%? Why stop there? Shut down every high school in the country except Stuyvesant and Bronx Science! Everyone will be doing calculus in polar coordinates by the time they turn 16 then!

This is demented. What’s going on? What am I missing?

He adds in another comment:

Further on survivorship bias. From the study:

Results for charter students in new schools mirror the 2009 findings: students
at new schools have significantly lower learning gains in reading than their TPS peers.
The new charter school results in math follow the
pattern seen for reading – the performance of the newcharter schools mirrors the 2009 results.

*New* charter schools continue to perform worse than public schools!! It seems clear that all of the improvement is the result of pure and simple survivor bias. You start a lot of charters. Some do worse, some do better, but overall they do somewhat worse than public schools. You shut down the bad ones (and kick the poor kids back to the public schools -lest we forget). You repeat the analysis with the non-terrible ones. Voila! You have improved!

And he adds in a third comment:

From the press release: “charters in the original 16 states have made modest progress in raising student performance in both reading and mathematics, caused in part by the closure of 8 percent of the charters in those states in the intervening years since the 2009 report”

So charter measurements the 8% of schools that had presumably worst outcomes??? Did the study do anything to compensate for this massive survivorship bias?