A reader, Enrique Diaz-Alvarez, offered the following critique of the new national study of charter schools by CREDO. The “improvement” seems to be a result of closing low-performing charter schools. Is this like kicking out the low-performing students and declaring that your test scores are great?
He writes:
Diane, you *must* read the scenario analysis that starts in page 89.
They actually think the survivorship bias is a *feature*, not a bug! They run a bunch of scenarios detailing the dramatic improvement that you get in results if you eliminate from the study different kinds of underperforming charters (in addition to the ones that were closed down, of course), and theyconclude:
“The purpose of these simulations is not to advocate for any particular approach. Rather, the different scenarios make obvious the fact that the impact on quality that accompanies closure is more dramatic and enduring than efforts to improve the current stock of schools. The glimpse of what the future holds provided by these scenarios should quicken the collective resolve to use closure policies where charter schools are clearly underperforming. If the commitment to quality is to be fully realized, everyone
needs to put the interest of students first and use all the resources at their disposal to ensure the best possible student outcomes.”
Hey! Why not close *every* charter school except the top 10%? Why stop there? Shut down every high school in the country except Stuyvesant and Bronx Science! Everyone will be doing calculus in polar coordinates by the time they turn 16 then!
This is demented. What’s going on? What am I missing?
He adds in another comment:
Further on survivorship bias. From the study:
Results for charter students in new schools mirror the 2009 findings: students
at new schools have significantly lower learning gains in reading than their TPS peers.
[…]
The new charter school results in math follow the
pattern seen for reading – the performance of the newcharter schools mirrors the 2009 results.
[…]
*New* charter schools continue to perform worse than public schools!! It seems clear that all of the improvement is the result of pure and simple survivor bias. You start a lot of charters. Some do worse, some do better, but overall they do somewhat worse than public schools. You shut down the bad ones (and kick the poor kids back to the public schools -lest we forget). You repeat the analysis with the non-terrible ones. Voila! You have improved!
And he adds in a third comment:
From the press release: “charters in the original 16 states have made modest progress in raising student performance in both reading and mathematics, caused in part by the closure of 8 percent of the charters in those states in the intervening years since the 2009 report”
So charter measurements the 8% of schools that had presumably worst outcomes??? Did the study do anything to compensate for this massive survivorship bias?
Diane Ravitch's blog 
Part of the charter idea is to close schools that after being given a chance to improve, don’t. Sometimes people comment on here that it’s too hard to close charters. This report says it’s good to close the lowest performing charters if they don’t show improvement.
The report some older schools have improved and some new schools are higher performing.
While I think some of the results are encouraging, I have my own concerns on the report, in part because it relies only on reading and math scores, and in part because I think there are vast differences among district schools and among charters. So I think it does not make sense to simply compare “charter’ and district schools as groups.
Here’s a link to a newspaper column that provides more details:
http://hometownsource.com/2013/06/26/joe-nathan-column-encouraging-but-limited-view-of-charter-public-school-progress/
Joe,
You probably have already seen this, but in case you have not, here is a study of the impact of early college high schools: http://www.air.org/reports-products/index.cfm?fa=viewContent&content_id=2640
That comment makes about as much sense as everything else you say. Is it good or bad to close charter schools based on test scores? You say it’s good to close “low performing” charters, but these schools are being closed based on test scores – which you say are being overemphasized. Your level of consistency is the same as your movement.
Wise charter contracts, like wise overall school use multiple measures including but not limited to test scores. Here’s a link to a report we did some years ago featuring outstanding work by some district & charters that were using multiple measures to measure progress.
http://centerforschoolchange.org/publications/assessments/
All this is interesting, but besides the point in my view. The charter movement fragments public dollars and, in many cases, diminishes the offerings at local public schools. When the student leaves and the “money follows”, the per pupil expenditures rise. These are the most damaging effects of charter schools, aside from the clever usurpation of local democratic control. Our community now funds more schools than our citizens want, thanks to the new for-profit charter.
And once again you offer a comment that doesn’t address the content of the post. Charters are making themselves look better by firing the lowest performing. Meanwhile the new schools replacing them are no better and sometimes worse. That is the new charter model – constant churn. But it’s the students and communities that are really being churned.
From my vantage point, it is hard to close down charters unless they go completely broke. I don’t know where you live but it is what is happening in other states.
The study that just came out on charter schools listed Utah’s charter schools as mediocre at best. 41% of charter schools did WORSE in math than regular public schools. However, there is NOT ONE person in power that has suggested closing charter schools. This “charter schools will close if they are failing” argument is incongruous. Once charter schools open, in Utah, anyway, they’re there to stay.
Thanks, TE.
??
This is the same game used in school districts and individual schools. Only what they do is get rid of the low performers such as behavioral problems, ESL, slow learners and special ed. What happens, why your scores go up even if those remaining do no better the average is better. Not really trick. They just pass the huge societal costs onto other sectors such as mental health and the criminal justice system at a huge increase in cost.
Those of us who’ve been following charter schools and the CREDO studies have been harping on the fact that low performing charter schools are being allowed to stay open, despite their poor performance. It would appear that this is no longer the case and low performing charter schools are being closed, though not rapidly enough. And now someone complains that closing these low performing charter schools is some sort of effort to improve the overall statistical picture? Something called “survivor bias”. I doubt individual schools, which probably fought to stay open would be agreeing to shutter their schools in order to improve some other schools’ chances to look good. Not human nature to do that sort of thing.
“Not human nature to do that sort of thing.”
Especially for the eduprenuerial types!!
I think it depends on the state. There are people out here making a lot of money while providing an inferior education. I don’t understand why they are allowed to stay open unless they have some sort of political power behind them. If you have massive teacher turnover and sometimes as many as six teachers in one classroom in one year, wouldn’t you think this school should be closed? This is why people know there are bad charters that are allowed to stay open. I’m sure it ranges from state to state.
Yes, closure rates vary across states and across communities. Some groups authorizing charters do a better job of supervising them than other groups.
A conservative group that supports charters has done a report describing the number and process of closing charters. An Atlanta Journal/Constitution journalist who I think is pretty fair and thoughtful has written the following on that report:
http://blogs.ajc.com/get-schooled-blog/2011/12/21/new-study-charter-schools-have-a-15-percent-closure-rate/