The Center for Research on Education Outcomes (CREDO) has positioned itself as the think tank that writes trustworthy evaluations of charter schools. In a sense, this is strange because CREDO’s leadership and its funding are not neutral on the subject of charter schools; they are decidedly pro-charter.

Freddie DeBoer argues on his blog that CREDO has consistently reported that charter schools find no gains for charter schools as compared to public schools. But it’s 2023 report was widely reported as the year of the Big Charter Breakthrough. He believes they are honorable in reporting their findings, but they pulled a fast one on the media and public by inventing a new metric that cannot be compared to other measures: “days of learning.”

DeBoer shows how the CREDO metric magnifies charter school results, which are actually small enough to be counted as “no effect.”

DeBoer begins:

In 2023 the Center for Research on Education Outcomes, known as CREDO, released its third national study of charter schools. The research covered 1.8 million students across twenty-nine states plus Washington D.C. and New York City. CREDO’s study purported to offer the most definitive evidence yet about the relative school quality of charter schools as compared to traditional public schools. Certainly our media, political establishment, and the nonprofit sector have acted as though what CREDO has to say is definitive; the 2023 study is, I believe, the most-cited piece of charter research in existence. Every pro-charter politician reaches for it, your Jonathan Chait neoliberal “reform” types treat its findings with near-mythic reverence, people who otherwise have no exposure to educational research wave at it as the final word on the subject. When the average person talks loosely as though there’s some sort of generic charter advantage, they’re usually referencing CREDO research, even if they don’t know they are.

This is all rather surprising, given that CREDO has not produced robust evidence in favor of charter schools, not in the 2023 study or its precursors. As we shall see, even if we assume that CREDO has solved the problem of the fundamentally non-random samples inherent to the task of comparing charter to public – which we should not, because they have not – the evidence supplied by CREDO in the favor of charter schools is so marginal and so noisy as to be meaningless from a policy perspective. And yet the message that the media took and shared from that report – the good news!, the vindication!, the headlines in Education Week and the New York Post!, the press releases from the National Alliance for Public Charter Schools! – was that the typical charter student was the beneficiary of a meaningful learning boost compared to their traditional public school counterpart. Why did this message spread so widely and resonate so loudly, if I’m right that even CREDO’s own numbers show very modest differences? Because of a remarkably savvy bit of marketing in how CREDO’s numbers were presented. The report did not show its findings in terms of effect size differences between charter and public, which would be the field’s standard way of sharing results. Rather, the outcome that was shared again and again was that a typical charter school student in the dataset had gained sixteen additional “days of learning” per year in reading and six in math. For a few months there you couldn’t read educational media without running into those numbers: sixteen additional days of learning in reading per year, six in math. It became a bit of holy writ in the endless quest to make bad students good with ACCOUNTABILITY and market reforms.

Now, me, I immediately gravitate to that last bit: what on Earth is a “day of learning,” and since when is that a statistical measure?
“Days of Learning”: A Jury-Rigged Metric Invented for the Purpose of Inflating the Effect
If you share my confusion over that metric, don’t blame yourself. “Days of learning” is not merely an unusual measure of student performance; it’s a boutique metric that CREDO invented themselves for their charter studies. It’s like YouTube changing how views are counted so the numbers look bigger.

This should raise a number of red flags, most importantly when it comes to comparison with other research. We have standardized metrics in social science research for a reason. One core reason among them is mutual interpretability, the ability to compare outcomes involving different interventions that are measured with different instruments and in different units. That’s why effect size exists, and various expressions of effect size have served as the dominant measure of interest in all manner of human research for decades precisely because effect size enables ease of comparison. If you want to compare the influence of the same intervention when one study uses grades as the DV and another used test scores, effect size enables you to do that. If you want to compare two different types of intervention, effect size enables you to do that too. There are some caveats, as there always are, but the basic, elegant logic of effect size makes the measure remarkably flexible and portable: effect size tells us how a given intervention moves average outcomes along the distribution of outcomes. If you want to know more there are plenty of simple explainers of effect size out there; here’s mine. There are some fields where effect size is used less often (economics, for example) but the interpretability and flexibility of effect size metrics like Cohen’s d are powerful advantages. Not surprisingly, then, it’s the dominant reported measure in education research.

So if effect size is both useful and common, and is in fact the dominant measure of interest in the relevant field, why would CREDO come up with its own boutique measure? Why invite the potential for confusion with such consequential research? Occam’s razor suggests a simple answer: because CREDO is a pro-charter organization that’s intent on convincing stakeholders that charters have meaningfully superior outcomes when compared to traditional public schools… and their own research demonstrates the opposite. Their own numbers, in fact, suggest that the returns on charter schools are meager even if we ignore the selection problems that continue to plague charter advocacy.

This is a much longer post about the inadequacy of research on charter schools. It’s well-written and thoughtful. Please open the link and finish reading.