Carol Burris is the Executive Director of the Network for Public Education. She was a much honored high school principal in New York State, following many years in the classroom. She earned her doctorate from Teachers College, Columbia University.
From my perspective, I think it always wise to pay attention to the funders of any study, especially when the funders have a strong point of view about the outcome. Just as we are wary when the tobacco industry releases a study that “proves” the safety of tobacco use, or the pharma industry funds a study claiming that opioids are not addictive, we should be wary of any study funded by the major sponsors of the charter school movement. “Follow the money” is a principle that should never be ignored.
Burris writes here about the new national CREDO study of charter schools, which was uncritically reviewed by Education Week and other publications, which simply quoted the press release.
She writes:
Last week the Center for Research on Education Outcomes (CREDO) released its third National Study on charter schools. The report was funded by two nonprofits that wholeheartedly support charter schools and generously fund them—the Walton Family Foundation and The City Fund. The City Fund, which was started and funded by pro-charter billionaires John Arnold and Reed Hastings, exists to turn public school districts into “portfolio” districts of charter schools and charter-like public schools.
Commenting on the report, Margaret “Macke” Raymond, founder and director of CREDO, told Ed Week’s Libby Stanford that the results were “remarkable.” Stanford claimed that “charters have drastically improved, producing better reading and math scores than traditional public schools.”
However, neither of those claims describes the reality of what the report found, as I will explain.
Let’s begin with what CREDO uses as its measure of achievement. In all of its reports, CREDO uses “days of learning” to attribute differences in student achievement between charter schools and district public schools. That measure creates dramatic bar graphs allowing CREDO to disguise the trivial effects on achievement those “days of learning” represent.
The overall math state score increase that CREDO attributes to a student attending a charter school is “six days of learning.” But what does that mean in the standard measures most researchers use, such as changes in standard deviations or effect sizes?
According to CREDO, 5.78 days of learning translates to only a 0.01 standard deviation. That means that the 6.0 “days of learning” increase in math translates to about a 0.0104 increase in standard deviations. Does that sound tiny? It is. For comparison, the negative impact on math scores of receiving a voucher in Louisiana was determined to be 0.4 standard deviations – more than 36 times greater magnitude.
After CREDO released its second national charter study in 2013, the National Education Policy Center (NEPC) reviewed it. You can find that critical review here, accompanied by a publication release titled CREDO’s Significantly Insignificant Findings.
As the authors of the review (Andrew Maul and Abby McClelland) note, a 0.01 difference (which the 2023 math gain only slightly exceeds) in a standard deviation means that “only a quarter of a hundredth of a percent (0.000025) of the variation” in the test scores could be explained by the type of school (charter or public) that the child attended.
Put another way, if a student gains six days of math and they originally scored at the 50th percentile on a standardized test, they would move to the 50.4th percentile. It’s as if they stood on a sheet of loose-leaf paper to stand taller—that’s how small the real difference is.
But what about the reported reading-score increase of 16 days? Sixteen CREDO days account for only a 0.028 standard deviation. Now we are increasing height by standing on two and a half sheets of looseleaf.
According to CREDO, those increases are statistically significant. Shouldn’t that count? As the NEPC reviewers state in their summary, “with a very large sample size, nearly any effect will be statistically significant, but in practical terms these effects are so small as to be regarded, without hyperbole, as trivial.”
To put all of this in a broader perspective, Maul and McClelland point out, “[Eric] Hanushek has described an effect size of 0.20 standard deviations for Tennessee’s class size reform as “relatively small” considering the nature of the intervention.” Hanushek is married to Macke Raymond, who found the much, much, much slighter results of her organization’s study to be “remarkable.”
Using CREDO’s conversion, in order to achieve 0.20 standard deviations of change, the difference would have to be 115.6 days of learning.
The only place in the report where there was an over 100-day difference was in online charter school students’ results in math. Compared with the public school students included in the study, online charter school students learned 124 fewer days of math. They may have something there.
CREDO Methodology
To draw its conclusions, CREDO matches charter students with what it calls “virtual twins” from public schools. But not all public schools were included, nor were all charter schools. The only public schools included were those in 29 states (for some odd reason, CREDO also includes NYC as a state) and the District of Columbia that met their definition of “feeder schools.” CREDO refers to 31 states, which include New York City and the District of Columbia, throughout the report.
According to page 35 of the report, in 2017-2018, there were 69,706 open public schools in their included “states,” and of those, fewer than half (34,792) were “feeder schools.” That same year, NCES Common Core of Data reports 91,326 non-charter public schools, 86,315 of which were in states that had charter schools.
From the chart, then, we can estimate that only about 38% of public schools and 94.5% of charter schools were included in the study, at least during the 2017 school year.
What, then, is a feeder school? The report claims that it is the public school the student would have attended if she were not in the charter school. But that is an inaccurate description. In the methodology report, CREDO explains how they identify feeder schools. “We identify all students at a given charter school who were enrolled in a TPS during the previous year. We identify these TPS as “feeder schools” for each charter school. Each charter school has a unique feeder school list for each year of data.”
While I understand why researchers want to use feeder schools for comparison, it produces an inherent bias in the sample. Feeder schools are, by definition, schools where parents disrupt their child’s schooling and place them in a charter school. They are not, as the report claims, “the school the student would have attended.” If a child starts in a charter school, her local school would not be a feeder school unless there was a parent who was so dissatisfied with the school that they were willing to pull their child out and place them in a charter, which may even be miles away in a neighborhood with very different demographics.
Virtual Twins
In 2013, Maul and McClelland also explained the virtual-twin method along with the problems inherent in its use.
“The larger issue with the use of any matching-based technique is that it depends on the
premise that the matching variables account for all relevant differences between students;
that is, once students are matched on the aforementioned seven variables [gender, ethnicity, English proficiency status, eligibility for subsidized meals, special education status, grade level, and a similar score from a prior year’s standardized test (within a tenth of a standard deviation), the only] remaining meaningful difference between students is their school type. Thus, for example, one must believe that there are no remaining systematic differences in the extent to which parents are engaged with their children (despite the fact that parents of charter school
students are necessarily sufficiently engaged with their children’s education to actively
select a charter school), that eligibility for subsidized meals is a sufficient proxy for
poverty when taken together with the other background characteristics.”
In addition to the above, special education students are not a monolith. Research has consistently shown that charters not only take fewer special education students but also enroll fewer students with more challenging disabilities that impact learning than public schools. English language learners, who are at different stages of language acquisition, are not a monolith as well. A few years ago, Wagma Mommandi and Kevin Welner filled an entire book (“School’s Choice”) with illustrations of how charter schools shape their enrollment – often in ways that the virtual-twin approach would not control. Therefore, even the included categories are rough proxies.
Virtual twinning (or “Virtual Control Record” or VCR) also results in an additional problem—large shares of charter school students going “unmatched” and therefore being excluded from the results. Again, I quote NEPC 2013.
“Even more troubling, the VCR technique found a match for only 85% of charter students.
There is evidence that the excluded 15% are, in fact, significantly different from the included
students in that their average score is 0.43 standard deviations lower than the average of
the included students; additionally, members of some demographic subgroups such as
English Language Learners were much less likely to have virtual matches.”
That was in 2013. In this new report, the problem is worse. The overall match rate dropped further to 81.2%. English-language learners had a match rate of 74.9%; multi-racial students had a rate of 58.1%; and the match rate for Native American students was only 38%.
And in some states, match rates were terrible. In New York, only 43.9% of charter school ELL students had a match, and 51% of special education students were matched. In the three categories that are most likely to affect educational outcomes—poverty, disability, and non-proficiency in English—New York rates were well below the average match rate for each category, which might at least partially explain the state’s above-average results.
The study itself notes, in a footnote, “Low match rates require a degree of caution in interpreting the national pooled findings as they may not fairly represent the learning of the student groups involved.”
Do Charters Cherry-Pick and Push Low-scoring Students Out?
Perhaps the most incredulous claim, however, in the study was its “proof” that charters do not cherry-pick or skim and, in fact, teach students who are lower initial achievers.
Here is the CREDO methodology on page 41 for making that claim.
“We compare students who initially enrolled in a TPS and took at least one achievement test before transferring to a charter school to their peers who enroll in the TPS. We can observe the distribution of charter students’ test scores across deciles of achievement and do the same for students in the feeder TPS.”
That may measure something, but not whether charter schools cherry-pick. First, it ignores potential differences in the majority of charter students who never enrolled in a public school. Second, it compares the scores of students whose parents withdrew them from the public school and then compares them with a more satisfied parent population. It’s far more likely that a withdrawal will occur if a student is doing poorly rather than doing well.
Given the CREDO dataset, it would have been relatively easy to explore the question of whether or not charter schools push lower-achieving students out, but that question was not explored.
Findings Regarding Charter Management Organizations
Although I did not review the study’s report that compared student achievement between standalone charters and charter management organizations (CMOS), I noticed that the CMOs of four states of the thirty-one were not included, one of which is Ohio, a state in which the vast majority of charters are run by CMOs (78%), with for-profits outnumbering nonprofits by 2 to 1.
CREDO used the same capricious definition as the National Alliance for Public Charter Schools—a CMO must control three or more schools to be included, which excludes many of the low-performing for-profit-run schools that NPE identified in our report, Chartered for Profit II. While it lists K12 online as a CMO, the equally low-performing Pearson’s Connections Academy was absent from the CMO list.
Conclusion
My review of the study found that the issues included in NEPC’s 2013 review were unaddressed in the newly released study, and new issues have emerged. Hopefully, those who are far more skilled in this type of regression analysis than I am will do a more comprehensive review. But given the bias introduced by the methods in matching and the additional biases created by charters’ shaping of their own enrollment, it’s easy to see how the 0.011 or 0.028 SD findings could be masking negative actual charter effects that are at least as large (in the other direction).
Moreover, based on the trivial topline increases combined with serious methodological issues, I think it is safe to say that despite the billions of tax dollars spent on growing charter schools, overall charter student achievement is about the same as the achievement of students in CREDO’s feeder schools and no conclusions can be drawn regarding the majority of public schools. As to the billionaire funders who financed the report that no doubt cost millions to produce—they got what they paid for. And reporters covering the report have thus far failed to ask the challenging questions that their readers deserve.
Have thus far failed to ask
the challenging questions…
Since one CAN NOT
reverse engineer the
truth out of
BULLSHIT,
Why try?
Have you been conditioned to
believe a score based avatar,
cast a glow of superiority
over less titled mortals?
Do you see through religious
mythology, but not through
the mythology of the state?
Was there a sub on any of those days of learning? Did the regular teacher have to proctor a test that day? Did the monthly active shooter drill interrupt a day of learning? Did the class have a permanent sub. that entire year? Were students transferred to another class suddenly to level overcrowding? And on…and on.
What a ludicrous measure of results. It’s the same crap TFA uses to skew their data by any means necessary.
Enough.
They use the same oligarch playbook as the Fordham Institute.
“…which was uncritically reviewed by Education Week and other publications, which simply quoted the press release.”
The way journalism gets done in the starved monetary condition of this modern day. No time for independent research. Gotta have that copy.
I reviewed that 2013 report and studied their virtual twins for an article I wrote on a CREDO study in 2020. I concluded that it was purposefully misleading for the reasons Burris listed. If this were an unbiased study, I believe they would have concluded that public schools still outperform charters.
One of the reasons behavioral and social studies fall short is that there are so many variables at play it is impossible to draw irrefutable conclusions from the results. Scientific studies in contrast study one variable at a time, which is not possible in “soft sciences” like education. Burris’ critique and methodological concerns are more than justified.
If I’m not mistaken, it was Jersey Jazzman who concluded that entering a charter lottery is one of the variables that impacts student achievement. A family who is motivated enough to take a chance at a lottery would likely have many of the usual attributes that could impact test scores. The same family/student would likely perform the same whether at a TPS or charter. Collectively these lottery students would “improve” scores at the charter, but as we all know, there is no secret sauce and it’s all about how you can drive certain populations to the charter, and at the same time keep certain populations out.
School choice means schools choose.
Thank you. I appreciate the effort. I felt the data was wrong and your report provides assurances that data can be misrepresented and important publication sources do not independently verify the data.
Patrick,
Anyone who follows charters has seen numerous ones that push out or don’t accept students with disabilities or students with learning issues. Any study that says such selection doesn’t exist is telling you to disbelieve what you have repeatedly observed.
In defense of traditional media reporters, newsrooms have been decimated and the reporters that are left are expected to do a job that several reporters handled before the birth of internet propaganda sites like FOX that destroyed common sense in millions of adults, who have been programed to be fascist loving MAGA RINOs.
The traditional media cannot compete with fake news sites like FOX where real reporters don’t exist. Real reporters used to investigate like ProPublica still does. Maybe a larger traditionally media site has one or two investigative reporters left, where there used to be dozens.
Most traditional news sites can’t afford to hire a staff of reporters that investigate. To make deadlines, many reporters who are overworked, write the news from those nonprofit or corporate news releases.
Fox doesn’t have reporters. Fox has propoganda writers who never leave their cubyholes as they poor over press releases and cherry pick the facts that fit the propaganda they are expected to write. They do not investigate anything. If they can’t find any facts to cherry pick, they make them up like pulling rabbits from hats.
This, and also the propaganda is free to access, but much true reporting is behind a paywall.
Fake science. How do these people look themselves in the mirror? I do not know.
And who exactly is bankrolling this in Ohio?
“Public Education Overhaul” reads the headline.
Ohio’s K-12 public schools would get a $1.3 billion increase under the Senate’s plan, but that was about $541 million less than the House proposed.
Instead, the Senate expanded eligibility for Ohio’s EdChoice scholarships to every student on a sliding scale basis, where children whose parents earn more receive smaller amounts.
“We wanted to provide parents throughout the entire state the opportunity to decide for themselves where their child should be educated,” Finance Chair Sen. Matt Dolan, R-Chagrin Falls, said. “Like the governor, like the House, we’ve expanded school choice. Unlike them, we have said that choice belongs to everyone.”
Republicans also added Senate Bill 1, which would put the governor’s office in charge of K-12 education by making the Department of Education’s director a cabinet appointee. The State Board of Education elects the department’s current leader.
Also added to the budget plan was Senate Bill 83, which would prohibit most mandatory diversity, equity and inclusion training, give greater oversight to partnerships with Chinese universities, restrict the kinds of public positions public colleges and universities could take, penalize teachers who fail to create classrooms “free from bias,” and ban professors from striking.
“This is bad for education and bad for Ohio,” Sen. Kent Smith, D-Euclid, said. “The anti-strike provisions make it the worst attack on collective bargaining this chamber has seen since Senate Bill 5 in 2011.”
https://www.dispatch.com/story/news/politics/2023/06/15/historic-education-changes-inside-budget-as-senate-prepares-to-vote/70317264007/
Kathy Irwin These are “up there” with the most dangerous words I have ever heard (quoted from your note). They are the building blocks for chaos and for the demise of democracy as a vibrant political system:
“’We wanted to provide parents throughout the entire state the opportunity to decide for themselves where their child should be educated,’ Finance Chair Sen. Matt Dolan, R-Chagrin Falls, said. ‘Like the governor, like the House, we’ve expanded school choice. Unlike them, we have said that choice belongs to everyone.’” CBK
That was the quote that caught my eye and my breath. Straight out of a propaganda playbook on how to weaponize words. And why are Chinese Universities at the forefront? Chilling. As always, follow the money.
Kathy Irwin Yes, follow the money. But with the Chinese, my thought is that it’s also very much about ideological power, slowly dropping poison on a democracy, and trade/trade/trade until we ripen like a fig on a fig tree. The money will follow. CBK
I am a retired “practitioner”, not a research specialist. Therefore, when I delve into the statistical realm of instructional practice I take the risk of showing my ignorance. However, as a practitioner, it occurs to me that much of the data produced and reported, especially from private interests pushing a particular bias, demonstrates little in the way of progress or decline. Since the beginning of the so called “Standards Movement”, I have seen no data that shows meaningful improvement in student achievement. Since the hyper focus on reading and math brought about by NCLB, the Common Core, or “Race to the Top,” no meaningful improvement has been identified in either subject. As a Principal I often found myself troubled by the lack of integrity shown in the reporting of education data to the general public. Charters were promoted as laboratories for innovative instructional practice until they weren’t. Vouchers are being promoted as choice where only the most astute or privileged actually get to choose. Little has changed in our measure or estimates of student progress. Meanwhile, teachers are leaving the profession because they have lost autonomy, and opportunity remains for the privileged while seldom available for the poor. We have been involved with this charade for over four decades now. At what point do politicians, policy makers, and corporatists admit that privatization and technology does not improve student outcomes? The results are in.
My sentiments exactly! I am so weary of this everlasting “debate” that has done nothing but serve as yet another thing to divide ourselves about instead of uniting to make our public education system first rate for every child. Wasn’t it about forty years ago that the “deindustrialization” of our country was hitting its peak resulting in the “rust belt” amongst other destruction and do you think that has any bearing on this seeming stampede into public education over the past few decades as an economic alternative? Like, if you can’t start a factory, you can start your own school or if you’re a college graduate but can’t find a job you can “do” TFA for a while? I’d appreciate your perspective on whether or not there’s a correlation there. Thank you!
There is a correlation to all of it. This age of privatization is no more than a realization among pyramid schemers that there is a lot of money that can be taken from government. As Steve Bannon often proclaims, they “fill the zone with shit” to create confusion and then pick up the pieces from this disruption. Our government no longer enforces anti-trust laws or campaign finance laws so billionaires believe they can do anything without impunity. So the public schools suffer because tech geeks, privatizers, and AMWAY legacies drain public coffers for their largess.