Charter school advocates were thrilled by a recent CREDO report celebrating the “remarkable improvement” in charter schools. Valerie Strauss of The Washington Post did some digging and found that the “remarkable improvement” was a chimera..

She wrote:

It seemed like good news for charter schools when a study released this summer declared that they get better student outcomes than do traditional public schools — at least from 2015 to 2019, the years for which researchers said they crunched the numbers. The Wall Street Journal editorial board hailed the results as showing “huge learning gains over union schools” (with “union schools” used as a pejorative reference to public schools in traditional school districts).

Education Week’s headline declared: “Charter Schools Now Outperform Traditional Public Schools, Sweeping Study Finds.”

But the study, it turned out, doesn’t show that at all. The headlines were wrong. For one thing, a close look at the results revealed only tiny improvements in charter schools. That, plus concerns critics have raised about the validity of the methodology and definitions used in the study, render moot the claims of besting traditional public schools.

The “not what they seem” theme of the study results reflect the uncertain position in which charter schools find themselves these days. The vanguard of the “school choice” movement when the first charter opened in 1992 in Minneapolis, these schools have been eclipsed in the national debate about “school choice” by programs that use public money for private and religious schools, including vouchers, tax credit programs and education savings accounts.

Robert Enlow, president and CEO of the Indianapolis-based EdChoice, a nonprofit that tracks and advocates for school choice policies, has declared 2023 the “year of universal choice” because of the proliferation of new state laws establishing or expanding programs that allow the use of public funds for private and religious education. According to EdChoice, North Carolina joined seven other states this year that have created a new “universal choice program” — meaning all families in the state have access to it — or expanded an existing one to include all students. The states are Iowa, Utah, Arkansas, Florida, Indiana, Oklahoma and Ohio. Arizona and West Virginia already had such programs.

At July’s 50th annual conference of the American Legislative Exchange Council — a network of conservative state legislators, philanthropies, donors and other groups, right-leaning advocacy groups, and private-sector businesses that drafts and disseminates “model bill” proposals for state legislation — the education talk was focused on programs that use public dollars for private and religious education, according to two Wisconsin Democratic state legislators who attended, Reps. Kristina Shelton and Francesca Hong. Both lawmakers said in interviews that charter schools were virtually not discussed.

Charter schools are publicly funded but privately operated, some of them as for-profit entities, and they educate about 7 percent of U.S. schoolchildren. The 30-year-old charter sector has been riddled with financial and other scandals over the years, though supporters say that the problems these schools face are expected growing pains and that they offer families an important option over schools in publicly funded districts.

Opponents say that charters are part of an effort to privatize public education, that there is little public accountability over many of them and that they drain resources from the traditional districts where the vast majority of children attend school.
Charters are permitted to operate in 45 states plus the District of Columbia; California has the most, with some 1,330, while other states have very few. Washington state, for example, has about a dozen.

As momentum for these programs grows, charter schools face existential questions about the “public” nature of “public charter schools.” That question was underscored most significantly with a recent, unfinished effort in Oklahoma to open what would be the nation’s first religious charter school. Publicly funded schools are not allowed to teach religious doctrine though they can teach about religion. The Catholic Archdiocese of Oklahoma City is seeking to open a virtual charter school that, according to the application, would serve “as a genuine instrument of the Church.” Republican Gov. Kevin Stitt backs the school’s application. Oklahoma’s Statewide Virtual Charter School Board voted 3-2 in June to allow the school to open, in 2024, but a lawsuit is challenging it, and state Attorney General Gentner Drummond (R) warned in February that allowing the school to open would create a dangerous precedent allowing any religious group to open a publicly funded charter school. Still, on Monday, a state board approved a contract with St. Isidore of Seville Virtual Charter School, bringing the institution one step closer to becoming the first publicly funded religious charter school in the nation.

Nina Rees, president and CEO of the National Alliance for Public Charter Schools, the charter sector’s major trade group, says charter schools are all public and have to follow laws, just like traditional school districts, that forbid the teaching of religious dogma. Some charter critics have questioned that “public” status for years — as have some judges — arguing that many are permitted by state laws to operate with little public accountability, that some operate as for-profit businesses, and that some charter schools claimed to be nongovernment entities when arguing against teacher unionization. Rees now acknowledges that “this notion of our public identity is going to be challenged.” The U.S. Supreme Court this year declined to accept a case that could have settled the issue.

Another bit of bad news for charter schools came recently in the form of a federal audit of the U.S. Education Department’s Charter Schools Program (CSP), which has provided more than $2.5 billion in grants to help open or expand charter schools. The audit by the department’s Office of Inspector General, released in August, was aimed at determining whether the department’s processes “provided reasonable assurance” that the program’s grantees were reporting “complete and accurate information” in their annual performance reports (APR) and spent grant money “only on allowable activities and in accordance with program requirements.”

Auditors wrote:

We concluded that the CSP office generally implemented these processes as designed. However, it did not always ensure that CSP program officers accurately and completely filled out APR review templates and notified grantees of issues or concerns identified during their reviews of APRs. As a result, the CSP office might not have had reliable information needed to make informed decisions about continuation funding. Additionally, the CSP office might not have provided timely assistance to grantees that needed assistance to meet their approved goals. Further, we determined that the Department and the CSP office also designed processes that should have provided reasonable assurance that Replication and Expansion grantees spent grant funds only on allowable activities and in accordance with program requirements. We concluded that the CSP office generally implemented these processes as designed. However, it did not always ensure that grantees implemented corrective actions to address significant compliance issues relevant to their uses of Replication and Expansion grant funds, fiscal control, and fund accounting. Lastly, the CSP office did not always retain records in official grant files. As a result, the CSP office could not find about 52 percent of the APR review forms that we concluded CSP program officers should have completed from Oct. 1, 2015, through June 30, 2021. Additionally, the CSP office could not find written correspondence with the grantees associated with about 10 percent of the APR review forms that we requested for review.

The audit included a response from the Education Department that said it was already implementing some of the recommendations made to improve processes but also said it did not concur with a few of the findings.

The federal program was the subject of several reports by the nonprofit Network for Public Education, an advocacy group that is highly critical of charter schools and advocates for legislative reform. It has published reports since 2019 on the federal program, revealing the waste of hundreds of millions of taxpayer dollars on charter schools that did not open or were shut down. The reports also showed that the Education Department did not adequately monitor federal grants to these schools. You can read about two of those reports here and here. A third report details how many for-profit management companies evade state laws banning for-profit charters.

Meanwhile, the boost charter schools seemed to get from the student results wasn’t authentic. The June study was the third in a series started in 2009 by CREDO, or the Center for Research on Education Outcomes, which was founded at the University of Rochester by Margaret “Macke” Raymond and her husband, Eric Hanushek, an economist. In 2000, they moved CREDO to Stanford University’s Hoover Institution, a conservative think tank that supports charter schools and school choice, according to CREDO’s 2001 annual report, giving it more “institutional credibility.” Raymond, who directs CREDO, is a research fellow and scholar at Hoover. Raymond still directs CREDO, which is funded in large part by foundations and individuals who have spent millions of dollars supporting charter schools.

The new CREDO report identifies two nonprofits as underwriters: the City Fund and the Walton Family Foundation. The City Fund is financially supported by a number of billionaires who support charter schools, including Bill Gates, John Arnold and Reed Hastings. The Walton Family Foundation was one of the first organizations to boost charter schools and calculates that it has supported about a quarter of them, spending hundreds of millions of dollars and pledging more than $1 billion.

The report looked at standardized test scores in 31 states between 2015 and 2019. It concludes that charter schools “produce superior student gains despite enrolling a more challenging student population than their adjacent” traditional public schools. It further says: “The benefit of attending charter schools during the period of study amounts to additional days of learning equivalent to six days in math (0.011) and 16 days in reading (0.028).” And it says there are more than 1,000 “gap-busting” charter schools that “have eliminated learning disparities for their students and moved their achievement ahead of their respective state’s average performance.”

Not really.

For one thing, Raymond said in an email that CREDO used standardized test scores as the basis for its calculations. In the United States, standardized test scores have for decades been a key measure for assigning quality to a school, despite the fact that assessment experts have long said that a single metric can’t reveal the full life of a school, or that the tests themselves are often substandard, or that the algorithms used to decide what the test scores really show are misused.

As for the specific findings of the study, Matt Barnum of the education news publication Chalkbeat put it this way: “CREDO found that attending a charter school for one year would raise the average student’s math scores from the 50th percentile to the 50.4 percentile and reading scores to the 51st percentile. By conventional research standards and common sense, these impacts are small.” Trivial, actually.

The nonpartisan National Education Policy Center at the University of Colorado at Boulder published a critique of CREDO’s report authored by Joseph J. Ferrare, an associate professor at the University of Washington at Bothell who studies education reform movements. He wrote: “Readers should maintain a healthy skepticism toward the findings of this report given its nonexperimental design … [I]t is simply not possible to rule out the potential that unobserved factors are, at least in part, driving differences in test score growth across sectors.”

The Network for Public Education issued a lengthy critique of the CREDO study about problems with data, reporting methods, conclusions and funders. One section detailed issues with CREDO’s calculation’s for “days of learning.” The report, written by the network’s executive director, Carol Burris, a former award-winning educator, says:
CREDO reports its top-line results in Days of Learning, a construct it developed based on the 2012 assumption of Eric Hanushek, Paul Peterson, and Ludger Woessman that “[o]n most measures of student performance, student growth is typically about one full standard deviation on standardized tests between 4th and 8th grade, or about 25 percent of a standard deviation from one grade to the next.”According to CREDO, 5.78 days of learning translates to a 0.01 standard deviation difference. That means the 6.0 “days of learning” average increase in math achievement between charter school students and their virtual twins translates to a 0.011 increase in standard deviation units, which is the accepted way of presenting such differences in scholarly literature. Sixteen CREDO days of learning in reading account for only 0.028 standard deviations (SDs).To ascertain whether or not differences in the range of 0.11 to 0.028 SDs are “remarkable,” I quote CREDO itself as it described its comparative findings between charter and public school students in 2009 on page 22 of the report. Note that the relative differences were similar, although reversed.”

In reading, charter students, on average, realize a growth in learning that is .01 standard deviations less than their TPS counterparts. This small difference — less than 1 percent of a standard deviation — is significant statistically but is meaningless from a practical standpoint. Differences of the magnitude described here could arise simply from the measurement error in the state achievement tests that make up the growth score, so considerable caution is needed in the use of these results.

In math, the analysis shows that students in charter schools gain significantly less than their virtual twin. Charter students on average have learning gains that are .03 standard deviations smaller than their TPS [traditional public school] peers.Unlike reading, the observed difference in average math gains is both significant and large enough to be meaningful. In both cases, however, the absolute size of the effect is small.”

In 2013, then director of the Brown Center on Education Policy at the Brookings Institution, Tom Loveless, used real-world examples to show the minimal impact of findings between .01 and .03 standard deviations in the second CREDO national study. In Charter School Study: Much Ado About Tiny Differences, he made the point that regardless of whether charter schools or public schools are up or down, the differences between the sectors were so small that “the two sectors perform about the same.”He refers to the 1969 guidelines provided by Jacob Cohen, a psychologist, and statistician best known for his work on effect sizes. Cohen categorized effect sizes as small if they meet the thresholds of 0.2, medium at 0.5, and large if it reaches 0.8 standard deviations.

To give real-world context, Loveless provided the following example:“You attend a public talk given by a close friend, a guy who is about 5’ 10” tall. He stands behind a podium on 7 sheets of paper. That’s a bit larger than a 0.01 SD addition to his height. Would you go up to him afterwards and say, ‘Wow, you look taller today.’ I doubt it. What if he stood on 20 sheets of paper (about 0.03 SDs)?

Nope. You wouldn’t notice a difference.”

It is worth noting that the new report uses the term “statistically significant” 39 times, a phrase researchers use to suggest that their findings have meaning, and, presumably, impact in the real world. But “statistically significant” findings often don’t have much meaning in the real world, a point made in a 2019 editorial by statisticians and published in the American Statistician, a journal of the American Statistical Association. The editorial called for ending the use of the term for reasons including this one: “Don’t believe that an association or effect exists just because it was statistically significant.”

Please open the link and read the rest of this important article.