An experienced researcher saw a story in the Economist about charter schools. It was, as is typical among news stories, incredibly naive. The writer didn’t ask the right questions. Maybe he already believed in the charter “miracle” story and didn’t ask any questions.

So my correspondent–who requires anonymity– decided that it would be helpful to reporters and members of the public to explain how to read stories about charter schools. Mainly it involves the ability to decipher false claims.

They do not have a “secret sauce,” the phrase once used by Mayor Rahm Emanuel to describe the Noble Network of Charter Schools, each of which is named to honor a very rich patron.

They do have a secret recipe, however, for manufacturing the illusion of success.

Be wise. Think critically. Read carefully.

Here is expert advice:

How to Read News Stories about Charter Schools

Reports and stories about charter schools are in the media every day. The majority of these stories praise charters, while often demeaning public schools. We propose that every reader of such stories ask the following questions before taking the claims of such articles seriously.

Does the story compare the demographics of the student population served by charter schools to the demographics of local public schools? Does it include data on the charter school attrition rate? Does it include data on how the students who leave the charters compare to students who leave public schools? Does it include numbers of students expelled? Does it include numbers of students suspended? Does the story focus exclusively on test scores? If so, has someone, with educational expertise, visited the school to determine if the school focuses on test prep at the expense of a rich curriculum? Are the test scores reported outside of school assessments such as the SAT/ACT or does the story only report test scores of exams that are proctored in-house? Does the story account for the fact that, due to the need to apply to the charter school, parents of the students at charters are, on average, likely to be more engaged in education than the parents of students at public schools? Does it exclusively or primarily cite reports funded by pro-charter or conservative think tanks? Does it include quotes from academic scholars or does it just cite charter school advocates? Does it identify advocates or simply call them “experts” or “researchers”? Does it compare the resources available to charter schools to those available to public schools? Let’s call this approach “identifying charters’ bogus statistics” or the ICBS strategy.

It grows tiresome to dispute every tendentious article written on charter schools.  But let’s see how the ICBS strategy would help us evaluate a sample story. The Economist recently ran an article praising charter schools and attacking Bill de Blasio for proposing to charge rent to charter schools that use public space in New York City.

The Economist presents the Noble Network of charter schools in Chicago as a paragon of charter school excellence. “Around 36% of the…children enrolled with Noble can expect to graduate from college, compared with 11%…city-wide.” What does the data actually tell us about the Noble Network?  As is, unfortunately, standard practice across many charter schools, the Noble Network does not serve equal proportions of the neediest students. In fact they serve 35% fewer English Language Learners and 22% fewer special education students than Chicago Public Schools. This lack of inclusivity extends to other areas too, such as their ban on a Gay Straight Alliance student group.

An op-ed by Congressman Danny Davis noted that the Noble Network suspends 51% of its students at least once during a school year. This includes suspending 88% of the African American students who attend its schools. It might be hard to understand why a school would want to suspend so many of its students…until you realize that this encourages students to leave. And it specifically encourages the more challenging students, the ones most likely to bring down test scores and college graduation rates, to depart. This is not the only such strategy they employ. One exposé revealed that the Noble Network’s “discipline system charges students $5 for minor behavior such as chewing gum, missing a button on their school uniform, or not making eye contact with their teacher, and up to $280 for required behavior classes. 90% of Noble students are low-income, yet if they can’t pay all fines, they are made to repeat the entire school year or prevented from graduating. No waivers are offered, giving many families no option but to leave the school.” The data show that this strategy works. The Noble Network loses over 30% of the students in each class that enters its schools.

As has become all too common, the public school district officials refuse to acknowledge these facts. The former CEO of the Chicago Public Schools told a reporter that he’d turn over data showing that charters don’t “have policies that systematically weed out weaker students.” But as the story notes “the district didn’t keep that promise. WBEZ did obtain an internal CPS memo. It’s titled “Memorandum on Charter School Myths.” The four-page report actually finds that traditional schools held onto more kids than charters did for the year CPS examined.”

The other set of Chicago charter schools praised by the Economist had their contract shortened from 3 years to 5 due to poor performance. Despite the Economist’s claim that “charters have worked well in Chicago,” the actual data show that charters are not working well. As reported by the Chicago Sun Times, “The overall passing rate at two city charter franchises — Aspira and North Lawndale — was below the city average at every campus those two groups operate. Four other chains — Betty Shabazz, Perspectives, North Lawndale and Chicago International — saw the majority of their campuses with over-all pass rates that were below the citywide average.” Even the Walton Foundation-funded CREDO report cited by the Economist, which did not account for the numbers-gaming we noted above, showed mixed outcomes by Chicago’s charters. “In reading, 21 percent of charters performed worse than traditional schools, while 20 percent did better and 59 percent showed no difference. In math, 21 percent of charters did worse, 37 percent performed better and 42 percent showed no difference. Black and Hispanic students continued to lag behind white students in reading, and received “no significant benefit or loss from charter school attendance” compared to students in traditional schools.”

And let us not even mention Chicago’s largest charter chain, called UNO, which received a state grant of $98 million to build new campuses. Its politically powerful CEO–who was co-chair of Mayor Emanuel’s election committee–resigned after revelations in the media of multiple conflicts of interest in the award of contracts and jobs.

But enough about Chicago. The Economist also claimed that “New York City’s charter schools generally outperform their neighbouring district schools.” The data do not support this. According to the data set on the New York City Department of Education’s website, when compared to similar elementary and middle schools, charter schools rank at the 46^th percentile in English growth, the 41^st percentile in English growth for students who start with scores in the bottom third, the 53^rd percentile in Math growth and the 45^th percentile in Math growth for students who start with scores in the bottom third. Not only do they not outperform they don’t even match. This past year charter schools saw bigger drops in performance on the Common Core exams than public schools. Additionally charter schools performed worse on average than public schools in English and the same as public schools in math. As do Chicago charter schools, New York City charter schools have extremely high suspension and alarming attrition rates. In fact a recent analysis by the NYC Independent Budget Office found that charter schools selectively attrite students with lower test scores. “The results are revealing. Among students in charter schools, those who remained in their kindergarten schools through third grade had higher average scale scores in both reading (English Language Arts) and mathematics in third grade compared with those who had left for another New York City public school.”

A school from the Success Academy network was singled out for praise by the Economist. What does Success Academy do? They seem to employ the same strategies as the Noble Network in Chicago. In one neighborhood, Success Academy serves 18% fewer impoverished students, 9% fewer English Language Learners, and 13% fewer team taught and self-contained special education students (at a negligible .01% of their student population) than the local public schools. What’s worse Success Academy seems to push out the few special education students that they do admit. Success Academy suspends students at rates well in excess of other public schools in the same district. According to one newspaper report “at Harlem Success 1… 22% of pupils got suspended at least once… That’s far above the 3% average for regular elementary schools in its school district.”

Success Academy has very, very high attrition rates. The data show that over half of each entering class disappears over time. The 2012 data reveal that there were 482 third grade students tested in 2012 but only 244 students were tested at the highest tested grade. The 2013 data reveal that 487 third grade students were tested in 2013, but only 220 students were tested at the schools’ highest testing grade. Assuming similarly sized entering classes at each school and only looking at schools for which we have data across years (i.e. excluding schools that have had only one testing grade which would not permit comparative analysis) over 55% of Success Academy’s students are lost from each grade. Success Academy’s strategy for “success” seems to be to get rid of students who are identified as not succeeding.

The Economist cites a report by “the Manhattan Institute, a conservative think-tank” on the rent question. Bruce Baker, of Rutgers University, has debunked that report. His conclusion, “it makes little sense for the district to heavily subsidize schools [i.e. charters] serving less needy children that already have access to more adequate resources. It makes even less sense to make these transfers of facilities space (or the value associated with that space) as city class sizes mushroom and as the state indicates the likelihood that its contributions will continue falling well short of past promises.”

Using the ICBS strategy it appears that the claims made by the Economist are unsupported by evidence. Stories like this will continue to be published but, armed with the ICBS strategy, readers should not fall prey to such propaganda.