I posted a summary of Professor Francesca Lopez’ review of a charter school meta-analysis published by the Center for Reinventing Public Education at the University of Washington. In my introduction, I referred to CPRE as a “leading proponent of charter schools.” Robin Lake wrote me to challenge that characterization. I associate CPRE, which receives extensive funding from the Gates Foundation, with the idea of portfolio districts, in which struggling public schools are replaced by a portfolio of privately managed schools. I invited Robin to send me any CPRE publications critical of charter schools, and I will post about them when I receive them.
Dr. Lopez writes:
Recently, I wrote a think tank review for NEPC on a CRPE report that was summarized on Diane Ravitch’s blog. I was contacted by Adam Gish, an English teacher at Garfield High School in Seattle, Washington, who had read the blog post and then asked CRPE’s Robin Lake about her opinion of the NEPC review. Mr. Gish sent me the exchange with Dr. Lake’s response, which he gave me permission to publish, with the thought being that a public exchange could help prompt a larger dialogue.
Here’s what Dr. Lake wrote:
“I patently disagree with the review. It seems to present statements out of important context and ignores what the authors say. For example the authors say that the time trend is positive but not statistically significant but the review cites the authors as having called the trend significant. That’s either a misunderstanding of basic statistical analysis or an intentional misrepresentation. There are numerous other inaccuracies and misinterpretations.
Julian Betts is one of the most cautious, rigorous, and respected analysts I know. That’s why we chose him to do this review. His analysis made minimal and evenhanded conclusions and was peer reviewed by one of the best statisticians in the country.
I really don’t see any legitimate critique here.
Hope this helps you know my view.
Best,
Robin”
Mr. Gish, in his note to me, asked “what [my] rebuttal would be,” so I would like to offer it here. It is Dr. Lake who is incorrect; nowhere in the NEPC review did I “cite the authors as having called the trend significant.” What I do point out is the authors’ claim that there is a positive trend, which is a misleading claim since they also (as I explain on pages 3 and 4 of the review) reported non-significant findings. To use the phrasing of Dr. Lake, this is “basic statistical analysis.” One cannot call something “positive” or “negative” when it is not significant. The point of the trend analysis was to determine if the trend was positive or negative. Because it was not, calling it “not significant” while at the same time calling it “positive” is inaccurate and misleading.
Dr. Lake did not offer sufficient details for a more elaborate rebuttal, but I welcome a discussion regarding what her perceived “numerous other inaccuracies and misrepresentations” might be.
Francesca López

Lopez is incorrect when she says “One cannot call something “positive” or “negative” when it is not significant.” For example, a regression coefficient can have a positive sign but the coefficient might not be statistically significant. Understanding the distinction between statistical and substantive significance here is important. Lake’s comments appear to engage that distinction while Lopez elides it.
LikeLike
What I’m taking away from the study and her review of the study isn’t what is “statistically significant” or not, rather that there are a lot of limitations to the study and those seemed to be skirted in the conclusion.
I still have not seen any strong evidence that just enrolling your student in a charter will make them perform better than a traditional school. There are so many factors involved and no two schools are alike that this will be hard to prove.
The strongest research still shows that parental involvement is still the #1 factor in student performance. If your kid is highly motivated and the parents support them, they will do well no matter what the school.
LikeLike
Nonsignificant results should not be further interpreted. To do so is to mislead.
LikeLike
I’m not sure what you mean by “further interpreted.”
What about statistically significant results that are not substantively significant? Should they be interpreted?
In general, your response here helps to illustrate my point. People often use the word “significant” very casually in these sorts of discussions. That ends up fostering misunderstandings rather than deeper understanding, especially for people who lack training in quantitative methods.
LikeLike
Well, somehow you beat me to the point, and your cogent reply wasn’t available to me (can’t understand why, based on the time difference) before I said, essentially, the same thing below. I totally agree, and I’m sick of seeing insignificant ‘trends’ touted. Clearly those who promote this nonsense are out to deceive.
LikeLike
Let me follow up on my previous comment. Those who knowingly use insignificant data to ‘suggest’ (promote?) a position are a HUGE danger to a Western Society based on rational discussion. They undercut honest science and inquiry and cut at the very foundations of our understanding. Whoever does this needs to be strung up by the thumbs.
LikeLike
If it ain’t statistically significant, I shouldn’t be reported. Doing so is intellectually dishonest, period!.
LikeLike
At this point in American history, illegitimate research results are moot. An organization’s total research work product, meshed with its funding sources, shows the embarrassing fact that research outfits, target and spin their results, for their benefactors.
The rationalization process provided by hired-gun researchers, is an unnecessary plutocratic expense. Simply said, politicians enact policies that make themselves rich.
Middle class workers can fund valid research and donate to politicians but, they don’t have the means to make politicians wealthy (or, to promote research findings). In a game devised so that the 99% will lose each time the game is played, minimizing losses is their only remaining strategy. An ignorant electorate preyed upon by oligarchs, explains the rigged game. The greatest generation abandoned their principles at the polls, after Eisenhower, and gave birth to successive “worst generations” of voters.
LikeLike
Paul Manna, my clarification was specifically in reference to the statistical test that was used to determine whether a trend was positive or negative. The test was not significant, which means one cannot call the trend positive.
The review is clear about what the original authors found, as well as what they did not find. The issue is in the exaggeration of claims that remain unsubstantiated by the results.
LikeLike
That’s not true. A coefficient can be positive without being statistically significant (because the estimate is noisy), but this does NOT mean that the best estimate of the coefficient is zero. The best estimate of the coefficient remains the positive estimate that you see before you, even if it is not “significant.”
LikeLike
You are ignoring the very “noise” you are referring to.
LikeLike