Yesterday, Patrick Wolf published a vitriolic attack on me and on the National Education Policy Center.

This was in response to a post I had published saying that vouchers had failed in Milwaukee. They are supposed to “save minority students from failing schools,” but they do no better and sometimes worse than public schools. I cited Wolf’s evaluation, state test scores (which showed no edge for voucher students), and the fact that 75% of the voucher students in his study did not remain in the voucher schools to graduate. The 75% attrition rate appears in Wolf’s report. I did not know that he subsequently lowered the attrition rate to 56%, although that too is a pretty staggering attrition rate. I also noted that Wolf recently wrote an editorial in the Minneapolis Star-Tribune chastising his home state of Minnesota for not requiring more school choice. This caused me to question his “independence” as an evaluator of school choice, since his article advocated for more school choice. And this is why he is so angry at me.

Kevin Welner, director of the National Education Policy Center, wrote this response to Wolf:

 PATRICK WOLF SHOULD APOLOGIZE

The Education Next website yesterday posted Patrick Wolf’s very personal and misguided attack on Diane Ravitch (http://educationnext.org/ravitch-blow-up-on-school-choice/). He also made inaccurate statements aimed at the National Education Policy Center (NEPC), which I direct. I will, below, extensively quote some of the vitriol, because I don’t want any of Patrick Wolf’s injudicious wrath to be lost in paraphrase. Then, after those quotes and a description of how they are false and misleading, I ask the piece’s author as well as the publishers of Education Next, to publicly apologize for the inaccurate statements about NEPC and about Dr. Ravitch.

The Education Next piece is written by Prof. Patrick Wolf of the University of Arkansas. He heads up the School Choice Demonstration Project (SCDP), and has been paid millions of dollars by several jurisdictions with voucher policies to “independently” evaluate those policies. The NEPC has, in turn, reviewed several of the publications that Wolf and his team produced. Those expert reviews show the evaluations to be reasonably well executed, but the reviewers have also consistently pointed to a pattern of presenting evaluation findings in ways that are misleadingly positive – as well as a pattern of minimizing clear limitations in the data.

This brings us to a review in 2012 of Wolf’s “Report #30” of the Milwaukee voucher program (see http://nepc.colorado.edu/thinktank/review-Milwaukee-Choice-Year-5). For the expert review, NEPC turned to Casey Cobb, who is Department Head and Professor in the Department of Educational Leadership at the University of Connecticut. He is also Director of the Center for Education Policy Analysis at UConn.

Prof. Cobb’s summary included the following:

[The report’s conclusions about improved graduation rates] should be considered alongside at least two important caveats, however. The first is a methodological concern. Roughly 75% of the original sample of 801 [Milwaukee Parental Choice (voucher) Program (MPCP)] 9th graders were not still enrolled in a MPCP high school in 12th grade. The inferences drawn about the effects of the MPCP on graduation rates compared with those in the [Milwaukee Public School (MPS)] are severely clouded by substantial sample attrition. A second concern lies in the report’s interpretation of the data. Among the most careful statistically controlled analyses, only one finding was statistically significant at conventional levels. These two limitations prevent broad conclusions being drawn about the relative effectiveness of the MPCP and the MPS on graduation and higher education continuation rates. (Emphasis added.)

This week, Diane Ravitch cited Cobb’s NEPC review, noting in particular the point about the 75% attrition. Wolf was none too pleased.

He first points to his stature as the go-to guy for people wanting their voucher programs evaluated: “I keep winning the competitions to perform the most important private school choice evaluations around the country, and regularly publish my results in the very best scientific peer-reviewed policy journals (see here, here, and here), Ravitch’s ad hominem attacks notwithstanding.”

(Regarding the ad hominem allegation here, Wolf seems to be responding to Ravitch pointing out that Wolf has publicly and vociferously advocated for vouchers, so she suggested that he might not be considered an “independent evaluator.” This doesn’t strike me as ad hominem any more than if were to suggest that his Education Next piece reads to me as smug. Readers can draw their own conclusions.)

Wolf then launches into an attack on Ravitch and NEPC, claiming that the 75% figure is incorrect. He even mocks Ravitch as innumerate and NEPC (and/or our expert reviewer) as unreliable or incompetent. Ravitch, he says, “claims that the similar Milwaukee finding of higher educational attainment from vouchers is questionable because ‘75% of the students who started in a voucher school left before graduation.’ For support, she cites a review of our study performed by the National Education Policy Center (NEPC).” He continues:

Now, professional historians cite original sources to make their claims, but, remember, we are talking about Diane Ravitch here. Is the NEPC claim credible? Let’s examine the original sources. From page 16 of our report, “the majority of students (approximately 56 percent) who were enrolled in 9th grade in MPCP were not enrolled there by the time they reached 12th grade.” Also, from page 163 of our article published in the prestigious scientific Journal of Policy Studies, “less than half (44 percent) of the original MPCP panelists examined were enrolled in a voucher school by the time they reached 12th grade.” I realize that Ravitch is no statistician but even she should know that 56 percent is not 75 percent and 44 percent is not 25 percent. It doesn’t excuse Ravitch that the factual error was first promulgated by NEPC. She should know better than to trust the accuracy of their “reviews” when primary source material clearly contradicts them.

Yesterday, after this was posted, I received an email from one of the EdNext readers, pointing me to Wolf’s critique. I immediately went to page 16 of Wolf’s report. Could we have made such a mistake?!  Actually … we didn’t. Here’s what it said on page 16: “A second caveat is that the majority of students (approximately 75 percent) who were enrolled in 9th grade in MPCP were not enrolled there by the time they reached 12th grade.”

So I followed the link in the Education Next piece and downloaded the same report. Here’s what it says on page 16: “A second caveat is that the majority of students (approximately 56 percent) who were enrolled in 9th grade in MPCP were not enrolled there by the time they reached 12th grade.”

That was certainly odd. Then on third page of the pdf I’d just downloaded, I found the following: “Updated and Corrected March 8, 2012.” It doesn’t say what specifically was updated or corrected, but clearly one change was on page 16.

So here’s the timeline:
1. February 2012: Wolf and his colleagues publishes the SCDP report, stating that “approximately 75 percent” of the voucher students  enrolled in 9th grade “were not enrolled there by the time they reached 12th grade.” (On February 24th, NEPC sent the report to Prof. Cobb for a review.)

2. March 8, 2012: The SCDP changes that sentence, substituting “56” for “75”.

3. April 19, 2012: NEPC publishes the Cobb review, pointing to (among other things) the 75% figure as evidence of the study’s limitations. Nobody had thought to go back and see whether Wolf or his colleagues had changed important numbers in the SCDP report.

4. April 1, 2013: Wolf attacks Diane Ravitch and NEPC for CORRECTLY quoting Wolf’s own report.

I will generously assume that, in writing this attack, Wolf had simply forgotten that he changed the 75 to a 56. In truth, every one of us makes mistakes, and our memories aren’t what they once were. Also, Wolf wasn’t first author of the SCDP report, so maybe he wasn’t privy to the change. But what makes this mistake come with such ill grace is that Wolf didn’t simply contend that NEPC or Ravitch made an error. Instead, he unloaded with an angry and self-righteous attack. His piece ends with this:  “It takes a lot of doing for a person to mislead so many about so much, but apparently Diane Ravitch is up to the job.”

Considering that Dr. Wolf has, in this Education Next piece, knowingly or unknowingly misled the public about the changes in the reporting of his results, the appropriate step for EdNext and Dr. Wolf to take would be the issuance of a public apology for the attacks based on this error. As is the collegial responsibility of academics, I also call upon Wolf to release his data. A re-analysis of the SCDP data is important in light of his critique and the circumstances and errors in the public reporting of his results.