In mid-December, Matt di Carlo of the Shanker Institute reviewed the year’s production of research about charters, teacher incentives, and other aspects of the market-based approach to schooling, that is, the use of incentives and sanctions to produce higher test scores.

Schools Matter has published critiques by John Thompson of di Carlo’s review. Di Carlo is known for his scrupulous nonpartisanship; although part of the AFT, he does not take the union’s side. He writes as a researcher with no skin in the game.

In part 1, Thompson asks

“Where is the Matt DiCarlo of corporate reform? Where is a reformer who is will break ranks, for instance, on Washington D.C.’s IMPACT? There are plenty of individual reformers who have open minds. Are there any who are allowed to be like DiCarlo and acknowledge the strengths of evidence on the teachers’ side?”

Thompson is hard on di Carlo for standing back and above the fray:

Matt DiCarlo voices agreement with Gates that teachers and students should continue to be lab rats for Gates’ theories before rejecting them. If DiCarlo doesn’t believe that the jury is in and that twenty years of “reform” has failed, that’s fine. Decent people can agree to disagree.

But, I’m growing more frustrated with DiCarlo’s timidity in the face of corporate reformers singing from the same hymnal. He doesn’t challenge their assumptions and he keeps letting them define the issues. Moreover, I’m getting upset at the way he equates the arguments of test-driven reformers and teachers who resist them. It was the Billionaires Boys Club that came into school reform, denied that they had the burden of proof to show that their hypotheses would do more good than harm, and employed scorched earth politics against teachers and unions. Now, educators are criticized when we raise our voices in protest.

Above all, I am perplexed by DiCarlo’s refusal to push back on policy papers that employ sophisticated quantitative methods, but make no effort to ground their models in reality. 

Thompson adds:

The Billionaires Boys Club has tried to replace the traditional peer review process, social science and education history with Big Data. In doing so, they are trying to drive the clash of ideas from public schools.

The accountability hawks’ utilitarianism, bordering on anti-intellectualism, helps explain why high-stakes testing has failed. They imposed a radical and risky experiment without bothering to study the evidence on teaching and learning. Old Blood and Guts Bill Gates, however, says we need to endure another decade of his bubble-in experiments to see if they work. Yeah. His guts; Our blood.

In part 2 of Thompson’s critique of market-based reforms, he reviews many of the research studies on value-added measurement, showing that many of them contain warnings about the inaccuracy of VAM (one says that 68% of the ratings produce false positives and false negatives, a heck of a way to fire a teacher).

He concludes:

I would add another point which applies to nearly every aspect of market-based reforms. In schooling, the feces rolls downhill. The venom dumped on adults flows down onto children. When principals are subject to not-ready-for-prime-time accountability schemes, the #1 priority often will be to make sure the patient doesn’t die on their operating table. The search for scapegoats takes off. That is why the blame game is the prime legacy of market-driven reforms.

Part 3 of Thompson’s critique analyzes “the pitfalls that were discovered after value-added systems were implemented.” Value-added studies consistently find that teachers in high-poverty schools get less VAM than those in low-poverty schools. Is it the teachers at fault or the model or the assumptions behind the model?

Thompson concludes:

The idea that value-added evaluations could address the teacher quality equity issues in high-challenge schools is based on three assumptions. Firstly, that overworked inner city principals (who in my experience always worked over 80 hours a week, but who went weeks or months at a time without being able to allow classroom instruction to enter their consciousness, much less their “to do” list) are so craven as to allow this situation to persist, and that principals could have confidence that qualified replacements for bad teachers would apply for jobs. The answer, it is assumed, is to dump far more work on those overwhelmed principals.The second assumption is that even though it is not hard to document bad performance and fire bad teachers, that collective bargaining agreements play a significant role in protecting those teachers. In the case of the problem documented by Xu, Ozek, Hansen, and Sass, one must assume that, when a termination approaches, that mild-mannered union officials (even in Right to Work states) step into a phone book, shed their moderate, collaborative demeanor, and emerge as supermen and save the jobs of the “lemons.”

The third assumption is that the federal government should treat teachers differently from any other employees, coerce states into repudiating good faith contracts, turn the evaluation process, which traditionally evaluates what employees do or don’t do, on its head and incorporate the opinions of a few billionaires into law. It assumes that the dismissal of bad teachers would not correlate with the dismissal of “ineffective” teachers. The corollary assumption is that stripping teachers of their rights is the way to make the profession more attractive to new talent.

Even if such bizarre assumptions proved to be true, I question whether such a policy, which is collective punishment of teachers who committed to schools where it is harder to meet test score growth targets, should be considered appropriate in a constitutional democracy. 

Another installment is forthcoming. Stand by for more analysis of the perils of VAM by John Thompson!