Last Friday, before the winter break, D.C. officials quietly released the news that the D.C. IMPACT evaluation system contained technical errors. It was the perfect time to reveal an embarrassing event, hoping no one would notice. Spokesmen minimized the importance of the errors, saying they affected “only” 44 teachers, one of whom was wrongfully terminated.

But Professor Audrey Amrein-Beardsley explains that what happened was “a major glitch,” not a “minor glitch.” It was not a one-time issue, but an integral part of a deeply flawed method of evaluating teachers. No amount of tinkering can overcome the fundamental flaws built into value-added measurement of teacher quality.

Beardsley writes:

VAM formulas are certainly “subject to error,” and they are subject to error always, across the board, for teachers in general as well as the 470 DC public school teachers with value-added scores based on student test scores. Put more accurately, just over 10% (n=470) of all DC teachers (n=4,000) were evaluated using their students’ test scores, which is even less than the 83% mentioned above. And for about 10% of these teachers (n=44), calculation errors were found.

This is not a “minor glitch” as written into a recent Huffington Post article covering the same story, which positions the teachers’ unions as almost irrational for “slamming the school system for the mistake and raising broader questions about the system.” It is a major glitch caused both by inappropriate “weightings” of teachers’ administrator’ and master educators’ observational scores, as well as “a small technical error” that directly impacted the teachers’ value-added calculations. It is a major glitch with major implications about which others, including not just those from the unions but many (e.g., 90%) from the research community, are concerned. It is a major glitch that does warrant additional cause about this AND all of the other statistical and other errors inherent not mentioned but prevalent in all value-added scores (e.g., the errors always found in large-scale standardized tests particularly given their non-equivalent scales, the errors caused by missing data, the errors caused by small class sizes, the errors caused by summer learning loss/gains, the errors caused by other teachers’ simultaneous and carry over effects, the errors caused by parental and peer effects [see also this recent post about these], etc.).

The “errors” cannot be corrected because the method itself is the problem. The errors and flaws are integral to the method. VAM is Junk Science, the use of numbers to intimidate the innumerate, the use of data to quantify the unmeasurable.