Some weeks back, the media reported that the District of Columbia’s infamous teacher evaluation program–known as IMPACT–was successful, based on a paper by researchers Thomas Dee and James Wyckoff. The takeaway allegedly was that VAM (value-added measurement) works and that DC is right to judge teacher quality by student test scores.
But Audrey Amrein-Beardsley, one of the pre-eminent national experts on VAM, says “not so fast. Don’t believe the hype.”
In this post, she dissects the DC research and says it was never peer-reviewed and is deeply flawed.
She identifies many errors, but this one is the most egregious:
“Teacher Performance:” Probably the largest fatal flaw, or the study’s most major limitation was that only 17% of the teachers included in this study (i.e., teachers of reading and mathematics in grades 4 through 8) were actually evaluated under the IMPACT system for their “teacher performance,” or for that which they contributed to the system’s most valued indicator: student achievement. Rather, 83% of the teachers did not have student test scores available to determine if they were indeed effective (or not) using individual value-added scores. It is implied throughout the paper, as well as the media reports covering this study post release, that “teacher performance” was what was investigated when in fact for four out of five DC teachers their “performance” was evaluated only as per what they were observed doing or self-reported doing all the while. These teachers were evaluated on their “performance” using almost exclusively (except for the 5% school-level value-added indicator) the same subjective measures integral to many traditional evaluation systems as well as student achievement/growth on teacher-developed and administrator-approvedclassroom-based tests, instead.
Thus, it is wrong to say that the paper vindicates IMPACT or its reliance on VAM when more than four of every five teachers in the study did not have value-added scores available.
“Unrestrained Bias”
I have not read a “reform” base report that included all of the data, analysis and tests that where used in the study. All of the “reports” where written in a form that would highlight the “reform” even though the results were insignificant and worse yet they would “poo-poo” any results that would contradict their desired results, use tactics that were statistically incorrect or only mention harmful side effects of the implementation of the “reform”!
The reports being issued have “Unrestrained Bias” from the author writing the Title to the conclusions. This is what the “reformers” push to the Newspapers and the Politicians, who never bother looking at the detail of the report!
Even if VAM models could be used on a higher percentage of teachers, they are so statistically “muddy” and volatile that they cannot be used for any substantive evaluation of teacher performance. As Diane has said before VAM is a sham.
I took a satirical look at the VAM in this post: http://russonreading.blogspot.com/2013/10/the-vam-moose-coming-to-school-near-you.html
How’s VAM going at the private schools? I’m sure parents paying 30K would want to know how their teachers rank, so they could pick the best and avoid the worst. Oh yeah, the don’t do it because it would destroy the school culture. Which is why it is a bad idea that should have gone from the desk to the wastebasket. Period.
Thank you for sharing this critique of a seriously flawed study and its false claims. Teacher evaluations should be peer-evaluations, the same as in other professions, not administration evaluations and especially not test-based evaluations.
The VAM song:
Lovely Lovely Vam, wonderful Va-a-m,
Lovely Vam, wonderful V Vpam,
Va-a-a-a-a-a-a-am,
Va-a-a-a-a-a-a-am,
VA-A-A-A-A-A-A-AM,
SPA-A-A-A-A-A-A-AM,
LOVELY VAM, LOVELY VAM,
LOVELY VAM, LOVELY VAM,
LOVELY VA-A-A-A-AM…
VA-AM, VA-AM, VA-AM, VA-A-A-AM!
Feel free to sing along! You can even substitute SCAM for VAM!!!
All apologies to Monty Python!
That one’s for you Bernie!
For all, don’t forget your Spam tomorrow for the TDay Dinner!
Happy Thanksgiving to all who post and read, but especially to Diane!