Value-added-measurement (VAM) produce ratings that are
inaccurate and unstable. In Florida, about half of teachers don’t
teach tested subjects, so they are assigned scores based on the
scores of their school, meaning they are rated in relation to the
scores of students they never taught and subjects they never
taught. This
Florida teacher explains why she was rated a 23.6583 out of 40,
even though she teaches a non-tested subject.
This is
irrational. Yet Arne Duncan has compelled almost every state to
develop VAM ratings because he believes in them, even though there
is no evidence for their value. How can a teacher’s quality be
judged by the test scores of students she never taught? If that is
not Junk Science, what is? Bill Gates gave Hillsborough County,
Florida, $100 million to evaluate teachers using
value-added-measurement. Here is the formula: Here’s how
the state’s Department of Education explains it, from
a department paper
methods2 I admit I don’t understand it. Many
people don’t understand it. But whoever wrote it understands it.
Bill Gates said recently it would take at least ten years to see if
this stuff “works.” I don’t think we have to wait ten years. “This
stuff” doesn’t work. It doesn’t even make sense. Teachers of the
gifted may be rated ineffective because their students have already
hit the top, and their scores can’t go up any higher. Their ratings
are Junk Science. When the same teacher gets a bonus one year, but
then is rated ineffective the next year, it shows how unstable the
ratings are. That means they are not science, they are Junk
Science. There is so much more to the art and craft of teaching
than standardized tests reveal. What matters most is not
quantifiable although peers and supervisors can indeed judge which
teachers are best and worst. If the measure is not valid, if the
measure in inaccurate and unstable, then it is wrong to use it to
give people bonuses or to fire them. In this post on her
VAMboozled, Audrey Amrein-Beardsley reviewed a study
of VAM which again identified the weaknesses of VAM. She writes:
Finally, these researchers conclude that, “even in the
best scenarios and under the simplistic and idealized
conditions…the potential for misclassifying above average teachers
as below average or for misidentifying the ‘worst’ or ‘best’
teachers remains nontrivial.” Accordingly, misclassification rates
can range “from at least seven to more than 60 percent” depending
on the statistical controls and estimators used and the moderately
to highly non-random student sorting practices and scenarios across
Now, think about it. If the VAM rating can be
wrong by as much as 60%, why would any school district use it to
fire teachers? No wonder teachers are suing for wrongful
termination! Call in the lawyers, VAM is Junk Science.