Audrey Amrein Beardsley is one of the leading experts in the nation in the field of value-added assessment and also one of the nation’s leading skeptics of the claim that teacher “effectiveness” can be measured by the test scores of their students.
Recently, a study was published by economists that purported to measure the effect of teachers’ on their students’ height. The study was a blatant lampoon of VAM (value-added modeling or measurement).
It turns out that I was one of about 25 people who promptly forwarded it to Amrein-Beardsley.
Beardsley reminds us of a paper written by economist Jesse Rothstein nearly a decade ago in which he lacerated VAM by showing that it could be twisted to show the effect of teachers on students’ past achievement, a feat that is clearly absurd.
When a policy idea like VAM becomes the target of satire, you know that it is well and truly dead. Now, if only someone would tell the state legislatures that.
Diane Ravitch's blog 
I’ve heard many great destructions of VAM, but I’m literally LOL (during student exams!) reading about teacher impact on past achievement. Too funny! Love it!
If only someone would tell Harvard prof Raj Chetty, former Harvard Prof John Friedman and Harvard dropout Bill Gates that.
Oh, wait, lots of people have but the above dishonestly cling to their support for VAM.
Meanwhile, the profs in the Hawvid stats department have remained utterly silent while their colleagues spread nonsense.
That’s the way they do it at Hawvid. It’s the way of a Hawvid don.
It’s like proving that a “perfect call” can exist.
An Argument for VAM
Teachers drive the student height
Impact students day and night
Really stretched both James and Shaq
Rationale for ranking stack
An Argument for DAM
The devalue added method
Devalues all the working class,
But Some DAM time the tested
Will stick it to the Billy Boys’ …behind.
And a DAM fine argument at that!
Fascinating.
I’ve taught sixth grade English for most of my career, in part because I don’t want to teach anyone taller than I am. (I’m about 5’10”, though I think I might be shrinking a bit. The ground seems a little closer these days.) This way it’s much easier for my students to . . . well, you know, “grow.”
It is hard to kill a zombie once and for all, but there must be a way to put VAM to rest.
VAM is just a symptom. The disease is the unwarranted belief that quality education can be quantified. Data driven instruction misses the very point of schooling.
Bingo Bangle Boingo. Give Rager a Kewpie Doll!
Agree w/DS – nailed it
A naive application is an understatement. Teachers have lost their careers over VAM scores.
Good point
The people behind VAM did huge damage and now are walking away Scott free.
People like Chetty and Friedman knew that what they were doing was highly questionable (at best) because they were shown as much (repeatedly) by people like Moshe Adler and Audrey Amrein-Beardsley.
But they were all too eager to ignore the warnings because they saw VAM as a way to advance their careers and make names for themselves. Chetty (aka VAManujan) even thought it would win him a (fake) Nobel prize. And he may get get one. The committee that awards the economics prize (which is not a real Nobel) is actually dumb enough to think VAM has merit.
After I criticized him on this blog, Raj Chetty called me. We politely disagreed. He told me that he expected to win a Nobel prize in economics for his VAM study.
He didn’t.
He may still get one.
Sometimes it takes years.
The committee that awards the Econ prize (which is a fraudulent Nobel prize) has awarded it to people who had been demonstrably wrong in their claims.
It’s really a joke and the scientists who get real Nobel prizes resent having to share the stage with a bunch of quacks.
Chetty acts like he’s Ramanujan reincarnated ,(hence the moniker VAManujan) but the truth is that he doesn’t understand very basic math.
His “famous” study was based on cherry picked data and utterly ridiculous extrapolation based on earnings for just 28 year olds ( as made clear by Moshe Adler in his devastating critique.)
Adler showed that Chetty’s findings disappeared for those over the age of 28.
Chetty found that the age 30 earnings differential was not significant, but nonetheless nonsensically claimed it was greater than that for age 28.
With this claim, Chetty demonstrated beyond ALL doubt that he does not understand statistical significance because if a result is not significant, one certainly can not claim it is greater than some other positive result. A result that is not statistically significant can not be claimed different from zero!
At least one person lost his life over VAM scores.
Tragic and true!
LMAO!!!
Every classroom teacher knows that there are important aspects of teaching and learning that cannot be quantified.
In formulating education policy, why do economists and statisticians have so much say? Schoolchildren are not commodities or data points. Neither are teachers.
I verily appreciate the scientific yet tongue in cheek comparisons of standard deviations between test scores and height, but like you wrote, it really should be just common sense that teachers and schools cannot be judged by student performance. When the NCLB was passed and signed, I was confused and concerned. I wondered how I was going to get my students to answer more multiple choice questions correctly. It wasn’t possible. There was no way to make test scores rise, no way at all. My world was taken over by nonsense. Nearly twenty years later, nonsense still rules.
It’s actually funny (in a tragic way) that many of the people who claim to be math experts don’t know the first thing about basic math.
The most basic thing is knowing when your results are nonsense and these people are so enamoured with their own equations and their own “genius” that they don’t know the difference between legitimate math and BS.
Doesn’t anyone know that the effectiveness of a teacher influences how much Madagascar vanilla and Indian cinnamon a student will consume before age 30, how likely the student will choose sweat pants over sweat shorts to work out in the gym, and to what extent a student will remain married or divorced?
In fact, why not put it out there that a teacher’s effectiveness correlates directly to how well a student handles medical debt after getting screwed by their insurance company while battling a grave illness or post-surgical procedure?
Teachers, NOW do you see what you’ve done?!!!
Teachers are EVERYTHING. They cause ALL the problems in this country. There are NO other factors. Politicians bought and paid for by the medical industrial complex (drug and medical equipment manufacturers and suppliers, some MDs., and insurance companies) are only trying to help us all. These politicians are free to focus on us, as they are relieved of the stress of having Lamborghini quality health insurance for the rest of their lives after they finish onut just one term.
Trust them. Get with the program.
I even know a prominent leader in education who blames the public school system for her brother’s fatal overdose instead of looking to the parents who raised him and a father who abandoned his family.
Excuses are easy. Blame the teachers. Not poverty . . .
“Blame the teachers. Not poverty . . .”
Just a slight correction: “Blame the teachers for poverty.” FIFY, you’re welcome!
For at least half a century teachers manage to avoid accountability for what they do, or often for what they do not do. They fight results-driven programs, in fact they detest anything that remotely looks like a solid curriculum, they think they know best, they cobble together their pitiful lessons only to find themselves teaching reading to six-grades. Obviously, they feel offended when someone suggests that teachers’ efforts can and should be analyzed and graded.If you cannot become an engineer, become a teacher. You are protected by the union and you don’t really need to follow a well-designed program, just come to class every day.
QuelleProf,
The teachers I know are devoted to their students and work tirelessly. They plan their lessons with care and spend hours grading every day, often in the evening and on the weekend. And when they’re not working, they’re thinking or worrying about their students.
You denigrate teachers, suggesting that teaching is a fall-back for those who would otherwise be an engineer. But I cannot think of a common job that is harder to do well than teaching.
Teachers do not “avoid accountability.” They are constantly being evaluated by their students and their parents, as well as by administrators.
The reason many teachers oppose “objective” measures of teaching effectiveness is that there are plenty of reasons to think them flawed. In particular, “objective” measures are almost always quantitative. But not everything of value can be reduced to a number.
You seem angry at teachers. Have you been, or are you, a teacher yourself? What experiences or observations of your own are you drawing on? I am trying to understand why you think as you do.
I usually don’t permit comments like this on my blog.
I did so that readers may respond, if they choose, to these unsubstantiated, groundless, insidious accusations against all teachers.
Why do people have to be so disrespectful of teachers? Why?
Those who can, teach.
Those who can’t teach become engineers.
Bethree
I worked in R&D with engineers for over a decade and from my experience , there actually is an “issue” with many engineers in that they tend to believe they know more than everyone else about subjects well outside engineering.( This “affliction” is particularly prevalent among computer engineers.)
This makes them believe, among other things that they know more about climate change based on a cursory look at the temperature data than the climate scientists who have studied it for their entire careers.
Visit the website Climate Audit and you will see what I am talking about. The host of the site is an engineer and so are many of the commenters and the gist of their (supposed) “analysis” is that climate change is either not human caused or not a serious problem.
Not all engineers are afflicted with the know it all disease, but many are and if I had to guess, I’d say that “QuelleProf” is some sort of engineer.
Incidentally, a computer engineer (Brian Davison — aka Virginia SGP) who used to comment here is a perfect example of an engineer with the know it all disease and used to bash teachers incessantly, despite knowing little if anything about education and teaching. Not coincidentally, he was also a did hard VAMbot.
SDP,
You can add our frequent (in the past) commenter Charles, who is an engineer, and considers himself an expert on everything to do with education, although he has never taught and has no children.
Yes, Charles the IT engineer has a chronic case of know it all disease.
There was also an engineer named “Raj” who used to comment here that also had the disease.
Another problem with engineers is that they believe that there is a quantitative “solution” to every issue, including human ones ,(whence the love of VAM.)
But some issues just don’t lend themselves to quantitative analysis and solution.
Well I don’t think we should make broad-brush characterizations of engineers any more than they should of us. The crowd I worked w/ [power generation/ transm/ distr] may be a tad more humble than the computer bunch, as they design & build IRL. They’re there to take the fall if risk calcs were off & something goes awry. And none of them that I know assume the learning of human children can be measured by the tools of engineering. I’m going to go out on a limb & challenge any that do: who was raising the kids in your household??
I’ve worked with many different types of engineers, including chemical, mechanical, nuclear electrical and computer.
I can only speak from my own work experience and from what I have seen elsewhere, but unfortunately, I’d have to say that the” know it all type” is definitely overrepresented among engineers.
It’s certainly more prevalent than it is among teachers, for example, with whom I have also worked and who generally seem to be a father humble lot.
But maybe my experience is anomalous.
And the important thing is not whether it is true in general but instead to call out the specific engineers (eg, who post here) when they make claims about teachers that are BS.
From my observation as the wife of an engineer & one who worked in an engrg firm for a decade: they got their start— encouragement of their love of learning and growing intellectually—from K12 teachers, and I’ve never heard one denigrate them. Many engineers over the course of their careers likewise mentor subordinates, drawing on those methods and values to the best of their ability. They cherish the role of teacher and admire those who can do it well. My husband envies the comparative job stability, health and retirement benefits that his career-teacher sister-in-law enjoys. But he recognizes that the erosion in his own field in that regard is a function of our govt system that is eroding public goods—and recognizes that those same factors have been greatly eroded for teachers coming up now.
OMG I love this study. If VAM ever comes to PreK specials like me, I’m going to insist I be graded on growth in height from age 2,5-5!
This “research” is a clear example of education establishment’s inability to produce anything of substance and meaning. One can use a car engine thermometer to measure human temperature and claim that because the “overheat” light does not come on, it is ok for the human body to have temperature of 110 degrees, in fact, the body is too cool and must be “warmed up” to boiling temperature. The point is, tools can be used only for what they are intended to be used. You don’t hammer in nails with a microscope.
This “study” has been intended to produce meaningless results, so garbage in – garbage out. Everyone knows that correlation does not mean causation. It is up to a researcher to judge the conditions and to decide whether the parameters of an experiment can plausibly affect each other. If yes, and if strong correlation is found, then the claim about causation can be made.
Strong correlation between students’ height and their teachers in no way discredits the VAM model. As computer engineers say, it is a “user error”.
By the way, the “research” notes that “while observational protocols which send observers to every teacher’s classroom require expensive training and considerable resources to implement at scale, VAMs use existing data and can be calculated centrally at low cost. Further, VAMs are arguably less biased than many other evaluation methods that districts might use instead.” Also, “it seems important to consider that value added measures which require multiple years of data to implement will likely permit identification of persistently bad teachers.”
VAMs have been proven to be inaccurate.
They have been tried and failed.
Bill Gates gave hundreds of millions to seven districts and charter chains to VAM their teachers, and the RAND=AIR report said that the result was nil: the scores did not identify the best or the worst teachers, the allegedly best teachers did not seek to teach the neediest students, teacher turnover was high. Nothing was gained. Zero. Nada.
A trial judge in Houston ruled that VAM was arbitrary and capricious since teachers could not see the secret formula by which they were rated to see if it contained errors.
A trial judge in New Mexico prohibited the application of VAM scores, and the newly elected Governor of New Mexico eliminated VAM.
It is a fraud and a waste of money.
Is your first name Raj, by chance?
VAMs (or SLOs) have now been used extensively since 2012 to evaluate teachers in almost every state. Still waiting for the headline regarding the success of VAMs (or SLOs) in identifying all those incompetent teachers you seem to think are out there. In most school districts in NYS, SLOs use shared or distributed Regents (HS) scores to evaluate all teachers. So 50% of teacher evaluations are based on the test scores of students and/or subjects that the majority of teachers do not teach. There is no fix to this problem because the majority of teachers do not teach “tested” subjects. Why don’t you get back to us when you have a clue.
Is it the fault of a tool, if the users of the tool use it incorrectly?
The tool is garbage.
Like trying to cook a meal with a hammer.
Like trying to drive a nail with a washrag.
Like trying to type using a pencil.
Did you ever read the 2014 statement of the American Statistical Association about the limited uses of VAM for individual teachers?
You should read it, because it says (five years ago!) that teachers contribute 1-14% of the variance in student test scores.
Click to access POL-ASAVAM-Statement.pdf
Since 2014, VAMS have been widely used and have found to be totally useless for the purpose of identifying the “best” and the “worst” teachers.
It is the fault of the tool because it can’t be used correctly.
@QuelleProf:
Why do you say that the research in question is meaningless, when the authors show that teachers have about as much effect on their students’ height as they do on their standardized test scores?
You seem so angry at teachers and the “education establishment” (whatever that is). Most people I know are sympathetic to teachers and are grateful for the work they do. (But maybe that is a reflection of my social circle: mostly teachers, lawyers, and academics.)
When you were a child, did you have a teacher who was unkind to you or somehow let you down? I am trying to understand why you feel the way you do. But perhaps you do not wish to say.
You can’t argue with people like Quelle Prof because they don’t know anything about the subject they are weighing in on (in this case VAM) and, worse still, don’t want to know.
That is perfectly clear from what Quelle Prof has said above.
It’s actually a mistake to engage with such people with the assumption that they are sincerely interested in learning because they are not.
They are a waste of time.