As is well known, the U. S. Department of Education zealously believes–like Michelle Rhee–that low test scores are caused by “bad” teachers. The way to find these ineffective teachers, the theory goes, is to see whose students get higher scores and whose don’t. That’s known as value-added measurement (VAM), and the DOE used Race to the Top to persuade or bribe most states to use it to discover who should be terminated.
As we also know, things have not worked out too well, as some Teachers of the Year were fired; some got a bonus one year, then got fired the next year. In many states, teachers are rated by the scores of students they never taught. The overall effect of VAM has been demoralization, even among those with high scores because they know the ratings are arbitrary.
For some reason, teachers don’t like to “win ” at the expense of their colleagues and they can spot a phony deal a mile away.
But the U.S. DOE won’t give up, so they released a research brief attempting to show that VAM does work!
But Audrey Amrein Beardsley deconstructs the brief and shows that it is a mix of ho-hum, old-hat and wrong-headed assumptions.
It’s true (but not new) that disadvantaged students have less access to the best teachers (e.g., NBCT, advanced degrees, expertise in content areas (although as Beardsley says, the brief doesn’t suggest such things matter).
It is true, that “Students’ access to effective teaching varies across districts. There is indeed a lot of variation in terms of teacher quality across districts, thanks largely to local (and historical) educational policies (e.g., district and school zoning, charter and magnet schools, open enrollment, vouchers and other choice policies promoting public school privatization), all of which continue to perpetuate these problems.”
She writes:
“What is most relevant here, though, and in particular for readers of this blog, is that the authors of this brief used misinformed approaches when writing this brief and advancing their findings. That is, they used VAMs to examine the extent to which disadvantaged students receive “less effective teaching” by defining “less effective teaching” using only VAM estimates as the indicators of effectiveness, and as relatively compared to other teachers across the schools and districts in which they found that such grave disparities exist. All the while, not once did they mention how these disparities very likely biased the relative estimates on which they based their main findings.
Most importantly, they blindly agreed to a largely unchecked and largely false assumption that the teachers caused the relatively low growth in scores rather than the low growth being caused by the bias inherent in the VAMs being used to estimate the relative levels of “effective teaching” across teachers. This is the bias that across VAMs is still, it seems weekly, becoming more apparent and of increasing concern.”
VAM in the real world is Junque Science.
Reblogged this on Rickarcher1959's Blog and commented:
It has always been a flawed formula!
One of the things I wish you had asked in the Bill Moyer interview is “how is it that extracting profits (management fees, etc.) from a non-profit enterprise like public education was supposed to make things better.”
Stephen, I keep thinking of the things I wish I had said.
dianeravitch: At one sitting, there will never be enough time to say everything that could and should be said. But you made excellent use of the time you were allotted.
Perhaps you were paying heed to Benjamin Franklin:
“Remember not only to say the right thing in the right place, but far more difficult still, to leave unsaid the wrong thing at the tempting moment.”
You made every word count. Thank you.
😎
This entire conversation about teacher evaluation is the second step in the war on teachers that is at the very base of the collapse of public education.
It will never end because no one speaks as or FOR the teachers who know what it takes to teach, or what learning looks like in the classrooms….their professional practices!
You know, Diane, that I was THERE, when the first step was in play, and across the country the veteran teachers, —despite credential or evidence of excellence, were thrown out of the schools and the conversation.
It is soooo simple: once the voice of the professional in the practice is GONE, then the profiteers who fill the schools with bogus learning materials and curricula GET TO CONTROL the national narrative. How else can Rhee or Klein, and DUNCAN be able to add anything to a genuine conversation about the way the human brain acquires skills and knowledge?
Anyone who knows who I was — that beyond a shadow of a doubt I met every criteria for excellence and was a celebrated educator at top academic levels and among the parents and children in NYC– knows that I was there when the first step in the assault was in full swing. My story, and that of David Pakter, Lorna Stremcha, Karen Horwitz, Dania Hall, Walter Porr, Pi Lian Tu, Ren Diedrich, Lenny Isenberg, Francesco Portelos… and tens of thousands of other professional practitioners of great quality, is THE story and THE REASON that the snake-oil charlatans can chatter about VAM instead of effective strategies like small class size, or genuine curricula practiced at the discretion of, and with the expertise by authentic professionals.
A totally controlled media and a press that lambasts teachers and teaching does the rest.
Your blog, is sensational, but you, despite your fame and access to Moyers and Stewart is preaching to the choir. That does not mean that the conversation here is not exceptionally valuable… it is just that it has yet to become THE national conversation. Maybe it will, someday, resonate with some journalists and get front page access. I hope so, because the truth that you speak here is crucial to change.
Here in greater Cincinnati there are two predictable rounds of news about education. The test prep extravaganzas and the league tables that rank schools by their test scores. Unions have been silenced, and the school board meetings are stuffed with agenda items that come from the state and feds. Charters quietly proliferate, aided by a local foundation with a CEO who came from doing Gates-funded work in Charlotte Mecklenberg. The former CEO was a banker with a big heart but no clue about how to invest money in education, or the foundation’s own resources, that shrank when the economy tanked. I also acutely aware that the higher education community seems to have little voice, or energy, or concern about the corporate takeover of public education, including new accountability and incentive bills for higher education that erode traditions of academic freedom and support for basic research. Not with a bang but a whimper.
If a research brief similar to this was completed in the field of Science or Medicine the authors would be laughed out of the profession!
How can these charlatans get away with this in the field of Education?
The US DOE should be ashamed of themselves!
This is called JUNK SCIENCE.
A thought experiment showing one of the flaws of using VAM for individual teacher evaluations.
Take the 50 best teachers in America and put them in the same school. Now add 1,000 smart, engaged kids from families in a wealthy American suburb that fully support their educational efforts, so we end up with a low class size of 20. Let’s further stipulate that this is a high school and that due to AP and honors classes, the average GPA in the school is 5.5 Long story short, a high performing school built with great students and teachers. At this point it is very important to note that this experiment also holds true for the 50 best teachers placed into a school where the kids have just as much potential as the first school but are suffering from the effects of poverty. Poverty is a diagnosis and as such cannot be ignored. We should all know by now it can never be an excuse.
Now, let’s use VAM to evaluate the contributions each teacher made to each students growth, how much each kid learned over the course of the year in this fantastic school. VAM grades teachers on a curve. No matter how well the kids and the teachers performed, being human, they are not all the same and hence there is a small, in fact in this case a tiny performance curve. VAM dictates that there be a cut point on that curve, a point below which a teacher is given an unsatisfactory rating that can lead to being fired. The actual quality of instruction is not considered or even seen by VAM, as it only makes a relative judgment of teacher effectiveness within an individual school, using the same group of students to compare and rank the teachers that instructed them, all based solely on test scores.
So, in this fantastic school, if the cut point is at 50% on the curve, then 1/2 of the 50 best teachers in the country are considered failures and could be fired, to be replaced by teachers who can’t be as good as the fired ones since at the outset, we stipulated that the school does in fact have the 50 best teachers in the nation working there. This is but one scenario illustrating the harm VAM can cause, removing highly effective teachers from the class room by declaring them to be failures.
Lets consider an additional absurdity, two teachers in this same school who straddle the cut point. Let’s further stipulate that their scores differ by less than 1/100th of a percent, in other words, they are virtually identical in their ratings. In spite of this, the teacher on the wrong side of the cut point is at risk of being fired
As previously noted,very similar effects occur for the 50 best teachers in schools populated by students who are low performing due to the effects of poverty, those who can least withstand the instability and disruption of constant teacher turnover inflicted by a flawed evaluation system.
This thought experiment deliberately ignores the use of observation systems as a crutch for VAM in multiple measures schemes. VAM has no capability to explain the why and how of student growth, it can only attempt to quantify it. Observation systems like PAR focus on both evaluating teacher performance and providing professional development & guidance for all levels of teachers. It leads to the removal by their peers of those who fail to improve. Good teachers do not wish to share their profession with poor performers. When student learning remains stagnant, we must ask what the real cause of that stagnation is, and the well validated answer is the effects of poverty. Basing teacher evaluations in large part on VAM diverts us from correctly asking let alone answering this question as it keeps the focus on teachers as the explanation of all problems within our schools. This in spite of the fact that VAM has as one of it’s core components an adjustment of sorts for students socio-economic status, AKA poverty.
Paul Krugman’s column today defines Zombie Ideas as ideas “…that should have been killed by evidence, but refuse to die.” VAM is a classic example of a Zombie idea… one of many that plague public education dialogue
I got my final “observation” results today that count towards a VAM score. It’s really sad when there seems to be no correlation to what you actually do and the observation score.. I just won my school district’s Innovative Educator Award for use of technology to promote student learning. I had paid on my own to have a programmer to make a program and a hardware person to make an interface to facilitate intellectually disbled students in using my state’s access points in my subject area. The evaluation tool requires teacher to supply evidence of their work. I had a great portfolio ready but was only verbally interviewed and the only question asked about how I use technology was what to list the websites I use. I was not allowed to present my portfolio or even mention other things I do for my students. What’s more, there doesn’t seem to be any criteria for what innovative is. Obviously what I’m doing isn’t valued or even recognized. It’s demeaning, demoralizing, devistating, and discouraged. After spending my own money and hours for development, why do above and beyone?
Only for the students’ quality of life. : ) Believe me not many people would bother.
deb: you have nailed a feature, not a bug, of VAMania—
What you’re actually “doing isn’t valued” or “even recognized.” Rather, VAM is meant to demean, demoralize, devastate and discourage.
How can I be so sure of this? Remember a common core objective of VAM as used on the gritty surface of Planet Reality and not in the hallucinatory mists of RheeWorld—is to sort, label and rank then punish [often] and reward [rarely] the frontline workers, in public education first and foremost teachers, but not management and owners [e.g., in education of charters].
Take Mr. Education Himself, Bill Gates [please! a la Henny Youngman] and consider an astounding exemption and assumption: that noxious mismanagement technique called stack ranking [aka forced ranking/rank-and-yank/burn-and-churn] did not apply to—drum roll please—
Bill Gates Himself and his equals or near equals in the Land of Microsoft.
Just how devastating was it that the actual “best and brightest” were judged quantitatively by the “dumb and dumbest” only because the latter were in superior positions of power and authority?
[start quote]
Eichenwald’s conversations reveal that a management system known as “stack ranking”—a program that forces every unit to declare a certain percentage of employees as top performers, good performers, average, and poor—effectively crippled Microsoft’s ability to innovate. “Every current and former Microsoft employee I interviewed—every one—cited stack ranking as the most destructive process inside of Microsoft, something that drove out untold numbers of employees,” Eichenwald writes. “If you were on a team of 10 people, you walked in the first day knowing that, no matter how good everyone was, 2 people were going to get a great review, 7 were going to get mediocre reviews, and 1 was going to get a terrible review,” says a former software developer. “It leads to employees focusing on competing with each other rather than competing with other companies.”
When Eichenwald asks Brian Cody, a former Microsoft engineer, whether a review of him was ever based on the quality of his work, Cody says, “It was always much less about how I could become a better engineer and much more about my need to improve my visibility among other managers.” Ed McCahill, who worked at Microsoft as a marketing manager for 16 years, says, “You look at the Windows Phone and you can’t help but wonder, How did Microsoft squander the lead they had with the Windows CE devices? They had a great lead, they were years ahead. And they completely blew it. And they completely blew it because of the bureaucracy.”
[end quote]
Link: http://www.vanityfair.com/online/daily/2012/07/microsoft-downfall-emails-steve-ballmer
And remember that when people suggest that their self-styled “education reformers” and their educract enablers subject themselves to the same data-driven “rigor” they ALWAYS avoid applying their own standards to themselves.
Just like they ensure for THEIR OWN CHILDREN an education far superior and vastly more enriched than what they mandate for OTHER PEOPLE’S CHILDREN.
As Dee Dee put it so well in her rhetorical question, “Have they no shame?????”
None. Absolutely none at all.
Thank you for your comments.
😎