Our wonderful reader Laura Chapman reports here on the origins of the laws that purport to measure teacher quality by the test scores of their students. The founding father of this methodology was the late William Sanders, an agricultural statistician who believed that the same productivity used to measure cows could be used to measure teachers. His ideas were adopted and promoted by Arne Duncan’s Race to the Top, which required states to adopt “value-added methodology” if they wanted to compete for a share of billions of federal dollars. The Gates Foundation also embraced test-based accountability. These methods proved to be ineffective at measuring teacher quality; they are inaccurate and demoralizing.
According to a 2019 report coauthored by Audrey Amrein-Beardsley, 15 states are still inflicting teacher evaluations by VAM (value added measures) and 28 are using the equally invalid process of writing up Student Learning Objectives (SLOs). SLOs require you to predict the end-of-year (or end of unit) achievements of students, among other ridicule-worthy feats. https://kappanonline.org/mapping-teacher-evaluation-plans-essa-close-amrein-beardsley-collins/
Vamboozled, the website of Audrey Amrein-Beardsley, is a great resource for anyone still being a victim of this method of estimating the “value you have added” to the test scores of your students.
But there is also a deeper and little known origin story for VAMs. That story was exposed to view in April, 2020, by Gene V. Glass, a Senior Researcher at the National Education Policy Center and a Regents’ Professor Emeritus from Arizona State University. Glass released a treasure trove of correspondence about VAM (value added measures), first used in education by the late William Sanders, an agricultural statistician. http://ed2worlds.blogspot.com/2020/04/an-archaeological-dig-for-vam.html
In his blog post “Archeological Dig for VAM,” Glass reveals how William Sanders borrowed statistical methods for calculating VAM, then began using those calculations to judge teacher productivity/quality, based on the test scores of their students, specifically in the Tennessee Value-Added Assessment System (TVAAS).
It turns out that Sanders’ TVAAS process (VAM) was “built on the formulation of the late C.R. Henderson, a Cornell statistician, a fellow in the American Statistical Association, known for his pioneering work in breeding animals, specifically herds of dairy cows. Henderson’s statistical methods of producing a “genetic evaluation of livestock have been widely accepted, utilized, and enhanced by animal breeders and statisticians.”
Until Henderson’s 1953 publication of “Estimation of variance and covariance components” in Biometrics,” no one had tackled the difficult problem of “estimating variance components from unbalanced data of cross-classified models, e.g., of milk production records of daughters of A.I. (Artificially Inseminated) sires in different herds – where sires are crossed with herds, and, for a large group of herds, each sire has daughters in many herds and each herd has daughters of many sires.” https://ecommons.cornell.edu/bitstream/handle/1813/31657/BU-1085-MA.pdf;sequence=1
If you have a background in statistics (mine is minimal and vintage), you may enjoy reading the extended “defense” of VAM/TVAAS by the late William Sanders who cites his debt to Henderson’s work. Sanders’ defense of using VAM with teachers and the test scores of their students is revealed in his answers to numerous questions from William Robert Saffold, Vanderbilt Institute for Public Policy Studies, who is well-informed about the results in TVASS in Tennessee and wanted more information to interpret the results of TVAAS for educators. The extended discussion reveals the many unwarranted assumptions Sanders made in constructing TVAAS.
I think the hoopla over the specifics of VAM (and SLO’s) is too often disconnected from the fact-based origin story on “how to cull herds of dairy cows to maximize their productivity.” VAMs and SLOs are designed to cull teachers based on their productivity in raising the test scores of their students.
Almost all of the accountability structures in education based on standardized test scores are designed to cull–select and discard–teachers who are not producing gains in test scores. In VAMs, test scores of students are not much different from measures of milk production, whether of individual teachers or the whole herd (school).
Some supporters of VAM’s are acting as if education geneticists. They seem to think that some teachers are destined to be more productive than others. They insist, for example, that Teach For America graduates with high GPA’s from selective colleges are good breeders of above average test scores in their students. Moreover, these potentially good breeders only need is a brief course in summer before they are ready to produce students who are high scorers on tests. That brief summer dose of instruction is analogous to providing artificial insemination in breeding females… or for males, perhaps a dose of Viagra.
VAMs and SLOs are flawed measured pushed by the Obama/Duncan administration’s Race to the Top. These measures are still present in many state ESSA plans. That may explain why Race to the Top testing resources are still available, even if developed under contract for Race to The Top by members of a “Reform Support Network.”
The Reform Support Network was nothing more than a huge marketing campaign for these flawed measures. Here, for example, is how they marketed SLOs as a substitute for subjects and grade levels for which there were no statewide standardized test scores for calculating VAM. One is the infamous collective measure where, as Diane notes, teachers “are assigned ratings for students they never taught in subjects they never taught.” https://www.engageny.org/sites/default/files/resource/attachments/rsn-slo-toolkit.pdf
I am not surprised that teachers are being treated and rated like cows by the billionaire class. Always Lying and Cheating TrumpEekThinlySkin revealed how these toxic ET’s think when he was talking about his method of waging war against COVID-10 and said “Herd Mentality.” He was supposed to say “Herd Immunity”.
To most billionaires and very wealthy multi-millionaires that think just like DT, the working class is a herd to control, manipulate, buy, and sell for profit and power.
The world would be a better place if we loaded DT and his think alikes into space ships and shot them to the sun with no return ticket.
Good morning Diane and everyone,
One of my favorite Carl Jung quotes:
“That we think in terms of statistical numbers is the contemporary prejudice, our particular insanity; we think how a certain individual problem could be solved as a sort of mass production, as if it were manufactured in a factory.”
And that was 90 years ago!
“Instead of using tools for measurement imposed in a commanding or consequential way, states have moved toward teacher evaluation systems that are informative and also supportive of teachers’ professional growth.”
Many states are moving away from high stakes testing, VAM and SLOs because the concept has been discredited. They assume that the teacher is the only reason students perform a certain way on a standardized test. Statisticians maintain that a teacher’s impact on a student’s test performance is somewhere between 1 and 14%, a far too low a number to be considered significant. Formative assessments, based on what people actually do, are far more meaningful and accurate for both students and teachers.
Many states and districts are quietly using the pandemic to wedge more testing into our curricula. I just had three more multiple choice tests by Renaissance Learning foisted on my students. I’ll be wasting my students’ time tomorrow with a test. The pols are claiming they need to keep track of the so-called learning loss so-called achievement gap. Disaster capitalism, when capitalism is a disaster on top of a disaster.
“Statisticians maintain that a teacher’s impact on a student’s test performance is somewhere between 1 and 14%, a far too low a number to be considered significant”
Because there is so much variance associated with VAM, it can have a highly disproportionate impact on teacher evaluation scores, even in cases where the VAM score contributes a relatively small percentage to the overall evaluation score.
That is because there is much less variance in overall teacher evaluation scores from other contributions like principal evaluations. Because most of the variance comes from VAM (because there are so many factors outside teacher control that affect the student test scores), the VAM score has highly disproportionate weight on the overall evaluation.
Cathy O’neil explains this on her blog Mathbabe
Reblogged this on David R. Taylor-Thoughts on Education.
Thou shalt not worship false teacher ratings.
All people are created equal. All people are not, however, created identical. Predicating formulas on the idea that the rise and fall of inaccurate, standardized “measures” of learning are due to teacher quality assumes that multitudinous differences in student neural traits and socioeconomic environmental variations are inexistent, or can be boiled down to simple digits in a formula. It’s junk science. The edu-profiteering industry will demand proof that their data are flawed. The proof is contained in four words: people, are, not, cows.
It is time to remove from government those who disrespect teachers and waste our resources to attack us. Not only do testing and tech industry sycophants not respect teachers; they don’t respect humanity. They worship money. They worship a cow breeding formula.
They worship the formula of the golden calf.
It is junk science. It is also another way to monetize students by selling their data without parents or students’ knowledge.
and that is the danger point: parents having no idea how much big data/big money now run school policy
Yes, and there was also the 12th commandment
Thou shalt not take the teacher’s job in VAM
🙂
Isaac Newton: If I have seen farther than others , it was by standing on the shoulders of giants
William Sanders: If I have seen farther than utters, it was by standing on the shoulders of cattle.
Farther than udders
Damn her self correct.
SDP, you are delightful.
I have a beef with VAM bull salespeople steering teachers right in the face while milking us dry, but I am not herd because VAM is a cash cow.
It’s called “herd immunity”
I guess that’s spelled “heard immunity”
The Word Herd
I’m a teacher
In a word
Not a creature
In a herd
Herd Immunity
I heard there was immunity
To ever getting covid
You simply gain impunity
By acting like a bovid
Udderly Ridiculous
Cattle word was uddered
Cattle word was heard
William Sanders muttered
“Teachers are a herd”
I definitely feel like I am affected by being heard immunity. Our puns are a little cheesy, but cheese is a natural part of the discussion when education policy mixes with, of all things, dairy policy.
Seriously, I thought the Agriculture Department was separate from the Education Department. I hope they don’t also start using pesticides on teachers.
Round Up
Round ’em up and fire ’em
VAM ’em till they’re gone
Don’t retread, retire ’em!
Simply say “so long”
Teacher Winnowing
Winnowing the wheat
To filter out the chaff
Is what we should repeat
To separate the staff
Isn’t agriculture fun?
Maybe TE has some more brilliant agricultural ideas. He’s an eageronomist, right?
Genetically Modified Teachers (GMTs)
To get the greatest teachers
You start with Crick and Watson
You modify the features:
Ferrari from a Datsun
Pest-aside
Let’s not be vague
A locust plague
Requires a pesticide
And so the teacher
Is also creature
That VAM will set aside
LOL
Here was my tribute to the stable genius of William Sanders (and others as well).
“The Maestro”
Chetty picked his VAMdolin
At Nobel-chasing speed
Duncan played the basket rim
And Rhee, she played the rheed
Coleman played his Core-o-net
Moskowitz, the lyre
Billy Gates played tête-à-tête
With Duncan and with higher
Sanders beat his cattle drum
Devalue Added Model
Pseudo-science weighted sum
Mathturbated twaddle
John King played the slide VAMbone
But Maestro was Obama
Who hired the band and set the tone
For current Grizzly drama
Make that “tribute to udders as well”
And in case anyone does not know what a stable genius is
How many states still require VAM? A quick search through EdWeek shows more and more states that have dropped it in the past four years.
I recall the first presentation I saw on VAM by the researchers from AIR.
Their best explanations (not rationale, just explanation) was baseball! Cybermetrics! Moneyball!
Take all the players in one year who had the same batting average (or any of the dozens of new statistics). The following year(s), cite that year’s batting average for each player. For those whose average went up – value added. For those who went down – no value added. They all started with the same prior-year average – but some went up – others donw. Then ask, what was different about the training of and regimens of those whose averages went up or down.
Ditto for kids.
Take all the kids who score X on a state test. Track them the following year – which ones improved, which ones dropped. And, do that math for every score and all the students with that score. However – with ZERO research or taking into account dozens of factors which contribute to learning, the reform boys saw this quantifying a good thing and developed policy to slap a label on the teachers whose kids’ scores went up and down.
In the extreme, sure. If a group of 50 kids all scored the same on a test that actually meant something got split into two groups the following year – and one group’s scores skyrocketed and kids in the other all dropped, well, maybe the teachers had an effect.
But… we already know that without all these tests or algorithms. And,, that logic and math do not extrapolate to thousands of teachers in a state and each of their hundreds students.
As for the cows and testing – remember – as the phrase goes – weighing the pigs more often does not make them fatter.
Bovid Model (BM)
Udderly impressive
On the cattle hung
Udderly defecative
Naught but cattle dung
Hanushek Addled Model (HAM)
The wings of boars
Are to the ham
What student scores
Are to the VAM
https://www.researchgate.net/publication/340608657_Putting_teacher_evaluation_systems_on_the_map_An_overview_of_states%27_teacher_evaluation_systems_post-Every_Student_Succeeds_Act
The preceding is a link to a scholarly study of the current use of VAMs, SLOs and other teacher evaluation schemes, dated April, 2020.
ESSA gave states some wiggle room but VAMs are still used in fifteen states and in some additional states where they are used not for high stakes judgments but for designing/assigning “professional development.” I think that use merely restates that test scores must shape how and what you teach.
This article also has information on lawsuits that have been filed on behalf of teachers who have been victims of high stake uses of VAMs, the uses of Marzano and Danielson’s observation schemes, and more.
Audrey Amrein-Beardsley, the expert on VAMs and other teacher evaluation schemes, is one of the authors of this downloadable article.
And thanks for all the witty spins on the initial post.
Speaking of udderly ridiculous, what ever happened to Virginia SGP?
He sure uddered a lot.
Uddering sweet nothings
Sweet nothings did he udder
Virginia SGP
Shear nonsense, like no other
For everyone to see
Virginia gave up. He went away.
He accomplished his goal, made his neighbors upset by the disruption he caused, and stopped. If you’re going to ruin public schools, it helps to have a billionaire backer who can sweep you up out of the mess you make and plop you somewhere new. Virginia did not.
Could this be what shut him up?
https://loudounnow.com/2017/09/01/supreme-court-rules-against-loudoun-parent-seeking-teachers-names-student-scores/
You know what they say about people who act as their own lawyer.
Looks like another stable genius just made an appearance
This does suggest a second question: can students be measured by the same methods used for cows?
Height. You can measure height and weight. You can count fingers and teeth, too, if that’s your fancy. You can’t measure learning. All summations of learning are subjective. If you want to use a cow formula to cull the student herd, you can base it on measurements of the circumferences of students’ pinky fingers because that’s quantifiable, but not on measurements of learning. Measuring learning is like measuring the number of angels on the head of a pin.
We do “cull the student herd”, as you so colorfully put it, using teacher assigned grades. They decide who gets to participate in extracurricular activities in high school, who graduates from high school, who is admitted to public universities and who is not, and which universities they attend.
Do you assign grades based on the circumferences of students’ pinky fingers?
Of Cowpies and Econopies
The measure of econopie
Could rank economist
Like measure of a cowpie
To rank which cow is best
And by that, TE would rank right up at the top
The word ain’t “economist” for nothing.
English is a Germanic language and “mist” is the German word for manure.
There is No No-bull Prize in Economics*
He’ll never get a “No-bull Prize”
Of that much, I am sure
Cuz No-bull Prize, in fact, denies
Economist manure
*There is no Nobel Prize in economics either
https://fivethirtyeight.com/features/the-economics-nobel-isnt-really-a-nobel/
There is no Nobel Prize in Economics
https://www.alternet.org/2012/10/there-no-nobel-prize-economics/
Brilliant, Laura!