Audrey Amrein Beardsley writes here about Houston’s experience with value-added evaluation of its teachers.
The Houston Independent School District (HISD) contracted with William Sanders’ SAS to provide a model to calculate the “value-added” of its teachers from 2007-2017.
Teachers objected that the method of calculating their scores was opaque. They couldn’t learn how to improve their practice because Sanders’ methodology was proprietary and secret.
Teachers were fired based on their VAM scores.
The Houston Federation of Teachers sued to stop the use of the “black box” method.
In 2017, a judge agreed and enjoined the use of VAM.
Thus by now, after a decade of VAMMING teachers, Houston should have identified and removed all the “bad” teachers and employ only “effective” or “highly effective” teachers.
But the state threatened to take over the entire district because one high school–Wheatley– has low test scores. Wheatley High School has a disproportionately large share of students who are poor and have special needs, has low scores, even though all of its teachers–like all of Houston’s teachers–were VAMMED for a decade.
If VAM were effective, HISD should be the best urban district in the nation.
All achievement gaps should have closed by now.
Why is the state–which has no expertise in running a large urban district–taking control away from the elected board?
Diane Ravitch's blog 
Because they would rather blame teachers or school board members, anyone but the real cause of low achievement: poverty, disrupted lives, and the failure of government to ensure equitable education, jobs, housing, and healthcare.
Lots of students were poor during the Great Depression, but still learned. Poverty, etc. can’t always be blamed for “everything”. 😐
Eddie who is responsible? When I was growing up I was an excellent student. Then, when my mother died and my father got remarried to an abusive alcoholic, I started struggling in school. Even after I ended up in foster care and I had some decent foster parents I struggled after having been moved around to so many schools/homes etc. When I graduated I was at the lower end of my class.
After going back to school at 28 and getting lots of help in remedial classes at community college I ended up getting into UC Berkeley and graduated with honors in my major and now I also have my Masters. I was very lucky to have a supportive boss who helped me by adjusting my work schedule to work around my school hours.
Along the way I had several wonderful teachers in my various high schools trying to help me. I had some great professors. I also had some that treated me like an annoyance when I fell asleep at my desk or I seemed lost in terms of our curriculum. As a teacher, although I tried to be very sensitive to my students’ needs, there were times where I too grew frustrated with the amount of students in my classes who were not making an effort and probably much had to do with childhood trauma. I also worked for a short time in a district with a lot of privileged kids and I’ll be honest… They were also extremely hard to teach but it was because of the entitlement and also some of them being spoiled. The difference between them and my traumatized kids from my former low income area school was that their parents pushed all obstacles out of the way to make sure that their offspring were taken care of. That did not occur in the impoverished area.
When I was growing up there wasn’t much sensitivity or awareness about how much trauma could affect a child. Now there is, so shouldn’t we be doing better?Regardless it’s no mystery that kids who experienced a lot of trauma are a lot harder to teach and it takes a lot more to help them. Now imagine an entire class of them. Eddie what do you do with your life to try to address this… I hope it’s not blaming all the teachers and sitting on your butt somewhere in a cushy job? Was it my teachers’ faults that I was failing? No. Was it mine? I don’t think so: I don’t think many 13 or 14-year-olds should have to deal with that kind of stress and abuse and show up with a smiling face ready to learn either. Regardless, I would hate to think that the staff at my school would’ve been punished for what was going on in my life.
It’s interesting how many people want to blame teachers. Were you a good student? Was it your teachers who did your work and deserve the credit? If you’re going to make the teachers responsible totally for a child’s success then you also need to make sure that you take no credit for your own successes… It was all your teachers’ efforts correct?
Thanks for your honesty. Lots of people do not understand how difficult it can be for young people that have to deal with poverty, disruption and dysfunction.
The tests do not accurately reveal what or how much people have learned in school, and the scores of students create a ridiculously false representation of what teachers taught in school. The overall test scores of large groups of students more accurately align with income level than with learning. Since the Depression was mentioned, take, for example, the tests given during the early to mid-twentieth century. Recent-immigrant groups had low scores and were assumed to have inferior intelligence. The same immigrant groups later had high test scores once they had overcome the initial poverty often associated with migration. They were smart! The tests were dumb. They still are.
Most likely, a recent immigrant would be poor, which isn’t much different than being poor in 1933 America. 😐
Does the immigrant speak English? Does (s)he understand the question? Does (s)he have prior exposure to the material?
😐
Most poor immigrants today do not speak English, and they often come from countries with poor educational systems, or their schooling was interrupted by war and chaos. Some of them are also undocumented. These folks cannot get any help from the government despite what the Republicans often claim. They cannot get any public assistance although some states may allow children to receive medical care under CHIP.
It’s not that individual scores rise and fall with economic levels. It’s that groups rise and fall. One individual’s score is not “statistically significant”. The tests suggested that Irish and Jewish people, for example, were stupid. Years later, the tests suggested that Irish and Jewish people were smart.
Poverty can’t be blamed for everything, but using tests to close schools and fire good teachers in impoverished communities can be blamed for, well, let’s be generous and just say not helping.
Eddie, I responded further down to get more margin space.
so succinctly said
Texas teacher, I very rarely blame teachers. 😐 I’m sorry about your life experiences. ☹️
As a Black American male from Chicago’s predominately Black Austin High School with two White male students, I don’t agree the various reasons given before us always or mostly the cause of students’ success. 😐
Blacks, with and without obstacles, have graduated from predominately Black high schools including Curtis Mayfield, Jerry Butler, Mary Flowers, Earlean Collins, Dempsey Travis, Harold Washington, Ernie Terrell, Jean Terrell and many others. The Terrell’s secondary school had/has Hispanic students too. 🙂
Historically, we Blacks were learning how to read with Bibles and candles, during slavery when it was illegal. 😐
During the Great Depression, some, probably most of us were educated with growling stomaches, bad housing and maybe poor indoor plumbing. 😐
retiredteacher, I realize that. The war, chaos, etc. is very sad and bad. ☹️
This report reflects a major achievement by Audrey Amrein Beardsley who has led the charge against the use of VAM, especially SAS’s proprietary formulas, to rate teachers.
Credit is also due to the judge whose keen understanding of the mathematical formulas extracted from the SAS black box showed how ridiculous these calculations are.
They are also ridiculous and offensive because the “originator” of SAS’s formulas, the late William S Sanders, borrowed his methods for calculating “value added” from statistical studies of the productivity of herds of dairy cows, as reported in my previous post on this blog.
The judge has a keen understanding of nonsense.
The Wisdom of Socrates
I know that I know nothing
And that’s what nonsense is
So nonsense is the one thing
That I can surely diss
“[w]hen a public agency adopts a policy of making high stakes employment decisions based on secret algorithms incompatible with minimum due process, the proper remedy [may be] to overturn the policy”
In case after case, judges have ruled that an opaque algorithm that may cause a teacher to lose due process rights and employment is capricious and unreliable. A big system like Houston enjoys the convenience of using VAM scores, but they do not accurately reflect the competency of the teachers. VAM is an unjust way to evaluate and remove teachers.
As far as the city of Houston, which has an overall ‘B’ rating from the state despite its diverse student body, using the scores of one school to justify a takeover, is a political move by Abbott and Patrick. It is an attempt to seize control of the public schools in order to privatize them. It is also a power grab to remove democratic governance from managing the Houston Public Schools.
Now, if those teachers were cows instead of humans, VAM should have worked.
After all, VAM was invented to assess cows, not humans.
“The process was created by William L. Sanders, a statistician in the college of business at the University of Knoxville, Tennessee. He thought the same kinds of statistics used to model genetic and reproductive trends among cattle could be used to measure growth among teachers and hold them accountable. You’ve heard of the Tennessee Value-Added Assessment System (TVAAS) or TxVAAS in Texas or PVAAS in Pennsylvania or more generically named EVAAS in states like Ohio, North Carolina, and South Carolina. That’s his work. The problem is that educating children is much more complex than feeding and growing cows. Not only is it insulting to assume otherwise, it’s incredibly naïve.”
https://nepc.colorado.edu/blog/top-10-reasons
So, if you are human and a teacher, the next time someone like a parent or admin-animal asks about your VAM score say, “I’m sorry, that assessment doesn’t apply to me. VAM was invented for model genetic and reproductive trends in cows, not humans.”
Minor point. Actually the formula was not for “cows, but for the productivity of herds of dairy cows…females. The VAM method of evaluating teachers has roots in genetic engineering for agriculture.
So, we should only use VAM on female teachers.
Luckily most teachers are female.
VAM accountability schemes were revealed for the shams they are in 2014 by the American Statistical Association. https://www.amstat.org/asa/files/pdfs/POL-ASAVAM-Statement.pdf
That my state uses any part of this sham is wrong– but at least the shammy stuff has only accounted for 15% of teacher evaluation for the last several years (it was more like 35% for the first year or two). 85% of NJ teacher performance is judged by classroom observations. HISD bases 50%-100% on black box computer spit-outs!!
“at least the shammy stuff has only accounted for 15% of teacher evaluation for the last several years”
Cathy O’neil (Mathbabe) pointed out why even if VAM scores have low weight in teacher evaluations, they comprise the vast majority of the variance in the overall teacher evaluations , which are then used for tenure and firing decisions.
https://mathbabe.org/2016/09/30/the-one-of-many-fallacy/#comments
So, the idea that if the weight is 15%, the VAM only accounts for 15% of the difference in overall evaluations is actually false.
Excellent reference, SDP! Immediately shows the flaw in my thinking. I stand corrected.
I’d already thought of her point that reducing the %age to a small one merely reveals that the original concept was all wet (should be eliminated, period.) But this rings true & really galls me: “The VAM was brought in precisely to introduce variance to the overall mix.” It was always a teacher-bashing concept intended to further undermine salaries & tenure [& thus taxes]. Teacher evaluations that scored them all OK, good or excellent? Put them on a bell curve so we can label % deadwood to be replaced by cheap newbies.
The thing I love about Cathy is that she tells it like it is.
A lot of people would be hesitant to say that the entire purpose of VAM was to introduce variance and thereby allow teachers to be ranked and hence culled.
As Cathy points out, math is all too often used in nefarious ways like this.
Sadly, the only people who recognize this are people who are actually trained in math, but all too often , they can’t be bothered to point out the BS.
And even when they do critique it, they are far too reserved.
Even the American Statistical Association critique of VAM was quite reserved.
What they should have said was “VAM should not be used to evaluate individual teachers period.” They implied as much but that is not the same as stating it quite categorically, leaving no wiggle room for people like Eric Hanushek and Raj Chetty
“Lots of students were poor during the Great Depression, but still learned. Poverty, etc. can’t always be blamed for “everything”. As always Eddie, you make provocative points. This one spurred me to research.
A huge number of poor children during the Depression were on the move with their families trying to find work. In between times, when they found it as migrant laborers, the children would be herded into garages or barns, as many as 125 to a teacher, with a schedule that would allow them to hit the fields at noon, and work until sunset. https://www.jstor.org/stable/4289275?seq=1 Can’t imagine much learning happened there.
When migrants found longer-lasting gigs in CA or TX, these poorest kids [a mix of white & Mexican-American] would be given a battery of tests designed to separate them from more privileged schoolchildren– tracking. Other tactics used for segregation included ‘school choice’ for whites, gerrymandering, Mex-Amer/ Okie- only schools, forced repatriation of Mex-Amers. At one point CA citizens even erected road blocks to keep Okies out. A CA Chamber of Commerce publication belatedly [1940 – migration to West had been decreasing for a decade] claimed “any further inflow of depression migrants would seriously disrupt the economy of CA, jeopardizing wage scales, living stds, and soc welfare programs.” Studies show the actual influx/ cost in taxes didn’t justify “the expensive option of segregating Okie and Mexican children from the mainstream.” https://nepc.colorado.edu/publication/children-harvest-the-schooling-dust-bowl-and-mexican-migrants-during-depression-era
“Poverty” in this country—in the context used here, i.e., low achievement per stdzd test scores reflects low-income/ lack of oppty—is shorthand for generational poverty/ institutional racism-classism. Going on longest in former slave states & sections of former Spanish territories, 2nd-longest in ‘white-trash’ areas in Carolinas & Appalachia; Depression-era ed issues mirror a long tradition.
bethree5, I don’t know CA and TX, so I’ll have to believe you. 🙂
Bethree5
Amazing info there. So much to think about. My dad was a depression era baby and I sometimes wonder if a lot of his emotional issues came from having a few choices in life and having to grasp the straws that were given to him. Lots of anger when people don’t have a fair shake.
Eddie, Thanks but I’m really not sorry about my experiences in life at all. It was really hard in my 20s and 30s to kind of get over some things but at this stage of my life I am so at peace with everything and I feel like that’s giving me more perspective in life. Now ironically I’m at that age where a lot of people assume that, when you are a mature looking woman who looks like they have it all together and educated, that you’ve never experienced hardship lol.
bethree5 I would extend that “going on longest” to the main recipients of the Great Migration: NYC, Cleveland, Detroit, Chgo. Also keep in mind Eddie that when you were in school there was no parallel system of publicly funded charters (College Preps) that essentially skim the highs and meds with steady families from your average neighborhood HS classrooms, such as Austin or Orr. The absence of that group is wholly under-estimated with regards to “grading” neighborhood High Schools in large minority communities. If you see yourself in an environment of last resort, you may act accordingly, teachers too.
Hi MSS. In my day, there were charters. Austinites were mostly choosing other Chicago Public Schools, for decades. My parents gave me a choice and I chose Austin. 🙂
You are so right, MSS. I see I have still not completely digested Wilkerson’s “The Warmth of Other Suns.” Which BTW provided necessary background to understand the early-20thC midwestern culture revealed to me in McRae’s “The Great Trials of Clarence Darrow.” That period is key to so much we see around us today.
My dad too was Depression-era, raised in then-KKK-dominated IN. But he was an outlier—called himself ‘black sheep of the family,’ who escaped to the liberal Northeast when still a young man. No anger. Slim choices plus luck worked for him: off the dirt farm to steel mills at 14, into Navy (WWII) at 17; made it from there on trade skills acquired (& that Navy friend from a Northeastern collegetown…)
Hi Eddie, Got ya you are younger than I imagined! FWIW my neighborhood HS was Crane. I chose SICP