Gary Rubinstein is an extraordinary math teacher who has a terrific blog.
His analysis of New York City’s teacher data reports shows that they are inaccurate, unreliable, and meaningless.
Any district or state official who is considering VAM should read Gary’s six posts.
If you do, you will discover there is no there there.
Please help this post go viral.
Every pundit–from Nick Kristof to David Brooks to the editorial writers–should read this analysis.
Diane Ravitch's blog 
Kudos to Gary Rubinstein for his sharp and relentless analyses.
In this particular post, he addresses the question: for the teachers who had the same grade level, subject, and student pretest score in 2008 and 2009, what is the relation between their 2008 and 2009 value-added scores?
The scatterplot (as I see it) shows them all over the map but does reveal a few patterns, which I pointed out in a comment.
The patterns are not strong enough or meaningful enough to give credibility to the value-added ratings. They are still close to meaningless–and anything that flawed, that unreliable, should not be used to evaluate or rank teachers.
I bring it up only because it’s important to use precise language when battling VAM. “Meaningless” strikes me as a slight overstatement (which the VAM proponents would frothingly knock down).
Diana,
“’Meaningless’ strikes me as a slight overstatement.”
Not an overstatement at all, except in the sense that the “meaning” to be taken from the VAM process is that it is completely invalid. Why is it “completely” invalid? As Wilson has proven, the whole process of making and using educational standards and standardized testing is so fraught with error that, even just one of the identified thirteen errors is enough to invalidate the process. So that VAM itself is based upon a logically invalid process. The saying “garbage in, garbage out” comes to mind (and I prefer the cruder way of saying it.)
The concept of VAM is also rooted in the UNETHICAL usage of student scores to evaluate the teacher. I’m not sure why educators aren’t screaming out about this fact. All the major testing organizations and even the test makers in the testing manual instructions point out that to use any standardized test results for anything other than what the test was developed for is UNETHICAL. A 5th grade math test supposedly assesses the student’s capabilities in 5th grade math, nothing more and really a whole lot less (keeping in mind the whole invalidity problem of the process.) That test was designed with the purpose of assessing the student’s math capability not the teacher’s teaching capability. To use said scores in the VAM process just heightens the errors involved rendering the whole process invalid and, in essence, “MEANINGLESS” and UNETHICAL or as Wilson states “vain and illusory”.
Come join me as I explore Wilson’s destruction of educational standards and standardized testing on my blog “Promoting Just Education for All” at: revivingwilson.org. Wilson’s “Educational Standards and the Problem of Error” should be required reading for all educators.
Duane
There’s a persistent myth that teachers don’t want to be evaluated. My teacher friend at Berkeley High weighs in:
I am in FAVOR of teacher evaluation. Teachers, like any profession and particularly one that has a focus on kids, should be subject to performance reviews. I also believe that achieving tenure should be a longer process and should involve more than “staying under the radar.” Principals have a huge responsibility to “non-reelect” teachers and they have only two years to do so (at least in CA). Having said that, I am a firm proponent of tenure and believe that tenure should not (and is not) the equivalent to being impossible to fire or replace. I believe that once a teacher has achieved tenure, they must still grow as an educator and continue to teach well and evaluations must be a part of their job.
The HOW EVALUATIONS ARE DONE is the bigger question. I AM NOT in favor of linking any kind of test scores (except possibly that teachers’ own test scores – a teacher who fails everyone or gives everyone A’s is failing basic probability) to a teachers’ evaluation. As I’d like to think, everyone knows that this is a bogus measure (and soul-sucking). I am lucky enough to be surrounded by amazing teachers and a supportive administration. Thus, pop-ins by principals are common (and whole-heartedly supported by the unionized faculty), evaluations are taken seriously, peer-coaching is happening, and teachers are often given the opportunity to do “alternative evaluations.” For my last evaluation, myself and 5 colleagues evaluated ourselves using a protocol called “critical friends.” It was like peer coaching except we had to report back to the faculty and our administrator regarding the process and what we learned about each other and our own teaching. It was one of the best learning experiences of my teaching career. I just sat in on a design team meeting last Wednesday to try and implement the protocol staff-wide. You’ve never seen teachers so excited. One of my favorite colleagues, an English teacher in her 11th year said, “Yes, please come in my class! Please evaluate me! Tell me how I’m doing!”
I’ve been at my current school for 6 years. The school is anything but perfect. However, I have seen two proactive administrations non-reelect dozens of new teachers instead of taking the chance to tenure them and I have seen 2 atrocious (and very senior) teachers get fired and I know of a couple current teachers that are working hard to show why they deserve to stay.
Teachers who have earned tenure (or at least received tenure by floating under the radar) can’t be fired willy-nilly. This is good. I’d like to think that educational institutions are built on scholarship, professionalism, and academic freedom. Anyone who has tenure, deserves due process. They deserve to have an opportunity to either defend their actions or right the ship. If they can’t do this, they should be canned. For the sake of kids everywhere, this is necessary.
Evaluate teachers? Yes…PLEASE!
“a teacher who fails everyone or gives everyone A’s is failing basic probability”
No, not at all. Considering that the assigning of grades, even by the classroom teacher, has all the errors, and therefore, invalidities that Wilson has shown, that the whole process is a falsehood then why do we insist upon “grading” students. I contend this sorting and separating out students using grades is just as evil of an educational practice, causing great harms to the students, as was “separate but equal”. Both practices are based on falsehoods, one that skin color should be a determining factor in the education one receives and the other, almost more pernicious, that one’s mental capacities should be a determining factor in the education one receives with some reaping the educational benefits and others being denied said benefits.
I don’t give formal tests to my level 3 and 4 Spanish students although I am constantly assessing them. They know that I have to play the grade game and that it really is their responsibility to learn the subject matter (but then again, they’ve had only me as their Spanish instructor and they know how I think education should be.) They are all hard working, self motivated students and I feel no need whatsoever to “grade” them.
So far this year, I’ve already been contacted by the prior two years worth of level 4 students (one from two years ago and the other three from last year) thanking me for helping them to learn not only Spanish but how to learn for themselves and how my methods help them now.
When will we, as educators finally understand just how pernicious this concept of assigning grades to/grading students is? When will we realize that we harm far more students by these practices than we help? When will we finally take up the oath of “Cause no harm”???
I agree with you, Duane. I am a also a devotee of Alfie Kohn.
Regarding the myth that teachers don’t want to be evaluated – it has been my experience that nobody is more intolerant of poor teachers than . . . other teachers. The few poor teachers in a given school tend to be personae non gratae, don’t they? The idea that we want to circle the wagons to coddle and protect the truly bad ones (not just the ones needing some remediation) is equally mythical.
Regarding Gary’s VAM analysis – this would be a wonderful project for math teachers to use in a stats class, wouldn’t it?
Furthermore, what Gary does not say in his analysis, is that in order for NYC to “predict” student’s ELA and Math scores, the students take 4 “predictive” tests during the school year – 2 for math and 2 for ELA. These tests are given using the same procedures as the “real” tests. We are on “lockdown”, students who are late must wait in the auditorium, our “testing coordinator” must arrange for all accommodations, etc. How much money and time are we going to spend on this nonsense before we realize that it has no value? Why do we give politicians like Bloomberg so much power in education policy? When are the teachers in NYC going to wake up and act like the teachers of Chicago?
“How much money and time are we going to spend on this nonsense before we realize that it has no value?” Well enough so the well heeled avaricious “friends” of the politicos can swoop in and grab all the more dollars off the backs of the students, and correspondingly the teachers.
Another post:
http://larrycuban.wordpress.com/2012/09/14/chicago-teachers-strike-performance-evaluation-and-school-reform-jack-schneider-and-ethan-hutt/
Diane’s comment re VAM that “there is no there there” is an understatement.
I just spent the last hour going over the six posts and the associated comments. It is not just that Gary’s postings show that the NYC TDRs are “inaccurate, unreliable, and meaningless” [Diane’s words]. Remember the “dog that didn’t bark” clue? If they were accurate, reliable, and meaningful, they would have been trotted out by the edubullies on every public forum they have at their disposal as the definitive answer to all the critics of VAM and edudeform.
Hence, the unintended value-added of the data is that they undermine those who insisted on providing them!
How ironic! The pineapple beat [down] the hare!
Mr. Rubenstein seems to be the perfect guest for The Daily Show with this type of analysis and questioning. In the posts, he asks us to try to publicize his findings. I did that on my facebook and with my email account. But I don’t know how I would get a suggestion like this through to the Daily Show people. You (Diane) might have more success making this recommendation than any of us.
Actually, all the politicians need to do is to listen to our best scientists and mathematicians. You know, those STEM professionals that we are trying to create with our education reforms. The National Academy of Sciences has released multiple reports proving the lack of validity of value-added measures in almost all contexts. Who are the academics who back value-added models? The economists who are now turning to education after proving the validity of their models by failing to predict our current economic decline.
Isn’t it ironic, by law we must use “research based” teaching methods, but we can be evaluated using methods that research has refuted as invalid. I can’t wait to see this nonsense defended in an open court. The experts that developed the methods will be present to testify that their theories are being misused. I only hope judges and juries are paying close attention. The end may not be near, but it is coming. This shows similarities to the problems with I.Q. testing and Terman at Stanford. A seemingly logical idea is in fact insidious and harmful.
K,
Well said! You are correct in that it will probably take legal action by a teacher who has been harmed by VAM to legally show the inherent fallacies. And/or by students who refuse to play their “role” in these “insidious and harmful” schemes.