Last Friday, officials at the central office of the District of Columbia Public Schools quietly released the news that the teacher ratings on its highly touted IMPACT system contained errors. It was not clear how many teachers were affected. If you want to bury a policy disaster, the best time to announce it is on a Friday before a long holiday, on the assumption it will be ignored and forgotten.
Researchers have warned for the past three years that grading teachers by the test scores of their students is error-ridden, inaccurate, and unstable. Earlier this year, the distinguished psychometrician Edward Haertel of Stanford warned in a major lecture that value-added scores should not be used as a fixed percentage when evaluating teachers and should have multiple safeguards to avoid error. Did anyone at the U,S. Department of Education or anywhere else take heed? Of course not.
As Valerie Strauss notes in the linked article, this inherently flawed and demoralizing process has been widely accepted (it is a major element of Race to the Top; in addition, states that want waivers from the impossible mandates of NCLB must agree to adopt this procedure, no matter how ill-conceived it is.)
Strauss writes:
“Testing experts have long warned that using test scores to evaluate teachers is a bad idea, and that these formulas are subject to error, but such evaluation has become a central part of modern school reform. In the District, the evaluation of adults in the school system by test scores included everybody in a school building; until this year, that even included custodians. In some places around the country, teachers received evaluations based on test scores of students they never had. (It sounds incredible but it’s true.)”
Only a few weeks ago, Audrey Amrein-Beardsley wrote on her blog that the ratings on the D.C. IMPACT system made no sense.
Now the company that created the rating system has acknowledged the errors.
Let’s see if some enterprising journalist digs into this fiasco.
Diane Ravitch's blog 
May this go nation-wide!
LikeLike
Experts warned that this makes no sense? Why listen to experts — when we can hire consultants with an agenda?
LikeLike
Hopefully it won’t be 60 minutes doing the digging — they would end up interviewing Michelle Rhee.
LikeLike
Give 60 minutes credit for covering Gulen charter schools. PBS has not given Gulen charters much coverage. As there are more Gulen charters than KIPP charters, this is an interesting choice for all the media outlets that have remained silent.
LikeLike
It’s good to see early signs like this of the disintegration of an workable system, not to say unfair to begin with.
LikeLike
HU,
Not sure what you mean by “an workable system”. Do you mean that VAM and/or SGP are valid indicators of teacher effectiveness in the teaching and learning process? Did you mean “an unworkable system”? which by the “an” makes sense.
LikeLike
I did mean “unworkable” system.
LikeLike
I am hoping for a nice little lawsuit as well by someone hurt by the faulty evaluations.
LikeLike
Because almost all testing experts agree that this evaluation system is not valid, I’m surprised some teachers haven’t taken their “evaluation” to court. Or have they?
LikeLike
More evidence this reform movement is built on sand.
LikeLike
More like quicksand for how it sucks students and teachers down into a bottomless pit>
LikeLike
This is awful news for DC teachers. Down here in Charleston, it’s the greatest Christmas gift imaginable.
We’re fighting VAM-based merit pay tooth and nail. Guess who our district hired to do the work?
Here’s the only question I have: was this what Mathematica had in mind in 2010 when they said that VAM has a 36% error rate?
Click to access 20104004.pdf
Is that before or after they foul up the data?
Tell you what, don’t ask Mathematica. I can tell you from personal experience: they REALLY don’t like talking about that study.
I know it was before Arne Duncan handed out nearly a billion dollars in grant funding for value-added systems.
When Mathematica published this, TIF grants were still comparatively small potatoes.
Funny thing is, Arne’s the one who picked up the tab for that study. His name appears on page 3. Go figure.
LikeLike
And, in other news, Michelle Not-in-Touch-with-Rheeality’s Students First (uh, Last) is sending out a holiday message to parents and educators and policy people here in Florida to the effect that PISA shows that the sky is falling and that we need to have “no more excuses.”
Of course, this message ignores the little fact that if you correct the scores for the socioeconomic class of kids taking the exam, U.S. students are at or very near the top.
LikeLike
Reposted from Huffington Post – “Michelle Rhee, then D.C. schools chancellor, instituted the use of IMPACT in 2009, making it one of the first teacher evaluation systems to treat students’ test scores as a significantly influential factor. The system uses “value-added measurement,” a complex algorithm that aims to remove the statistical effects of factors like students’ socioeconomic status to uncover how much teachers truly affect their students’ test scores.”
IMPAC does more that ignore socioeconomic facts, it was designed to remove socioeconomic factors – to shift all of the responsibility onto teachers.
LikeLike
This just in: The Common Core Curriculum Commissariat and Ministry of Truth (CCCMT) has determined that elderly people, after retirement, have “little value add” because they are not working.
However, they note that studies of Platyhelminthes (flatworms) show that if you teach these worms to turn left in a T-maze (by electrifying the right side) and then grind them up and feed them to untrained worms, the untrained worms turn left without having to be trained.
The CCCCMT is therefore recommending that elderly past retirement age be ground up and fed to toddlers.
After all, it’s all about the value add. One cannot argue with THE DATA.
Freedom is slavery.
War is peace.
Ignorance is strength.
Truth is data.
Learning is checking off the bullet list.
Teaching is punishment and reward.
Arbeit macht frei.
LikeLike
I bet if we grind up reformers and feed them to children, they would regress in their learning.
Nic post, Robert!
LikeLike
I got the good news that my score went up from minimally effective to slightly less minimally effective!
At least I don’t have to change my URL.
LikeLike
@robert…that was really funny:)
LikeLike
And sad and true at the same time . . .
LikeLike
“. . . the District of Columbia Public Schools quietly released the news that the teacher ratings on its highly touted IMPACT system contained errors.”
Those are the errors that they are grudgingly willing to concede. They and the edumetricians will never acknowledge the myriad errors as shown by Noel Wilson that render the whole process invalid and any results “vain and illusory”. To understand just how ludicrous it is to even begin to lend a thimble full of credence to these educational malpractices and the very many harms caused now not only to students but teachers read and understand “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Join the Quixotic Quest bandwagon to rid the world of these educational malpractices, first by reading and understanding what Wilson has to say. There is no jumping off once one understands the sheer idiocy of these practices.
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A quality cannot be quantified. Quantity is a sub-category of quality. It is illogical to judge/assess a whole category by only a part (sub-category) of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as one dimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing we are lacking much information about said interactions.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it measures “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
LikeLike
But hasn’t this been the central issue all along — the post-modern critique of testing. The entire accountability/privatization movement has been about power and money, not educating children better. The design of the SAT/ACT has always been about a students background knowledge, what a group of test makers has determined to be privileged knowledge — that privilege knowledge “measures” narrow band of cognitive skills that elite universities have determined to be knowledge of most worth. The question educators should be asking of any reform movement is: 1) Who is benefitting from this reform; and 2) Who designed the reform. A close look at the accountability/privatization movement and it becomes clear that the benefits of these reform movements can be generous to those who now have the power to keep designing policies that further enrich their companies and consulting fees. There was a period when private sector left educators alone because they though that there was no money in the public sector — not anymore. As these blogs have pointed out, the private sector sees big money in education, which then translates into acquiring the right political influence and we end up with the current definition of schooling —a language system, a teaching system, an assessment system all designed to transfer money from the public to the private sector — all in the name of racing to the top (the corporate metaphor of the decade).
LikeLike
Your comment assumes that the reform and test-fetish crowd give a rat’s bladder about facts. This is tough reading, even for a well-educated person; would someone so self-absorbed and thoughtless [as the high-stakes crowd] be able to penetrate that these tests are no better than fertilizer?
LikeLike
BW,
Actually, my comment is directed to those who do care about the teaching and learning process and to give them the “nukular weapons” to obliterate those malpractices.
The edudeformer crowd won’t give one second of thought to what I have to say, they know they are right, just as the religious know they are right in their beliefs, never mind that those beliefs can be rationally, logically shown to be false.
And the edumetricians, well let’s just say Sinclair said it way back when (paraphrased) “It’s hard to get an edumetrician to understand his machinations are illogical and invalid when his salary depends upon him/her believing and proffering that psychometrics is a true science.”
LikeLike
Duane:
Thanks for directing me to that article on standards and error. Just completed reading the paper and found his comments on assessment, and even schooling in general, to be a perspective that we don’t often see explored in educational journals or books.
LikeLike
And having independent, critical and free thinkers is a threat to the current socio-economic structure of society, otherwise known as Unregulated CAPITALISM.
I’m NOT pushing
“Pinko” here, but UNREGULATED CAP kills the “Hosts”.
“The Democratic Party and the Republican Party are not mortal enemies, as portrayed in the corporate media; they are in collusion against the world’s working class and the poor. Together, they are raping and pillaging the Earth Mother and repressing workers through economic violence and imposed austerity. Like costumed wrestlers performing on television, the acrimony is not real; it is vitriolic political theater, an enthralling puppet show for diehard believers.
We must somehow move beyond party politics, beyond the simplicity of liberal versus conservative dichotomy, beyond left against right, and see things as they really are rather than as we wish them to be.
Voting doesn’t change anything in a system flush with corporate money. The structures that put the money into politics cannot be used to extract it. Without proportional representation or corporate money, third parties are not a viable option in state and federal elections. They are just another distraction from reality, a mild form of symbolic protest. Voting for justice does nothing to actually attain it. Direct action directly applied to a problem offers the best hope for revolutionary change.
Conversely, political dualism keeps us fighting the wrong people. It has us believing in people and institutions that do not promote justice and do us harm. These institutions are not what they purport to be. They are at best a mirage; something that appears real but only exists in the mind of the beholder.
Belief in the American Dream and perverted systems of power as a means to justice provides a method for directing and cajoling the masses to do the biding of the super-wealthy and all-powerful corporate state. Faith, hope, and belief in phony people and bogus institutions function as a form of mass hypnosis that keeps the people from organizing in class struggle against a common oppressor—UNREGULATED CAP.
Despite reams of contradictory historical evidence, most people in the U.S. continue to associate democracy with capitalism. It is reckless of us to allow anyone to use these terms interchangeably without contesting them at every opportunity. Let me be clear: Democracy is the antithesis of capitalism! But capitalism is the product the U.S. government, the Pentagon, and the commercial media are marketing to us as democracy. And thus the inequality gap, the disparity between rich and poor, is growing wider rather than shrinking.
The nemesis of all working people, regardless of where they live or their political affiliation, is UNREGULATED capitalism and its linear, hierarchal, male-dominated power structures. This is why we must have a truthful critique of UNREGULATED capitalism and patriarchy and create alternatives that promote the public wellbeing above corporate profits.
When the richest and most powerful people on earth, the primary beneficiaries of unregulated capitalism, invest so many resources into demonizing and subverting ANY economic and philosophic alternatives to unregulated capitalism, inquisitive minds want to know why.
If workers understood unregulated capitalism, not one in ten thousand would voluntarily accept their performative role in this exploitative economic system. There would be widespread conflict and social upheaval. There would be global revolution. The power elite spends trillions of dollars to maintain the façade of unregulated capitalism as a manifestation of democracy. In fact, I would argue that nothing could be more opposed to democracy than unregulated capitalism.
The key point to understand is that unregulated capitalism, a system based upon the ruthless exploitation and commodification of workers and the relentless rape of our Earth stifles and represses democracy. Unregulated Capitalists abhor all forms of egalitarianism and regulation.
The mere possibility of an empowered work force troubles the unregulated capitalist’s sleep, as did the possibility of slave rebellion, albeit it small, distress the slaveholder.
Consider the vitriol, not to mention counter revolutionary forces that are levied against the alternatives to unregulated capitalism. What is their source? Who but wealthy unregulated capitalists fund America’s propaganda apparatus? Working people in the U.S. are conditioned to reflexively recoil against ideas they do not understand. They are psychologically programmed to detest that which could potentially set them free. American workers are led to believe that economic servitude and wage slavery is freedom.
Why does a government that calls itself a democracy systemically spy on its citizens? Why does it punish its whistle blowers but materially reward the vilest white-collar criminals? Why is the majority of the U.S. budget spent on funding an insatiable war machine? Why do we raise classrooms of meat puppets rather than critical thinkers and political dissidents? It is all done for the benefit of unregulated capitalists at the expense of society.
It is by these means that unregulated capitalism survives and spreads like an aggressive malignancy to every organ of the planet. Furthermore, the majority of the wealth produced by labor is subverted to prop up the unregulated capitalist system and to indoctrinate and oppress the worker. To the detriment of us all, freethinking and critical analysis are discouraged and often reprimanded in academia and elsewhere. And thus hundreds of millions of human beings are transformed into herd animals that are led to slaughter in the military and the world’s sweat shops. We celebrate our freedom and patriotism on our march to the scalding pots, singing “God Bless America.” There is no fight in us. We go too quietly and too obediently into the good night of eternity.
Yet, despite everything and the repressive weight of history, Americans still have a propensity to believe in myths and fairy tales. Hope and faith in phony leaders and bogus institutions keep us servile and docile. Irrational faith requires nothing from us. Delusion has become the norm because too many of us are incapable of grappling with reality. We can and must do better than unregulated capitalism or we are doomed to an ignominious fate.”
Again, I’m not pushing “Pinko” here, but REGULATED Capitalism, “Seemed” to work
in the past, more so than today.
Dead fish swim with the stream as people sink in the Quicksand of their thought…
LikeLike
And who is to do the regulating? YOU?
LikeLike
There are way too many variables in human behavior and cognition to be reduced to a numerical spreadsheet. Teachers are losing their jobs and having their careers ruined. Students are being put under unnecessary pressure. Parents are worrying about the wrong things.
LikeLike
As a special education teacher, I have been opposed to the IMPACT system since day one. I feel teachers should be evaluated and I am all for holding us to high teaching standards and pedagogy. But, IMPACT, is literally setting teachers up to fail. As a special education teacher I am judge on either of two rubrics, IMM (if i am co-teaching) or TLF (if I am the lead or only teacher – my case in self contained class); neither work, because I am not a content teacher, but I am graded on a rubric that talks about delivery of content, building UBD and lesson plans. Things which did not fall under my job description. But I adopted them or I would not have been found effective. What about judging me on my IEP quality (removed from rubric), IEP timeliness, improvement in IEP goals, behavior management, deescalation strategies or differentiation. All the things I was taught as a SPED teacher. Should I really be evaluated on teaching an AP Calculus class? I have had to teach myself the content before I can even attempt to differentiate for my special needs students. The system is flawed on so many levels. It’s sad. I recently received an email from DCPS HR asking about teacher retention and how to get teachers to stay in the DCPS system. How about reevaluating your own judgement system for a start. And then actually doing something to fix the problem?
LikeLike
My simple-minded belief is that I’ve always felt it manifestly unfair to judge any person on the peformance of *somebody else.*
The idea that my teaching evaluation is based on the good test-taking skills of a student runs against common sense. It also runs counter to much of what we believe in [a student’s] self-motivation, free will, and personal autonomy.
A small percentage of my [Indiana] teaching evaluation is based on the school’s letter grade. (“This gives teachers *ownsership* (emphasis added) in the performance of their school.” –Mr. Accountability himself, Tony Bennett) I have argued that this would base my evaluation on a school grade which is largely determined by students I’ve never seen or taught.
Although it might be a faulty comparison, I once compared it to giving a surgeon a low rating because the cafeteria in his hospital faiiled a health inspection.
LikeLike