Do you want to know why it is wrong to evaluate teachers based on test scores of students?
A new book by scholars explains it here.
They also show better, research-based methods of evaluation, intended to support, not fire, teachers.
There are books soon to be published by other eminent scholars that show why VAM is flawed and demoralizes teachers without helping kids.
Diane Ravitch's blog 
VAM is the FACTORY model. Standardizing humans is an exercise in futility. The deformers really want robots.
This is why I don’t understand why our unions are embracing it and why many teachers who did have the right to vote on an evaluation plan, unlike the teachers in NY, voted for something that is proven to be an invalid measure. People better stop voting their pocketbooks and start reading the fine print because when you are out of a job based on VAM (unless you decide to be a cheater), the district will find a reason to no longer give you a salary even if you are the best teacher in the world. They are looking for reasons to rid the system of experienced teachers and replace them with newbies. It’s cost effective for them.
Love this video….http://www.youtube.com/watch?v=uONqxysWEk8
I cannot speak for how VAMs are implemented throughout various districts, but I can speak on behalf of how it’s implemented in DCPS. An honest opinion from a teacher on the frontlines: Teaching in a high poverty public middle school in a high poverty neighborhood. http://wp.me/p3Lk1s-5F
You can’t objectively measure teacher “quality.” It is strictly subjective, and evaluations are based strictly on the opinion and biases of one person, a principal, unaccountable at all for his or her actions.
Learning is strictly up to the student and his or her ability or desire to learn. A teacher has NO control over that; a teacher can only deliver content.
I have not read the book yet but this study helped to clarify the shortcomings of VAM fairly well. http://epaa.asu.edu/ojs/article/view/810
“VAM is invalid”
Wild bears crap in the woods.
The sun comes up in the east.
The only for sure thing in life is death.
Like
and Taxes
Kidding aside there are myriad reasons that any conclusions drawn from standardized tests are invalid. Noel Wilson has pointed out 13 sources of error, any one of which renders completely INVALID the process of educational standards, standardized testing and the grading of students, and by extension, e.g., VAM teachers. It is also UNETHICAL to use the results of any test for any other purpose other than what it was designed.
Doing the wrong thing righter, crap in-crap out is what the above mentioned educational malpractices are.
Join the Quixotic Quest to rid the world of these amazingly pernicious malpractices by reading and understanding Wilson’s never refuted, never rebutted “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A quality cannot be quantified. Quantity is a sub-category of quality. It is illogical to judge/assess a whole category by only a part (sub-category) of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as one dimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing we are lacking much information about said interactions.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it measures “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
You know Duane, people are starting to quote you on message boards that I browse. You are starting to get popular you know.
Thanks for letting me know. If I may ask what sites? I might not know of some of them and sounds like they should be good reads.
But I take no credit as it is Noel Wilson who has so elegantly destroyed the standardized testing “logic” or perhaps better said “illogic”. I know I’ve gotten my share of the “blame” for helping Wilson expose the fat old emperor who goes by the name educational standards and standardized testing. Those in power, even, or should I say especially, at the “lower” local level do not like having their schemes challenged. And I sometimes do feel quite Quixotic in my Quest as many have a hard time breaking out of such ingrained educational practices.
And I would also like to give credit to two education professors whom I was fortunate to have had, Drs. James Walthers and Charles Fazzaro, who urged their administration grad students to practice what they call “Critical Enquiry” and to think about unspoken and unwritten practices that permeate our schools. I can’t seem to find a description of Critical Enquiry. I’ll have to get back with you on that.
Yes, love it when Duane posts the Wilson list. (Seriously).
VAM makes business people feel better that teachers are not getting a good deal. The fact that THEY need us to have VAM is about them. How to get their persuasion back in check? I have no clue. But the Wilson list does help.
Swacker are you sure that is not Critical Inquiry?
Getting back to you with the Critical Enquiry info:
What is Critical Enquiry?
The capital “C” in Critical emphasizes social criticism at the most fundamental level of what ought to constitute an ideal just social structure. Enquiry emphasizes the self-conscious use of all forms of analysis and interpretation of actions and discourses that create, maintain, and justify social structures. To this end:
• Critical Enquiry is suspicious of al “isms” offered as The ideal social structure because like all “isms” they purport to transcend human subjectivity, that they are constituted in nature, outside the boundaries of human consciousness.
• Critical Enquiry fully recognizes the political nature of social structures, and seeks to reveal the power embedded in all forms of historically contextualized discourses to condition popular thought to accept a particular ideology, an “ism,” as natural and inevitable. Of particular concern are those “isms” that attempt to justify socioeconomic power differentials as inevitable, as normal.
• Critical Enquiry works dialectically in an unremitting search for contradictions between existing social arrangements and the Enlightenment ideals of natural rights, such as those embodied in the Founding Documents of the United States.
• Critical Enquiry is particularly concerned with those contradictions which systematically exclude individuals and groups from sociopolitical power or from the free access to information that is used to both condition and justify the status quo.
• Critical Enquiry is based on the belief that emancipation comes only to individuals that increase their understanding and self-reflective analysis of their social conditions. Such an analysis depends on the free and open exchange of knowledge and information uncontaminated by authoritative privilege and sanctions. Only after meeting these conditions regarding knowledge can citizens in a democratic society be sufficiently prepared to make ethical and moral judgments. (emphasis in original)
Joanna,
I definitely agree that Wilson’s work comes under the Fazzaro and Walthers umbrella of Critical Enquiry which can be seen as a derivative of Critical Theory**. I like to say that the “E” of Enquiry also stands for education and educational practices.
Duane
And there are a number of folks here who practice Critical Enquiry. You can probably guess the usual suspects, eh! Yourself included!
**Critical Theory has a narrow and a broad meaning in philosophy and in the history of the social sciences. “Critical Theory” in the narrow sense designates several generations of German philosophers and social theorists in the Western European Marxist tradition known as the Frankfurt School. According to these theorists, a “critical” theory may be distinguished from a “traditional” theory according to a specific practical purpose: a theory is critical to the extent that it seeks human emancipation, “to liberate human beings from the circumstances that enslave them” (Horkheimer 1982, 244). Because such theories aim to explain and transform all the circumstances that enslave human beings, many “critical theories” in the broader sense have been developed. They have emerged in connection with the many social movements that identify varied dimensions of the domination of human beings in modern societies. In both the broad and the narrow senses, however, a critical theory provides the descriptive and normative bases for social inquiry aimed at decreasing domination and increasing freedom in all their forms.
In all humility, whether it is this book or others, I urge all supporters of public education and a “better education for all” to inform themselves about Value Added Measurement/Modeling and high-stakes standardized testing.
As a non-math person myself, I can readily understand the reluctance to do so, in part due to the mathematical intimidation employed by the accountabully underlings of the edufrauds. However, don’t despair! Yes, there are some arcane mathematics behind VAM and high-stakes standardized testing, but a great deal of what goes into the tortured numbers and misleading statistics employed by the edubullies is readily understandable to the general public.
First, for complete beginners, go to the websites of deutsch29, Gary Rubinstein, GFBrandenberg, Dr. Bruce Baker, or Jersey Jazzman, and see how they explain—in plain language with a minimum of mathematical jargon—the latest examples of contorted mathematical claims by the edufrauds.
A very recent and excellent example: http://garyrubinstein.teachforus.org/2013/09/20/how-i-teach-2-6-months-more-of-math-in-a-year-than-the-rest-of-you-slackers/
Second, when designing and producing VAM and high-stakes tests there are a number of decisions made about what to include and what to exclude, what to do and what not to do. For example, fundamental to the whole process is defining what you want to measure [ever try to get a roomful of “experts” on anything to agree about everything?]. Then you have to decide if and how you can measure it. Turns out there are no magic feathers, silver bullets, pixie dust, or elixirs of excellence. Just to complicate matters further, there are inevitably trade-offs depending on what you are trying to accomplish. So human judgment is not just critical, but absolutely essential, to the whole process of conceiving, designing, producing, using/administering and evaluating VAM and high-stakes tests.
As if ll that is not enough: such undertakings have some uncertainty built-in!
In the end, you will find that mathematical obfuscation is the evil twin of mathematical intimidation. When you develop the ability to shine a little light on what is obscured by Rheeality Distortion Fields, you won’t need a PhD in math to figure out that someone is trying to run a scam on you.
We still need KrazyMathLady (aka deutsch29 aka Dr. Mercedes Schneider) and Dr. Bruce Baker and the other numbers/stats folks to act like the professionals they are and show us where the accountabullies go off the rails.
But we can do a lot of it ourselves—and then point out to others that the little man behind the curtain is really the ‘Great and Powerful Wizard of Oz.’
Want to know the dirty little secret of the leading charterites/privatizers? Almost to a person they know nothing about the Holy EduMetrics they present to others as the cure-all for what ails education.
In addition to the book mentioned in this posting, I urge everyone to start saving now to get Dr. Mercedes Schneider’s book [mentioned in a comment under another posting today].
Even in this Most Innovative Twenty First Century of Cagebusting EduExcellence, Mark Twain’s advice still holds:
“A person who won’t read has no advantage over one who can’t read.”
🙂
The Mismeausure of Education by Jim Horn and Denise Wilborn draws the same conclusion after an exhaustive history and study of William Sanders’s “Value Added” stuff out of Tennessee.
With its detailed (almost excruciatingly so) legislative history (complete with quotes from legislative debates in Tennessee), The Mismeasure of Education is a tough read, but worth the effort. What it shows is that Sanders — who proclaimed his VAM method as a “paradigm shift” back in the days when everyone was paradigm shifting to prove their genius — was intensely marketing his stuff from the beginning, often to his profit. Every time the stuff was actually measured by any reasonable scholarly or psychometric standards, it was proved to be junk science. I guess that it fits in a long sad history of such stuff going all the way back to the side Tennessee chose during the Civil War, continuing through the Scopes “Monkey Trial”, and continuing to that odious “Confederate Park” honoring Nathan Bedford Forrest…
Don’t get me started.
Have fun reading The Mismeasure of Education along with “… Moving beyond the failure of school reform.”
George Schmidt: I second your favorable remarks about THE MISMEASURE OF EDUCATION (2013) by Jim Horn and Denise Wilburn.
For example, the VAManiacs have been so successful with their multibillionaire-backed campaign of miseducation that even those who oppose their destructive policies have adopted much of their terminology and a bit of their mindset, see pp. 184-185:
One of the most unfortunate and miseducative consequences of the early twenty-first century’s “orgy of tabulation” in schools continues to be the constriction of what is taught and how it is taught so that both match more closely the way that learning is assessed by thoroughly inadequate models. A scan of the educational literature relevant to assessment, in fact, finds the word “learning” often replaced by the psychometrically correct “achievement,” or the more theatrical “student performance,” as is evident in all the critiques above. Sadly, the value of every sort of educational experience, concept, technique, or intervention gets regularly weighed against its capacity for increasing achievement or raising student performance, rather than the more expansive and pragmatic goal of improving student learning. [end of quote]
That said, I caution all readers that the excellence of much of the content is not matched by the editing. It even extends to such distracting annoyances as “(Ravitch, 2011)” [p. 215] when a previous citation [p. 185] and the index indicate that “(Ravitch, 2010)” is intended.
Yes, this is petty, but the book deserved better. And for those like myself who are reading Ravitch 2013, the writing itself and the editing match the high quality of the content.
Lastly, thank you for your efforts on behalf of a “better education for all.”
I think Mark Twain had folks like you in mind when he wrote
“Do the right thing. It will gratify some people and astonish the rest.”
Color me gratified and astonished.
🙂
Test scores do not equal “achievement”