The Los Angeles Times explains today that California has stubbornly resisted Arne Duncan’s demand that teachers be evaluated by junk science.
Despite the fact that researchers overwhelmingly agree that “value-added assessment” is flawed and unstable, that it reflects whom you teach, and that good teachers may be rated ineffective, Duncan blithely insists that it is essential.
Was Duncan successful in Chicago? Is Chicago a national model of school reform? Did Duncan’s Renaissance 2010 create a renaissance?
Why is this man allowed to tell every state in the nation how to evaluate teachers?
How awkward for California Democrat George Miller, one of the lead authors of NCLB, a favorite of DFER, and senior Democrat on House Education committee
Bravo, Governor Jerry Brown!
Bravo, Tom Torlakson.
Stay strong. Don’t let Arne bully you.
Diane Ravitch's blog 
But arbitrary measures are so essential to maintaining arbitrary power that the self-appointed elite can hardly hope to stay in power without them.
You can bet they will keep fighting, tooth and nail, hook and crook …
“But arbitrary measures are so essential to maintaining arbitrary power . . . ”
Twas the beauty of the Stalinist system, eh!
Like, like, like your post Diane. You tell ’em — shame on you George Miller, Obama, Duncan. Shame on you all.
Years ago John Naisbett wrote in Megatrends that what happens in CA spreads eastward… lets hope that is the case…
I wish the other states would follow suit. RTTT is nothing but a bait-and-switch anyway, with the switch far worse than before.
Junk science would imply that VAM’s are wholly inaccurate and not useful. That is certainly not the case because they can provide some useful information that can help schools and educators identify strengths and weaknesses. They just should NOT be used to make high-stakes judgments about teachers or principals. Even at the school-level, they should be interpreted with caution. But they are not junk science, especially when you consider the past metrics states and districts have used to assess “growth” such as changes in proficiency level. Now, those metrics are truly junk.
Junk science is as junk science does.
You have to disassociate measures from their use. A measure can be good for one use and bad for another. Making blanket statements about VAM being junk science and inappropriate for all uses makes us just as guilty as those who believe VAMs are accurate indicators of teacher effectiveness.
“Making blanket statements about VAM being junk science. . . ”
It can’t help but be labeled junk science as the basis for VAM scores are standardized test scores which have been proven to be completely invalid by Wilson. When one starts with falsehoods one, by definition ends up with false conclusion 99.99999% of the time with the other .00001% falling under the category of blind and anosmic squirrel finding an acorn or a more crass version, crap in crap out. See “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700 .
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A quality cannot be quantified. Quantity is a sub-category of quality. It is illogical to judge/assess a whole category by only a part (sub-category) of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as one dimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing we are lacking much information about said interactions.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms shit in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it measures “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
“But they [VAMs] are not junk science, . . . ” Ha, ha, ha ha ha, ha ha, ha, ha, he, he he!!! and ¡¡¡Je, je je je je, je je, je, je, je!!!
Ed, you made me laugh in two languages with that one. Good thing I wasn’t having a Modelo especial or the monitor and keyboard would be soaking.
Reblogged this on David R. Taylor-Thoughts on Texas Education.
So is this a threat?
All states need to pull out…RTT costs more to implement than you receive so why bother?
R E V O L U T I O N !
See here:
“There are elements in the California educational establishment that don’t want any accountability,” he said, and he warned there would be a price to pay.
“The administration is trying to provide relief for districts who want to move to the future,” Miller said. “For districts who don’t want to do anything, they will have to live with the consequences.”
Linda,
George Miller means he is willing to accept the closure if almost every public school in Califirnia if the state doesn’t do what Arne wants. Some representative!
When I was first elected president of my local, I attended a labor breakfast with some other teachers. When they announced the different elected officials, George Miller was the only one in which the people sitting with me got up and booed. We had many people who were not happy with us.
I maintained then as I do now, that these policies were one of the first shots fired in this war against teachers and their unions. I also maintain that this sort of assault is what has led to the attacks on other unions in states like Wisconsin.
He was the congressman who represents the district I work in and I have tried over the course of these past four years to meet with him to discuss the damaging impact of NCLB on our school district. He has refused.
It’s really sad – when the CRS (Congressional Research Service) looked at whether Duncan in fact has the authority to offer waivers, and, as a separate question, whether those waivers could have conditions attached to them, they responded that they believed he could, because the states could choose to not apply for the waiver and not accept the conditions.
However, this comes as a way of forcing them to comply – do it voluntarily or we’ll take you over and force you to do it.
Why doesn’t congress step in and take away some of this authority? How can they allow their local authority to be usurped in this way?
I can not tell you how heavy my heart is about George Miller. 20 years ago he was the champion of teachers and public schools. Now he has nothing but disdain. What happened?
It’s not just about CTA’s influence in our state. Some might forget that California has actually had two Broad Prize winners, Long Beach and Garden Grove, who did NOT use test scores to evaluate their teachers…Chicago, on the other hand, has never even been a finalist for the Broad Prize, which suggests that Arne Duncan doesn’t really know what he’s talking about.
The tone of the article made it sound like California is resisting because of teachers’ unions and Democratic party politics rather than because the proposed reforms are demonstrably bad ideas.
Yes but the Tmes misses the point and once again frames the issue as a labor dispute instead of a reasonable decision based on research and professional expertise.
The Times misses the point and once again frames the issue as a labor dispute instead of a reasonable and informed decision based on research and professional expertise.
Robin,
There are excellent reasons to reject VAM, and the Times should have mentioned these. However reason is not power. Money is power. And fortunately, CTA has enough members to accrue a big war chest that enables it to play ball with the plutocrats. This war chest is imperiled however: year after year the Right puts initiatives on the the ballot that would make it harder for the union to collect dues or use dues for political purposes. And new non-union charter schools are replacing union jobs with non-union jobs: CTA’s membership roster has dwindled by tens of thousands in the last few years (yes, some of this was due to budget cuts and the end of class-size reduction, but charter schools are a significant and worrisome factor, esp. in LA). CTA is one of the last strong tribunes of the common teacher, but it is under constant assault.
VAM is a sham! In most schools, students are tracked, unofficially. Often, the best teachers get the most challenging students. It takes years to close an achievement gap. Any teacher whose students’ scores rise 100+ points must be investigated.
I urge people to click on the link provided by Diane. **Unless otherwise noted, all citations are from the linked article.”
Notice that the only solid lead to “grading the performance of teachers and using those measurements to reward the best teachers and punish the worst” comes under the mention of “standardized tests.”
Now peruse the following:
“Duncan, on a visit to California a few weeks ago, expressed bewilderment that the state could find the plan objectionable.
“I actually use the California model, and not in a good way, as I travel the country,” he said. “There are about 300,000 teachers in California. The top 10% arguably are among the best in the world. The bottom 10% maybe shouldn’t be teaching. No one in California that I have met can tell me who is in that top 10% and that bottom 10%.””
Where would such [improbably] precise figures about the “top 10%” and the “bottom 10%” come from if not from high-stakes standardized tests? And even if other measurements are involved, remember that the criteria used to come up with such figures [not excluding standardized tests themselves!] ultimately rely on human judgments about how much more weight to give certain metrics and how less weight to others.
Now on to Duncan’s recent speech to the 2013 annual meeting of the American Educational Research Association For the full text, go to: http://www.ed.gov/news/speeches/choosing-right-battles-remarks-and-conversation
“State assessments in mathematics and English often fail to capture the full spectrum of what students know and can do. Students, parents, and educators know there is much more to a sound education than picking the right answer on a multiple choice question.
Many current state assessments tend to focus on easy-to-measure concepts and fill-in-the-bubble answers. Results come back months later, usually after the end of the school year, when their instructional usefulness has expired.
And today’s assessments certainly don’t measures qualities of great teaching that we know make a difference—things like classroom management, teamwork, collaboration, and individualized instruction. They don’t measure the invaluable ability to inspire a love of learning.
Most of the assessment done in schools today is after the fact. Some schools have an almost obsessive culture around testing, and that hurts their most vulnerable learners and narrows the curriculum. It’s heartbreaking to hear a child identify himself as “below basic” or “I’m a one out of four.””
Perhaps the Secretary of Education should have heeded the following advice: “If you keep your mouth shut you will never put your foot in it.” [Austin O’Malley]
🙂
This is a huge, huge move by California. If they don’t get smacked down severely by the Dept of Ed, other states will be likely to follow and this entire thing will unravel.
Or the Dept of Ed will just wait it out until the insertion of a poison pill in the new NCLB law that is up for renewal, forcing CA to comply. Then this defiance will mean nothing.
Hopefully its the former and not the latter.
State Superintendent Tom Torlakson is a former teacher. When he gathered a group of educators to hammer out the blueprint for the future of CA schools, he wisely selected classroom teachers to be on the task force. I was honored to be co-chair of the Teacher Evaluation committee.
We believed then and do now that evaluating a teacher via an algorithm is a poor and cheap way to do the hard and time-consuming work of evaluating a teacher effectively. Students deserve more.
I am proud to call Tom Torlakson my leader.
Martha Infante
Teacher
Los Angeles
Very nice work Martha! I’m proud of you and the rest of the task force.
This is a great example of you scratch my back and I’ll scratch yours from the reformers. NCTQ trumps up the Teacher Preparation report and the Texas Public Policy Foundation runs with it as if it was gospel.
http://www.yourhoustonnews.com/pasadena/opinion/golsan-re-thinking-texas-teaching-encouraging-and-rewarding-excellence/article_710acb0c-82d5-553a-9a2e-47a4c728b4fa.html#user-comment-area
NCLB is a bad law that needs to be murdered/killed and making other laws to support a really bad one is silly. California can lead a national revolt against this stupidity. I’m proud to be a part of this state and this revolt.
Why is no one in Congress standing-up against Duncan and Obama? This must be the liberal media’s fault. Maybe if they spent more time on education issues and less time on the socks of King George the First, they’d serve the public. Too bad there are no spines left in the House or in the Senate. Go Jerry Brown!
Most members of Congress have no idea about education. The key members, like Sen. Harkin, have staff from Gates or TFA.
Does this mean that LAUSD is going to give up on this new ridiculous evaluation that seems to have strong connections to Arne’s desired evaluation style?
When I clicked on the link, I hoped to find an article about one state’s courageous decision to take a stand against this insanity of federal Education overreach. What I read was another diatribe about teacher unions.
Pat, you are right about the mainstream media. The only story line that most know how to tell is “bad teachers, bad unions.” They don’t understand that many who are not union members hate what the federal government is doing to our nation’s schools.
We need to get rid of Deasy in L.A.
Good for California! I hope that other states take notice and follow!