Update: an earlier post reported that UTLA would accept evaluations based on test scores.
I relied on a story in the Los Angeles Times.
The lawyer for UTLA informed me this is not true.
When I have more details, I will post them.
Archives for category: Teacher Evaluations
A story in the Los Angeles Times says that the United Teachers of Los Angeles has agreed to permit test scores to be part of teachers’ evaluations.
This is in response to a lawsuit brought by EdVoice on behalf of anonymous parents. EdVoice is one of those organizations funded by the Broad Foundation, the Walton Foundation and other members of the billionaire boys’ club who will never leave teachers alone until they teach to the test.
I hope this is not true. As we have seen again and again, judging teachers by the test scores of their students is harmful to the quality of education as it places too much emphasis on testing. It incentives narrowing of the curriculum, teaching to the test, cheating, gaming the system, score inflation.
Value-added modeling, which would be used here, is junk science.
Even Eric Hanushek, the favorite economist of the VAM crowd, says that teachers account for only 7 1/2-15 percent of the variation in students’ test scores.
What about the 60 percent that is usually attributed to the influence of family, especially family income?
If Los Angeles goes down this path, it may well fire the wrong teachers (Houston fired one of its teachers of the year based on VAM data).
Surely there is a better, more constructive way to evaluate teachers than to rely on unstable and inaccurate measures.
Yesterday, Secretary of Education Arne Duncan had a story in the Huffington Post extolling his work in building respect for the teaching profession.
He has accomplished this, he says, by insisting that teachers be evaluated based on the test scores of their students.
Exhibit A of his success, he says, is Tennessee. Mr. Duncan relies on a report by Kevin Huffman, the state commissioner of education (former PR director for TFA, now employed by one of the nation’s most conservative governors).
The report says that since Tennessee won Race to the Top funding in 2010, it has seen remarkable results because it is now using test scores as 50% of teachers’ evaluations.
Leave aside for the moment the fact that leading researchers (like Linda Darling-Hammond of Stanford University and the National Academy of Education and the American Educational Research Association) say that these value-added measures are inaccurate, unreliable, and unstable.
It is simply bizarre to boast about a one-year change in state test scores. It has long been obvious that state test scores are less reliable than NAEP and that any real change requires more than one year of data as evidence of anything.
According to NAEP, the scores for Tennessee in both reading and math were flat from 2009-2011. Perhaps Secretary Duncan should wait for the release of the 2013 NAEP before boasting about the dramatic gains in Tennessee.
In the meanwhile, I urge Secretary Duncan and his staff, and Commissioner Huffman, to read the joint statement of the National Academy of Education and the American Educational Research Association on value-added testing and its misuse in evaluating teachers. It is called “Getting Teacher Evaluation Right.” I am sure that the Secretary agrees that policy should be informed by research.
Here is the executive summary:
Consensus that current teacher evaluation systems often do little to help teachers improve or to support personnel decision making has led to a range of new approaches to teacher evaluation. This brief looks at the available research about teacher evaluation strategies and their impacts on teaching and learning.
Prominent among these new approaches are value-added models (VAM) for examining changes in student test scores over time. These models control for prior scores and some student characteristics known to be related to achievement when looking at score gains. When linked to individual teachers, they are sometimes promoted as measuring teacher ―effectiveness.‖
Drawing this conclusion, however, assumes that student learning is measured well by a given test, is influenced by the teacher alone, and is independent of other aspects of the classroom context. Because these assumptions are problematic, researchers have documented problems with value-added models as measures of teachers‘ effectiveness. These include the facts that:
1. Value-Added Models of Teacher Effectiveness Are Highly Unstable: Teachers‘ ratings differ substantially from class to class and from year to year, as well as from one test to the next.
2. Teachers’ Value-Added Ratings Are Significantly Affected by Differences in the Students Who Are Assigned to Them: Even when models try to control for prior achievement and student demographic variables, teachers are advantaged or disadvantaged based on the students they teach. In particular, teachers with large numbers of new English learners and others with special needs have been found to show lower gains than the same teachers when they are teaching other students.
3. Value-Added Ratings Cannot Disentangle the Many Influences on Student Progress: Many other home, school, and student factors influence student learning gains, and these matter more than the individual teacher in explaining changes in scores.
Other tools have been found to be more stable. Some have been found both to predict teacher effectiveness and to help improve teachers’ practice. These include:
- Performance assessments for licensure and advanced certification that are based on professional teaching standards, such as National Board Certification and beginning teacher performance assessments in states like California and Connecticut.
- On-the-job evaluation tools that include structured observations, classroom artifacts, analysis of student learning, and frequent feedback based on professional standards.
In addition to the use of well-grounded instruments, research has found benefits of systems that recognize teacher collaboration, which supports greater student learning.
Finally, systems are found to be more effective when they ensure that evaluators are well-trained, evaluation and feedback are frequent, mentoring and coaching are available, and processes, such as Peer Assistance and Review systems, are in place to support due process and timely decision making by an appropriate body.
And here is a short summary of the report by Linda Darling-Hammond.
A teacher faces the state’s new evaluation with anxiety, uncertainty, fear, and confusion. She has a great idea about how to game this invalid, meaningless, absurd and punitive system:
It’s not just in LA that teachers are afraid. VA is implementing new eval requirements that must include student progress worth 40% of a teacher’s overal eval. The plan at my former school (might be at the new school I will be working for in the fall as well) was to develop pre and post tests to show growth over the year for non-tested subjects (and first year of a subject wouldn’t count, since obviously students would be starting at zero. I wasn’t really sure about the logic there). Yet, I was unable to ever get clear answers on how they would be implented for teachers of multiple levels/preps. One idea was that only one level would count. After I pointed out that this would incentivize focusing on only the level that counted to the detriment of the non-counted levels, I was met with silence. When I asked who was picking, I was also met with confusion. Moreover, it also occurred to me how easy it would be for some teachers to game the system. I teach Latin, which is something that most people have little familiarity with. So, I could create an insanely difficult pre-test and a cake post test so that I showed awesome growth, and really who would know, except me, and maybe the kids (certainly those of my bosses who have no familiarity with the subject would not be able to figure it out). Moreover, they wouldn’t be able to break down how students do on partiular sections and what that means, nor figure out what sections represented higher level thinking and which represented rote memorization. Finally, I began to question how one evaluated the ability to translate Latin to English by only using a multiple choice test. I wondered why the ability to translate Latin to English wasn’t measured by actually having students translate Latin to English. For the first time, I felt the fear and the pressure that teachers of the core had been feeling for years. I dread these new eval requirements with a sick feeling in the pit of my stomach that never quite goes away.
I try to be careful in terms of what I write and what I repost from readers. If readers express their personal experiences and strong views, I have no problem with that, and I never repost anything that uses epithets or goes beyond the bounds of civility or fair disagreement. If I err, it is not knowingly.
I just received a comment about Tennessee’s evaluation system and its provenance. This was in response to my post today about “legal fraud.” I would appreciate readers’ responses, not opinions, but citations and facts:
| TN bought TEAM/TAP, a teacher evaluation system from a Milken owned company called NIET. http://www.tapsystem.org/about/about.taf?page=nietbio_lmilken The 1-5 scoring rubric met the requirement in Race to the Top to evaluate teachers with “objective” measures.The Milkens have been marketing TEAM/TAP since the mid 1990’s. If it is so effective, shouldn’t there be dozens of studies replicating its success? There are none. Peer-review constrains bad science and practices, and protects those from harm who are subjected to its application. It’s not an overstatement to say that, absent critical review, the TEAM evaluation can identify teaching quality about as well as rolling dice. That the Milken family foundation has bypassed peer review and critical analysis and sold its “product” to the taxpayers is very curious. This use of our public funds deserves further scrutiny. Recall that co-founder Michael Milken is a convicted felon. Rudy Giuliani successfully prosecuted him for a massive fraud in 1989 that destroyed his company and cost the taxpayers millions in the ensuing cascade of savings and loan failures. Milken’s crimes were so egregious, president G.W. Bush refused to give him a presidential pardon. Caution and study here would seem wise. http://articles.latimes.com/2009/feb/03/business/fi-milken3 Outsourcing to contractors with a history of fraud has the potential to be a spectacular boondoggle. Our students will be the victims. Who will be held accountable? |
After my last book was published, I did some radio interviews and got some interesting feedback.
One of the most informative responses came from a distinguished professor emeritus at the University of Michigan, Harry Frank, who has written textbooks about measurement and evaluation.
His observations about testing and evaluation were brilliant. What he wrote helped me understand why NCLB had failed. As I re-read this letter, I understood better why Race to the Top will fail. For one thing, it assumes that the same tests may be used both to evaluate the teacher and to counsel the teacher. What this does, he says, is to promote cheating and teaching to the test.
And Professor Frank explains why student evaluations distort the educational process.
Professor Frank gave me his permission to reprint his letter.
|
.
|
|
I am by training a social psychologist, with a subspecialty and one-time consulting practice in testing and measurement. When the Flint campus sought its first accreditation independent of the main (Ann Arbor) campus, the provost established an ad-hoc committee to develop assessment procedures. I spent nine years on the committee, my last couple as its chair. The procedures we developed became something of a model for the North Central Association of Schools and Colleges. It has worked extremely well precisely because it conformed to some very fundamental principles of validation, which No Child Left Behind blatantly (if not intentionally) ignored.
The first principle is that no assessment can be used at the same time for both counseling and for administrative decisions (retention, increment, tenure, promotion). As you emphasized (and as every organizational psychologist with an ounce of brains wailed when No Child was first described), all this does is promote cheating and teaching to the exam (much as does the staatsexamen in Germany). This principle is so basic that it’s often covered in the very first chapter of introductory texts on workplace performance evaluation. Accordingly, in the very first meeting of the committee, we established an absolute firewall. Department chairs, deans, and executive committees would never be permitted to see individual raw data; they would see only departmental pooled data. This action did not immediately eliminate faculty resistance, but it went further in that regard than even you might imagine. The same should apply to K-12 teachers’ unions. Like you, I don’t think the problem is testing–any more than the problem with a badly built house is with the hammers and saws. The problem in both cases is how potentially useful tools should be used. Many of the current difficulties would be reduced or eliminated if it were clear that (1) K-12 education is a developmental process, so assessment in schools is a developmental measure not a terminal measure. The concern should be with change not simply “score.” (2) Assessment should be a counseling resource, not a source of extrinsic motivation, i.e., rewards and punishments for teachers, administrators, and school districts. (3) Student evaluations are worse than useless; they are egregiously misleading. A 10-year study by the American Psychological Association indicated that student evaluations are correlated with only two factors: i. Students’ expected course grades compared with their expected grades in other courses. ii workload (negative correlation). For untenured faculty, course evaluations–if used for administrative decisions–therefore have the effect of motivating both grade inflation and the dumbing down of course content. (4) Instruments and procedures must be national in scope and standardized in their administration and reportage (cf. your interview comments concerning the superior validity of the national examination vs. state examinations). (5) Data should be clustered rather than pooled. That is, performance of mainstream students, students whose first language is not English, and developmentally disabled students should be examined separately. It is clearly inappropriate to compare overall scores for students in, say, Birmingham, Michigan, where an overwhelming majority are native speakers of English, with students in Taos, New Mexico, where English as a first language falls behind both Spanish and Tiwa. (6) Teachers should never have access in advance to test questions or even precise content. They should be given global guidelines–general areas in which student competence is expected. (7) Ideally, the procedures should make no attempt to be exhaustive. They should represent a random sampling of content, and the sample should change annually so that past tests cannot be used to prep students but can and should be used to familiarize students with the form of the questions, the level of detail expected, and so on. |
Since Arne Duncan became Secretary of Education and unleashed the Race to the Top, almost every state has adopted laws to evaluate teachers by the test scores of their students. Most teachers know that this is unfair because the factors that have the greatest influence on students’ test scores are not within the control of teachers. Reformers tell us that teachers are the most important influence within the school on student scores, and that is right. But the teacher contribution to scores is dwarfed by the influence of family and other out of school factors.
It is also obvious to everyone but the U.S. Department of Education that when testing becomes the determinant of teachers’ evaluation, their reputation, and their careers, the results are predictable: narrowing of the curriculum, teaching to the test, gaming the system, and cheating. None of this improves education. Why would any responsible public official want to promote such behavior?
The eminent mathematician John Ewing, who is the president of Math for America, wrote a concise and slashing attack on the misuse of mathematics in value-added methodology. He writes about how teachers in Los Angeles were bullied by journalists who ranked them and then confronted them with their low scores. The journalists warned that value-added should not rely on a single measure, but they themselves relied on a single measure to create their rankings.
Ewing says that the public is being subjected to “mathematical intimidation” by policymakers and education “experts,” and that mathematicians have a duty to speak out and tell them to stop misusing their field for political ends.
Indiana is one of the states where the governor and the state commissioner of education seem determined to put public education out of business. They are implementing vouchers, expanding charters, and given the green light to for-profit online charter schools. They do not have a shred of evidence that any of this will improve the education of children in Indiana, but that doesn’t slow them down. They are in love with the ideology of choice and competition and the glories of the marketplace, and that’s the end of the discussion. Plenty of entrepreneurs will get rich off taxpayers’ dollars in Indiana.
Fortunately, there is strong resistance from parents and educators in Northeast Indiana. When I spoke in Indiana last fall, I met some of the parent leaders. They were in despair about the destructive policies being pushed through the legislature. I am glad to say that they organized and are speaking out. They can serve as a model for other concerned citizens.
They have drafted a statement in opposition to what Governor Mitch Daniels and State Superintendent Tony Bennett are doing. They not only oppose these harmful policies, but they offer a platform describing the positive steps that must be taken to save public education in the state of Indiana.
Congratulations to these courageous, thoughtful, and concerned citizens of Indiana!
I hope that others will take this statement of principles and adapt it to their own community and state. Help it go viral, as the Texas anti-high-stakes testing resolution has gone viral. Join with your friends and neighbors to awaken the American public to support good education policies that strengthen our public schools and our democracy.
A reader writes in response to the post about New Jersey Governor Christie:
When you solve this mystery please come and help us in Ohio to uncover why Gov. Kascich has made teachers public enemy number one. We may have defeated his infamous House Bill 5, but he and his cronies are managing to slip in most of the laws and regulations in the back door .
Are there any Republican governors who are not at war with the teachers in their state? If so, please let me know. Maybe something happened at the Republican Governors’ conference in 2011.
A reader from Pennsylvania asks whether charter schools are public schools if they seek to avoid transparency and if their teachers are not subject to the same evaluation scheme as public school teachers:
Charters insist on being called “public” schools.
Yet in Pennsylvania charters are in court trying to prevent laws requiring them to be transparent about their operations, as public schools are required to do.
The state legislature just passed a law requiring 50% of teacher evaluations to be based test scores. The law EXEMPTS charter teachers from this new evaluation system.
In the ALEC rush of legislation at the close of its session last week, a bill was introduced in the PA legislature to EXEMPT charters from the state’s Sunshine Law which requires public institutions receiving state money to be transparent about their contracts. It received 120 favorable votes in the House and failed by a few votes in the Senate.
In Philadelphia we have a charter operator, Universal, which was given Audenreid High School, which was made a charter as soon as a new facility was built at tax payer expense, operating for the past year rent and maintenance cost free. Next year they will have to pay $500,000 which just a quarter of the expense for rent and maintanence. The SRC will cover the rest. This is in a School District which has a $265 million deficit, plans to close 65 public schools over the next few years, and is threatening to unilaterally cut the wages and benefits of public school employees.
So I take back what I said at the beginning of this thread. Charter schools are not open to public scrutiny.
