Archives for category: Teacher Evaluations

President Obama chose Robert Gordon, who served in key roles in the first Obama administration, as assistant secretary for planning, evaluation, and policy development in the U.S. Department of Education. This is a very important position in the Education Department; he will be the person in charge of the agency that basically decides what is working, what is not, and which way to go next with policy.

 

When he worked in the Office of Management and Budget, Gordon helped to develop the priorities for the controversial Race to the Top program. Before joining the Obama administration, he worked for Joel Klein in the New York City Department of Education.

 

An economist, Gordon was lead author of an influential paper in 2006 that helped to put value-added-measurement at the top of the “reformers” policy agenda. That paper, called “Identifying Effective Teachers Using Performance on the Job,” was co-authored by Thomas J. Kane and Douglas O. Staiger. Kane became the lead adviser to the Gates Foundation in developing its “Measures of Effective Teaching,” which has spent hundreds of millions of dollars trying to develop the formula for the teacher who can raise test scores consistently. Gordon went on to Obama’s Office of Management and Budget, which is the U.S. government’s lead agency for determining budget priorities.

 

The paper co-authored by this triumvirate championed VAM (value-added measurement, i.e., the use of student test scores to judge teacher “effectiveness”) as one of the key policy levers of reform. Here is the abstract:

 

Traditionally, policymakers have attempted to improve the quality of the teaching force by raising minimum credentials for entering teachers. Recent research, however, suggests that such paper qualifications have little predictive power in identifying effective teachers. We propose federal support to help states measure the effectiveness of individual teachers—based on their impact on student achievement, subjective evaluations by principals and peers, and parental evaluations. States would be given considerable discretion to develop their own measures, as long as student achieve- ment impacts (using so-called “value-added” measures) are a key component. The federal government would pay for bonuses to highly rated teachers willing to teach in high-poverty schools. In return for federal support, schools would not be able to offer tenure to new teachers who receive poor evaluations during their first two years on the job without obtaining district approval and informing parents in the schools. States would open further the door to teaching for those who lack traditional certification but can demonstrate success on the job. This approach would facilitate entry into teaching by those pursuing other careers. The new measures of teacher performance would also provide key data for teachers and schools to use in their efforts to improve their performance.

 

This paper, based on economists’ speculation about what works, became a justification often cited for the importance of minimizing teacher certification (“paper qualifications”) and factoring student test scores into teachers’ evaluations, which are a major–if not THE major–component of Race to the Top. The papers’ advocacy of opening the door to uncertified teachers has become a government priority, as shown by Arne Duncan’s award of $50 million to Teach for America (Gordon’s wife worked for TFA), although there is no evidence that TFA can replace the nation’s 3 million teachers and a growing body of evidence that TFA teachers are not more effective than other new teachers or veteran teachers. And since they are usually gone in two years, they have little lasting impact except to increase churn in the teaching staff.

 

Much has happened since Gordon, Kane, and Staiger speculated about how to identify effective teachers by performance measures such as student test scores. We now have evidence that these measures are fraught with error and instability. We now have numerous examples where teachers are evaluated based on the scores of students they never taught. We have numerous examples of teachers rated highly effective one year, but ineffective the next year, showing that what mattered most was the composition of their class, not their quality or effectiveness. Just recently, the American Statistical Association said: “Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.”). In a joint statement, the National Academy of Education and the American Educational Research Association warned about the defects and limitations of VAM and showed that most of the factors that determine test scores are beyond the control of teachers.  Numerous individual scholars have taken issue with the naive belief that teacher quality can be established by the test scores of their students, even when the computer matches as many variables as it can find.

 

 

What we don’t know is this: Has Robert Gordon changed his mind in light of evidence undermining his belief in VAM?

 

Or will the Obama administration continue on its now well-established course, demoralizing veteran teachers, lowering standards for entry-level teachers, dismissing the professional preparation of teachers, and creating new opportunities for the inexperienced, ill-trained recruits of TFA?

 

Having met Robert Gordon and knowing him to be a very smart person, I am betting that he will help the Obama administration change course and inject the wisdom of experience into its policies. That’s my hope.

 

 

Secretary Arne Duncan recently announced his plan to judge teacher education programs by their “results,” including the test scores of the students taught by their graduates. If the Ed Schools can’t produce teachers who can raise test scores, Duncan said, they should go out of business. Spoken like a true businessman.

Mike Rose, celebrated author and professor emeritus at UCLA, has six questions for Arne.

He writes:

Six Questions for Secretary Duncan

1. Will you be evaluating with the same metrics all teacher preparation programs, alternative as well as traditional, Teach for America as well as California State University at Northridge or UCLA?

2. If the Department of Education will use close to $100 million per year on grants to forward its agenda, where will that money come from? From other educational programs that serve needy populations? If so, what services or funding will be cut or discontinued because of this reallocation?

3. Policy formation emerges out of staff research, consultation with experts, and political deliberation. What research and consultation leads you to the current project? I ask because your statement about teacher preparation programs needing to improve “or go out of business” as well as your general approach echoes last year’s report from the National Council of Teacher Quality, a report that has been roundly criticized by a wide range of experts.

4. The National Academy of Education recently issued a comprehensive report on evaluating teacher education programs that recommends an approach very different from yours. Have you read it or consulted its authors?

5. There is an increasing number of respected scholarly organizations—the National Academies Board on Testing and Assessment, AERA, the National Academy of Education, the American Statistical Association—that are advising caution in the use of procedures like value-added to evaluate teacher effectiveness. These organizations point to technical, logistical, and conceptual problems in doing so. One conceptual problem imputing causality between teachers’ activity and a test score, for so many other variables come into play. Your stated plan will use student test scores to not only judge teachers, but also the institutions from which they come, introducing another level of questionable causal attribution in your model. You will have a putative causal chain that goes from the student test score to the teacher to the teacher’s training institution. How do you plan to address this basic conceptual problem?

6. The implication in your plan that bad schools will go out of business assumes that all prospective teachers are the economist’s idealized free agents who can go wherever a highly rated program exists. But a number of prospective teachers from lower income backgrounds do not have the finances to travel—or cannot travel because of family obligations and expectations. How will you address the possible unintended consequence of your program placing burdens on this segment of the population?

Thanks, Mike. If I hear from Secretary Duncan, I will post his answers.

Audrey Amrein-Beardsley has just published a new book that explains value-added measurement (VAM). It is now available for pre-order.

Having been a classroom teacher and now a university scholar, Beardsley takes a highly critical view of simplistic approaches to teacher evaluation.

Here are her chapter headings.

“Paperback: 256 pages; Chapters: 8 and titled as follows:

Socially Engineering the Road to Utopia
Value-Added Models (VAMs) and the Human Factor
A VAMoramic View of the Nation
Assumptions Used as Rationales and Justifications
Test-Based, Statistical, and Methodological Assumptions
Reliability and Validity
Bias and the Random Assignment of Students into Classrooms
Alternatives, Solutions, and Conclusions”

As readers of this blog know, using value-added-assessment is a very poor way to gauge teacher quality. The research and evidence do not support this methodology. Teacher quality cannot be judged by computers. Test scores reflect only a small part of what teachers do. And as the American Statistical Association recently said, teachers account for about 1-16% of the variation in test scores, yet their rating will be heavily influenced by test scores.

Here is a press release about the lawsuit filed in federal court in Houston, challenging VAM:

 

For Immediate Release
May 1, 2014

 

 

 

Contact:
Gayle Fallon, HFT
713/623-8891
Janet Bass, AFT
301/502-5222
jbass@aft.org

Seven Houston Teachers, Houston Federation of Teachers File Lawsuit
Challenging Constitutionality of Value-Added Measure for Evaluations
HOUSTON—Highly recognized, good teachers are receiving poor evaluations because of a grossly flawed value-added algorithm that should be changed, seven Houston teachers and the Houston Federation of Teachers said today in an unprecedented lawsuit filed in U.S. District Court for the Southern District of Texas.
 
The lawsuit details numerous problems with the Houston Independent School District’s Education Value-Added Assessment System, or EVAAS. Its statistical methodology uses a student’s performance on prior standardized tests to predict academic growth in the current year, though what is considered a sufficient level of growth is not defined. A teacher’s EVAAS score is supposed to measure the effect, or added value, of a teacher on a student’s academic growth over the school year. The school district uses this deeply flawed methodology for decisions about teacher evaluation, bonuses and termination, yet it is a “black box” system in which the methodology is considered proprietary and confidential.
 
“Due to a faulty, incomprehensible and secret formula, good teachers like the ones filing this suit are being labeled failures and our entire education system is being reduced to a numbers game,” said American Federation of Teachers President Randi Weingarten. “Testing isn’t aligned with the purposes of public education. It doesn’t measure big-picture learning, critical thinking, resilience, creativity or curiosity, yet those are the qualities that great teaching brings out in a student. The fixation on testing has literally drained the joy out of learning. We’ve always been leery of value-added models, and we have enough evidence to make clear that not only has VAM not worked, it’s been really destructive and it in no way helps improve teaching and learning.”
 
Daniel Santos, an award-winning sixth grade social studies teacher at Jackson Middle School, said the evaluation system is failing him, his students and his profession.

“It’s dispiriting and insulting to be told I’m ineffective, a judgment that doesn’t mesh with my classroom performance or the time and effort I devote to my students. Texas is using a broken evaluation system that isn’t properly identifying who really needs help to improve,” Santos said, adding, “My students are being tested on material that is not aligned with our curriculum.”

While there have been other suits challenging VAM, the Houston suit is unique because it was brought by highly recognized teachers who contend their poor EVAAS ratings do not correlate with their actual performance nor take into account socioeconomic or demographic variables in predicting student performance. The complaint also charges that the district directed and/or pressured school administrators to “manufacture deficiencies or otherwise find fault with the instructional practices” by teachers who received low EVAAS ratings. Those teachers would be placed on a growth plan to correct the supposed faulty instructional practice.
 
Myla Van Duyn, who teaches ninth grade biology at Davis High School, received an unwarranted low EVAAS score and contends her classroom observation scores were artificially lowered to be aligned with the EVAAS scores.
 
“This is demoralizing,” said Van Duyn, who is leaving the school district at the end of the school year, a decision motivated in part by the flawed evaluation system.

“EVAAS is driving out great Houston teachers because they’d rather work in a place that respects teachers. Everybody can do better, but the EVASS method does not accurately account for all of the variables that go into a child’s ability to answer questions on a multiple choice test,” Van Duyn said.
 
Andy Dewey, another plaintiff, is a history teacher at Carnegie Vanguard High School who developed the curriculum for two history courses and has been consistently rated well by his administrators. “Houston’s evaluation system is sold as a system to support and strengthen teaching, but it’s actually a bait and switch sham that’s weakening instruction and not helping teaching or learning at all,” Dewey said. “Teachers are told their scores are low but are not given information about what they did or did not do to cause their students to perform less than predicted.”
 
Gayle Fallon, president of the Houston Federation of Teachers, said: “The district is using a woefully inaccurate, educationally destructive evaluation system. If a car had as many design defects as the EVAAS system, it would be recalled as a lemon. EVAAS should be replaced with a fair and meaningful system that will actually help improve teaching and learning.”
 
AFT’s Weingarten said the real culprit is school districts’ fixation on testing, noting none of the top-performing countries subjects their students to as much testing as the United States.
 
“The testing obsession has turned kids into test scores and teachers into algorithms. Rote memorization and testing don’t prepare our students for the 21st century. We need to help kids problem solve, think critically, learn to be persistent and work in teams,” Weingarten said. “This country has spent billions on accountability, not on the improvement of teaching and learning at the classroom level, and value-added models are the leading edge of this misguided effort,” she said.
 
Weingarten said districts should develop, with teachers, a comprehensive evaluation system based on multiple measures that accurately identifies teachers needing improvement and that provides targeted help. To reclaim the promise of public education, she said there should be safe and welcoming neighborhood schools, access to early childhood education, wraparounds services at schools to address health, social and emotional issues, and project-based instruction.

Weingarten first questioned the fairness and accuracy of value-added metrics in 2007. It also has been criticized as inaccurate and an unstable measure of teacher performance by, most recently, the American Statistical Association, as well as RAND Corp. researchers, the National Academy of Sciences and the Economic Policy Institute. The ASA said the majority of the variation in test scores is attributable to factors outside the teachers’ control.
 
The federal lawsuit states teachers’ due process constitutional rights are being violated because the EVAAS system is not an accurate or reliable indicator of teachers’ performance and because “a cloak of secrecy” prevents teachers from verifying or challenging their EVAAS score. The suit also contends teachers’ equal protection constitutional rights are being violated because the school district directs and/or pressures school administrators to align teachers’ instructional practice scores with their EVAAS scores—two separate evaluation components—so those with below-average EVAAS scores receive arbitrary, harsher instructional practice scores. The suit also contends that the standards for acceptable growth are arbitrary, vague and constantly being recalibrated.
 
In addition to the Houston Federation of Teachers, Daniel Santos, Myla Van Duyn, and Andy Dewey, the other plaintiffs are Ivan Castillo (a fourth-grade bilingual teacher who has taught at  Briscoe Elementary School since 1994, was the school’s bilingual teacher of the year in 2013-14 and is a finalist for bilingual elementary teacher for the Houston Independent School District) ; Paloma Garner (a ninth-grade biology teacher at David High School, has received several national awards recognizing her mentoring skills and has a consistent record of strong performance); Araceli Ramos (a ninth-grade English teacher at Austin High School who was selected in 2014 by a school official to tutor teachers on the Texas English Language Proficiency Assessment System) and Joyce Helfman (an eighth-grade English teacher at Johnston Middle School who has had a consistent record of strong performance and is well respected by her colleagues).

 

 

Here is the story of the Houston Seven, the teachers suing to invalidate the evaluations based on student tests scores.

How nutty is this?

“Andrew Dewey is an award-winning history teacher at Carnegie Vanguard High School in Houston. In 2011-12, he earned the top merit pay award that his school district gives out and had “most effective” teacher status through a controversial evaluation system that uses student standardized test scores. The next year, after teaching similar students in the same way, he went from being one of the district’s highest-performing teachers to one that made “no detectable difference” for his students.

“Dewey is one of seven high-achieving teachers who, along with the Houston Federation of Teachers, filed a lawsuit in federal court in Texas late Wednesday alleging that the Houston Independent School District uses a badly flawed method of evaluating teacher effectiveness, known as the “Educational Value-Added Assessment System.” The teachers argue that the EVAAS is inaccurate and unfair but that it still plays a large role in determining how much teachers are paid and whether they can keep their jobs.
The method, generically known as “value added measures,” or VAM, is increasingly in use around the country — with the support of the Obama administration — after Michelle Rhee pioneered the method when she ran D.C. public schools several years ago. The result of this lawsuit could affect evaluation systems well beyond Texas.”

Just think: if the Houston teachers win, and the evidence is on their side–they take down the central theory of Race to the Top and Rhee, as well as laws in dozens of states that will face similar lawsuits.

“Randi Weingarten, the president of the American Federation of Teachers, said that the obsession with standardized testing that has driven education policymakers to make standardized test scores the key metric of accountability for students, educators and schools, is bastardizing public education.

“This country has spent billions on accountability, not on the improvement of teaching and learning at the classroom level, and value-added models are the leading edge of this misguided effort,” she said.”

Start with Race to the Top. $5 billion wasted.

Seven teachers in Houston are suing the district over the use of test-score-based evaluations.

Good for them!

As a K-12 graduate of HISD, I am proud of these teachers for standing up for their profession.

I hope they will introduce as evidence the recent statement of the American Statistical Association cautioning about the limitations of VAM, as well as the joint statement of the National Academy of Education and the American Educational Research Association, warning that VAM produces results that are inaccurate and unstable.

Here is a good list of references the plaintiffs can use.

VAM is junk science when used to rate individual teachers. The ratings change if a different test is used. VAM says more about the composition of the class than the quality of the teacher.

Jersey Jazzman heard NPR describe the reason that Washington State refused to bow to Arne Duncan’s demand that the sate use test scores to evaluate teacher quality.

It wasn’t because the methodology has no evidence behind it.

It wasn’t because the method has been questioned by theNational Academy of Education, the American Educational Research Association, the American Statistical Association, and leading scholars.

No, Washington State said no to our omnipotent, omniscient Secretary of Education because of those terrible unions who are afraid of being evaluated.

Or could this explain NPR’s rationale:

“So, as I was sitting at the kitchen table this evening, my ears perked up at the 5:30 break for WNYC, the NPR outlet here in the greater New York area. The announcer let us know that All Things Considered was proudly sponsored by the Walton Family Foundation, which was supporting (I’m paraphrasing here) educational “choice” for families.”

– See more at: http://jerseyjazzman.blogspot.com/2014/04/corporate-education-reform-buys-public.html#sthash.DkIcHS3q.dpuf

Anthony Cody points out that for the past dozen years or so, Bill Gates has had his fun experimenting with education reform. Obsessed as he is with measurement and data, he imagined that he could impose his narrow ideas on American public schools and bring about a magical transformation.

Does American education need reform and improvement? Absolutely. Stuck as it is in the paradigm of testing and punishment, it sorely needs a revival of humanism and attention to the needs of children, families, and communities. It needs teachers who are well-prepared. It needs a recommitment o the health and happiness of children and to a deeper love of learning.

Yet Gates used HS vast wealth to steer national policy to the dry and loveless task of higher scores on tests of dubious value.

He wanted charter schools, and Arne Duncan, his faithful liege, demanded more charter schools,even if it was central to the Republican agenda.

He wanted national standards and quite willingly paid out over $2 billion to prove that one man could create the nation’s academic standards by buying off almost every group that mattered.

He wanted teachers to be evaluated based on test scores, and Ducan gave that to him too.

But says Cody, everything failed.

Cody writes:
.

“Last September Bill Gates said,

“It would be great if our education stuff worked, but that we won’t know for probably a decade.”

But, says Cody,

“I think we already know enough to declare the experiment a failure.

Value Added is a disaster. Any “reformer” who continues to support giving significant weight to such unreliable indicators should lose any credibility.

“Charter schools are, as a sector, not better than public schools, and are expanding segregation, and increasing inequality.

“The Common Core and the high stakes accountability system in which it is embedded is on its way to the graveyard of grand ideas.

“The only question remaining is how long Gates and his employees and proxies will remain wedded to their ideas, and continue to push them through their sponsored advocacy, even when these policies have been proven to be ill-founded and unworkable.

“Part of the problem with market-driven reform is that when you introduce the opportunity to make money off something like education, you unleash a feedback loop. Companies like the virtual charter chain K12 Inc can make tremendous profits, which they can use to buy off politicians, given our Supreme Court’s “Corporations are people and money is speech” philosophy. There are no systemic brakes on this train. The only way turn this around is for people to organize in large enough numbers, and act together in ways that actively disrupt and derail the operation.

“Along those lines, activists in Seattle are organizing a demonstration on June 26th, protesting the Gates Foundation at their headquarters. It has been a year and a half since I engaged the Gates Foundation in dialogue. Given the rather poor aptitude for learning Gates and company have shown, I will be joining this protest, and perhaps if enough of us are there, we can take the dialogue to the next level.”

Reader Michael Fiorillo deciphers the corporate reformers’ game plan:

The Final Solution to the Teacher Question:

– Proclaim austerity for the public schools, while continuing to expand charters.

– Create incentives for non-educators to be in positions of power, from Assistant Principal on up.

– Maintain a climate of scapegoating and witch hunting for “bad teachers,” who are posited as the cause of poverty and student failure, doing everything possible to keep debate from addressing systemic inequities.

– Neutralize and eventually eliminate teacher unions (the first part largely accomplished in the case of the AFT). As part of that process, eliminate tenure, seniority and defined benefit pensions.

– Create and maintain a climate of constant disruption and destabilization, with cascading mandates that are impossible to keep up or comply with.

– Create teacher evaluations based on Common Core-related high stakes tests for which no curriculum has been developed. Arbitrarily impose cut scores on those exams that cast students, teachers and schools as failing, as was done by NYS Education Commissioner John King and Regent Meryl Tisch.

– Get teachers and administrators, whether through extortion (see RttT funding) threats or non-stop propaganda, to accept the premises of “data-driven” everything, even when that data is irrelevant, opaque, contradictory, or just plain wrong.

– Get everyone to internalize the premises and language of so-called education reform:

– Parents are not citizens with rights, but “customers” who are provided “choices”
that are in fact restricted to the decisions of those in charge, based on policies
developed by an educational industrial complex made up of foundations,
McKinsey-type consultants and captive academics.

– Students are “valuable assets” and “products,” whose value is to be enhanced
(see the definition of VAM) before being offered to employers.

– Teachers are fungible units of “human capital,” to be deployed as policy-makers
and management see fit. Since human capital depreciates over time, it
needs to be replaced by fresh capital, branded as “the Best and Brightest.”

– Schools are part of an investment “portfolio,” explicitly including the real estate
they inhabit, and are subject to the “demands” of the market and the preferences
of policy-makers and management.

– Create an intimidating, punitive environment, where the questions and qualms are either disregarded or responded to with threats.

– Get the university education programs on board under threat of continuing attack. Once they are on board, go after them anyway, and deregulate the teacher licensing process so that it’s easier to hire temps.

– Eliminate instruction that is deemed irrelevant to the most narrowly-cast labor market needs of employers, getting rid of art, music, dance, electives, etc., thereby reducing the focus of education to preparation for passive acceptance of low-wage employment.

– Embed software and electronic gadgets in every facet of the classroom and school, from reading to test-taking, with the intention of automating as much classroom input and output as possible.

– Use the automation of the classroom to enlarge class size – something explicitly promoted by Bill Gates – and transform teachers into overseers of student digital production that is connected to massive databases, so that every keystroke is data to be potentially monetized.

– Cash your bonus checks, exercise your stock options, and declare Excellence and Civil Rights achieved.

Despite the fact that major scholarly organizations have debunked value-added measurement as a way of identifying and quantifying teacher quality, there are still a few lonely defenders of VAM.

There is the U.S. Department of Education, which bet nearly $5 billion on VAM.

There is the Gates Foundation, which has bet hundreds of millions on VAM.

There are stragglers here and there.

And then there is the Center for American Progress, which says that despite all the research to the contrary, they are sticking with VAM.

Just a few weeks ago, the American Statistical Association stuck a pin in the VAM bubble.

The National Academy of Education and the American Educational Research Association had earlier expressed their skepticism about the utility of VAM.

Nonetheless, the CAP still wants to believe. They really truly want to believe, no matter what the statisticians and researchers say.

Probably they are just showing their loyalty to Arne Duncan and the Obama administration.

So Audrey Amrein-Beardsley decided to stick a pin in CAP’s ideological bubble. 

She writes:

Their research is notably a small subset of the actual research out there on VAMs, research that was used to rightfully construct the aforementioned position statement released by the ASA, and research that for decades has evidenced that teachers account for, or can be credited for, approximately 10% of the variance in student test scores, while the other 90% is typically due to factors outside of teachers’ control.

Regardless, while the Center for American Progress briefly acknowledges this, they spin this into their solution: The reason this percentage is so low is because we have not yet been accounting for growth in student achievement over time; that is, via value-added models (VAMs). In other words, using more sophisticated models of measurements (i.e., VAMs) will help to illuminate the “real” results we know are out there, but simply have not been able to capture given our archaic models of measurement and teacher accountability.

Not to worry, though, as they write that these “[n]ew measures of teacher effectiveness, determined by evidence of teacher practice and improvements in student achievement, are now available [emphasis added] and provide strong markers for assessing teaching quality and the equitable distribution of the most capable teachers.”

CAP wants to believe in VAM, therefore it does believe in VAM, no matter what the evidence may show.

This should be laughable but this skit, she says, is not funny.