Archives for category: Teacher Evaluations

We now know that William Sanders, the creator of Value-Added Measurement, which measures teacher quality with the methods of an agricultural statistician, is revered in the think tank world of D.C.

But Peter Greene, who teaches in a Pennsylvania high school, doesn’t think much of VAM.

He recounts how Sanders was inspired to think about measuring teachers based on his studies of radioactivity in cows that were downwind from a nuclear explosion (really!).

To which Sanders writes:

Oh, let’s tell the truth. VAM systems have also been limited by the fact that they’re junk, taking bad data from test scores, massaging them through an opaque and improbable mathematical model to arrive at conclusions that are volatile and inconsistent and which a myriad educators have looked at and responded, “Well, this can’t possible be right.”

You’ll never find me arguing against any accountability; taxpayers (and I am one) have the right to know how their money is spent. But Sander’s work ultimately wasted a lot of time and money and produced a system about as effective as checking toad warts under a full moon– worse, because it looked all number and sciencey and so lots of suckers believed in it. Carey can be the apologist crafting it all into a charming and earnest tale, but the bottom line is that VAM has done plenty of damage, and we’d all be better off if Sanders had stuck to his radioactive cows.

The New York Times published a tribute by Kevin Carey of the New America Foundationto William Sanders, “the little-known statistician who taught us to measure teachers.”

One hates to speak ill of the dead, but accuracy requires that we note that Sanders’ statistical model for “measuring” teachers was flawed, inaccurate, and damaged the lives of thousands of teachers based on Sanders’ obscure algorithms. Sanders was an agricultural statistician before he found a goldmine in education. Measuring teacher quality really is not akin to measuring cattle or crops. Every analysis of the influences on students’ test performance gives far more weight to family income and education than to the teachers who see her or him for an hour or five hours a day. Sanders tried to remove human judgment from the equation and ended up creating a profitable business that distorted teaching and learning into a struggle for higher test scores. If the tests themselves are invalid, then any accountability measures based on them will be invalid.

No one knows William Sanders’ works and its flaws better than Audrey Amrein-Beardsley. She has studied Sanders’ value-added measures for years and testified against them in court. She comments on the New York Times’ article here. Amrein-Beardsley points out that Sanders’ methods have been faring poorly in court because it is unfair to judge a teacher based on a mysterious algorithm that no one can understand or explain.

In my book Reign of Error, I wrote about the fallacy behind Sanders’ reasoning by quoting a song from “The Fantasticks.” I paid $1200 for the right to reprint the lyrics. It is the one that goes “Plant a radish, get a radish, not a sauerkraut./That’s why I love vegetables, they know what they’re about.”

No one can say the same about children. Children from the same parents are different, even when their upbringing is as identical as those parents can make it. They look different, they act different, they have different interests, they have different goals.

Sanders never understood that.

The blog poet makes the assertion that economists are like weathermen, but in this case, I disagree respectfully. Economists of education express a certainty that no weatherman would. Weathermen (meteorologists) give you different scenarios, warn you that the track of the storm might shift, recognize that there are many uncertainties. Economists of education (not all of them! Think Helen Ladd of Duke) claim that they can use test scores to measure teacher effectiveness, and that they have taken all contingencies into account. Hundreds, probably thousands, of educators have lost their careers because of the specious claims of economists like Raj Chetty, who assert that they can judge the value-added of teachers by the test scores of their students, that third-grade teachers affect the life-time earnings of their students, and that teachers who don’t hit arbitrary test scores marks should be fired, sooner rather than later.

“Economists are like weathermen”

Economists are like weathermen
This cannot be denied
Cuz if, by chance, they get it right
It’s greatly AMPLIFIED!!

But mostly, they just get it wrong
And utter not a word
For them to actually point this out
Would really be unheard

And when their goof’s so blatant
They really can’t ignore it
They simply claim they found a “flaw”
But “markets” will restore it

This story appeared on the blog of Patrick Hayes’ EdFirstSC.

In Charleston, South Carolina, Principal Jake Rambo was ordered to evaluate his teachers based solely on the test scores of their students. Not multiple measures. Standardized tests.

He refused.

He was told that he was being transferred to another school because of his school’s test scores.

He said he didn’t want to leave his school.

He was told to tell the school community that he requested a transfer.

He said he wouldn’t lie.

He resigned rather than lie.

Dear Ms. Darby and CCSD Board of Trustees, Ms. Belk and District 2 Constituent Board,

It is with a heavy heart and out of a sense of moral obligation that I write to share with you my concerns about our District, its students, the James B. Edwards community, and specifically the events that have occurred over the last month. Alongside my wife, I have prayed about the decision to compose this letter and have acknowledged my fear continuing to work in a district that seems to often misrepresent the truth and punish those who question anything about its direction. It is my hope that you are truly unaware of what’s occurring in CCSD.

On the evening of April 24, I received a call from the Executive Director of the Elementary Learning Community, informing me that although I would receive a “Principal” contract for 2017-18, it would not be at JBE. I was shocked, as my tenure as principal of the school began less than two short years prior.

Not one time throughout this school year has any CCSD administrator, including the Superintendent, the Associate Superintendent of Schools, the Executive Director of the Elementary Learning Community, or the Director of the Elementary Learning Community visited our building. Neither had any leader shared with me a single concern about my performance, the performance of our teachers, or the performance of our students. Not one time, if it existed, were any community concerns conveyed to me, and not one time was even the “threat” of a potential move shared with me privately, publicly and/or in a group setting.

I was devastated, as I love our community, its parents and most importantly, its students. While we have lots left to accomplish, during my 1 ½ years at the school, we have increased student enrollment, put community programs in place to close the opportunity gap, increased physical activity opportunities, and improved parent, teacher, and student satisfaction.

The following morning, April 25, I met with the Executive Director of the Elementary Learning Community. The first thing he did was take a torn half-sheet of paper on which several rows of numbers were scribbled, handed me the paper and said, “Pick which scores are yours.” After asking for further
clarification about his request, I identified JBE students’ winter MAP scores. He then said, “That’s why you’re being moved.”

Perplexed again, I remained silent. My study of the NWEA website suggests that the intent of MAP is not to penalize students, teachers and/or principals. Instead, its purpose is to be used as a formative tool to help teachers know in what areas they need to focus their instruction throughout the following semester to grow students.

I sat silent, much as I had less than a month earlier on March 29, after being told in a principal allocation meeting that I should use testing data to place educators on improvement plans. I later spoke honestly to the teachers at JBE about my hesitation to do this and indicated that I would never use
student test scores in isolation to place teachers on improvement plans. Make no mistake: teachers and principals welcome accountability, but they want it to be fair, consistent, and student-centered.

After my meeting with the Executive Director on the morning of April 25, I met with the Superintendent later that day, per my request, after which I was more stunned than ever before. She indicated that the school’s data supported I did not have experience working in a “low income” school and said, “You are a young guy. You’ve not had experience working under a strong principal leader, have you?” Raised to respect authority, I did not respond that the principal underneath whom I worked for several years and who mentored me is currently appointed by the Superintendent herself as the Interim Director of Administrator Hiring and Leadership Development for CCSD.

I shared with the Superintendent that no one had visited our school or shared with me any performance concerns throughout the school year and that I was baffled why all of this was occurring. She indicated that because I received a principal contract and wasn’t being “demoted,” this wasn’t an issue. I remained silent. I was shocked to hear that such could actually happen in a school system where due process, best practice, and mentorship should exist for all teachers and administrators.

I changed the subject and shared with the Superintendent that I believed my work at JBE to be unfinished. As a result, I indicated that my community would likely be upset when it learned of this decision.

She then responded, “Your future in CCSD depends on how you handle this situation.” I sat silent.

She continued, “You could either play the victim, or you could tell your community it was your decision to leave JBE and that you’ve been ‘called’ to lead a school with students who need you more.”

Stunned it would be suggested I misrepresent “my” intentions to a community I love because of what appeared to be student test scores, my speaking out against a plan to put teachers on improvement plans, and my inexperience as a leader-I quickly explained that while I would “always respect my employer, I would not lie.”

She then said, “This is your truth to tell.”

On Thursday, April 27, I met with the Associate Superintendent of Schools. She confirmed the reasons I was being moved as student test data and a lack of diverse experience. During this meeting, she discussed the timeline with me for sharing this information with the JBE community. She approved the letter I had written to send to parents with the exception of a few sentences that identified the reasons for my move as student test scores and level of experiences. She requested that I remove those lines as “people don’t need that much information.”

I complied with her request and removed the information.

It was at this time that I made the one special request I’ve made throughout this process to the Associate Superintendent of Schools, to allow my 7 year old son to transfer from JBE and be placed at Sullivan’s Island Elementary School for 17-18. I explained that his remaining at JBE would be too hard
emotionally on him and our family as the school community and its teachers are very special to us all. This would also relieve a hardship on my wife and me in regards to transportation, as my sister is a teacher at the school.

The Associate Superintendent of Schools quickly and confidently approved this request, indicating it “would be no problem at all.” Afterwards, I spoke to the SIES principal and arranged the transfer.

I then notified my community of the transfer. When parents began sharing concerns with the Board of Trustees and district office, I received a call from the Associate Superintendent of Schools, indicating my 7 year old son would no longer be approved for a transfer and able to attend SIES. Instead, she indicated, “He’s been placed on the waiting list.”

Since then, parents and community members have been told by the Superintendent and Board members that it is because “of an outstanding skill set” that I’m being moved and that it has nothing to do with the aforementioned reasons. They even told a select group of parents in a closed-door meeting
that my transfer was unequivocally not about test scores.

If this decision was indeed based on an “outstanding skill set,” which would benefit students at another school, why could it not benefit the students at JBE, the most diverse elementary school in Mt. Pleasant? Who is advocating for all of these children in our community?

When the Associate Superintendent of Schools visited JBE for the first time this year, Friday, May 5, to meet with the faculty without me present, she, once again, informed staff that this decision was not about test scores. After the meeting, she said to me, “The Superintendent is NOT happy with this.”

Meanwhile, all of our students sit silent and wait. They wait for us to value them over test scores. They wait for us to value things like learning through play and physical activity. They wait for us to value real-world experiences over test preparation. They wait for us to empower their teachers to be creative and engage them in meaningful learning. Most of all, they wait for us to value doing what is right.

After nearly 10 years in CCSD, it is apparent that I now have a fundamental, philosophical difference with its leadership. Therefore, please accept this as my official letter of resignation, effective June 30, 2017.

Sincerely,
Jake Rambo
Principal
James B. Edwards Elementary School

I now place the name of Jake Rambo on the blog’s honor roll for his principled resistance to unconscionable policies that harm teachers and students.

Cathy O’Neill is a mathematician who wrote a wonderful book called “weapons of Math Destruction,” in which she showed how math and big data can be misused to reach very bad decisions.

She recently wrote an article for Bkloomberg News in which she explained why VAM is a terrible way to evaluate teachers.

She writes:

“For more than a decade, a glitchy and unaccountable algorithm has been making life difficult for America’s teachers. The good news is that its reign of terror might finally be drawing to a close.

“I first became acquainted with the Value-Added Model in 2011, when a friend of mine, a high school principal in Brooklyn, told me that a complex mathematical system was being used to assess her teachers — and to help decide such important matters as tenure. I offered to explain the formula to her if she could get it. She said she had tried, but had been told “it’s math, you wouldn’t understand it.”

“This was the first sign that something very weird was going on, and that somebody was avoiding scrutiny by invoking the authority and trustworthiness of mathematics. Not cool. The results have actually been terrible, and may be partly to blame for a national teacher shortage.

“The VAM — actually a family of algorithms — purports to determine how much “value” an individual teacher adds to a classroom. It goes by standardized test scores, and holds teachers accountable for what’s called student growth, which comes down to the difference between how well students performed on a test and how well a predictive model “expected” them to do…

“Fundamental problems immediately arose. Inconsistency was the most notable, statistically speaking: The same person teaching the same course in the same way to similar students could get wildly different scores from year to year. Teachers sometimes received scores for classes they hadn’t taught, or lost their jobs due to mistakes in code. Some cheated to raise their students’ test scores, creating false baselines that could lead to the firing of subsequent teachers (assuming they didn’t cheat, too).

“Perhaps most galling was the sheer lack of accountability. The code was proprietary, which meant administrators didn’t really understand the scores and appealing the model’s conclusions was next to impossible. Although economists studied such things as the effects of high-scoring teachers on students’ longer-term income, nobody paid adequate attention to the system’s effect on the quality and motivation of teachers overall.”

Happily, she says, VAM is on the way out. The new federal law does not require it, and courts have been ruling against. She cites Sheri Lederman’s court victory in New York and the recent court victory in Houston, where the judge said the algorithm was so opaque that it should not be used at all.

VAM has ruined the careers, the reputations, and the lives of many educators. One teacher, Rigoberto Ruelas, committed suicide soon after his VAM rating was published online by the Los Angeles Times. This is an example of a deadly use of math to damage real people, not just a game played by economists. Whoever participated willingly in this sham exercise should do penance.

The value-added assessment model that was forced on states by Arne Duncan’s Race to the Top is starting to fall apart, in the courts and in the experience of every state compelled to use it.

In this post, evaluation expert Audrey Amrein-Beardsley explains the top ten reasons why large scale, standardized tests should not be used to evaluate teachers.

She faced off against the rightwing National Council on Teacher Quality, which has consistently supported VAM and high-stakes testing. It is funded by, among others, the Gates Foundation.

Tim Slekar, dean of education at Edgewood College in Milwaukee, warns of the strategy that corporate reformers are using to undermine and destroy the teaching profession.

They say there is a shortage. They ignore the fact that the “shortage” is caused by the exodus of experienced teachers due to policies that create intolerable working conditions. They then say that the “solution” to the manufactured shortage is to eliminate entry requirements for teachers, thus lowering the bar to anyone with a college degree.

He writes:

It is happening all across the country. Policy makers, pundits and idiots keep screaming teacher shortage. And, in the same breath advocating and putting forth policies that do away with teacher licensing. Why?

First the BS answer: Allowing license flexibility or doing away with teaching licenses altogether will fix the shortage and if “we” don’t do something fast children will face empty classrooms.

The TRUTH: Softening teacher license policies or doing away with the license altogether will kill the profession of teaching.

Its that simple. Yet trying to get media and policy makers to understand this seems almost impossible. Trust me. I have been trying for 4 months.

Now it’s up to you. Take what you learn below. Call your legislators. Call your local media. Chain yourself to a tree naked. Oops. That was supposed to say “chain yourself to a naked tree.” Do whatever you can to get this ALEC backed attack on the profession of teaching to the people.

This report comes from a parent activist in Dallas, which held its school board election on Saturday (yesterday).

UPDATE: Kirkpatrick beat Marshall by 300 votes but fell short of 50%, and there will be a runoff. The future of Dallas’s failed corporate reform hinges on this race. Great that Kirkpatrick came in ahead of businessman Dustin Marshall. And fabulous that dedicated board member Joyce Foreman was re-elected!

https://www.dallasnews.com/news/education/2017/05/06/pivotal-dallas-isd-trustee-race-close-call

Read what follows with knowledge that Lori Kirkpatrick came in first and is going into a runoff with Marshall.

“Great news from Dallas, Texas to report. Public school advocate, and tireless Dallas ISD trustee Joyce Foreman has retained her seat on the school board in today’s elections.

Also, public school advocate and parent Lori Kirkpatrick has won a seat on the Dallas ISD school board. In a runoff for the same seat last year, Dustin Marshall won by only 42 votes against Mita Havlick, another parent and public school advocate.

Marshall, a business person who lost this time around, is in favor of the district’s pay for performance (TEI) initiative, the proposed Texas Education Agency A-F campus grading system, expanded school choice, and, would you believe it, vouchers. He has also been heavily involved with Uplift Education, the largest charter operator in Texas. A textbook deformer.

Marshall, of course won the endorsement of the Dallas Morning News Editorial Board for being a “vocal supporter of the Teacher Excellence Initiative, the district’s evaluation system, as an effective way to measure effective teachers and hold them accountable for improving student outcomes.
https://www.dallasnews.com/opinion/editorials/2017/04/12/recommend-dustin-marshall-disd-trustee-district-2 The DMN editorial writers still don’t get it, will probably never get it. But this time around, their candidate lost. Yay!

Kirkpatrick, the winner, states in a Dallas Morning News questionnaire, that “I am running for office because public education is under fierce attack. I expect my trustee to be committed to DISD and public education. I am 100% committed to DISD as evidenced by the fact that I send my daughter to DISD. This is in stark comparison to my opponent who has school-aged children all of whom are in private school. Additionally, I am opposed to diverting public money to private schools, unlike my opponent who voted against a resolution opposing vouchers and the A-F grading system.”

Further, she states, “Teachers with whom I have met feel very deflated due to the TEI evaluation system. I understand from them that many of their colleagues have left and won’t return due to this system. Teachers deserve to be paid fairly for the extraordinarily difficult job of educating our children. I will work to ensure we provide a fair evaluation system and thus pay so that we can maintain a quality educator at the helm of every classroom…..I think (TEI) is deeply flawed and needs a major overhaul. It is a factor in poor teacher morale, teacher turnover and hurts DISD when it comes to attracting new teachers. Education must remain a collaborative endeavor and should not artificially cap the number of teachers that can reach the top ratings, thus incentivizing those with less experience and those just becoming experienced while leaving the truly experienced teachers without the same opportunity to advance and gain fair compensation.”

https://www.kirkpatrick4disd.com/single-post/2017/05/03/The-Debate-between-Kirkpatrick-Marshall

This powerfully written article by John Connally appears on Kirkpatrick’s website. It deserves to be quoted in its entirety.

“I attended the recent debate between the two main candidates for Dallas ISD District 2 at Mata Montessori School.

On one side of the stage, Lori Kirkpatrick, a physician assistant at Parkland Hospital; on the other, businessman and incumbent Dustin Marshall. The debate was quite brief but still revealed a striking, if by now familiar, distinction between two visions for public education.

Kirkpatrick spoke of the gift of public education to society, conveyed an empathy for schoolteachers working under hostile conditions, and underlined the cost to society of not providing teachers and students with the necessary resources and support.

In contrast, Marshall expressed concern over an apparent mismatch between teacher evaluations and student test scores, and focused on the need to craft incentives to drive below average teachers out of the profession and expose “failing” schools. (According to this logic, parents would then have the knowledge required to choose between public schools — as if the choice of where to locate one’s family is comparable to choosing between two different colored apples at the grocery store.)

This hard-nosed business approach to overseeing schools actually has a long-failed history. Yet it’s an irony that, no matter the facts and evidence, this same approach is pursued relentlessly by those very people who portray themselves as objective and rational.

Why is it that this business-driven approach to public education has such a failed history? One reason is that treating teachers as self-serving individuals driven only by monetary incentives to achieve high class test scores can lead some to respond in kind by gaming the system to save their jobs. Notorious and extreme examples of this have been documented in places like Baltimore, Washington, and Atlanta.

But the more general answer to this question was given by the renowned scholar, James Q. Wilson. Public schools are not “production” or “procedural” organizations but what Wilson called “coping” organizations. This means that their operational activities and outputs are not easy to observe or measure. This is an intrinsic characteristic of public schools. To think of a public school as some kind of black box with well-defined measurable inputs and outputs is a pretense; indeed, a dangerous and dehumanizing pretense given all the students in danger of being tagged as failures at an early stage in life.

There is a further irony here. All this emphasis on test scores, rote learning, and impersonal teaching, is only advocated for students in public schools. For students in private schools it’s often just the opposite: intramural sports, Shakespeare, and joyful inquiry, sometimes taught by outstanding former public school teachers who reluctantly fled the system to escape the mind-numbing obsession with constant assessment, monitoring and micro-management.

It’s therefore not surprising that the issue of public v. private schools has come up in the race between Kirkpatrick and Marshall; in particular, concerning why the incumbent, Marshall, chooses to send his own children to a private school while promoting himself as the best qualified person to be public school trustee for Dallas ISD District 2.

Marshall took umbrage at the suggestion his decision had any kind of broader significance, explaining that he sent his children to the same private school he attended and of which he had such fond memories, claiming that one of his motivations in running for office is to help others experience the same positive start that he had at a private school.

Of course, from the perspective of a private citizen, where one chooses to send one’s children is one’s own business, and there are plenty of circumstantial reasons one can think of as to why parents may choose not to send their child to the local public school.
But what does it mean — as a matter of public policy — to view one’s private school experience as a kind of ideal to which public schools should aspire? Public schools differ in crucial ways from private schools.

Unlike private schools, public schools are subject to elected school boards, class size requirements, building regulations, as well as all kinds of state regulations, such as being required to cater to students with special needs. Teachers in public schools must have state certification and public schools must comply with a state-approved curriculum.

Private schools are not subject to these constraints. Further, private schools are not bound by the U.S. Constitution, including the Bill of Rights. In contrast, public schools have no discretion on such weighty matters.

So while one should not begrudge the incumbent his fond memories of private school, it does not necessarily seem the appropriate kind of experience one should be looking for in a public school trustee.

Interestingly, in his debate with Kirkpatrick, Marshall sought to allay any fear that he was some kind of educational extremist, stating that he disapproved of the recent appointment of Betsy DeVos as United States Secretary of Education.

But why is it so many people agree that DeVos is unqualified? The question was recently put in an interview to Diane Ravitch, former Under Secretary of Education to George H.W. Bush, and a leading national thinker on public education. Ravitch responded (talking about DeVos): “Well, she does not understand anything about education except for escaping from public schools. She’s never taught. She’s never supervised. She’s never attended public schools. Her children did not attend public schools. She thinks that public schools everywhere are just awful …”

If these kinds of criticisms are appropriate of DeVos, are they not also relevant to other candidates for public education posts (like this District 2 Trustee seat) when their experience, both as a parent and student, is limited to that of private schools?

Of course, unlike DeVos, Marshall has not explicitly advocated for vouchers. Indeed it would be foolhardy to do so — the public is strongly against these ideologically-driven social experiments. It is for this reason that Marshall’s opponent, Lori Kirkpatrick, is undoubtedly correct in emphasizing that one needs to look beyond words to specific actions in assessing where one comes down on this highly charged political issue. In this regard, Marshall’s recurring advocacy of competition and school-choice as the panacea to the problems of public schools is significant — student against student, teacher against teacher, school against school. Reward the winners and drop the losers — precisely the kind of thinking that led to the original idea of vouchers.

How did this Darwinian survival-of-the-fittest paradigm come to be seen as appropriate for public education? And how is the system supposed to replace all these supposed “underperforming” teachers that Marshall is so keen to drive out the system? What better and more experienced people are going to choose teaching as working conditions become ever more hostile?

But perhaps that’s not something we should be concerned with. Diane Ravitch points out that many of the “school choice” advocates seem to think that computers can do much of the work formerly done by “inefficient” teachers. Again though, the plan is selective. As Ravitch puts it: “the poor will get computers, the rich will get computers and teachers.”

The truth is that there has always been a battle over two alternative visions for public education. One sees it as essentially about knowledge and enrichment, as education for life as a citizen through the cultivation of independent critical minds, and therefore crucial to a functioning democracy.

The alternative perspective sees public education as serving quite different ends: the sorting of students at an early age to determine their place in society and role in the workforce; the promotion of deference to authority, conformity, passivity, and docility.

The two visions are incompatible. Take your pick.”

_________________________________________________________________________
John Connolly lives in East Dallas. He has a Ph.D. in Political Science and has several articles published on law, politics, and education.

Postscript to Diane, normally information has been sent about Dallas ISD elections before the vote, but the results have not been favorable for the pro public school candidates. So, the results information is being sent after the vote because there was in fear of jinxing the election. Today was a great day for Dallas ISD.

[I guess my correspondent in Dallas jinxed the outcome by declaring victory before all the votes were counted! Here is hoping that Lori Kirkpatrick can maintain her lead in the runoff and became a member of the DISD board.]

Audrey Amrein-Beardsley of Arizona State University is one of the nation’s most prominent scholars of teacher evaluation. She is especially critical of VAM (value-added measurement); she has studied TVAAS, EVAAS, and other similar metrics and found them deeply flawed. She has testified frequently in court cases as an expert witness.

In this post, she analyzes the court decision that blocks the use of VAM to evaluate teachers in Houston. The misuse of VAM was especially egregious in Houston, which terminated 221 teachers in one year, based on their VAM scores.

This is a very important article. Amrein-Beardsley and Jesse Rothstein of the University of California testified on behalf of the teachers; Tom Kane (who led the Gates’ Measures of Effective Teaching (MET) Study) and John Friedman (of the notorious Chetty-Friedman-Rockoff study) testified on behalf of the district.

Amrein-Beardsley writes:

Of primary issue will be the following (as taken from Judge Smith’s Summary Judgment released yesterday): “Plaintiffs [will continue to] challenge the use of EVAAS under various aspects of the Fourteenth Amendment, including: (1) procedural due process, due to lack of sufficient information to meaningfully challenge terminations based on low EVAAS scores,” and given “due process is designed to foster government decision-making that is both fair and accurate.”

Related, and of most importance, as also taken directly from Judge Smith’s Summary, he wrote:

HISD’s value-added appraisal system poses a realistic threat to deprive plaintiffs of constitutionally protected property interests in employment.

HISD does not itself calculate the EVAAS score for any of its teachers. Instead, that task is delegated to its third party vendor, SAS. The scores are generated by complex algorithms, employing “sophisticated software and many layers of calculations.” SAS treats these algorithms and software as trade secrets, refusing to divulge them to either HISD or the teachers themselves. HISD has admitted that it does not itself verify or audit the EVAAS scores received from SAS, nor does it engage any contractor to do so. HISD further concedes that any effort by teachers to replicate their own scores, with the limited information available to them, will necessarily fail. This has been confirmed by plaintiffs’ expert, who was unable to replicate the scores despite being given far greater access to the underlying computer codes than is available to an individual teacher [emphasis added, as also related to a prior post about how SAS claimed that plaintiffs violated SAS’s protective order (protecting its trade secrets), that the court overruled, see here].

The EVAAS score might be erroneously calculated for any number of reasons, ranging from data-entry mistakes to glitches in the computer code itself. Algorithms are human creations, and subject to error like any other human endeavor. HISD has acknowledged that mistakes can occur in calculating a teacher’s EVAAS score; moreover, even when a mistake is found in a particular teacher’s score, it will not be promptly corrected. As HISD candidly explained in response to a frequently asked question, “Why can’t my value-added analysis be recalculated?”:

Once completed, any re-analysis can only occur at the system level. What this means is that if we change information for one teacher, we would have to re- run the analysis for the entire district, which has two effects: one, this would be very costly for the district, as the analysis itself would have to be paid for again; and two, this re-analysis has the potential to change all other teachers’ reports.

The remarkable thing about this passage is not simply that cost considerations trump accuracy in teacher evaluations, troubling as that might be. Of greater concern is the house-of-cards fragility of the EVAAS system, where the wrong score of a single teacher could alter the scores of every other teacher in the district. This interconnectivity means that the accuracy of one score hinges upon the accuracy of all. Thus, without access to data supporting all teacher scores, any teacher facing discharge for a low value-added score will necessarily be unable to verify that her own score is error-free.

HISD’s own discovery responses and witnesses concede that an HISD teacher is unable to verify or replicate his EVAAS score based on the limited information provided by HISD.

According to the unrebutted testimony of plaintiffs’ expert, without access to SAS’s proprietary information – the value-added equations, computer source codes, decision rules, and assumptions – EVAAS scores will remain a mysterious “black box,” impervious to challenge.

While conceding that a teacher’s EVAAS score cannot be independently verified, HISD argues that the Constitution does not require the ability to replicate EVAAS scores “down to the last decimal point.” But EVAAS scores are calculated to the second decimal place, so an error as small as one hundredth of a point could spell the difference between a positive or negative EVAAS effectiveness rating, with serious consequences for the affected teacher.

Hence, “When a public agency adopts a policy of making high stakes employment decisions based on secret algorithms incompatible with minimum due process, the proper remedy is to overturn the policy.”

Today, teachers in Houston won a major court victory against the discredited teacher evaluation method called VAM, or “value-added measurement.” The court battle was led by the AFT and the Houston Federation of Teachers.

VAM was originally developed by an agricultural statistician, William Sanders, who believed that the rise or fall of student test scores can be attributed to the students’ teachers. This theory was incorporated into the Race to the Top program, which led many states to adopt it, despite the fact that it had never been proven to Wotan in a real-world situation. Seventy percent of teachers do not teach tested subjects, which led to bizarre strategies of evaluating teachers by scores of students they never taught in subjects they never taught.

Here is the press release from the AFT about the decision:

May 4, 2017

AFT, Houston Federation of Teachers Hail Court Ruling
on Flawed Evaluation System

Statements by American Federation of Teachers President Randi Weingarten and Houston Federation of Teachers President Zeph Capo on U.S. District Court decision on Houston’s Evaluation Value-Added Assessment System (EVAAS), known elsewhere as VAM or value-added measures:

AFT President Randi Weingarten: “Houston developed an incomprehensible, unfair and secret algorithm to evaluate teachers that had no rational meaning. This is the algebraic formula: 𝑦𝑖𝑗𝑘𝑙= 𝜇𝑗𝑘𝑙+ (Σ𝑘∗≤𝑘Σ𝑤𝑖𝑗𝑘∗𝑙∗𝑡 × 𝜏𝑖𝑗𝑘∗𝑙∗𝑡𝑇𝑖𝑗𝑘∗𝑙∗𝑡=1)+ 𝜖𝑖𝑗𝑘𝑙

“U.S. Magistrate Judge Stephen Smith saw that it was seriously flawed and posed a threat to teachers’ employment rights; he rejected it. This is a huge victory for Houston teachers, their students and educators’ deeply held contention that VAM is a sham.

“The judge said teachers had no way to ensure that EVAAS was correctly calculating their performance score, nor was there a way to promptly correct a mistake. Judge Smith added that the proper remedy is to overturn the policy; we wholeheartedly agree. Teaching must be about helping kids develop the skills and knowledge they need to be prepared for college, career and life—not be about focusing on test scores for punitive purposes.”

HFT President Zeph Capo: “With this decision, Houston should wipe clean the record of every teacher who was negatively evaluated. From here on, teacher evaluation systems should be developed with educators to ensure that they are fair, transparent and help inform instruction, not be used as a punitive tool.”

###