Archives for category: VAM

The Ohio State Senate wants to drop changes in test scores from teacher evaluations. However, the Cleveland district objects because the superintendent clings stubbornly to standardized tests of students as a reasonable measure of teacher quality. The fact that value-added measurement has flopped nationally doesn’t matter to him.

”District CEO Eric Gordon isn’t happy about the change and still wants to use test scores as a major part of teacher ratings. He looks at student scores — particular the “value added” measure of how much students learn in a year — as an important part of gauging whether teachers are doing well or not.”

Maybe no one told him that VAM is a sham.

Kevin Welner of the National Education Policy Center has written a thoughtful (and optimistic) commentary on the Gates Foundation’s latest big bet on reforming education. The new one will invest $1.7 billion in networks of schools in big cities, in the hopes that they can work together to solve common problems.

Welner, K. (2017). Might the New Gates Education Initiative Close Opportunity Gaps? Boulder, CO: National Education Policy Center. Retrieved [date] from

Welner notes that the previous big initiatives of the Gates Foundation failed, although he believes that Gates was too quick to pull the plug on the small schools initiative in 2008, into which he had poured $2 billion. Gates bet another $2 billion on the Common Core, and that was sunk by backlash from right and left and in any case, has made no notable difference. Gates poured untold millions into his plan for teacher evaluation (MET), but it failed because it relied too much on test scores.

Welner says that Bill Gates and the foundation he owns suffer from certain blind spots: First, he believes in free markets and choice, and he ends up pouring hundreds of millions into charters with little to show for it; second, he believes in data, and that belief has been costly without producing better schools; third, he believes in the transformative power of technology, forgetting that technology is only a tool, whose value is determined by how wisely it is used.

Last, Welner worries that Gates does not pay enough attention to the out of school factors that have a far greater impact on student learning that teachers and schools, including poverty and racism. These are the factors that mediate opportunity to learn. Without addressing those factors, none of the others will make much difference.

Welner is cautiously optimistic that the new initiative might pay more attention to opportunity to learn issues than any of Gates’ other investments.

But he notes with concern that Gates continues to fund charters, data, technology, and testing. He continues to believe that somewhere over the rainbow is a magical key to innovation. He continues to believe in standardization.

It seems to me that Kevin Welner bends over backwards to give Gates the benefit of the doubt. With his well-established track record of failure, it is hard to believe he has learned anything. But let’s keep hoping for the best.

Gary Rubinstein knows Michael Johnston from his days in Teach for America. He wishes Mike would stop telling tall tales about the school he briefly ran.

Mike said that the school he ran had a 100% graduation rate and college acceptance rate. Gary points out that 44 seniors graduated and got accepted to college, but there were 73 students in tenth grade two years earlier. That’s a 60% graduation rate, not 100%.

Now Mike Johnston is running for Governor of Colorado. He has built a reputation in the state as an education “reformer.” After graduating from Yale, he taught in Mississippi as a member of Teach for America, earned a degree at the Harvard Graduate School of Education, then a law degree, then was principal of a small school in Colorado where he claimed the school had a graduation rate of 100% and all were accepted into college. Based on this record, he ran for and was elected to the State Senate at the age of 35.

I met Mike Johnston in 2010, when I visited Denver to talk about my then-new book “The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education.” I was scheduled to debate Johnston at a luncheon before about 100 of Denver’s civic leaders. At the very moment I was in Denver, the Legislature was debating Johnston’s legislation to evaluate teachers and principals by the test scores of their students. Johnston called his law, SB 10-191, the “Great Teachers, Great Principals Act.” It required that test scores would count for 50% of every teacher and principal’s evaluation.

On the day we were to debate, Johnston was late. I spoke. Minutes later, Johnston arrived, not having heard anything I said about choice and testing. He spoke with great excitement about how his new legislation would weed out all the bad and ineffective teachers in the state and would lead to a new era of great teachers, great principals, and great schools.

Johnston, as Gary Rubinstein points out, is very much an Obama Democrat. Arne Duncan, whose Race to the Top squandered $5 Billion, has endorsed Johnston’s candidacy for governor.

Seven years later, even Colorado reformers acknowledge that Mike Johnston’s grandiose promises fell flat. In an article in Education Week, Colorado reformer Van Schoales admitted that the punitive SB 10-191 didn’t have much, if any effect.

He wrote:

“Implementation did not live up to the promises.

“Colorado Department of Education data released in February show that the distribution of teacher effectiveness in the state looks much as it did before passage of the bill. Eighty-eight percent of Colorado teachers were rated effective or highly effective, 4 percent were partially effective, 7.8 percent of teachers were not rated, and less than 1 percent were deemed ineffective. In other words, we leveraged everything we could and not only didn’t advance teacher effectiveness, we created a massive bureaucracy and alienated many in the field.

“What happened?

“It was wrong to force everyone in a state to have one ‘best’ evaluation system.”

“First, the data. We built a policy on growth data that only partially existed. The majority of teachers teach in states’ untested subject areas. This meant processes for measuring student growth outside of literacy or math were often thoughtlessly slapped together to meet the new evaluation law. For example, some elementary school art-teacher evaluations were linked to student performance on multiple-choice district art tests, while Spanish-teacher evaluations were tied to how the school did on the state’s math and literacy tests. Even for those who teach the grades and subjects with state tests, some debate remains on how much growth should be weighted for high-stakes decisions on teacher ratings. And we knew that few teachers accepted having their evaluations heavily weighted on student growth.

“Second, there has been little embrace of the state’s new teacher-evaluation system even from administrators frustrated with the former system. There were exceptions, namely the districts of Denver and Harrison, which had far fewer highly effective teachers than elsewhere in the state. Both districts invested time and resources in the development of a system that more accurately reflects a teacher’s impact on student learning. Yet most Colorado districts were forced to create new evaluation systems in alignment with the new law or adopt the state system, and most did the latter. This meant that these districts focused on compliance (and checking off evaluation boxes), rather than using the law to support teacher improvement.

“Third, we continue to have a leadership problem. Research shows that teacher evaluators are still not likely to give direct and honest feedback to teachers. A Brown University study on teacher evaluators in these new systems shows that the evaluators are three times more likely to rate teachers higher than they should be rated. This is a problem of school and district culture, not a fault with the evaluation rubric.

“Fourth, all of Colorado’s 238 charter schools waived out of the system.

“We wanted a new system to help professionalize teaching and address the real disparities in teacher quality. Instead, we got an 18-page state rubric and 345-page user guide for teacher evaluation.

“We didn’t understand how most school systems would respond to these teacher-evaluation laws. We failed to track implementation and didn’t check our assumptions along the way.”

Unfortunately, when the time came to change the law, Sen. Mike Johnston joined with five Republicans on the State Senate Education Committee to defeat a proposal to fix his failed law.

The rejected proposal, “originally introduced with bipartisan sponsorship, would have allowed school districts to drop the use of student academic growth data in teacher evaluations. It also would have eliminated the annual evaluation requirement for effective and highly effective teachers.”

But Johnston preferred to keep his law in place, despite its failure. It remains today as the most regressive teacher evaluation law in the nation. And it has had seven years to demonstrate its ineffectiveness.

Gary Rubinstein calls on Mike Johnston to stop making the false claim in his campaign literature that his high school’s graduation rate was 100%.

I call on him to renounce and denounce SB 10-191.

Make a clean break of it, Mike. Set things right. Show you are man enough to admit you were wrong.

After a long court fight in Houston, the school district agreed not to use value-added scores to evaluate teachers, because it was unable to explain what the algorithms for evaluating teacher performance meant or how they were calculated. The district also agreed to pay the lawyers’ fees for the Texas AFT, which fought the use of VAM.

What is the purpose of unions? To fight for the rights of teachers. No individual teacher (unless married to a lawyer) could have pursued this remedy on his or her own. The union had the resources to protect teachers from an unfair, nonsensical, illegitimate way of evaluating their teaching.

By the way, the courts in Houston were a lot wiser than the courts in Florida, which upheld the practice of evaluating teachers based on the test scores of students they do not teach in subjects they do not teach. The court in Florida said it was “unfair,” but constitutional. How can it be constitutional to have your teaching license depend on the work that others do, in which you have no part at all?

For Immediate Release
October 10, 2017

Zeph Capo

Janet Bass

Federal Suit Settlement: End of Value-Added Measures
for Teacher Termination in Houston

HOUSTON—In a huge victory for the right of teachers to be fairly evaluated, the Houston Independent School District agreed, in a settlement of a federal lawsuit brought by seven Houston teachers and the Houston Federation of Teachers, not to use value-added scores to terminate a teacher as long as the teacher is unable to independently test or challenge the score.

Value-added measures for teacher evaluation, called the Education Value-Added Assessment System, or EVAAS, in Houston, is a statistical method that uses a student’s performance on prior standardized tests to predict academic growth in the current year. This methodology—derided as deeply flawed, unfair and incomprehensible—was used to make decisions about teacher evaluation, bonuses and termination. It uses a secret computer program based on an inexplicable algorithm: = + (Σ∗≤Σ∗∗ × ∗∗∗∗=1)+ .

In May 2014, seven Houston teachers and the Houston Federation of Teachers brought an unprecedented federal lawsuit to end the policy, saying it reduced education to a test score, didn’t help improve teaching or learning, and ruined teachers’ careers when they were incorrectly terminated. Neither HISD nor its contractor allowed teachers access to the data or computer algorithms so that they could test or challenge the legitimacy of the scores, creating a “black box.” In May 2017, the federal district court in Houston issued a decision stating that, “HISD teachers have no meaningful way to ensure correct calculation of their EVAAS scores, and as a result are unfairly subject to mistaken deprivation of constitutionally protected property interests in their jobs.”

HFT President Zeph Capo said: “This victory should mark the end of a destructive era that put tests and a broken evaluation system over making sure our students leave school well prepared for college, career and life. As a practical matter, this ends the use of value-added to terminate teachers in HISD because the district does not have a contractor that is willing or able to meet the constitutional due process standards spelled out by the court.”

Daniel Santos, one of the plaintiffs and an award-winning sixth-grade teacher at Navarro Middle School who was rated ineffective by the flawed EVAAS method, was elated with the settlement.

“I have always been devoted to my students and proud of my teaching skills. Houston needs a well-developed system that properly evaluates teachers, provides good feedback and ensures that educators will receive continuous, targeted professional development to improve their performance,” Santos said.

American Federation of Teachers President Randi Weingarten said the agreement not to use value-added measures for this purpose is the latest nail in the coffin of using tests as a punitive tool. The Every Student Succeeds Act, the federal education law that replaced the No Child Left Behind Act, eliminated the emphasis on test scores.

“Testing and EVAAS don’t measure critical or analytical thinking skills, don’t allow for engaging learning, and certainly don’t improve or create joy in teaching or learning. Instead of value-added methods, let’s value what kids really need: attention to their well-being, engaging and powerful learning, a collaborative school environment, and opportunities for teachers to build their skills throughout their careers,” Weingarten said.

In addition to agreeing to restrict its use of value-added measures, including EVAAS scores, the school district agreed to create an instructional consultation panel—with representatives from the district and the faculty—to discuss and make recommendations on the district’s teacher appraisal process. The settlement also requires HISD to pay Texas AFT $237,000 for attorney’s fees and expenses related to the lawsuit.

Here is the amended summary judgment opinion.

Perhaps our blog poet was thinking of Leonard Cohen’s great song “Anthem” when he wrote this:

“The Fall of the House of Reform”

A crack, a crack in outer wall
The House of Reform, about to fall
A house of test and house of VAM
Of fake “Success” and charter scam
A house of standards built on sand
With Core arranged by Coleman hand
A house infused with sickly air
The flatulence of billionaire
The House was doomed from very start
An empty place without a heart
Expanding crack, lets in the light
As daylight breaks the longest night

It is called VAM. Value-added-measurement, or value-added-modeling. It means measuring the effectiveness of teachers by the rise or fall of the test scores of their students.

Rachel M. Cohen, writing in The American Prospect, documents the slow but steady retreat from evaluating teachers by the test scores of their students. Only a few years ago, VAM was lauded by Secretary of a Education Arne Duncan as the ultimate way to determine which teachers were succeeding and which were failing; Duncan made it a condition of competing for Race to the Top billions, and more than 40 states agreed to adopt it; Bill Gates spent hundreds of millions of dollars promoting it; a team of economists led by Raj Chetty of Harvard claimed that the actions of a teacher in elementary school predicted teen pregnancy, adult earnings, and other momentous life consequences, and earned front-page status in the anew York Times; and thousands of teachers and principals were fired because of it.

But time is the test, and time has not been kind to VAM.

Cohen reviews the role of the courts, with some refusing to get involved, and others agreeing that VAM is arbitrary and capricious. She credits Duncan and Gates for their role in creating this monstrous and invalid way of evaluating teachers. The grand idea, having cut down many good teachers, is nearing its end. But not soon enough.

We now know that William Sanders, the creator of Value-Added Measurement, which measures teacher quality with the methods of an agricultural statistician, is revered in the think tank world of D.C.

But Peter Greene, who teaches in a Pennsylvania high school, doesn’t think much of VAM.

He recounts how Sanders was inspired to think about measuring teachers based on his studies of radioactivity in cows that were downwind from a nuclear explosion (really!).

To which Sanders writes:

Oh, let’s tell the truth. VAM systems have also been limited by the fact that they’re junk, taking bad data from test scores, massaging them through an opaque and improbable mathematical model to arrive at conclusions that are volatile and inconsistent and which a myriad educators have looked at and responded, “Well, this can’t possible be right.”

You’ll never find me arguing against any accountability; taxpayers (and I am one) have the right to know how their money is spent. But Sander’s work ultimately wasted a lot of time and money and produced a system about as effective as checking toad warts under a full moon– worse, because it looked all number and sciencey and so lots of suckers believed in it. Carey can be the apologist crafting it all into a charming and earnest tale, but the bottom line is that VAM has done plenty of damage, and we’d all be better off if Sanders had stuck to his radioactive cows.

The New York Times published a tribute by Kevin Carey of the New America Foundationto William Sanders, “the little-known statistician who taught us to measure teachers.”

One hates to speak ill of the dead, but accuracy requires that we note that Sanders’ statistical model for “measuring” teachers was flawed, inaccurate, and damaged the lives of thousands of teachers based on Sanders’ obscure algorithms. Sanders was an agricultural statistician before he found a goldmine in education. Measuring teacher quality really is not akin to measuring cattle or crops. Every analysis of the influences on students’ test performance gives far more weight to family income and education than to the teachers who see her or him for an hour or five hours a day. Sanders tried to remove human judgment from the equation and ended up creating a profitable business that distorted teaching and learning into a struggle for higher test scores. If the tests themselves are invalid, then any accountability measures based on them will be invalid.

No one knows William Sanders’ works and its flaws better than Audrey Amrein-Beardsley. She has studied Sanders’ value-added measures for years and testified against them in court. She comments on the New York Times’ article here. Amrein-Beardsley points out that Sanders’ methods have been faring poorly in court because it is unfair to judge a teacher based on a mysterious algorithm that no one can understand or explain.

In my book Reign of Error, I wrote about the fallacy behind Sanders’ reasoning by quoting a song from “The Fantasticks.” I paid $1200 for the right to reprint the lyrics. It is the one that goes “Plant a radish, get a radish, not a sauerkraut./That’s why I love vegetables, they know what they’re about.”

No one can say the same about children. Children from the same parents are different, even when their upbringing is as identical as those parents can make it. They look different, they act different, they have different interests, they have different goals.

Sanders never understood that.

This story appeared on the blog of Patrick Hayes’ EdFirstSC.

In Charleston, South Carolina, Principal Jake Rambo was ordered to evaluate his teachers based solely on the test scores of their students. Not multiple measures. Standardized tests.

He refused.

He was told that he was being transferred to another school because of his school’s test scores.

He said he didn’t want to leave his school.

He was told to tell the school community that he requested a transfer.

He said he wouldn’t lie.

He resigned rather than lie.

Dear Ms. Darby and CCSD Board of Trustees, Ms. Belk and District 2 Constituent Board,

It is with a heavy heart and out of a sense of moral obligation that I write to share with you my concerns about our District, its students, the James B. Edwards community, and specifically the events that have occurred over the last month. Alongside my wife, I have prayed about the decision to compose this letter and have acknowledged my fear continuing to work in a district that seems to often misrepresent the truth and punish those who question anything about its direction. It is my hope that you are truly unaware of what’s occurring in CCSD.

On the evening of April 24, I received a call from the Executive Director of the Elementary Learning Community, informing me that although I would receive a “Principal” contract for 2017-18, it would not be at JBE. I was shocked, as my tenure as principal of the school began less than two short years prior.

Not one time throughout this school year has any CCSD administrator, including the Superintendent, the Associate Superintendent of Schools, the Executive Director of the Elementary Learning Community, or the Director of the Elementary Learning Community visited our building. Neither had any leader shared with me a single concern about my performance, the performance of our teachers, or the performance of our students. Not one time, if it existed, were any community concerns conveyed to me, and not one time was even the “threat” of a potential move shared with me privately, publicly and/or in a group setting.

I was devastated, as I love our community, its parents and most importantly, its students. While we have lots left to accomplish, during my 1 ½ years at the school, we have increased student enrollment, put community programs in place to close the opportunity gap, increased physical activity opportunities, and improved parent, teacher, and student satisfaction.

The following morning, April 25, I met with the Executive Director of the Elementary Learning Community. The first thing he did was take a torn half-sheet of paper on which several rows of numbers were scribbled, handed me the paper and said, “Pick which scores are yours.” After asking for further
clarification about his request, I identified JBE students’ winter MAP scores. He then said, “That’s why you’re being moved.”

Perplexed again, I remained silent. My study of the NWEA website suggests that the intent of MAP is not to penalize students, teachers and/or principals. Instead, its purpose is to be used as a formative tool to help teachers know in what areas they need to focus their instruction throughout the following semester to grow students.

I sat silent, much as I had less than a month earlier on March 29, after being told in a principal allocation meeting that I should use testing data to place educators on improvement plans. I later spoke honestly to the teachers at JBE about my hesitation to do this and indicated that I would never use
student test scores in isolation to place teachers on improvement plans. Make no mistake: teachers and principals welcome accountability, but they want it to be fair, consistent, and student-centered.

After my meeting with the Executive Director on the morning of April 25, I met with the Superintendent later that day, per my request, after which I was more stunned than ever before. She indicated that the school’s data supported I did not have experience working in a “low income” school and said, “You are a young guy. You’ve not had experience working under a strong principal leader, have you?” Raised to respect authority, I did not respond that the principal underneath whom I worked for several years and who mentored me is currently appointed by the Superintendent herself as the Interim Director of Administrator Hiring and Leadership Development for CCSD.

I shared with the Superintendent that no one had visited our school or shared with me any performance concerns throughout the school year and that I was baffled why all of this was occurring. She indicated that because I received a principal contract and wasn’t being “demoted,” this wasn’t an issue. I remained silent. I was shocked to hear that such could actually happen in a school system where due process, best practice, and mentorship should exist for all teachers and administrators.

I changed the subject and shared with the Superintendent that I believed my work at JBE to be unfinished. As a result, I indicated that my community would likely be upset when it learned of this decision.

She then responded, “Your future in CCSD depends on how you handle this situation.” I sat silent.

She continued, “You could either play the victim, or you could tell your community it was your decision to leave JBE and that you’ve been ‘called’ to lead a school with students who need you more.”

Stunned it would be suggested I misrepresent “my” intentions to a community I love because of what appeared to be student test scores, my speaking out against a plan to put teachers on improvement plans, and my inexperience as a leader-I quickly explained that while I would “always respect my employer, I would not lie.”

She then said, “This is your truth to tell.”

On Thursday, April 27, I met with the Associate Superintendent of Schools. She confirmed the reasons I was being moved as student test data and a lack of diverse experience. During this meeting, she discussed the timeline with me for sharing this information with the JBE community. She approved the letter I had written to send to parents with the exception of a few sentences that identified the reasons for my move as student test scores and level of experiences. She requested that I remove those lines as “people don’t need that much information.”

I complied with her request and removed the information.

It was at this time that I made the one special request I’ve made throughout this process to the Associate Superintendent of Schools, to allow my 7 year old son to transfer from JBE and be placed at Sullivan’s Island Elementary School for 17-18. I explained that his remaining at JBE would be too hard
emotionally on him and our family as the school community and its teachers are very special to us all. This would also relieve a hardship on my wife and me in regards to transportation, as my sister is a teacher at the school.

The Associate Superintendent of Schools quickly and confidently approved this request, indicating it “would be no problem at all.” Afterwards, I spoke to the SIES principal and arranged the transfer.

I then notified my community of the transfer. When parents began sharing concerns with the Board of Trustees and district office, I received a call from the Associate Superintendent of Schools, indicating my 7 year old son would no longer be approved for a transfer and able to attend SIES. Instead, she indicated, “He’s been placed on the waiting list.”

Since then, parents and community members have been told by the Superintendent and Board members that it is because “of an outstanding skill set” that I’m being moved and that it has nothing to do with the aforementioned reasons. They even told a select group of parents in a closed-door meeting
that my transfer was unequivocally not about test scores.

If this decision was indeed based on an “outstanding skill set,” which would benefit students at another school, why could it not benefit the students at JBE, the most diverse elementary school in Mt. Pleasant? Who is advocating for all of these children in our community?

When the Associate Superintendent of Schools visited JBE for the first time this year, Friday, May 5, to meet with the faculty without me present, she, once again, informed staff that this decision was not about test scores. After the meeting, she said to me, “The Superintendent is NOT happy with this.”

Meanwhile, all of our students sit silent and wait. They wait for us to value them over test scores. They wait for us to value things like learning through play and physical activity. They wait for us to value real-world experiences over test preparation. They wait for us to empower their teachers to be creative and engage them in meaningful learning. Most of all, they wait for us to value doing what is right.

After nearly 10 years in CCSD, it is apparent that I now have a fundamental, philosophical difference with its leadership. Therefore, please accept this as my official letter of resignation, effective June 30, 2017.

Jake Rambo
James B. Edwards Elementary School

I now place the name of Jake Rambo on the blog’s honor roll for his principled resistance to unconscionable policies that harm teachers and students.

Cathy O’Neill is a mathematician who wrote a wonderful book called “weapons of Math Destruction,” in which she showed how math and big data can be misused to reach very bad decisions.

She recently wrote an article for Bkloomberg News in which she explained why VAM is a terrible way to evaluate teachers.

She writes:

“For more than a decade, a glitchy and unaccountable algorithm has been making life difficult for America’s teachers. The good news is that its reign of terror might finally be drawing to a close.

“I first became acquainted with the Value-Added Model in 2011, when a friend of mine, a high school principal in Brooklyn, told me that a complex mathematical system was being used to assess her teachers — and to help decide such important matters as tenure. I offered to explain the formula to her if she could get it. She said she had tried, but had been told “it’s math, you wouldn’t understand it.”

“This was the first sign that something very weird was going on, and that somebody was avoiding scrutiny by invoking the authority and trustworthiness of mathematics. Not cool. The results have actually been terrible, and may be partly to blame for a national teacher shortage.

“The VAM — actually a family of algorithms — purports to determine how much “value” an individual teacher adds to a classroom. It goes by standardized test scores, and holds teachers accountable for what’s called student growth, which comes down to the difference between how well students performed on a test and how well a predictive model “expected” them to do…

“Fundamental problems immediately arose. Inconsistency was the most notable, statistically speaking: The same person teaching the same course in the same way to similar students could get wildly different scores from year to year. Teachers sometimes received scores for classes they hadn’t taught, or lost their jobs due to mistakes in code. Some cheated to raise their students’ test scores, creating false baselines that could lead to the firing of subsequent teachers (assuming they didn’t cheat, too).

“Perhaps most galling was the sheer lack of accountability. The code was proprietary, which meant administrators didn’t really understand the scores and appealing the model’s conclusions was next to impossible. Although economists studied such things as the effects of high-scoring teachers on students’ longer-term income, nobody paid adequate attention to the system’s effect on the quality and motivation of teachers overall.”

Happily, she says, VAM is on the way out. The new federal law does not require it, and courts have been ruling against. She cites Sheri Lederman’s court victory in New York and the recent court victory in Houston, where the judge said the algorithm was so opaque that it should not be used at all.

VAM has ruined the careers, the reputations, and the lives of many educators. One teacher, Rigoberto Ruelas, committed suicide soon after his VAM rating was published online by the Los Angeles Times. This is an example of a deadly use of math to damage real people, not just a game played by economists. Whoever participated willingly in this sham exercise should do penance.