Archives for category: VAM

This story appeared on the blog of Patrick Hayes’ EdFirstSC.

In Charleston, South Carolina, Principal Jake Rambo was ordered to evaluate his teachers based solely on the test scores of their students. Not multiple measures. Standardized tests.

He refused.

He was told that he was being transferred to another school because of his school’s test scores.

He said he didn’t want to leave his school.

He was told to tell the school community that he requested a transfer.

He said he wouldn’t lie.

He resigned rather than lie.

Dear Ms. Darby and CCSD Board of Trustees, Ms. Belk and District 2 Constituent Board,

It is with a heavy heart and out of a sense of moral obligation that I write to share with you my concerns about our District, its students, the James B. Edwards community, and specifically the events that have occurred over the last month. Alongside my wife, I have prayed about the decision to compose this letter and have acknowledged my fear continuing to work in a district that seems to often misrepresent the truth and punish those who question anything about its direction. It is my hope that you are truly unaware of what’s occurring in CCSD.

On the evening of April 24, I received a call from the Executive Director of the Elementary Learning Community, informing me that although I would receive a “Principal” contract for 2017-18, it would not be at JBE. I was shocked, as my tenure as principal of the school began less than two short years prior.

Not one time throughout this school year has any CCSD administrator, including the Superintendent, the Associate Superintendent of Schools, the Executive Director of the Elementary Learning Community, or the Director of the Elementary Learning Community visited our building. Neither had any leader shared with me a single concern about my performance, the performance of our teachers, or the performance of our students. Not one time, if it existed, were any community concerns conveyed to me, and not one time was even the “threat” of a potential move shared with me privately, publicly and/or in a group setting.

I was devastated, as I love our community, its parents and most importantly, its students. While we have lots left to accomplish, during my 1 ½ years at the school, we have increased student enrollment, put community programs in place to close the opportunity gap, increased physical activity opportunities, and improved parent, teacher, and student satisfaction.

The following morning, April 25, I met with the Executive Director of the Elementary Learning Community. The first thing he did was take a torn half-sheet of paper on which several rows of numbers were scribbled, handed me the paper and said, “Pick which scores are yours.” After asking for further
clarification about his request, I identified JBE students’ winter MAP scores. He then said, “That’s why you’re being moved.”

Perplexed again, I remained silent. My study of the NWEA website suggests that the intent of MAP is not to penalize students, teachers and/or principals. Instead, its purpose is to be used as a formative tool to help teachers know in what areas they need to focus their instruction throughout the following semester to grow students.

I sat silent, much as I had less than a month earlier on March 29, after being told in a principal allocation meeting that I should use testing data to place educators on improvement plans. I later spoke honestly to the teachers at JBE about my hesitation to do this and indicated that I would never use
student test scores in isolation to place teachers on improvement plans. Make no mistake: teachers and principals welcome accountability, but they want it to be fair, consistent, and student-centered.

After my meeting with the Executive Director on the morning of April 25, I met with the Superintendent later that day, per my request, after which I was more stunned than ever before. She indicated that the school’s data supported I did not have experience working in a “low income” school and said, “You are a young guy. You’ve not had experience working under a strong principal leader, have you?” Raised to respect authority, I did not respond that the principal underneath whom I worked for several years and who mentored me is currently appointed by the Superintendent herself as the Interim Director of Administrator Hiring and Leadership Development for CCSD.

I shared with the Superintendent that no one had visited our school or shared with me any performance concerns throughout the school year and that I was baffled why all of this was occurring. She indicated that because I received a principal contract and wasn’t being “demoted,” this wasn’t an issue. I remained silent. I was shocked to hear that such could actually happen in a school system where due process, best practice, and mentorship should exist for all teachers and administrators.

I changed the subject and shared with the Superintendent that I believed my work at JBE to be unfinished. As a result, I indicated that my community would likely be upset when it learned of this decision.

She then responded, “Your future in CCSD depends on how you handle this situation.” I sat silent.

She continued, “You could either play the victim, or you could tell your community it was your decision to leave JBE and that you’ve been ‘called’ to lead a school with students who need you more.”

Stunned it would be suggested I misrepresent “my” intentions to a community I love because of what appeared to be student test scores, my speaking out against a plan to put teachers on improvement plans, and my inexperience as a leader-I quickly explained that while I would “always respect my employer, I would not lie.”

She then said, “This is your truth to tell.”

On Thursday, April 27, I met with the Associate Superintendent of Schools. She confirmed the reasons I was being moved as student test data and a lack of diverse experience. During this meeting, she discussed the timeline with me for sharing this information with the JBE community. She approved the letter I had written to send to parents with the exception of a few sentences that identified the reasons for my move as student test scores and level of experiences. She requested that I remove those lines as “people don’t need that much information.”

I complied with her request and removed the information.

It was at this time that I made the one special request I’ve made throughout this process to the Associate Superintendent of Schools, to allow my 7 year old son to transfer from JBE and be placed at Sullivan’s Island Elementary School for 17-18. I explained that his remaining at JBE would be too hard
emotionally on him and our family as the school community and its teachers are very special to us all. This would also relieve a hardship on my wife and me in regards to transportation, as my sister is a teacher at the school.

The Associate Superintendent of Schools quickly and confidently approved this request, indicating it “would be no problem at all.” Afterwards, I spoke to the SIES principal and arranged the transfer.

I then notified my community of the transfer. When parents began sharing concerns with the Board of Trustees and district office, I received a call from the Associate Superintendent of Schools, indicating my 7 year old son would no longer be approved for a transfer and able to attend SIES. Instead, she indicated, “He’s been placed on the waiting list.”

Since then, parents and community members have been told by the Superintendent and Board members that it is because “of an outstanding skill set” that I’m being moved and that it has nothing to do with the aforementioned reasons. They even told a select group of parents in a closed-door meeting
that my transfer was unequivocally not about test scores.

If this decision was indeed based on an “outstanding skill set,” which would benefit students at another school, why could it not benefit the students at JBE, the most diverse elementary school in Mt. Pleasant? Who is advocating for all of these children in our community?

When the Associate Superintendent of Schools visited JBE for the first time this year, Friday, May 5, to meet with the faculty without me present, she, once again, informed staff that this decision was not about test scores. After the meeting, she said to me, “The Superintendent is NOT happy with this.”

Meanwhile, all of our students sit silent and wait. They wait for us to value them over test scores. They wait for us to value things like learning through play and physical activity. They wait for us to value real-world experiences over test preparation. They wait for us to empower their teachers to be creative and engage them in meaningful learning. Most of all, they wait for us to value doing what is right.

After nearly 10 years in CCSD, it is apparent that I now have a fundamental, philosophical difference with its leadership. Therefore, please accept this as my official letter of resignation, effective June 30, 2017.

Sincerely,
Jake Rambo
Principal
James B. Edwards Elementary School

I now place the name of Jake Rambo on the blog’s honor roll for his principled resistance to unconscionable policies that harm teachers and students.

Cathy O’Neill is a mathematician who wrote a wonderful book called “weapons of Math Destruction,” in which she showed how math and big data can be misused to reach very bad decisions.

She recently wrote an article for Bkloomberg News in which she explained why VAM is a terrible way to evaluate teachers.

She writes:

“For more than a decade, a glitchy and unaccountable algorithm has been making life difficult for America’s teachers. The good news is that its reign of terror might finally be drawing to a close.

“I first became acquainted with the Value-Added Model in 2011, when a friend of mine, a high school principal in Brooklyn, told me that a complex mathematical system was being used to assess her teachers — and to help decide such important matters as tenure. I offered to explain the formula to her if she could get it. She said she had tried, but had been told “it’s math, you wouldn’t understand it.”

“This was the first sign that something very weird was going on, and that somebody was avoiding scrutiny by invoking the authority and trustworthiness of mathematics. Not cool. The results have actually been terrible, and may be partly to blame for a national teacher shortage.

“The VAM — actually a family of algorithms — purports to determine how much “value” an individual teacher adds to a classroom. It goes by standardized test scores, and holds teachers accountable for what’s called student growth, which comes down to the difference between how well students performed on a test and how well a predictive model “expected” them to do…

“Fundamental problems immediately arose. Inconsistency was the most notable, statistically speaking: The same person teaching the same course in the same way to similar students could get wildly different scores from year to year. Teachers sometimes received scores for classes they hadn’t taught, or lost their jobs due to mistakes in code. Some cheated to raise their students’ test scores, creating false baselines that could lead to the firing of subsequent teachers (assuming they didn’t cheat, too).

“Perhaps most galling was the sheer lack of accountability. The code was proprietary, which meant administrators didn’t really understand the scores and appealing the model’s conclusions was next to impossible. Although economists studied such things as the effects of high-scoring teachers on students’ longer-term income, nobody paid adequate attention to the system’s effect on the quality and motivation of teachers overall.”

Happily, she says, VAM is on the way out. The new federal law does not require it, and courts have been ruling against. She cites Sheri Lederman’s court victory in New York and the recent court victory in Houston, where the judge said the algorithm was so opaque that it should not be used at all.

VAM has ruined the careers, the reputations, and the lives of many educators. One teacher, Rigoberto Ruelas, committed suicide soon after his VAM rating was published online by the Los Angeles Times. This is an example of a deadly use of math to damage real people, not just a game played by economists. Whoever participated willingly in this sham exercise should do penance.

The value-added assessment model that was forced on states by Arne Duncan’s Race to the Top is starting to fall apart, in the courts and in the experience of every state compelled to use it.

In this post, evaluation expert Audrey Amrein-Beardsley explains the top ten reasons why large scale, standardized tests should not be used to evaluate teachers.

She faced off against the rightwing National Council on Teacher Quality, which has consistently supported VAM and high-stakes testing. It is funded by, among others, the Gates Foundation.

Audrey Amrein-Beardsley of Arizona State University is one of the nation’s most prominent scholars of teacher evaluation. She is especially critical of VAM (value-added measurement); she has studied TVAAS, EVAAS, and other similar metrics and found them deeply flawed. She has testified frequently in court cases as an expert witness.

In this post, she analyzes the court decision that blocks the use of VAM to evaluate teachers in Houston. The misuse of VAM was especially egregious in Houston, which terminated 221 teachers in one year, based on their VAM scores.

This is a very important article. Amrein-Beardsley and Jesse Rothstein of the University of California testified on behalf of the teachers; Tom Kane (who led the Gates’ Measures of Effective Teaching (MET) Study) and John Friedman (of the notorious Chetty-Friedman-Rockoff study) testified on behalf of the district.

Amrein-Beardsley writes:

Of primary issue will be the following (as taken from Judge Smith’s Summary Judgment released yesterday): “Plaintiffs [will continue to] challenge the use of EVAAS under various aspects of the Fourteenth Amendment, including: (1) procedural due process, due to lack of sufficient information to meaningfully challenge terminations based on low EVAAS scores,” and given “due process is designed to foster government decision-making that is both fair and accurate.”

Related, and of most importance, as also taken directly from Judge Smith’s Summary, he wrote:

HISD’s value-added appraisal system poses a realistic threat to deprive plaintiffs of constitutionally protected property interests in employment.

HISD does not itself calculate the EVAAS score for any of its teachers. Instead, that task is delegated to its third party vendor, SAS. The scores are generated by complex algorithms, employing “sophisticated software and many layers of calculations.” SAS treats these algorithms and software as trade secrets, refusing to divulge them to either HISD or the teachers themselves. HISD has admitted that it does not itself verify or audit the EVAAS scores received from SAS, nor does it engage any contractor to do so. HISD further concedes that any effort by teachers to replicate their own scores, with the limited information available to them, will necessarily fail. This has been confirmed by plaintiffs’ expert, who was unable to replicate the scores despite being given far greater access to the underlying computer codes than is available to an individual teacher [emphasis added, as also related to a prior post about how SAS claimed that plaintiffs violated SAS’s protective order (protecting its trade secrets), that the court overruled, see here].

The EVAAS score might be erroneously calculated for any number of reasons, ranging from data-entry mistakes to glitches in the computer code itself. Algorithms are human creations, and subject to error like any other human endeavor. HISD has acknowledged that mistakes can occur in calculating a teacher’s EVAAS score; moreover, even when a mistake is found in a particular teacher’s score, it will not be promptly corrected. As HISD candidly explained in response to a frequently asked question, “Why can’t my value-added analysis be recalculated?”:

Once completed, any re-analysis can only occur at the system level. What this means is that if we change information for one teacher, we would have to re- run the analysis for the entire district, which has two effects: one, this would be very costly for the district, as the analysis itself would have to be paid for again; and two, this re-analysis has the potential to change all other teachers’ reports.

The remarkable thing about this passage is not simply that cost considerations trump accuracy in teacher evaluations, troubling as that might be. Of greater concern is the house-of-cards fragility of the EVAAS system, where the wrong score of a single teacher could alter the scores of every other teacher in the district. This interconnectivity means that the accuracy of one score hinges upon the accuracy of all. Thus, without access to data supporting all teacher scores, any teacher facing discharge for a low value-added score will necessarily be unable to verify that her own score is error-free.

HISD’s own discovery responses and witnesses concede that an HISD teacher is unable to verify or replicate his EVAAS score based on the limited information provided by HISD.

According to the unrebutted testimony of plaintiffs’ expert, without access to SAS’s proprietary information – the value-added equations, computer source codes, decision rules, and assumptions – EVAAS scores will remain a mysterious “black box,” impervious to challenge.

While conceding that a teacher’s EVAAS score cannot be independently verified, HISD argues that the Constitution does not require the ability to replicate EVAAS scores “down to the last decimal point.” But EVAAS scores are calculated to the second decimal place, so an error as small as one hundredth of a point could spell the difference between a positive or negative EVAAS effectiveness rating, with serious consequences for the affected teacher.

Hence, “When a public agency adopts a policy of making high stakes employment decisions based on secret algorithms incompatible with minimum due process, the proper remedy is to overturn the policy.”

Today, teachers in Houston won a major court victory against the discredited teacher evaluation method called VAM, or “value-added measurement.” The court battle was led by the AFT and the Houston Federation of Teachers.

VAM was originally developed by an agricultural statistician, William Sanders, who believed that the rise or fall of student test scores can be attributed to the students’ teachers. This theory was incorporated into the Race to the Top program, which led many states to adopt it, despite the fact that it had never been proven to Wotan in a real-world situation. Seventy percent of teachers do not teach tested subjects, which led to bizarre strategies of evaluating teachers by scores of students they never taught in subjects they never taught.

Here is the press release from the AFT about the decision:

May 4, 2017

AFT, Houston Federation of Teachers Hail Court Ruling
on Flawed Evaluation System

Statements by American Federation of Teachers President Randi Weingarten and Houston Federation of Teachers President Zeph Capo on U.S. District Court decision on Houston’s Evaluation Value-Added Assessment System (EVAAS), known elsewhere as VAM or value-added measures:

AFT President Randi Weingarten: “Houston developed an incomprehensible, unfair and secret algorithm to evaluate teachers that had no rational meaning. This is the algebraic formula: 𝑦𝑖𝑗𝑘𝑙= 𝜇𝑗𝑘𝑙+ (Σ𝑘∗≤𝑘Σ𝑤𝑖𝑗𝑘∗𝑙∗𝑡 × 𝜏𝑖𝑗𝑘∗𝑙∗𝑡𝑇𝑖𝑗𝑘∗𝑙∗𝑡=1)+ 𝜖𝑖𝑗𝑘𝑙

“U.S. Magistrate Judge Stephen Smith saw that it was seriously flawed and posed a threat to teachers’ employment rights; he rejected it. This is a huge victory for Houston teachers, their students and educators’ deeply held contention that VAM is a sham.

“The judge said teachers had no way to ensure that EVAAS was correctly calculating their performance score, nor was there a way to promptly correct a mistake. Judge Smith added that the proper remedy is to overturn the policy; we wholeheartedly agree. Teaching must be about helping kids develop the skills and knowledge they need to be prepared for college, career and life—not be about focusing on test scores for punitive purposes.”

HFT President Zeph Capo: “With this decision, Houston should wipe clean the record of every teacher who was negatively evaluated. From here on, teacher evaluation systems should be developed with educators to ensure that they are fair, transparent and help inform instruction, not be used as a punitive tool.”

###

Alex Molnar, research professor and publications director at the National Education Policy Center at the University of Colorado in Boulder, writes here about the privatization movement and its sustained attack on public education.

He writes:

Today, politicians in thrall to neoliberal ideology seek to subordinate the democratic mission of public education to a theory of market-driven economic development and social organization. Policy deliberations are now dominated by of econometric modeling and production function research. This modeling and research is often used, inappropriately, to make decisions about the value of education reforms. The mathematical models used by researchers are made to “work” only by assuming away much of the real world in which people live and students learn. The phantasmagorical belief in neutral “scientific” expertise as the primary basis for policymaking has, therefore, profoundly antihuman as well as antidemocratic implications — a topic Sheila Dow takes up in “People Have Had Enough of Experts.”[5]

The major education reforms of the past 35 years — education vouchers, charter schools, tuition tax credits, and education savings accounts — all seek to remove public schools from the control of elected bodies; to subject them to the “laws” of the “market”; and to put them at the service of the economic elite. The world being called into existence is based on the belief that anyone, but not everyone, can succeed—a world of winners and losers, each of whom has earned his or her fate.

Of course, if the privatizers actually believed in science or evidence, they would have already abandoned vouchers, which has no research to support it, and whose results have been shown in some places to actually harm students. In effect, students are given a low-cost voucher to spend in a school where teachers are usually uncertified and the curriculum is based on 19th century ideas that have been long disproven. It is ideology, not science, that drives the voucher movement, and its wicked stepsisters, tax credits and education savings accounts.

Those who believe in evidence would also demand transparency and accountability from privately managed charter schools, which in many states are excused from such inconveniences and use their freedom to kick out and exclude students they don’t want.

Molnar examines the policies of the past 25 years and their neglect of the lives of people affected by them.

He writes:

Over the past two and a half decades, the poor in privatized urban schools have been successfully harnessed to the delivery of reliable profits to investors and munificent salaries to executives. At the same time, the working class has discovered that schools in their communities often cost more than they can afford to pay. The decades of wage stagnation, unemployment, and tax shifting have taken their toll. Teachers and the unions that had won them the relatively high wages, job security, and benefits that are a distant memory for many blue collar workers became a useful target for the ideologues and politicians pursuing neoliberal reforms.

The neoliberal argument is that public schools cost too much (the largest item in a school budget is for teacher salaries) and performed too poorly to justify the tax dollars they commanded. If “star” teachers could be freed from the union wage scale to earn what they were worth, the resulting competition would create incentives for better teacher performance. Mediocre teachers would earn less, and low performing teachers would be fired. The mechanism proposed for measuring teacher performance was assessing the performance of their students on standardized tests. So began the policy embrace of “Value Added Assessment” (VAA). In the kind of methodologically sophisticated, intellectually fatuous study that has become all too common, Chetty, Friedman, and Rockoff claim to have found long-term economic benefits for students whose teachers have higher “value added” scores.[22]

This is a valuable overview of the recent past, the present, and the likely future. Unless we fight back hard.

The Good Old Days

Don’t you miss the good old days?
The days of school deforming ways?
When Arne ruled with iron hand
With Common Core and test and VAM?
And Cuomo plotted night and day
The way to make the schools obey?
And Rhee was riding on her broom
And closing schools and spreading doom?
And charter schools in neighborhoods
Were popping up like shrooms in woods
And billionaires were here and there
And all about and everywhere?
Don’t you miss reformy times
Immortalized by someDAM rhymes?
Well, good old days of yesteryear
Have never left, are still right here
The good old days were never gone
The school deform lives on and on

Our blog poet writes about Senator Franken’s statement that no Democrat will vote to confirm DeVos, highlighting contradictions:

“No Democrat will vote”

No Democrat would vote
For testing and for VAM
No Democrat would vote
For any other sham
No Democrat would vote
For Gates and Common Core
No Democrat would vote
For David Coleman lore
No Democrat would vote
For closing public schools
No Democrat would vote
For Arne Duncan rules
No Democrat would vote
For voucher and for charter
No Democrat would vote
To focus on the “smarter”
No Democrat will vote
For rule by billionaire
No Democrat will vote
Tautology is there