Archives for category: VAM (value-added modeling)

John Ewing, mathematician, is CEO of Math for America, an organization that supports teachers of mathematics.

In this post, he reviews some of the recent ill-considered efforts to “respect” teachers and offers advice about the minimum conditions necessary to assure that teachers have the respect, autonomy, and trust that professionals deserve.

You will also enjoy reading John Ewing’s brilliant takedown of teacher evaluation by test scores, which he called “Mathematical Intimidation.”

Melinda Gates told the National Conference of State Legislatures that the Gates Foundation has no intention of backing away from their agenda of Common Core, teacher evaluations that include test scores, charter schools, and digital learning.

No matter how controversial, no matter how much public pushback, they are determined to stay the course. For some reason, she thinks that the foundation is a “neutral broker,” when in fact it is an advocate for policies that many teachers and parents reject. She also assumes that the Gates Foundation has “the real facts,” when in fact it has a strong point of view reflecting the will of Bill & Melinda. There was no reference to evidence or research in this account of her position. Her point was that, no matter what the public or teachers may say, no matter how they damage the profession and public education, the multi-billion dollar foundation will not back down from its priorities. The only things that can stop them are informed voters and courts, such as the vote against charter schools in Nashville and the court decision in Washington State declaring that charter schools are not public schools.

The question that will be resolved over the next decade is whether the public will fight for democratic control of public schools or whether the world’s richest man can buy public education.

Melinda Gates said she and her husband, Microsoft co-founder Bill Gates, learned an important lesson from the fierce pushback against the Common Core State Standards in recent years. Not that they made the wrong bet when they poured hundreds of millions of dollars into supporting the education standards, but that such a massive initiative will not be successful unless teachers and parents believe in it.

“Community buy-in is huge,” Melinda Gates said in an interview here on Wednesday, adding that cultivating such support for big cultural shifts in education takes time. “It means that in some ways, you have to go more slowly.”

That does not mean the foundation has any plans to back off the Common Core or its other priorities, including its long-held belief that improving teacher quality is the key to transforming public education. “I would say stay the course. We’re not even close to finished,” Gates said.

The Bill & Melinda Gates Foundation has helped shape the nation’s education policies during the past decade with philanthropic donations that have supported digital learning and charter schools and helped accelerate shifts not only to the new, common academic standards, but to new teacher evaluations that incorporate student test scores.

The Obama administration shared and promoted many of the foundation’s priorities, arguing that they were necessary to push the nation’s schools forward and close yawning achievement gaps. Now that a new federal education law has returned authority over public education to the states, the foundation is following suit, seeking to become involved in the debates about the direction of public schools that are heating up in state capitals across the country.

Speaking here at a meeting of the National Conference of State Legislatures, Melinda Gates told lawmakers on Wednesday that the new federal education law, the Every Student Succeeds Act, gives them a chance to grapple with whether “we are doing everything in our power to ensure that students are truly graduating ready to go on to meaningful work or to college.”

“I want the foundation to be the neutral broker that’s able to bring up the real data of what is working and what’s not working,” Gates said in an interview afterward.

She went on to say that the foundation would continue to pursue its priorities.

“I think we know what the big elements are in education reform. It’s how do you support the things that you know work and how do you get the whole system aligned behind it,” Gates said. “I’m not telling you it’s going to be easy. There are now 50 states that have to do it, and there isn’t this federal carrot or the stick, the push or pull, to help them along.”

The agenda she described is not one that everyone considers neutral. It includes supporting the Common Core standards and developing lesson-planning materials to help teachers teach to those standards; promoting personalized learning, or digital programs meant to target students’ individual needs; and, above all, improving the quality of teachers in the nation’s classrooms, from boosting teacher preparation to rethinking on-the-job professional development.

In an interview published in The Hechinger Report, Randi Weingarten expresses her belief that Hillary Clinton will change course from the Obama education policies. She expects that a President Clinton would select a new Secretary of Education, one who shares her expressed belief in strengthening public schools and supporting teachers.

Emmanuel Felton, who conducted the interview, writes:

While teachers unions have long been a key pillar in Democratic Party, they’ve been on the outs with President Barack Obama’s education department. The administration doubled down on Republican President George W. Bush’s educational agenda of holding schools accountable for students’ test scores. Under the administration’s $3 billion School Improvement Grant program, for example, struggling schools had options to implement new accountability systems for teachers, remove staff, be closed or converted into charter schools, the vast majority of which employ non-unionized staff.

These policies devastated some local teachers unions, including Philadelphia’s, which lost 10,000 members during the Obama and Bush administrations. Weingarten expects Clinton to totally upend this agenda, and hopes she won’t reappoint Education Secretary John King, who was just confirmed by the senate in March.

From the day he was elected, President Obama decided to maintain the punitive policies of George W. Bush’s No Child Left Behind and made standardized testing even more consequential. He and his Secretary of Education Arne Duncan pressed for higher standards, tougher accountability, and more choices, especially charter schools. They used Race to the Top to promote the evaluation of teachers by their students’ test scores, a policy that cost hundreds of millions, perhaps billions of dollars, with nothing to show for it.

Let’s all hope that Hillary Clinton, if elected, will recognize the damage done by the Bush-Obama education agenda and push the “reset” button for a federal policy that helps children, educators, and public schools.

Our blog poet has not commented lately. Poet, we miss you! Come back!

“The night they drove Statricksy down” (parody of “The night they drove Old Dixie down)

Thomas Kane is my name and I drove on the VAMville train

‘Til Audrey Beardsley came and tore up the tracks again.

After the ASA* paper knife , we were hungry, just barely alive.**

By twenty-fourteen, Rich man had fell.
It’s a time I remember, oh so well.

(*American Statistical Association, **only had $ 45 million from the “Rich man”, Bill Gates)

The night they drove statricksy down

And all the bells were ringing,

The night they drove statricksy down

And all the people were singing

They went, “Na,na,na.na,

Na na na na na na na na na.”

Back with my colleague, Raj Chet-ty, when one day he called to me,
“Thomas, quick, come see, there goes the Gatesly Billee!”
Now I don’t mind I’m choppin’ stats, and I don’t care if I’m paid by the brats
You take what you need and leave the rest,
But they should never have taken the VAMmy best.

The night they drove statricksy down
And all the bells were ringing,
The night they drove statricksy down
And all the people were singing
They went, “Na,na,na.na,
Na na na na na na na na na.”

Like my “Father”*** before me, I will work the VAM
And like my colleague before me, I took a junk-stat stand.
He was just 34, proud and brave,
but the ASA put him in his grave.
I swear by the mud below my feet
You can’t raise a Kane back up when he’s in defeat

(***William Sanders, Father of VAM)

The night they drove statricksy down
And all the bells were ringing,
The night they drove statricksy down
And all the people were singing
They went, “Na,na,na.na,
Na na na na na na na na na.”

The night they drove statricksy down
And all the bells were ringing,
The night they drove statricksy down
And all the people were singing
They went, “Na,na,na.na,
Na na na na na na na na na.”

Harvard Magazine published a noteworthy article about the failure of pay-for-performance plans in hospitals, written by Marina N. Bolotnikova. It is worth your time to read. It is devastating.

The logic of pay-for-performance systems is simple enough: pay doctors and hospitals based on how well their patients are doing, rather than on the number of medical services they provide. The payment structure was designed to fix a central problem in American healthcare. The United States spends far more per person on healthcare than any other country, yet has the poorest health outcomes in the advanced world. Pay-for-performance, also known as value-based purchasing, was meant to encourage doctors to optimize the welfare of patients while discouraging spending on unnecessary care.

There was never any evidence that pay-for-performance works, said Li professor of international health Ashish Jha, who investigates the effectiveness of economic incentives in the healthcare market. A recent paper published this April in the BMJ by Jha finds that the federal pay-for-performance program under Medicare, the Hospital Value-Based Purchasing (HVBP) program, hasn’t had any impact on mortality rates. Nor do HVBP-participating programs show any statistically significant advances over the small number of hospitals that don’t participate in the program. No hospitals, not even those with the worst mortality rates before the program was implemented, showed an improvement that could be attributed to pay-for-performance. “We looked across all different conditions and couldn’t find a single one where it seemed to have a meaningful effect,” Jha explained.

In the early 2000s, the Bush administration ran a pilot version of pay-for-performance at about 200 hospitals that agreed to tie their payments to certain quality measures. Even then, Jha said, “the broad consensus in the community was that it didn’t really work.” Despite this, the system was implemented on a national scale by the Affordable Care Act (ACA), which created the HVBP program at more than 90 percent of U.S. hospitals. “There was a sense that we needed to begin somewhere,” he explained, “and that this program would be good to test out nationally.”

Medicine and education could learn from one another. Why has there been so little attention to the persistent failure of pay-for-performance plans in education? States and districts and the federal government pour hundreds of millions, billions, into developing incentives, despite the fact that the National Academy of Sciences studied the issue in education and said that such plans don’t work, especially when the bonuses are tied to test scores.

I wrote a response to an editorial that appeared in the Boston Globe, which advocated for using test scores to judge teacher quality.

My response explained why that idea doesn’t work.

I cited evidence and experience.

But people who live in Massachusetts who don’t read the Globe online won’t see it.

Please forward to friends, elected officials, and policymakers.

Open the article to see the links to sources.

Here are some excerpts:

Evaluating teachers by test scores has not raised scores significantly anywhere. Good teachers have been fired by this flawed method. A New York judge ruled this method “arbitrary and capricious” after one of the state’s best teachers was judged ineffective.

Test-based evaluation has demoralized teachers because they know it is unfair to judge them by student scores. Many believe it has contributed to a growing national teacher shortage and declining enrollments in teacher education programs.

A major problem with test-based evaluation is that students are not randomly assigned. Teachers in affluent suburbs may get higher scores year after year, while teachers in urban districts enrolling many high-need students will not see big test score gains. Teachers of English-language learners, teachers of students with cognitive disabilities, and teachers of children who live in poverty are unlikely to see big test score gains, even though they are as good or even better than their peers in the suburbs. Even teachers of the gifted are unlikely to see big test score gains, because their students already have such high scores. Test scores are a measure of class composition, not teacher quality.

Seventy percent of teachers do not teach subjects that have annual tests. Schools could develop standardized tests for every subject, including the arts and physical education. But most have chosen to rate these teachers by the scores of students they don’t know and subjects they never taught.

Scholarly groups like the American Educational Research Association and the American Statistical Association have warned against using test scores to rate individual teachers. There are too many uncontrolled variables, as well as individual differences among students to make these ratings valid. The biggest source of variation in test scores is not the teacher, but students’ family income and home environment.

The American Statistical Association said that teachers affect 1 percent to 14 percent of test score variation. The ASA is an impeccable nonpartisan, authoritative source, not influenced by the teachers’ unions.

The Gates Foundation gave a grant of $100 million to the schools of Hillsborough County, Florida (Tampa), to evaluate their teachers by gains and losses in student test scores. It was an abject failure. The district drained its reserve funds, spending nearly $200 million to implement the foundation’s ideas. Gates refused to pay the last $20 million on its $100 million pledge. The superintendent who led the effort was fired and replaced by one who promised a different direction.

Should Massachusetts cling to a costly, failed, and demoralizing way to evaluate teachers? Should it ignore evidence and experience?

Common sense and logic say no.

Should teachers be judged “subjectively”? Of course. That is called human judgment. Is it perfect? No. Can it be corrected? Yes. Most professionals are judged subjectively by their supervisors and bosses. Standardized tests are flawed instruments. They are normed on a bell curve, guaranteeing winners and losers. They often contain errors — statistical errors, human errors, random errors, scoring errors, poorly worded questions, two right answers, no right answers. No one’s professional career should hinge on the answers to standardized test questions.

Massachusetts is widely considered the best state school system in the nation. The hunt for bad teachers who were somehow undetected by their supervisors is fruitless. The Legislature is right to return the decision about which teachers are effective and which are not to the professionals who see their work every day.

Diane Ravitch is president of the Network for Public Education, a nonprofit that advocates on behalf of public education. She is the author of “Reign of Error: The Hoax of the Privatization Movement and the Danger to America’s Schools.”

Surprise: I wrote the article.

Nothing I wrote will be a surprise to readers of this blog, but may be new to the readers of The Boston Globe.

I was moved to write it because the Globe published an editorial calling for the opposite.

Last week, the Houston Independent School Board deadlocked in a 3-3 tie vote on whether to renew its contract with the vendor supplying the teacher evaluation program.

Audrey Amrein-Beardsley explains their decision here.

At least three board members realized that five years of this program had not moved the needle by an inch. If performance matters, then EVAAS was a failure.

Beardsley is one of the nation’s leading researchers in the study of teacher evaluation.

She writes:

Seven teachers in the Houston Independent School District (HISD), with the support of the Houston Federation of Teachers (HFT), are taking HISD to federal court over how their value-added scores, derived via the Education Value-Added Assessment System (EVAAS), are being used, and allegedly abused, while this district that has tied more high-stakes consequences to value-added output than any other district/state in the nation. The case, Houston Federation of Teachers, et al. v. Houston ISD, is ongoing.

But just announced is that the HISD school board, in a 3:3 split vote late last Thursday night, elected to no longer pay an annual $680K to SAS Institute Inc. to calculate the district’s EVAAS value-added estimates. As per an HFT press release (below), HISD “will not be renewing the district’s seriously flawed teacher evaluation system, [which is] good news for students, teachers and the community, [although] the school board and incoming superintendent must work with educators and others to choose a more effective system.”

Open the link, read the full article, and read her links. This is excellent news.

The bad part of her post is the news that the federal government is still giving out grants that require districts to continue using this flawed methodology, despite the fact that it hasn’t worked anywhere.

Apparently, HISD was holding onto the EVAAS, despite the research surrounding the EVAAS in general and in Houston, in that they have received (and are still set to receive) over $4 million in federal grant funds that has required them to have value-added estimates as a component of their evaluation and accountability system(s).

So Houston will have to find a new vendor of a failed methodology.

The Boston Globe seems to be the Rip van Winkle of the mainstream media. It recently published an editorial that insists that teachers should be evaluated by the test scores of their students. Really. Apparently it is still 2010 in the offices of the Globe, when Arne Duncan claimed that this was the very best way to determine which teachers were effective or ineffective.

 

But it is no longer 2010. The U.S. Departnent of Education handed out $5 billion to states to promote test-based evaluation. The Gates Foundation gave away hundreds of millions of dollars to states to use test scores to evaluate teachers. This method has had negative results everywhere. It has demoralized teachers everywhere. It has contributed to a growing national teacher shortage and declining enrollments in education programs.

 

Scholarly groups like the American Educational Research Association and the American Statistical Association have warned against using test scores to rate individual teachers. There are too many uncontrolled variables, as well as individual differences among students. The American Statistical Association said that teachers affect 1-14% of test score variation. Surely the Boston Globe editorial board must be aware of that report by an impeccable nonpartisan authoritative source. Surely the Boston Globe editorial board must know that teachers in affluent districts are likely to produce high test scores, while teachers of children with disabilities, English language learners, impoverished children, and homeless children are likely to get low test scores. Even teachers of the gifted will receive low ratings because their students get small test score gains since they are already at the top of the scale.

 

The Boston Globe editorial board should learn about the disastrous experience with Gates-style test-based evaluation in Hillsborough County, Florida. The district accepted a $100 million award from the Gates Foundation to rate its teachers by test score gains and losses. It was an abject failure. The district drained its reserve funds. It concluded that it would cost the district $52 million a year to sustain the Gates program. The superintendent who led the effort, MaryEllen Elia, was fired. Gates cut its ties to the county and stopped the payout after wasting $80 million.

 

Should Massachusetts cling to a costly, expensive, failed way to evaluate teachers? Should it ignore evidence and experience?

 

Common sense and logic say no. Will someone send this post to the editorial board of the Boston Globe?

 

 

 

 

 

 

 

 

 

2010 was the high watermark of the corporate reform movement.

In spring 2010, the entire staff at Central Falls, Rhode Island, was fired because of low test scores, which created a national sensation. Arne Duncan and President Obama hailed the courage of Deborah Gist, the state superintendent, and Frances Gallo, the city superintendent, who ordered and confirmed the strategy. Duncan said the firings showed that the administrators were “doing the right things for kids.”

Thus began the reformers’ war against teachers.

In September 2010, “Waiting for Superman,” debuted with a multimillion dollar campaign to promote it: the cover of TIME, appearances by the “stars” on Oprah (Michelle Rhee, Geoffrey Canada, Bill Gates, etc.), and NBC’s Education Nation, focused on promoting the film and its advocacy for charters. “Superman” was a hit job on unions, teachers, and public schools. Its data were skewed, and some of its scenes were staged. It was denied an Academy Award. But Bill Gates put up at least $2 million for public relations.

Thus launched the reformers’ fraudulent fight for privatization as a “civil rights” issue.

Into this fray came the Los Angeles Times, with its own evaluation of thousands of teachers in Los Angeles, created by an economist who employed the methods approved by the Gates Foundation. Teachers were rated on a scale from least effective to most effective. One of those teachers, a dedicated fifth grade teacher named Rigoberto Ruelas, jumped off a bridge and committed suicide after he was publicly labeled as one of the least effective teachers in math and average in reading. Who knew that becoming a teacher would be a hazardous profession?

Anthony Cody delves into the journalistic responsibility of the Los Angeles Times in this important post. The LA Times hired an economist who created VAM ratings and used test scores to rank teachers. Its reporters, Jason Felch and Jason Song, warned against using test scores as the only measure to rank teachers, then proceeded to use test scores as the only measure to rank teachers. The two Jasons, as they were known, hoped to win a Pulitzer Prize. They didn’t. They did come in second in the Education Writers Association choice of the best reporting of the year. Felch was subsequently fired for an ethical breach that involved inappropriate relations with a source.

Cody is concerned about the ethics of journalists who cloak their advocacy and partisanship behind the charade of journalistic independence.

Now, it turns out that the Hechinger Institute at Teachers College, Columbia University, funded the LA Times’ rating scheme. And who do you think funded the Hechinger Report: the Gates Foundation.

We know more about VAM now. We know that it has been rejected by numerous scholars and scholarly associations as invalid, unstable, and unreliable.

Who killed Rigoberto Ruelas?