Archives for category: VAM (value-added modeling)

Peter Greene recently read a blog debate in the “Néw York Times” on the topic of how to improve teaching. He reacted strongly to the contribution by Eric Hanushek, an economist at the Hoover Institution. Hanushek is well known for his belief that the best way to tell which teachers are best is to see which ones get the highest test score gains; that raising scores will eventually produce trillions of dollars in economic growth; and that teachers who can’t produce higher scores should be “deselected.” That is, fired.

Here is the beginning of Greene’s critique of the economists’ contribution to education policy;

“When you want a bunch of legit-sounding baloney about education, call up an economist. I can’t think of a single card-carrying economist who has produced useful insights about education, schools and teaching, but from Brookings to the Hoover Institute, economists can be counted on to provide a regular stream of fecund fertilizer about schools.

“So here comes Eric Hanushek in the New York Times (staging one of their op-ed debates, which tend to resemble a soccer game played on the side of a mountain) to offer yet another rehash of his ideas about teaching. The Room for Debate pieces are always brief, but Hansuhek impressively gets a whole ton of wrong squeezed into a tiny space. Here’s his opening paragaph:

“Despite decades of study and enormous effort, we know little about how to train or select high quality teachers. We do know, however, that there are huge differences in the effectiveness of classroom teachers and that these differences can be observed.”

“This is a research puzzler of epic proportions. Hansuhek is saying, “We do not know how to tell the difference between a green apple and a red apple, but we have conclusive proof that a red apple tastes better.” Exactly what would that experimental design look like? Exactly how do you compare the red and green apples if you can’t tell them apart?

“The research gets around this issue by using a circular design. We first define high quality teachers as those whose students get high test scores. Then we study these high quality teachers and discover that they get students to score well on tests. It’s amazing!

“Economists have been at the front of the parade declaring that teachers cannot be judged on qualifications or anything else except results. Here’s a typical quote, this time from a Rand economist: “The best way to assess teachers’ effectiveness is to look at their on-the-job performance, including what they do in the classroom and how much progress their students make on achievement tests.”

“It’s economists who have given us the widely debunked shell game that is Valued Added Measuring of teachers, and they’ve been peddling that snake oil for a while (here’s a research summary from 2005). It captures all the wrong thinking of economists in one destructive ball– all that matters about teachers is the test scores they produce, and every other factor that affects a student’s test score can be worked out in a fancy equation.”

A few years ago, I engaged in an Internet debate with Rick Hanushek on the Eduwonk website. Here is the exchange:
Hanushek
My Response
Hanushek
My response

I agree with Peter Greene that economists have had far too much influence on educational policy. The attempt to quantify teaching and learning is ruinous to education and buries any consideration of the purpose of education. Children are not widgets. Learning is far too complex to be measured by standardized multiple-choice tests. Education includes many goals other than test scores. Teachers are professionals and should not be treated as interchangeable low-wage workers.

Seven outstanding teachers wrote a letter to Governor Cuomo. It was published in the Albany Times-Union, where there is a good chance he and members of the Legislature might read it. Unfortunately, it is behind a paywall. Maybe by now the paywall has disappeared. I hope so as everyone in every state should read this excellent letter.

The teachers write:

The following article was written by seven New York state Teachers of the Year: Ashli Dreher (2014, Buffalo); Katie Ferguson (2012, Schenectady); Jeff Peneston (2011, Syracuse); Rich Ognibene (2008, Rochester); Marguerite Izzo (2007, Malverne); Steve Bongiovi (2006, Seaford); and Liz Day (2005, Mechanicville)

Dear Governor Cuomo:

We are teachers. We have given our hearts and souls to this noble profession. We have pursued intellectual rigor. We have fed students who were hungry. We have celebrated at student weddings and wept at student funerals. Education is our life. For this, you have made us the enemy. This is personal.

Under your leadership, schools have endured the Gap Elimination Adjustment and the tax cap, which have caused layoffs and draconian budget cuts across the state. Classes are larger and support services are fewer, particularly for our neediest students.

We have also endured a difficult rollout of the Common Core Standards. A reasonable implementation would have started the new standards in kindergarten and advanced those standards one grade at a time. Instead, the new standards were rushed into all grades at once, without any time to see if they were developmentally appropriate or useful.

Then our students were given new tests — of questionable validity — before they had a chance to develop the skills necessary to be successful. These flawed tests reinforced the false narrative that all public schools — and therefore all teachers — are in drastic need of reform. In our many years of teaching, we’ve never found that denigrating others is a useful strategy for improvement.

Now you are doubling down on test scores as a proxy for teacher effectiveness. The state has focused on test scores for years and this approach has proven to be fraught with peril. Testing scandals erupted. Teachers who questioned the validity of tests were given gag orders. Parents in wealthier districts hired test-prep tutors, which exacerbated the achievement gap between rich and poor.

Beyond those concerns, if the state places this much emphasis on test scores who will want to teach our neediest students? Will you assume that the teachers in wealthier districts are highly effective and the teachers in poorer districts are ineffective, simply based on test scores?

Most of us have failed an exam or two along life’s path. From those results, can we conclude that our teachers were ineffective? We understand the value of collecting data, but it must be interpreted wisely. Using test scores as 50 percent of a teacher’s evaluation does not meet this criterion.

Your other proposals are also unlikely to succeed. Merit pay, charter schools and increased scrutiny of teachers won’t work because they fundamentally misdiagnose the problem. It’s not that teachers or schools are horrible. Rather, the problem is that students with an achievement gap also have an income gap, a health-care gap, a housing gap, a family gap and a safety gap, just to name a few. If we truly want to improve educational outcomes, these are the real issues that must be addressed.

Much is right in public education today. We invite you to visit our classrooms and see for yourself. Most teachers, administrators and school board members are doing quality work. Our students and alumni have accomplished great things. Let’s stop the narrative of systemic failure.

Instead, let’s talk about ways to help the kids who are struggling. Let’s talk about addressing the concentration of poverty in our cities. Let’s talk about creating a culture of family so that our weakest students feel emotionally connected to their schools. Let’s talk about fostering collaboration between teachers, administrators and elected officials. It is by working together, not competing for test scores, that we will advance our cause.

None of these suggestions are easily measured with a No. 2 pencil, but they would work. On behalf of teachers across the state, we say these are our kids, we love them, and this is personal.

Audrey Amrein Beardsley, a national authority on teacher evaluation, reviews here the latest scholarly research on value-added measurement” or VAM. This is the practice of evaluating teachers by changes in their students’ test scores. It was made into a national issue by Race to the Top, which required states to make VAM a significant part of teacher evaluation. In granting waivers to states from the Draconian sanctions of NCLB, Arne Duncan required states to adopt VAM.

The research keeps building against the usefulness of VAM. The latest study concludes that VAM is highly unreliable. Children are not randomly assigned, and teachers face widely varying challenges.

Beardsley writes:

“In a recent paper published in the peer-reviewed journal Education Finance and Policy, coauthors Cassandra Guarino (Indiana University – Bloomington), Mark Reckase (Michigan State University), and Jeffrey Wooldridge (Michigan State University) ask and then answer the following question: “Can Value-Added Measures of Teacher Performance Be Trusted?…

“From the abstract, authors “investigate whether commonly used value-added estimation strategies produce accurate estimates of teacher effects under a variety of scenarios. [They] estimate teacher effects [using] simulated student achievement data sets that mimic plausible types of student grouping and teacher assignment scenarios. [They] find that no one method accurately captures true teacher effects in all scenarios, and the potential for misclassifying teachers as high- or low-performing can be substantial.”

She adds:

“They found…

“No one [value-added] estimator performs well under all plausible circumstances, but some are more robust than others…[some] fare better than expected…[and] some of the most popular methods are neither the most robust nor ideal.” In other words, calculating value-added regardless of the sophistication of the statistical specifications and controls used is messy, and this messiness can seriously throw off the validity of the inferences to be drawn about teachers, even given the fanciest models and methodological approaches we currently have going (i.e., those models and model specifications being advanced via policy).

“[S]ubstantial proportions of teachers can be misclassified as ‘below average’ or ‘above average’ as well as in the bottom and top quintiles of the teacher quality distribution, even in [these] best-case scenarios.” This means that the misclassification errors were are seeing with real-world data, we are also seeing with simulated data. This leads us to more concern about whether VAMs will ever be able to get it right, or in this case, counter the effects of the nonrandom assignment of students to classrooms and teachers to the same.

“Researchers found that “even in the best scenarios and under the simplistic and idealized conditions imposed by [their] data-generating process, the potential for misclassifying above-average teachers as below average or for misidentifying the “worst” or “best” teachers remains nontrivial, particularly if teacher effects are relatively small. Applying the [most] commonly used [value-added approaches] results in misclassification rates that range from at least 7 percent to more than 60 percent, depending upon the estimator and scenario.” So even with a pretty perfect dataset, or a dataset much cleaner than those that come from actual children and their test scores in real schools, misclassification errors can impact teachers upwards of 60% of the time….

“In sum, researchers conclude that while certain VAMs hold more promise than others, they may not be capable of overcoming the many obstacles presented by the non-random assignment of students to teachers (and teachers to classrooms).

“In their own words, “it is clear that every estimator has an Achilles heel (or more than one area of potential weakness)” that can distort teacher-level output in highly consequential ways. Hence, “[t]he degree of error in [VAM] estimates…may make them less trustworthy for the specific purpose of evaluating individual teachers” than we might think.”

I sent this to each Senate Committee member:

 

 

Dear Sen. xxxx
I am a TN educator and I’d like to ask that you consider some facts about public education reform in TN generally and the proliferation of charter schools in particular.

 

The testing & accountability measures in TN were written by ALEC and by for-profit entities that have an interest in privatizing public education.

 

The value-added model (TN version is TVASS), marketed as an indicator of teacher quality, is junk science according to the American Statistical Association and by a majority of independent researchers: The lit review is here:

 

http://vamboozled.com/recommended-reading/value-added-models/

 

How can an education system improve if Congress allows junk science to dictate the direction of our education system? Test scores are designed to sort & rank. Testing is not learning- it’s a tool that teachers know when & how to use. Congress doesn’t dictate to any other profession how to use the tools of their profession. Why should teaching be any different?

 

All around the country VAM & standardized test scores are being misused to close schools, disperse, destabilize poor communities, sort out high needs (e.g. expensive children in SPED or at-risk) and privatize. The Dept of Education is now promoting VAM junk-science for colleges of Education.

 

Accountability has been in short supply for TN’s charter authorizer Achievement School District (ASD) and for outside consultants sucking up our tax dollars for invalid teacher evaluations and useless standardized tests(e.g., TEAM/TAP was developed by convicted felon Michael Milken & his brother and has no valid research line to support it’s claims)

 

Here are some persistent problems with charter schools & education privatizaion that deserve greater accountability and compliance.

 

1. Increased Segregation

 

• The vast majority of high-poverty charters fail due to racial & socio-economic segregation. The high-poverty model has not met with success at a national level.

 

• The most comprehensive study of charter schools completed to date found that only 17% of charter schools outperformed comparable traditional pubic schools.83% of public schools are better than charters. New Orleans Charter Schools have the lowest ACT scores in the country.

 

• Many families now believe- as do virtually all leading colleges & universities- that racial, ethnic, & income diversity enriches classrooms.

 

• The main problem with American schools in not their teachers or their unions, but poverty & economic segregation.

 

Reference:

 

Kahlenberg (2013). From all walks of life: New hopes for school integration. American Educator. Winter 2012-2013, pp. 2 – 40.

 

2. Sanctioned Discrimination or Whose Choice?

 

• The first choice of most parents is to send their child to a high-quality neighborhood school; it is unclear how this bill supports that choice. In fact, we have seen how the rapid expansion of the charter sector has undermined neighborhood schools, drawing resources from them and at the same time expecting them to serve our most at-risk students. –

 

• Charters take public money yet have the legal status of private schools.

 

• Charter organizations have gone to court to protect themselves from educating & retaining ALL children.

 

• Charters discriminate against children with disabilities, children who do not test well, or who do not fit into inflexible discipline policies. Such children may be admitted to bolster enrollment but are expelled or counseled out after BEP funds are distributed, Public schools lose $6,000/child and face class overloads near testing time.

 

• Charters advertise ‘choice’ but overwhelmingly exclude parent voice.

 

• Parents have no legal recourse to challenge harmful charter school practices. Charters may legally ignore the key aspect of parent involvement: school level decision- making.

 

• Parents and the public are consistently misled about the community desires for a charter school. Charter waitlists cannot be confirmed and many records are slipshod.

 

• In New Orleans where all public schools have disappeared, the most difficult to teach children have been abandoned.

 

References:

Green, P. C., III, Baker, B. D., & Oluwole, J. O. (2013) Having it both ways: How charter schools try to obtain funding of public schools and the autonomy of private schools. Emory Law Journal, Vol. 63.303.

 

Parents Across America (PAA) http://parentsacrossamerica.org/parents-america-hr2218-%e2%80%9cempowering-parents-quality-charter-schools-act%e2%80%9d/#sthash.Ch0TKntq.dpuf

 

Welner, K. G. & Miron, G., (2014). Wait,wait. Don’t mislead me! Nine reasons to be skeptical about charter waitlist numbers. National Education Policy Center, University of Colorado, Boulder. http://nepc.colorado.edu/publication/charter-waitlists

 

Gabor, A. (2013) The great charter tryout. The Investigative Fund. http://www.theinvestigativefund.org/investigations/politicsandgovernment/1848/

 

What we support:

 

More community schools just like the highly successful Pond Gap in Knoxviile, TN.

 

To improve the schools we have, rather than shutting down or turning around traditional schools to make way for more charter schools.

 

All charter schools to have neighborhood boundaries and accept all children from within those boundaries whose parents choose to enroll their child at the charter school. Charter school enrollment processes should be consistent with and as simple as those of neighborhood public schools.

 

Charter schools should be held accountable for their enrollment, discipline, transfer, and other practices.

 

Charter schools and all other schools receiving public funds must be equally transparent and accountable to the public.

 

Finally, TN has a shameful 45% child poverty rate. My state has one of the highest rates of low wage & minimum wage jobs in the country. Our public schools in TN need resources- not privatization- to compensate for failed political & economic policies.

 

Thank-you for your work & consideration,

 

 

Joan Grim

Jersey Jazzman calls out journalist Jon Chait for being against political correctness except when it serves his purpose.

It seems Chait was deeply offended when I said that former CNN anchor Campbell Brown is no educational expert. Her campaign to eliminate teacher tenure won’t improve education, I dared to say. We might, as a test case, compare the academic performance of states that have tenure with states that don’t, but that involves a rudimentary knowledge of actual research.

It is highly offensive to those bashing teachers to suggest that their campaign to remove teacher tenure and to provide merit pay has no evidence behind it and is illogical. They don’t like it when you point out that VAM sounds good but doesn’t work. If they read the statement of the American Statistical Association, they would be informed, but that requires research, or if you open the link, reading.

Despite the lack of evidence for tying teacher evaluation to student test scores, despite the hundreds of millions spent to implement it without success, this is Arne Duncan’s line in the sand. He insists on mandated annual testing, because without it, his idea of teacher evaluation crashes. He doesn’t care that most teachers don’t teach tested subjects. It is not the annual tests he loves, it is the teacher grades based in annual test scores.

 

In this thoughtful article in Education Week, Alyson Klein explains the dilemma of states. They need an NCLB waiver, but to get it they must follow Duncan’s orders on teacher evaluation. If the new Congress reauthorizes NCLB, all of this might be swept away. So the US DOE is trying to lock states into plans that last until 2018, long after this administration is gone. Once Duncan is gone, most states will abandon his mandates if they can.

 

 

Klein writes:

 

 

Congress is moving full steam ahead on a rewrite of the No Child Left Behind Act that could undo nearly of the Obama administration’s K-12 policy priorities, including state goals for student achievement, dramatic school turnarounds, and evaluating teachers through test scores—and maybe even the tests themselves.

 

But, even the most optimistic prognosticators don’t expect the final legislation to make it across the finish line until the summer.

 

That means states with waivers from the No Child Left Behind law—42 plus the District of Columbia—will still have to negotiate the finer points of their accountability plans with the department for waiver renewals that could last through 2018-19, well beyond the end of the Obama administration.

 

Already states, including Texas and Maine, have been told they need to make changes to their teacher rating systems—or provide the department with much more information—before submitting their renewal applications at the end of March. Neither state’s waiver has been put on high risk status just yet. (More below.)

 

The administration, though, may be entering into the waiver-renewal process with a severely weakened hand, especially when it comes to holding states’ feet to the fire on the policy that seems nearest and dearest to its heart: crafting teacher evaluation systems that take state test scores into account, and align with the administration’s vision.

 

“I think there’s going to be so much state pushback on that that the department may have to be open to negotiations on what states put in for teacher evaluation,” said Terry Holiday, Kentucky’s education commissioner who, coincidentally enough, is testifying at a Senate NCLB reauthorization hearing on Tuesday on teacher quality…

 

What’s more, once the waivers are a thing of the past—either because NCLB has been reauthorized or because a new president has gotten rid of them—states aren’t likely to continue with teacher evaluation through outcomes on assessments, Holliday said.

 

I think we’d all quickly abandon all the work on tying teacher evaluation to test scores,” he said.

You have until February 2 to post your comment about whether the US Department of Education should impose VAM on teacher education. Test-happy DOE wants to evaluate colleges of education by the test scores of students taught by their graduates.

http://thewire.k12newsnetwork.com/2015/01/27/value-added-teacher-preparation-regulations-changes-your-comments/

Secretary Duncan wants to rate colleges of education by the test scores of students taught by their graduates.

Read this post by VAM expert Audrey Amrein Beardsley to learn why this is a very bad idea and how you can register your protest against it.

Does Louisiana need more value-added modeling? Douglas Harris says yes; Audrey. Amrein-Beardsley and Mercedes Schneider say no.

In a report to the State Board of Education, Harris proposed VAM for schools to increase accountability.

Beardsley reviews the research on VAM as well as her past exchanges with Harris and her specific critique of this proposal.

Beardsley writes:

“Harris concludes that “With these changes, Louisiana would have one of the best accountability systems in the country. Rather than weakening accountability, these recommendations [would] make accountability smarter and make it more likely to improve students’ academic performance.” Following these recommendations would “make the state a national leader.” While Harris cites 20 years of failed attempts in Louisiana and across all states across the country as the reason America’s public education system has not improved its public school students’ academic performance, I’d argue it’s more like 40 years of failed attempts because Harris’s (and so many others’) accountability-bent logic is seriously flawed.”

Schneider questions the practical value of VAM in her dissent. Schneider writes:

“Point systems for “grading” the teacher-student (and school-teacher-student) dynamic will always fall short because the complex nature of that dynamic defies quantifying. If test-loving reformers insist upon imposing high-stakes quantification onto schools and teachers, it will backfire, a system begging to be corrupted by those fighting to survive it.

“It is not that I cannot be evaluated as a teacher. It’s just that such evaluation is rooted a complex subjectivity that is best understood by those who are familiar with my reality. This should be true of the administrators at one’s school, and I am fortunate to state that it is true in my case.

“There are no numbers that sufficiently capture my work with my students. I know this. Yes, I am caught in a system that wants to impose a numeric values on my teaching. My “value” to my students cannot be quantified, nor can my school’s value to my students, no matter what the Harrises of this world might suggest in commissioned reports.”