Mathematica Policy Research released a study that proves that experience matters.

Some readers thought the study was about merit pay, but it was not. Merit pay has never worked.

Merit pay studies usually compare one group of teachers matched to a similar group. One group is offered a bonus if they can raise test scores, the other is not. The bonus is supposed to incentivize the teachers to push their students to achieve higher test scores.

But that is not what happened in this study.

In this study, the the bonus was awarded for transferring to the low-performing school for two years, not for getting higher test scores.

What the study demonstrates is that if you offer a bonus of $20,000, you might attract the top talent in the district to teach in low-performing schools, and these older, experienced teachers will get better results than regular teachers, many of whom are brand new to teaching.

In her story about the study,  Dana Goldstein noted:

It’s also worth pointing out that these transfer teachers were far from the Teach for America archetype of a young, transient Ivy League grad. Their average age was 42, and they had an average of 12 years of experience in the classroom. They were also more likely than control group teachers to be African-American, to be homeowners, and to hold a master’s degree. In short, they were stable adults with deep ties to the cities in which they worked.

Audrey Amrein-Beardsley, one of the nation’s leading scholars of value-added measurement, points out the dissimilarity of the experimental group and the control group:

The high value-added teachers who were selected to participate in this study, and transfer into high-needs schools to teach for two years, were disproportionately National Board Certified Teachers and teachers with more years of teaching experience. The finding that these teachers, selected only because they were high value-added teachers was confounded by the very fact that they were compared to “similar” teachers in the high-needs schools, many of whom were not certified as exemplary teachers and many of whom (20%) were new teachers…as in, entirely new to the teaching profession! While the high value-added teachers who choose to teach in higher needs schools for two years (with $20,000 bonuses to boot) were likely wonderful teachers in their own rights, the same study results would have likely been achieved by simply choosing teachers with more than X years of experience or choosing teachers whose supervisors selected them as “the best.” Hence, this study was not about using “value-added” as the arbiter of all that is good and objective in measuring teacher effects, it was about selecting teachers who were distinctly different than the teachers to whom they were compared and attributing the predictable results back to the “value-added” selections that were made.

What the study really shows is the foolishness of the many states that are changing salary scales to discourage experienced teachers, removing stipends for masters degrees, and making other policies that discourage the very teachers that this study salutes. States like Tennessee and North Carolina, among others, are enacting laws to discourage or push out the very teachers that are considered “the best” in this study.

As Amrein-Beardsley observes:

Related, many of the politicians and policymakers who are advancing national and state value-added initiatives and policies forward are continuously using sets of false assumptions about teacher experience, teacher credentials, and how/why these things do not matter to advance their agendas forward. Rather, in this study, it seems that teacher experience and credentials mattered the most. Results from this study, hence, contradict initiatives, for example, to get rid of salary schedules that rely on years of experience and credentials, as value-added scores, as evidenced in this study, do seem to capture these other variables (i.e., experience and credentials) as well.

The takeaway? Blogger Steve Strieker of Wisconsin put it this way in an email to me:

Experience, education, age, and teacher willingness to participate seemed to matter in this case. The program also seems to have eyes on the eight ball.  Teacher accountability and stack-ranking evaluation systems are not part of the program. Unlike other merit pay studies, this was a low-stakes study. Testing scores were not connected to the bonus payout. Teachers chosen were paid the bonus for their service regardless of student performance.

If we want to see improvement and results, we should have policies and extra pay to recruit top teachers to work in hard-to-staff schools, and we should place high value on experience and education.