Perhaps you recall the hoopla that surrounded the release in late 2011 of a study by economists Chetty, Friedman, and Rockoff, in which they claimed that a great teacher would produce a huge increase in lifetime earnings, fewer pregnancies, and other wonderful life outcomes. It was reported on the front page of the New York Times, where one of the authors said that the lesson of the study was that it was best to fire teachers who couldn’t produce big gains sooner rather than later. The study was discussed reverentially on the Newshour and President Obama referenced its conclusion in his 2012 State of the Union address.

The central claim was that the great teacher produced a lifetime gain of $266,000 for a typical classroom.

First out of the box to challenge the study was Bruce Baker of Rutgers. He pointed out that if a class had 26 students, each of them would see an increment of $1,000 per year, or about $20 a week, or less than $5 a day. Maybe. Or maybe not.

Recently he realized that the Chetty study was being used to sell test-based evaluation of teachers, and no one among the policymakers seemed aware of the defects and critiques of the study.

He decided that the study seems to have gotten a special kind of treatment, which he refers to as “Mountain-Out-of-a-Molehill-inator.” Once again, here views the skewing of the results to show that policymakers would be foolish to draw conclusions and apply them to real schools and real teachers.

He concludes:

“What really are the implications of this study for practice – for human resource policy in local public (or private schools)? Well, not much! A study like this can be used to guide simulations of what might theoretically happen if we had 10,000 teachers, and were able to identify, with slightly better than even odds, the “really good” teachers – keep them, and fire the rest (knowing that we have high odds that we are wrongly firing many good teachers… but accepting this fact on the basis that we were at least slightly more likely to be right than wrong in identifying future higher vs. lower value added producers). As I noted on my previous post, this type of big data – this type of small margin-of-difference finding in big data – really isn’t helpful for making determinations about individual teachers in the real world. Yeah… works great in big-data simulations based on big-data findings, but that’s about it.

“Indeed it’s an interesting study, but to suggest that this study has important immediate implications for school and district level human resource management is not only naive, but reckless and irresponsible and must stop.”