On May 3, I received an email from Professor Raj Chetty of Harvard University, informing me that his famous paper on value-added assessment of teachers was being published by the American Economic Review. The paper has three authors: in addition to Chetty, the other authors include John Friedman and Jonah Rockoff, also at Harvard. When the paper was first released, it was reported on the front page of the New York Times, one of the authors discussed it on the PBS Newshour, and President Obama referred to it in his 2012 State of the Union address.

The New York Times story appeared on January 6, 2012. it began thus:

“WASHINGTON — Elementary- and middle-school teachers who help raise their students’ standardized-test scores seem to have a wide-ranging, lasting positive effect on those students’ lives beyond academics, including lower teenage-pregnancy rates and greater college matriculation and adult earnings, according to a new study that tracked 2.5 million students over 20 years.”

The reporter noted that the effect of a single “high-value” teacher was actually quite modest: “The average effect of one teacher on a single student is modest. All else equal, a student with one excellent teacher for one year between fourth and eighth grade would gain $4,600 in lifetime income, compared to a student of similar demographics who has an average teacher. The student with the excellent teacher would also be 0.5 percent more likely to attend college.” But think of the aggregate effect on an entire classroom: “Replacing a poor teacher with an average one would raise a single classroom’s lifetime earnings by about $266,000, the economists estimate. Multiply that by a career’s worth of classrooms.” President Obama cited the aggregate income gain for a classroom in his State of the Union address 18 days later.

This was the takeaway from the authors, as reported in the New York Times:

“The authors argue that school districts should use value-added measures in evaluations, and to remove the lowest performers, despite the disruption and uncertainty involved.

“The message is to fire people sooner rather than later,” Professor Friedman said.

“Professor Chetty acknowledged, “Of course there are going to be mistakes — teachers who get fired who do not deserve to get fired.” But he said that using value-added scores would lead to fewer mistakes, not more.

“Still, translating value-added scores into policy is fraught with problems. Judging teachers by their students’ test scores might encourage cheating, teaching to the test or lobbying to have certain students in class, for instance.”

The Chetty, et al, study supported VAM, which was the central feature of Race to the Top. Fire teachers sooner rather than later. One great teacher can produce lifetime gains.

Over the past few years, as more districts have implemented VAM, it has turned out to be far more complicated than the economists predicted to determine which teachers would produce great scores year after year, and which would not. Teachers were rated effective one year, ineffective the next year. Those who taught English learners, the gifted, and students with disabilities were less likely to get big gains. It turned out that VAM is affected by the composition of the classroom, since students are not randomly assigned.

But their paper continues to be the lodestar of VAM research.

Whereas it had originally appeared as a single paper published by the National Bureau of Economic Research, the editors suggested the paper was so important that it should be split into two papers and published separately. The last time this had happened was in 1971, for papers on taxation that had won two Nobel Prizes.

Here are the papers.

Click to access w19423.pdf

Click to access w19424.pdf

Professor Chetty’s email was addressed to me and Audrey Amrein-Beardsley, who has written extensively and critically about value-added assessment. In addition to her recently published book on VAM—Rethinking Value-Added Models in Education: Critical Perspectives on Tests and Assessment-Based Accountability—she writes a blog called VAMboozled that I often cite.

For the record, I have never met Raj Chetty, and I have met Beardsley once, when she interviewed me for an oral history archive.

I asked Beardsley if she would be willing to review the latest iteration of this now famous study of VAM, and she did, here on her blog.

Beardsley notes that there is a divide between econometricians, like Chetty, Friedman, and Rockoff, and educational researchers, who often feel some obligation to visit classrooms and see the effects of policies, not just analyze data from a great distance, without reference to context or something like reality.

Professor Chetty and I exchanged several emails. I asked for his permission to post our exchange. He said that he preferred that I not post his comments, which were invariably polite, but of course I was free to post my comments to him.

So here goes. This was my first response:

Dear Professor Chetty,

I certainly agree that teachers are valuable. I had some wonderful teachers
as I was growing up, also some mediocre ones, and a few really bad ones. I
went to an ordinary public school system in Houston, not an elite private
school.

I wish that this sentiment about the value of teachers was all that came
from your vast publicity machine.

Instead, we get more high-stakes testing, more test prep, more phony claims
that the work of my fourth grade or fifth-grade teacher was responsible for
my not getting pregnant when I was 15. Maybe my lifetime income was
increased by my sixth-grade teacher, though I doubt it. Funny, I was one of
eight children. We all had the same teachers, and we all turned out
differently. Some of us did well in school, others nearly flunked out. Was
it the fault of our teachers?

I know you love your celebrity–and hobnobbing with Obama and Duncan and
supporting their emphasis on testing and firing teachers sooner rather than
later—but think of the harm that you do to millions of children and their
teachers by the way you publicize your work. Do you feel good every time you
read about a teacher who is graded based on the work of children she never
taught? Or the “highly effective” teacher who was rated ineffective the next
year based on test scores? Or the precipitous decline in the number of
people who want to be teachers because of the non-stop attacks on teachers?
I don’t think your positive message is getting through. All people hear is
that you want those lousy teachers whose kids get low scores to be fired.
Now.

Diane Ravitch

On May 5, I wrote to both Raj and Audrey (we had reached a first-name basis):

Raj and Audrey,

I don’t know whether my thoughts advance or retard this informed discussion.

I look at the Chetty, etc. study as comparable to a pilot in a bomber
dropping a bomb on a city 30,000 feet below. He didn’t construct the bomb,
he doesn’t know how it hurts the people below, he can’t be held responsible
if his good intentions went wrong.

I invite you to read this blog by a teacher in Oklahoma:
http://bluecerealeducation.blogspot.com/2014/05/ms-bullens-data-rich-year.html

The odds are that he never heard of Raj Chetty. But look what Raj Chetty has
done to the quality of education, the students, and the teachers in
Oklahoma. Is this something to be proud of?

Your work–not yours alone, of course–has encouraged a technocratic
approach to education that would never be tolerated in our nation’s elite
private schools.

The pursuit of higher test scores on stupid multiple-choice standardized
tests does not improve education: it corrupts it.

Those who care deeply about humanistic education, about the life of the
mind, about deep learning, find your work–no matter how technically
perfect–utterly appalling. It drains education of joy and discovery and
makes everyone a slave to Pearson.

I would love to discuss this further with you over a glass of wine. I can’t
believe you do not understand the pernicious effects of your famous study,
featured on the first page of the New York Times, on the PBS Newshour, and
in President Obama’s State of the Union Address.

It seems to be my life work to insist that education is far, far more than a
score on a standardized test. Somehow, I suspect you agree. You are far too
intelligent not to.

Diane

Later on the same day, May 5, Raj responded, and I wrote:

Thanks, Raj,

A question and a comment.

My question: Could I publish our exchange on my blog? I get about 25,000-40,000 readers daily. But I would publish nothing without your permission.

My comment: Race to the Top has incentivized the use of VAM in most states. Your study has been cited by Obama and Duncan as evidence that they are on the right track, that it is “bad teachers,” not poverty, that cause low test scores.

Based on the real-world effects of VAM on real children and real teachers, I conclude that VAM has limited use, perhaps informative in looking at the effects of policies and programs (faithfully enacted, which they seldom are) in a school or a district, but of zero value in assessing individual teacher quality. As you must know by now, the ratings for individual teachers are unstable, and may change if a different test is used or unstable for no apparent reason at all. Teachers intuitively know that their ratings reflect the composition of the class, not their “quality” or efficacy as teachers. Even if VAM did work–and it does not–it would keep every teacher singularly focused on standardized tests, which narrow the curriculum, encourage schools and teachers to avoid the neediest students, promote test prep and cheating, and have other perverse effects.

At the end of the day, I as a mother and grandmother would not want my offspring to be enrolled in a school where standardized tests dominate teaching and learning. And that is precisely what VAM is doing to our nation’s public schools.

My third grandson enters third grade in a New York City public school next September. I hope by then that the opt out movement has grown so strong that teachers cannot be subjected to unfair and inaccurate VAMs. I will do whatever I can to encourage parents in every school district in the U.S. to keep their children home on testing day. That seems to be the only way that the giant standardized testing machine can be stopped.

Your work has been crucial in promoting standardized testing as the measure of teacher quality, even though major scholarly organizations disagree (the American Educational Research Association, the National Academy of Education, the American Statistical Association).

If you have modified your views (message: “fire teachers sooner rather than later”); if you have learned anything new since you first introduced your findings, I would love to know about it.

I repeat that I do not have the technical ability to argue algorithms with you. Your study may be technically brilliant. But its consequences for the quality of education and the lives of children and teachers have been disastrous. In its current application, it is Junk Science. Since I feel certain you don’t want to be remembered in history as the economist who sponsored Junk Science and treated children as data points, I hope you will give me reason to believe that you have rethought the conclusions of your study and provided clear warnings about the limitations and misuses of VAM.

Diane

We ended with the understanding that I would not quote his words or paraphrase them. I think I was true to that understanding.

Now, as I told him, I am not an economist, and I lack the technical proficiency to critique his paper. Maybe it will win two or three Nobel prizes. If all it says is that teachers are valuable, I agree. If it says that teacher affect eternity, I agree.

But if he really expects me to believe that my fifth-grade teacher (or was it my fourth-grade teacher) caused me to get higher test scores, and that because of her and my higher test scores, I did not get pregnant when I was 15, I think this is just plain silly.

This strikes me as the kind of study that brings huzzahs from economists for its technical precision, but is unrelated to the messiness of real life. The numbers may all add up, but there are no living, breathing students or teachers here, just data.

It is so incredibly frustrating to me to see economists and policymakers playing with the lives of children and teachers as if they were ants seen from a far distance or merely data points. I recommend to my new friend Raj a book by Yale Professor James C. Scott titled “Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed.” It changed my life. Maybe it will change his too.