Search results for: "Raj Chetty's"

Jan Resseger writes here about Raj Chetty’s return to Harvard to start a new project, reviving the American dream. l

In the past, we have known Raj as the prime author of a widely doubted study that concluded that one effective teacher (who raised test scores) would have a significant impact on lifetime earnings, pregnancy rates, and other important life outcomes. In the years since that study was published, the efforts to apply its lessons have universally failed, because–as social scientists have long known–the influence of teachers on student test scores is small as compared to the influence of the family, its income, its level of education, its devotion to education.

Jan’s post begins:

Raj Chetty, the superstar, big-data economist, has returned to Harvard from Stanford to establish his own Opportunity Insights research and policy institute, a project seeded with funding from the Bill & Melinda Gates Foundation and the Chan-Zuckerberg Initiative. Chetty’s research has created an Opportunity Atlas, which maps neighborhoods of opportunity and other neighborhoods where children are unable to move up the economic ladder.

After reviewing some of his ideas, Jan asks why Gates and CZI don’t advocate for a higher minimum wage, which doesn’t need a lot of new research to establish the premise that the best way to reduce poverty is to increase family incomes.

This is a good opportunity to recommend two very valuable books about the paradox of super-rich people saving the world:

Anand Girihiradhas, Winners Take All

Michael Edwards: Small Change: Why Business Won’t Save the World

Edwards makes this salient point: Why ask capitalists (or philanthrocapitalists, as he calls them) to fix the problems created by capitalism?

John Thompson, historian and teacher in Oklahoma, has reviewed the work of economists Raj Chetty. You may recall that Chetty, a Harvard professor, was co-author of a study that purported to show that teachers could be evaluated by the test scores of their students. An effective teacher, one who raised test scores, would raise lifetime income, increase high school graduation rates, prevent teen pregnancies, and have lifelong effects on students. Raj and his colleagues John Friedman and Jonah Rockoff were cited on the first page of the New York Times (before the study was peer-reviewed), appeared on the PBS NewsHour, and were hailed by President Obama in his State of the Union speech in 2012. Their study became the #1 talking point for those who thought that using test scores–their rise and fall–would be the best way to identify effective and ineffective teachers. As Professor Friedman told the New York Times, “The message is to fire people sooner rather than later.”

 

Critics thought the findings were fairly modest. Even the Times said:

The average effect of one teacher on a single student is modest. All else equal, a student with one excellent teacher for one year between fourth and eighth grade would gain $4,600 in lifetime income, compared to a student of similar demographics who has an average teacher. The student with the excellent teacher would also be 0.5 percent more likely to attend college.

 

That works out to about $105 a year for a 40-year career, or $2 a week. But the Times then looked at the results in the aggregate and calculated that the aggregate of gains for an entire class would be $266,000 over the lifetimes of the entire class, or millions of dollars in added income when multiplied by millions of classrooms. Pretty great stuff, even though it means only $2 a week for one student.

 

The Obama administration bought into the Chetty-Friedman-Rockoff thesis whole-heartedly. Fire teachers sooner rather than later. Use test scores to find out who is a great teacher, who is a rotten teacher. It all made sense, except that it didn’t work anywhere. The scores bounced around. A teacher who was great one year was ineffective the next year; and vice versa. Teachers were rated based on the scores of students they never taught. Tests became the goal of education rather than the measure. It was a plague of madness that overcame public education across the land, embedded in Race to the Top (2009) and certified by Ivy League professors.

 

Thompson writes:

 

As it becomes more clear that value-added teacher evaluations are headed for the scrap heap of history, true believers in corporate reform continue to respond with the same old soundbites on the ways that their statistical models (VAMs) can be valid and reliable under research conditions. But, they continue to ignore the real issue and offer no evidence that VAMs can be made reliable and valid for evaluating real individuals in real schools.

 

Gates Foundation scholar Dan Goldhaber recently replied to the American Educational Research Association (AERA) statement which “cautions against VAM being used to have a high-stakes, dispositive weight in evaluations.” His protest recalls the special pleading of VAM advocates Raj Chetty, John Friedman, and Jonah Rockoff in reply to the American Statistical Association’s (ASA) 2014 statement warning about the problems with using VAMs for teacher evaluations.

 

Goldhaber criticizes the AERA by citing a couple of studies that use random samples to defend the claim that they can be causally linked to a teacher’s performance. Using random samples makes research easier but it also makes those studies irrelevant to real world policy questions. Goldhaber then cites Chetty et. al and their claim that low-stakes 1990s test scores resulted in the increased income of individuals during the subsequent economic boom in New York City during the 2000s.

 

Interestingly, Chetty’s rebuttal of the ASA cited the same two random sample studies, as well as his own research that was cited by Goldhaber. Like Goldhaber and other value-added proponents, he acknowledged the myriad of problems with value-added evaluations, but added, “School administrators, teachers, and other relevant parties can be trained to understand how to interpret a VAM estimate properly, including measures of precision as well as the assumptions and limitations of VAM.”

 

That raises two other concerns. First, if educators should be trained in the arcane methodologies, assumptions, and limitations of regression studies in order to use VAMs, should economists not be trained in the logistics of schools so they can conduct research that is relevant to education policy? Secondly, even if they ignore the nuts and bolts of schools, isn’t it strange that Chetty and his colleagues ignore economic factors when explaining economic effects? Why are they so sure that education – not economic forces – explains economic outcomes?

 

These questions become particularly interesting when reading Chetty’s web site. If he was really committed to the use of his Big Data methodology to help improve schools and students’ subsequent economic outcomes, would he not engage in a conversation with practitioners, and ground his methods in reality, so information from his models could be used to improve schools? After all, architects run plenty of quantitative structural analyses of their construction projects but they also interview their clients and listen to how they will use their buildings.

 

Chetty could have gone back and learned what he didn’t know about schools before he joined in the social engineering experiment known as school reform. Instead, he is rushing off to promote policies for problems which seem to be equally beyond his realm of knowledge. And, he seems equally uncurious about the new people he wants to “nudge” into better behavior. His method for studying anti-poverty policy is to ignore what actually happens in schools and communities and to “treat behavioral factors like any other modeling decision, such as assuming time-separable or quasi-linear utility.” The goal of his new project is to create incentives so that policy-makers can rid poor people, especially, of their “loss aversion, present bias, mental accounting, [and] inattention” so they will move to better places.

 

I’m not an expert on Chetty’s new The Equality of Economic Opportunity Project but my reading of the evidence is that Robert Putnam, who combines qualitative and quantitative research to document the decline of social mobility, makes a much stronger case than Chetty, who believes social and economic mobility hasn’t declined. It seems to me that Putnam is right and that we must take a generational view in order to show that economic opportunity for the poor has been reduced. I also believe that Derek Thompson nails the case that each generation since the first half of the Baby Boom is seeing an economic deterioration.

 

I can’t help but wondering why Chetty doesn’t stop scurrying around complex social issues, pontificating on simplistic quick fixes, and study issues in depth. He seems more intent on promoting his Big Data methods, and defeating traditional social science, than actually solving real-world problems. Chetty (and other VAM true believers?) appear preoccupied with academic combat against traditional social scientists who still respect falsifiable hypotheses and peer review. Education and child poverty appear to be just the battlegrounds for academic combat with researchers.

 

Traditional school improvement was based on the imperfect process of drawing upon the scientific method to diagnose problems, policy debates, and the imperfect democratic process known as compromise. To do that, educators and researchers studied the history and the nature of the causes and effects of underperformance. Corporate reform sought the opposite. Rather than study and debate the nature of our schools’ shortcomings, problems and solutions, the contemporary reform movement attempted a series of bank shots. Ignoring their actual targets, they sought incentives and disincentives that would prompt others to devise solutions. The job of economists’ regression studies was to suggest rewards and punishments that would make educators improve.

 

An illustration of Chetty’s disdain for evidence-based, collaborative conversations about school improvement is the first graph on his web site. It shows the surge in student test score growth which occurs when a “High VA Teacher Enters,” and replaces a low performer. If Chetty sought to articulate a hypothesis or discuss how his hypothesis, if proven, could improve teacher quality, he would have addressed some issues. But, the graphic resembles a political attack ad more than a presentation of evidence for school improvement.

 

Chetty’s graphic is strangely opaque about what he means by “high VA” teachers or how many of them there are. In fact, those gains he showcased are the educational equivalent of a White Rhinoceros.
Chetty emphasizes the incredible size of his database.  His data spans the school years 1988-1989 through 2008-2009 and covers roughly 2.5 million children in grades 3-8. Because there are 974,686 unique students in the dataset, his Power Points seem impressive. But, it is extremely difficult to find the key number which a traditional social scientist would have volunteered at the beginning of a study. Chetty’s graphs that illustrate such dramatic gains are based on samples as small as 1135. In other words, about 12 to 17 of these top-performing New York City teachers transferred, per year, into low value-added classrooms.
Chetty doesn’t ask why such transfers are so rare. Moreover, he makes it extremely difficult for a reader to learn the most important facts that would prompt that essential question and a constructive discussion of solution. Instead, he indicates that the answer is using VAMs to fire low-performing teachers and, without evidence, he implies that there are enough top 5% teachers who would respond to modest incentives and transfer to those low value-added classrooms. Otherwise, Chetty’s work on transfers might earn him academic awards but it is just theory, irrelevant for real world policy.
Sadly, it looks like Chetty’s new studies are equally simplistic. The problem, he implies, is not that the economic ladder out of poverty is broken. The problem is getting poor families to move from places without opportunity to places where there is opportunity. So, we in Oklahoma City should forget that Supply Side economics incentivized the mass transfer of good-paying jobs to the exurbs. In Oklahoma County, where poor children’s economic opportunity is in the bottom 17% of the nation, we should incentivize the movement of poor families to Cleveland County where social mobility hasn’t declined.
Presumably, the additional good-paying jobs for the influx of poor families would magically appear. In other words, Chetty’s logic on moving to opportunity is the first cousin of his faith that top teachers will flock to the inner city because they want to be evaluated with an algorithm which is biased against inner city teachers.
I wish I didn’t feel compelled to sound so sarcastic. I really do. But, for every complicated question, there is an answer that is quick, simple, and wrong. Why are Chetty et. al so quick to conclude that it is schools – not the totality of market and historical forces – that drive economic outcomes? Even though the market has undermined the futures of poor families, why does he remain convinced that it can fix schools?
And, the inconsistencies of Chetty and other corporate reformers drive me up the wall. He now proclaims, “We find that every year of exposure to a better environment improves a child’s chances of success.” Were he consistent, Chetty might understand that exposure to education environments might improve his chance of studying education in a way that improves his chances of successfully helping students.
Why does Chetty not take the time to understand the environments of poor children, and build better school environments? Why not help create learning environments that would attract high value-added teachers, not drive them out of the profession? Rather than demand that teachers and poor families learn to look at their worlds the way Chetty does, why not listen to the people who he says he wants to help?

On May 3, I received an email from Professor Raj Chetty of Harvard University, informing me that his famous paper on value-added assessment of teachers was being published by the American Economic Review. The paper has three authors: in addition to Chetty, the other authors include John Friedman and Jonah Rockoff, also at Harvard. When the paper was first released, it was reported on the front page of the New York Times, one of the authors discussed it on the PBS Newshour, and President Obama referred to it in his 2012 State of the Union address.

The New York Times story appeared on January 6, 2012. it began thus:

“WASHINGTON — Elementary- and middle-school teachers who help raise their students’ standardized-test scores seem to have a wide-ranging, lasting positive effect on those students’ lives beyond academics, including lower teenage-pregnancy rates and greater college matriculation and adult earnings, according to a new study that tracked 2.5 million students over 20 years.”

The reporter noted that the effect of a single “high-value” teacher was actually quite modest: “The average effect of one teacher on a single student is modest. All else equal, a student with one excellent teacher for one year between fourth and eighth grade would gain $4,600 in lifetime income, compared to a student of similar demographics who has an average teacher. The student with the excellent teacher would also be 0.5 percent more likely to attend college.” But think of the aggregate effect on an entire classroom: “Replacing a poor teacher with an average one would raise a single classroom’s lifetime earnings by about $266,000, the economists estimate. Multiply that by a career’s worth of classrooms.” President Obama cited the aggregate income gain for a classroom in his State of the Union address 18 days later.

This was the takeaway from the authors, as reported in the New York Times:

“The authors argue that school districts should use value-added measures in evaluations, and to remove the lowest performers, despite the disruption and uncertainty involved.

“The message is to fire people sooner rather than later,” Professor Friedman said.

“Professor Chetty acknowledged, “Of course there are going to be mistakes — teachers who get fired who do not deserve to get fired.” But he said that using value-added scores would lead to fewer mistakes, not more.

“Still, translating value-added scores into policy is fraught with problems. Judging teachers by their students’ test scores might encourage cheating, teaching to the test or lobbying to have certain students in class, for instance.”

The Chetty, et al, study supported VAM, which was the central feature of Race to the Top. Fire teachers sooner rather than later. One great teacher can produce lifetime gains.

Over the past few years, as more districts have implemented VAM, it has turned out to be far more complicated than the economists predicted to determine which teachers would produce great scores year after year, and which would not. Teachers were rated effective one year, ineffective the next year. Those who taught English learners, the gifted, and students with disabilities were less likely to get big gains. It turned out that VAM is affected by the composition of the classroom, since students are not randomly assigned.

But their paper continues to be the lodestar of VAM research.

Whereas it had originally appeared as a single paper published by the National Bureau of Economic Research, the editors suggested the paper was so important that it should be split into two papers and published separately. The last time this had happened was in 1971, for papers on taxation that had won two Nobel Prizes.

Here are the papers.

Click to access w19423.pdf


Click to access w19424.pdf

Professor Chetty’s email was addressed to me and Audrey Amrein-Beardsley, who has written extensively and critically about value-added assessment. In addition to her recently published book on VAM—Rethinking Value-Added Models in Education: Critical Perspectives on Tests and Assessment-Based Accountability—she writes a blog called VAMboozled that I often cite.

For the record, I have never met Raj Chetty, and I have met Beardsley once, when she interviewed me for an oral history archive.

I asked Beardsley if she would be willing to review the latest iteration of this now famous study of VAM, and she did, here on her blog.

Beardsley notes that there is a divide between econometricians, like Chetty, Friedman, and Rockoff, and educational researchers, who often feel some obligation to visit classrooms and see the effects of policies, not just analyze data from a great distance, without reference to context or something like reality.

Professor Chetty and I exchanged several emails. I asked for his permission to post our exchange. He said that he preferred that I not post his comments, which were invariably polite, but of course I was free to post my comments to him.

So here goes. This was my first response:

Dear Professor Chetty,

I certainly agree that teachers are valuable. I had some wonderful teachers
as I was growing up, also some mediocre ones, and a few really bad ones. I
went to an ordinary public school system in Houston, not an elite private
school.

I wish that this sentiment about the value of teachers was all that came
from your vast publicity machine.

Instead, we get more high-stakes testing, more test prep, more phony claims
that the work of my fourth grade or fifth-grade teacher was responsible for
my not getting pregnant when I was 15. Maybe my lifetime income was
increased by my sixth-grade teacher, though I doubt it. Funny, I was one of
eight children. We all had the same teachers, and we all turned out
differently. Some of us did well in school, others nearly flunked out. Was
it the fault of our teachers?

I know you love your celebrity–and hobnobbing with Obama and Duncan and
supporting their emphasis on testing and firing teachers sooner rather than
later—but think of the harm that you do to millions of children and their
teachers by the way you publicize your work. Do you feel good every time you
read about a teacher who is graded based on the work of children she never
taught? Or the “highly effective” teacher who was rated ineffective the next
year based on test scores? Or the precipitous decline in the number of
people who want to be teachers because of the non-stop attacks on teachers?
I don’t think your positive message is getting through. All people hear is
that you want those lousy teachers whose kids get low scores to be fired.
Now.

Diane Ravitch

On May 5, I wrote to both Raj and Audrey (we had reached a first-name basis):

Raj and Audrey,

I don’t know whether my thoughts advance or retard this informed discussion.

I look at the Chetty, etc. study as comparable to a pilot in a bomber
dropping a bomb on a city 30,000 feet below. He didn’t construct the bomb,
he doesn’t know how it hurts the people below, he can’t be held responsible
if his good intentions went wrong.

I invite you to read this blog by a teacher in Oklahoma:
http://bluecerealeducation.blogspot.com/2014/05/ms-bullens-data-rich-year.html

The odds are that he never heard of Raj Chetty. But look what Raj Chetty has
done to the quality of education, the students, and the teachers in
Oklahoma. Is this something to be proud of?

Your work–not yours alone, of course–has encouraged a technocratic
approach to education that would never be tolerated in our nation’s elite
private schools.

The pursuit of higher test scores on stupid multiple-choice standardized
tests does not improve education: it corrupts it.

Those who care deeply about humanistic education, about the life of the
mind, about deep learning, find your work–no matter how technically
perfect–utterly appalling. It drains education of joy and discovery and
makes everyone a slave to Pearson.

I would love to discuss this further with you over a glass of wine. I can’t
believe you do not understand the pernicious effects of your famous study,
featured on the first page of the New York Times, on the PBS Newshour, and
in President Obama’s State of the Union Address.

It seems to be my life work to insist that education is far, far more than a
score on a standardized test. Somehow, I suspect you agree. You are far too
intelligent not to.

Diane

Later on the same day, May 5, Raj responded, and I wrote:

Thanks, Raj,

A question and a comment.

My question: Could I publish our exchange on my blog? I get about 25,000-40,000 readers daily. But I would publish nothing without your permission.

My comment: Race to the Top has incentivized the use of VAM in most states. Your study has been cited by Obama and Duncan as evidence that they are on the right track, that it is “bad teachers,” not poverty, that cause low test scores.

Based on the real-world effects of VAM on real children and real teachers, I conclude that VAM has limited use, perhaps informative in looking at the effects of policies and programs (faithfully enacted, which they seldom are) in a school or a district, but of zero value in assessing individual teacher quality. As you must know by now, the ratings for individual teachers are unstable, and may change if a different test is used or unstable for no apparent reason at all. Teachers intuitively know that their ratings reflect the composition of the class, not their “quality” or efficacy as teachers. Even if VAM did work–and it does not–it would keep every teacher singularly focused on standardized tests, which narrow the curriculum, encourage schools and teachers to avoid the neediest students, promote test prep and cheating, and have other perverse effects.

At the end of the day, I as a mother and grandmother would not want my offspring to be enrolled in a school where standardized tests dominate teaching and learning. And that is precisely what VAM is doing to our nation’s public schools.

My third grandson enters third grade in a New York City public school next September. I hope by then that the opt out movement has grown so strong that teachers cannot be subjected to unfair and inaccurate VAMs. I will do whatever I can to encourage parents in every school district in the U.S. to keep their children home on testing day. That seems to be the only way that the giant standardized testing machine can be stopped.

Your work has been crucial in promoting standardized testing as the measure of teacher quality, even though major scholarly organizations disagree (the American Educational Research Association, the National Academy of Education, the American Statistical Association).

If you have modified your views (message: “fire teachers sooner rather than later”); if you have learned anything new since you first introduced your findings, I would love to know about it.

I repeat that I do not have the technical ability to argue algorithms with you. Your study may be technically brilliant. But its consequences for the quality of education and the lives of children and teachers have been disastrous. In its current application, it is Junk Science. Since I feel certain you don’t want to be remembered in history as the economist who sponsored Junk Science and treated children as data points, I hope you will give me reason to believe that you have rethought the conclusions of your study and provided clear warnings about the limitations and misuses of VAM.

Diane

We ended with the understanding that I would not quote his words or paraphrase them. I think I was true to that understanding.

Now, as I told him, I am not an economist, and I lack the technical proficiency to critique his paper. Maybe it will win two or three Nobel prizes. If all it says is that teachers are valuable, I agree. If it says that teacher affect eternity, I agree.

But if he really expects me to believe that my fifth-grade teacher (or was it my fourth-grade teacher) caused me to get higher test scores, and that because of her and my higher test scores, I did not get pregnant when I was 15, I think this is just plain silly.

This strikes me as the kind of study that brings huzzahs from economists for its technical precision, but is unrelated to the messiness of real life. The numbers may all add up, but there are no living, breathing students or teachers here, just data.

It is so incredibly frustrating to me to see economists and policymakers playing with the lives of children and teachers as if they were ants seen from a far distance or merely data points. I recommend to my new friend Raj a book by Yale Professor James C. Scott titled “Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed.” It changed my life. Maybe it will change his too.