Remember the study that claimed that teachers who produce higher test scores in fourth or fifth grade have miraculous lifetime effects on students? Among other things, the students are less likely to get pregnant and an entire class will increase its lifetime earnings by $250,000 a year!
The study was reported on the first page of the Néw York Times, where one author said the lesson was to fire teachers sooner, based on value-added test scores.
President Obama was so impressed–since the study echoed Race to the Top’s focus on using VAM for firing teachers and giving bonuses –that he mentioned the study in his State of the Union Address. Of course, Sidwell Friends, where his daughters are enrolled, wouldn’t do any such thing as VAM.
Critics quickly pointed out that the Chetty study, as it was known for its lead author Raj Chetty, had not been peer reviewed and its findings were overstated.
Bruce Baker explained that an increase of $250,000 for a class works out to about $5 a week, maybe a cup of coffee at Starbucks or a couple more plus a donut at Dunkin Donuts.
Now, Audrey Amrein-Beardsley provides a valuable service by pointing out that the study has been released a THIRD time, still not peer reviewed.
Diane Ravitch's blog 
Wonder if Chetty has submitted it for review, and it was rejected.
Hey, if the study says what you want it to say, why bother fact checking it? Ed reformers do not dwell in a data based reality. Investors only look at the monetary impact.
I am curious if you found flaws in the design or execution of the research reported in the papers. That would be fact checking.
If you go to the link provided near the end of the above article, you will find links to peer reviews that do get into design and execution questions.
Thanks for pointing that out. I have just a few minutes to look through the University of Colorado comments, but the conclusion is fairly straightforwardly stated in the last section of the paper:
“The report (Chetty et al) offers an impressive set of data and analyses that add substantially to our knowledge base. The report includes several persuasive tests that substantiate the claim that teachers with high measured value-added are in fact raising students’ test scores: it is not merely an observed association, but a causal impact. However, the report’s findings with regard to students’ long-term success do not include adequate tests of this sort. Thus, the report’s key findings linking teacher value-added scores to outcomes such as later earnings are not sufficiently validated; more evidence on that point needs to be presented before policy and practice are shaped in response to this report.
The report raises a question that is both important and timely. If the report’s conclusions regarding long-term effects can be substantiated, it would strongly suggest that high value-added teachers do more than simply raise test scores. To at least a modest extent, these teachers would be shown to be transformative, changing students for the better in ways that do not show up for years to come. Given the number of students with whom a teacher comes into contact over the course of a career, these modest impacts could have a large cumulative effect.”
I would characterize the comments from the center at UC Boulder as appropriately cautionary rather than critical. They suggest that it might be that another factor, like good parenting, leads to both students being assigned to high value added teachers and lower teenage pregnancy rates and the other markers of better long term outcomes. This is always a possibility with empirical work.
Ill take a look at the other comments on the paper as time allows.
TE,
Suppose you found that the design and method only were absolutely first-rate but the results were sheer nonsense?
Suppose I design a risk aversion study in which I threaten to kill the children of teachers if they can’t produce higher test scores? Then they produce higher test scores? Would that be a valuable study?
Rather than contemplate all the possible ways an empirical analysis can be done incorrectly, I think it would be a more effective to see if the work in question was done correctly.
And they want to make high stakes decisions on their understanding of statistics?!
The damage is done. The whole country heard the President. They say if you tell a lie long enough it becomes the truth.
My, my, how clumsy of Harvard. Of course, the school stands to benefit financially if more folks attend their graduate school for education—-a mixture of some ed administration and business courses. Rotten leaves among the Ivy League??
Amen!
How low can the Harvard name go?
Two important people are refuting Chetty, casting doubt on fourth grade scores predicting earning power.
Deasy cautioned, “I believe the statistics correlate, but I don’t believe in causality” And Obama confidently proclaimed, “grade school assessments should not determine destiny”.
Clearly fourth grade determines future earnings, fifth grade isn’t necessary.
This information should be made more widely available.
Reformers have a sophisticated set of resources and strategies for churning out propaganda, especially studies that depend on the aura of authority more than expertise in education. Statisticians and economists at Harvard, Dartmouth, Stanford and other high-profile universities get invited to testify about their grand inferential leaps from scores on standardized tests to the fate of the economy, nation and so on. Informed critics of these non-sense studies do not often get a hearing. if you want informed reviews of high-profile reports from think tanks that are not peer reviewed get your alerts from this fabulous website and also from http://nepc.colorado.edu/thinktank/review
VAM is a sham!
That’s being nice!
Personally I find it to be 100% USDA Grade AA Bovine Excrement.
Yuck.. he’s coming to our campus next week.. I can’t tell if he intends to talk about education, the topic of his talk is upward mobility.
Which would mean that the reformers’ reason the US is so low in upward mobility is the teachers. Not the sucking of the wealth from the middle class.
Reformers have a sophisticated set of resources and strategies for churning out propaganda, especially studies that depend on the aura of authority more than expertise in education. Statisticians and economists at Harvard, Dartmouth, Stanford and other high-profile universities get invited to testify about their grand inferential leaps from scores on standardized tests to the fate of the economy, nation and so on. Informed critics of these non-sense studies do not often get a hearing. if you want informed reviews of high-profile reports from think tanks that are not peer reviewed get your alerts from this fabulous website and also from http://nepc.colorado.edu/thinktank/review
My comment above (@ 1:14 p.m.) quotes extensively from the NEPC comments on Chetty et al. Those comments seem to me to be more cautionary than critical
That’s because you don’t understand academic papers.
Possibly. Could you point out the harsh criticism of Chetty?
What I could find in the NEPC review was a concern that “plausible alternative explanations—for example, that parents who did the most to promote their offspring’s long-term success also endeavored to secure high value-added teachers for their children” had not been ruled out as an alternative explanation for the observed good outcomes.
With VAM having a margin of error up to 50%, the probability of a student having three high teachers in a row is akin to three flips of a coin. Even with lower margin of errors, you’re making predictions with a pretty random error. Which means statistical significance for the study would be nebulous.
But what does matter to a Harvard researcher?
Have you read the paper? The data set covers about 2.5 million students. How many runs of three heads do you think you might get in 2.5 million flips?
TE, thanks to my fourth-grade teacher in the Houston public schools, I did not get pregnant before marriage, and I had an additional boost of $5 a week in my lifetime earnings. Or was it my fifth-grade teacher? Or sixth-grade?
Sigh.
I would have thought that someone who (correctly I think) argues that poverty causes poor school performance for poor children but that does not mean every poor child performs poorly in school would understand the difference between population moments and individual observations. It seems I was mistaken.
Lets try a simple example. Suppose I have an unfair coin that is more likely to land on heads than tails. Does it mean that every flip will be heads? No. Does it mean that I might not observe a long string of tails? No. Does it mean that after sufficiently many observations I can detect with an arbitrary level of certainty that the coin is not fair and is more likely to land heads than tails? Yes.
If it’s all down to the fourth grade teacher, then we don’t need no fifth grade 🙂
If the problem is fourth grade, skip it. But if it’s a great fourth grade teacher, increase the class size and make kids repeat the grade. Twice. You listening, Arne?
TE, for critiques of Chetty, google Bruce Baker’s wonderful analyses of the flaws of his study. Or thank your fifth-grade public school teacher for making you an economist.
If you are referring to this post http://schoolfinance101.wordpress.com/2012/01/07/fire-first-ask-questions-later-comments-on-recent-teacher-effectiveness-studies/), the concern seems to be with the headlines written in newspapers, not the validity of the work done in the paper.
If I can quote the most critical section of that post:
“These are interesting findings. It’s a really cool academic study. It’s a freakin’ amazing data set! But these findings cannot be immediately translated into what the headlines have suggested – that immediate use of value-added metrics to reshape the teacher workforce can lift the economy, and increase wages across the board! The headlines and media spin have been dreadfully overstated and deceptive.”
With 2.5 million students time the number of teachers they had, that’s a lot of flips and a lot of runs. Good question for my stats students.
But which run really represent three truly good teachers in a row, (a success) and which just appear to be a success.. And which non-runs were really successes.
This is why paper need peer review.
Diane, it was probably none of those teachers. Or a flip of the coin.
What is most scary about this is that the president of our country is willing to “believe” in this unsupported information because it supports some very shaky ground!!! Scary on so many levels.
It is very troubling that these “researchers” are unwilling to subject their paper to peer review or/and respond to the issues raised by other scholars. Their resounding silence and unwillingness/refusal to follow accepted scholarly practice suggests that they have no response to give, and are motivated solely by a desire to arrive at the results their funders wanted to hear and not by the spirit of honest inquiry.
Harold,
Publication lags in economics tend to be a couple of years. If you look at Chetty’s CV (http://obs.rc.fas.harvard.edu/chetty/cv.pdf) both part I and part II are in the editorial review process at the American Economic Review, arguably the most important and influential economics journal in the world. If they are accepted for publication, it will likely be another year before you see them in print.
I have read Chetty’s CV. He is a very young man. Some would say it was unethical to release a study three times (since 2011) that had not been found acceptable by a peer-review publication — though you seem to say that it just now it has been.
Harold, I don’t care if Chetty’s study was or was not peer reviewed. It doesn’t make sense. I really liked my teachers, but 1) I don’t see the connection between what my 4th grade teacher did and my pregnancies; and that big boost in income was $5 a week.
And God does not play dice with the universe, so quantum mechanics must be wrong.
I don’t believe it is about ethics, rather it is the requirements of the referees at AER. From the looks of it, the referees suggested an additional econometrics technique be used to analyze the data, and perhaps they (or the editor) suggested the original article be broken into two parts. Journal pages are scarce, and shorter articles have a better chance of publication than longer ones.
Perhaps you are not familier with academic publication. An author submits an article for potential publication to a journal (only one at a time, as simultaneous submition are considered unethical), the journal farms this out to three unpaid anonymous referees thought to be qualified to review the work by an editor of the journal. The referees eventually write a report about the quality of the research, submit it to the editor who communicates this back to the original author (the longest time period for this stage my spouse experienced was 18 months). The reports will typically make suggestions that might improve the work, and suggest the author revise and resubmit the paper. This seems to have been the case for the paper in question. The author(s) will revise the paper in accordance with the referees criticism, send a letter to the editor explaining the changes had been made, and the revised work and letter is sent back to the same set of referees to see if they are satisfied with the changes. If they are satisfied and the editor concures, the article is accepted for publication, but given the backlog of accepted articles, it might be a year or more before it comes out in print.
Of course, if the first journal turns it down, the process starts again at the beginning with the second journal.
“This is not about ethics” — truer words were never spoken. I know enough about academic papers to know that you don’t repeatedly “pre-publish” one. You may read it at a conference. But then you take it home and answer the objections that have been made to it (at the conference) before letting it out again into the light of day. Stonewalling is not academic procedure.
I don’t know what you mean by “stonewalling”. The authors will provide you with their data, their SAS program, the original working paper, the latest version of if their work which splits the original paper into two parts and changes the econometric model a bit, all at no cost to you. How could they be more transparent?
You know very well what I am talking about when I say stonewalling. Either that or you can’t or won’t read what I have written.
I think you may be using the word in a way I don’t understand. When I think of “stonewalling” I think that a person is not being forthcoming in how they reached a conclusion. The authors of this study seem to be doing the exact opposite of that by making all the data, all the drafts, and all the econometric programing freely available.
What do you mean by “stonewalling?
Teaching economist’s definition of stonewall: ” When I think of stonewalling, I think that a person is not being forthcoming in how they reached a conclusion.”
===
Dictionary definition: “Stonewall” = to delay or block (a request, process, or person) by refusing to answer questions or by giving evasive replies, esp. in politics.
“the highest level of bureaucracy stonewalled us”
==
I think of it as ignoring and refusing to answer questions about the content of one’s work.
Harold,
I don’t know what you want the authors of this study to do. They made the initial working paper available for free to anyone that wants to read it. They made the new versions of the paper for free to anyone who wants to read it. They made the data they used available for free to anyone who wants it. They made the SAS program they used to extract the data and estimate the relationships available for free to anyone that wants to use it.
You are free to replicate their study down to the last decimal point if you wish. You can examine every line computer code, every word they wrote. There are no secrets, you can see absolutely everything that they did.
What else would you have them do?
It is not simply a question of “Here is my work. Good bye!”
As you should be aware, scholars are obligated to justify their methods and respond to questions and criticisms.
In any case, the study was an analysis done on data from 1992 and, since time travel is impossible, conditions cannot be replicated. It is not like a chemistry lab.
You are also stonewalling when you pointedly ignore the dictionary definition of stonewall (which means “refuse to answer questions”) and simply repeat your erroneous and misleading words over and over. This is a good tactic for a trial lawyer or someone perhaps engaging in coercive propaganda. It is not honest argument (on your part) nor honest scholarship on Chetty’s part. I imagine some of the delay in publication acceptance, by the way, is due to Chetty et al.’s problems with the disclosure agreements.
Which questions have they refused to answer?
They told you exactly what they did, give you the data that they used, gave you the exact recipe they used for the analysis of data. what else would you have them do?
You are, of course, free to imagine anything that you wish.
This study will be used as part of the evidence for removing teacher rights in the California Lawsuit in LA. No Peer review needed for lawyers to declare it is true. Just throw enough darts at the board and eventually some will stick.