Judge Rolf M. Treu, who decided the Vergara case , declared that he was shocked, shocked to learn from Professor Raj Chetty and Professor Thomas Kane of Harvard about the enormous harm that one “grossly ineffective” teacher can do to a child’s lifetime earnings or to their academic gains.
How did he define “grossly ineffective” teacher? He didn’t. How did these dreadful teachers get tenure? Clearly, some grossly incompetent principal must have granted it to them. What was the basis–factual or theoretical–that the students would have had high scores if their teachers did not have the right to due process? He didn’t say.
The theory behind the case–as I see it–is that low test scores are caused by bad teachers. Get rid of the bad teachers, replace them with average teachers, and all students will get high test scores. You might call it the judicial version of No Child Left Behind–that is, pull the right policy levers–say, testing and accountability–and every single child in America will be proficient by 2014. Congress should hang its collective head in shame for having passed that ridiculous law, yet it still sits on the books as the scorned, ineffective, toxic law of the land.
You might also say that Judge Treu was regurgitating the unproven claims behind Race to the Top, specifically that using test scores to evaluate teachers will make it possible to weed out “bad teachers,” recruit and reward top teachers, and test scores will rise to the top. Given this theory, a concept like tenure (due process) slows down the effort to fire those “grossly ineffective” teachers and delays the day when every student is proficient.
Relying on Chetty and Kane, Judge Treu is quite certain that the theory of universal proficiency is correct. Thus, in his thinking, it becomes a matter of urgency to eliminate tenure, seniority, and any other legal protection for teachers, leaving principals free to fire them promptly, without delay or hindrance.
Set aside for the moment that this decision lacks any evidentiary basis. Another judge might have heard the same parade of witnesses and reached a different conclusion.
Bear in mind that the case will be appealed to a higher court, and will continue to be appealed until there is no higher court.
It is not unreasonable to believe that the California Teachers Association might negotiate a different tenure process with the Legislature, perhaps a requirement of three years probationary status instead of two.
The one thing that does seem certain is that, contrary to the victory claims of hedge fund managers and rightwing editorial writers, no student will gain anything as a result of this decision. Millions more dollars will be spent to litigate the issues in California and elsewhere, but what will students gain? Nothing. The poorest, neediest students will still be in schools that lack the resources to meet their needs. They will still be in schools where classes are too large. They will still be in buildings that need repairs. They will still be in schools where the arts program and nurses and counselors were eliminated by budget cuts.
If their principals fire all or most or some of their teachers, who will take their places? There is no long line of superb teachers waiting for a chance to teach in inner-city schools. Chetty and Kane blithely assume that those who are fired will be replaced by better teachers. How do they know that?
Let’s be clear. No “grossly ineffective” teacher should ever get tenure. Only a “grossly ineffective” principal would give tenure to a “grossly ineffective” teacher. Teachers do not give tenure to themselves.
Unfortunately, the Vergara decision is the latest example of the blame-shifting strategy of the privatization movement. Instead of acknowledging that test scores are highly correlated with family income, they prefer to blame teachers and the very idea of public education. If they were truly interested in supporting the needs of the children, the backers of this case would be advocating for smaller classes, for arts programs, for well-equipped and up-to-date schools, for after-school programs, for health clinics, for librarians and counselors, and for inducements to attract and retain a stable corps of experienced teachers in the schools attended by Beatriz Vergara and her co-plaintiffs.
Let us hope that a wiser judicial panel speedily overturns this bad decision and seeks a path of school reform that actually helps the plaintiffs without inflicting harm on their teachers.

Cross posted at Oped-news.
http://www.opednews.com/Quicklink/What-Was-the-Evidence-in-t-in-Best_Web_OpEds-Education_Privatization_Teacher-140611-508.html#comment494195
I would like to see some of your comments there. No one follows education news there, and it considers itself a ‘progressive site.’
Wow. I follow LAUSD and have, over a decade. I met Lenny Isenberg online, when he put up perdaily.com and imagined that he would break the curtain of silence that allowed the corruption to flourish and the teachers to become fodder for the administration to burn at the stake. If you have never read his site, do go. This is to an essay he put up YEARS AGO, but you should read others. HE is brilliant and a tireless fighter.
http://www.perdaily.com/2014/06/lausds-treacherous-road-from-reed-to-vergara–its-never-been-about-students-just-money.html
Would teachers prefer the appeal be heard by a Democratic or Republican judge?
Are there Green Party judges, in California?
This needs to go to the SUPREME COURT!
SCOTUS has a Republican majority.
Ironically, it is poor kids who are most likely to lose.
The loss of tenure is a very high profile blow to an already publicly demonized job.
That public demonization has already led to far fewer students entering teacher prep programs and faster teacher burnout.
This decision will either accelerate or at best, not change that trend.
If there are fewer people coming into the job, are the “great” teachers likely to end up in the inner city or the wealthy suburbs?
That trend alone might kill the education reform movement: even if they buy all the politicians and get all their policies enacted, what will they do if no one wants to be a teacher?
http://news.nationalpost.com/2014/06/11/b-c-teachers-on-brink-of-all-out-strike-after-union-votes-in-favour-of-walk-out/
Meanwhile, in Canada. . . .
The war against the middle class has diminished the role of the unions, which were the working man’s only defense.
sigh!
The responses to the articles are almost uniformly anti-teacher. It really is scary how little people understand about what teachers do.
We are coming off more than a decade of anti-teacher rhetoric from Education Deformers. So, this is not surprising. Disgusting, horrifying, dispiriting, but not surprising.
2 Decades!
Well, we could probably go back three. Starting with A Nation at Risk, that earliest of the Rheeformish scriptures (1983).
Insidious.
But the anti-teacher rhetoric has been extreme in recent years.
Witness the crowning of the Michelle Rhee as queen of Rheeform, the Time magazine cover of her with the broom. Time to sweep out all that teacher trash!
When I was a boy, I was taught to honor and respect my teachers.
I still do.
By now, you know that I was caught unaware by the top-down revolution… where those I once trusted to support my practice , want dmd GONE! There is no describing the kind of teacher-trashing that went on in NYC. During my time in the rubber room, I met teachers of great quality, whose students and their parents were furious at their disappearance. I saw them sob and witnessed their trauma. What you read about int the press does not begin to tell the tale of the assault on America’s veteran teachers. You should read Lorna Stremcha’s book when it comes out. It’s title is not yet settled, but hers is a story you cannot miss. She testified before Congress, by the way.
The decision was solely political with no hard evidence connecting due process to ineffective teaching. I hope it moves up the ladder
Michelle Rhee: “I think this is an absolutely huge win for public school students across the country.”
She was being interviewed today on The Take Away: http://www.thetakeaway.org/story/california-judge-rules-against-teachers-unions-sparks-education-reform-debate/
I fought the urge to switch to a different channel on my NPR News App. The interviewer seemed to buy most of the tired rhetoric and shaky evidence she was spouting. One problem is that Michelle Rhee is a polished performer, able to deliver complete sentences that make sense to the uninformed. She and her fellow “reformers” really know how to get ink and air time (and funding, which helps pay for both).
I’m getting the impression that the defendants’ attorneys weren’t very effective. The judge seems to have bought the entire tissue-thin case presented by the plaintiff’s attorneys. It looks like we’re going to have to offer an authentic and truly gripping counter-narrative. In this venue at least, the “reform” story pummeled the true story. Why wasn’t the defense able to have the case thrown out before it even went to trial? I think they must have told a weak story, or no story at all.
Randal Hendee: Michelle Rhee fits into this perfectly.
Why? Because an important part of due process is about defining and acting, i.e., setting clear and reasonable limits and fairly enforcing them. If anything even resembling ethical due process had been in place and acted upon by competent and ethical administrators while she was doing her brief three-year teaching stint during the early 1990s—and made the mouths of dozens of her students bleed—she would have been questioned, judged and rightfully terminated. With prejudice. And with lots of documentation to back it up.
Google “Michelle Rhee,” “masking tape” and “students.”
Here is one of many many examples: http://www.huffingtonpost.com/peter-zucker/michelle-rhee-and-masking_b_801339.html
Then why is she tooting that “absolutely huge win” horn? The SOP of the leaders of the self-styled “education reform” movement is to say one thing and do another.
A very dead, very old, and extremely Greek guy knew the type in his day:
“Hateful to me as are the gates of hell, Is he who, hiding one thing in his heart, Utters another.” [Homer]
Thank you for your comments.
😎
P.S. Watch out for the masking tape!
😡
May their own actions curse them! Years from now, when there are an insignificant number of experienced teachers left will they sue because all they are getting are ineffective TFA recruits non stop? It would serve them right, the end result will be far worse than the current situation. The parents and students will find out they were used.
But they won’t. Didn’t you know? TFA will SMASH the achievement gap. And it won’t matter to the billionaires. Their kids don’t go to public schools. I loved Edushyster’s comment about the “billionaires who care more about poor children than the people who work with them every day.”
Of course we hope that only effective teachers get tenure. But one of the problems in California is that tenure must, by law, be granted after two years, when the teacher is still learning. (A recent proposal to extend that to three years in San Jose, agreed to by the local union, was blocked by the state union). But sometime principals make mistakes. Sometimes a teacher burns out and becomes ineffective. Sometimes a change in technology or school programs render them ineffective. Tenure serves to protect teachers, but it also serves to lock them in place, because no one wants to leave tenure behind, so fewer jobs are available for teachers who want a change.
I agree that teachers need to know that they can’t be fired at the whim of an outraged parent or trend-chasing administrator. But the idea that someone’s employment should be guaranteed for life and unrelated to their performance is no longer seen as reasonable. Lifetime tenure should be replaced with guaranteed multi-year contracts, which would give teachers stability and latitude without the downsides of lifetime tenure.
We teachers do not have “lifetime” tenure, we have due process rights. There is a difference. What change do these teachers that want a change you allude to want? Merit pay, the teachers at low scoring schools will continue to lose and be blamed for things they can not control. “Tenured” teachers are fired all the time, administration has to do their job and show they have a legitimate reason for firing them. Clearly you do not understand the contractual agreement in this matter. As a teacher I only have a job as long as I do my job.
For more discussion of the ruling, and the idea of “strict scrutiny” that put a heavy burden on the defense rather than the plaintiffs, I invite you to read:
http://accomplishedcaliforniateachers.wordpress.com/2014/06/11/vergara-ruling/
“The most important labor law case SCOTUS has considered in decades.” – Harris vs. Quinn, according to the Center for Media and Democracy. In the article, they link the case to Koch money and to ALEC.
I have no strong opinion on the merits of this case, I do not know too much about Michelle Rhee who seems to be routinely maligned on this blog, yet I have grown progressively sympathetic to her intentions. She may be going about it in a completely wrongheaded way — I do not know enough to judge, but unlike so many of you here (along with the Teacher Union you seem so staunchly defend) she actually identifies a *huge* problem and is trying to do something about it. And no, I am not a Fox News-viewing tea-partier and I am not paid by the Koch brothers. I am a science professor at the University of California, Riverside and I tend to consider myself a liberal. I am a scientist though and I am both trained and paid to deal with the objective reality and not substitute my wishes and biases for the hard data.
And here is what the objective reality looks like to me. A large number of students who come to my university are woefully unprepared or under-prepared. That is, a large number of students have been under-served by the present school system which so many of you are so eager to defend. I routinely spend a large portion of my introductory classes teaching them what I knew coming out of a high school a mere quarter a century ago. Translation: I have no choice but to do the job that *should have been* done by the high school teachers. And it’s not just me; I hear similar sentiments from my friends and colleagues in other departments ranging from Math to English.
So let me repeat: I frequently do the job a school teacher has failed to do. And the students that come our way were presumably near the top of their class. Can you imagine how badly these teachers fail those who don’t go to college?
So remind me please, what do you have against people trying to reform this system?
And before you accuse me in engaging in double standards while enjoying my own tenure, let me make it clear: I would not fight tooth and nail to preserve the tenure system in academia. It’s definitely a perk and an incentive of sorts. Were tenure to disappear though, I would expect our salaries to rise in order to compensate for that lost intensive. Believe it or not, we have other employment opportunities.
The same is true of school teachers, of course. Good teachers should be paid more. Ineffective teachers should go. The future of our children trumps seniority.
Your experiences are consistent with mine. I am surprised that you find it as much of a problem at a UC school.
I think you should be asking why NCLB was instituted and teaching to the test had to be done. When I first began teaching in AZ in 1998, NCLB had not been fully instituted. I was allowed to be creative, allowed independent thinking in my classroom, took my students on at least three field trips per year, held classroom meetings to teach my students communication skills, etc. This was a Title I school, and I enjoyed teaching my students. Then NCLB took over, and I could no longer teach creatively. I would be put on an improvement plan and fired if I didn’t do it “their” way. I don’t know of any teachers that thought that teaching to the test was okay for our students or taking away our creativity was okay. In the beginning, if I had students I suspected of learning disabilities, I could go directly to the school psychologist. She would schedule a time to come and observe, and then she would tell me if she thought the student needed additional evaluation, etc. A meeting was scheduled with the parents and the child was helped. Not anymore. If a teacher suspects a child has disabilities, you must schedule a child study. You meet with the principal, social worker, counselor, other teachers. You are usually given strategies to try. By the time you ask for another meeting, there is usually no funding left to evaluate the student, so you are told they will revisit the situation next year. Many of my students fell through the cracks this way. I don’t defend the current system. I think we should revamp the current system and offer more choices to our parents in all schools–not just wealthy schools. We should start putting more funding into our low income schools and stop putting so much money into prisons. Many of our low income students are going to these prisons. The current “reformation” is not doing the things needed to help our public schools or to help you, professor, when you receive these students. They are helping to create the problems, so there will be no more public schools.
Dottie – No question that NCLB should probably get some sort of prize for one of the stupidest and most quixotic pieces of legislation ever passed. It’s professed goal was to bring the entire US student population up to or above what was that time the 70th percentile.
The ability of our political system to produce policies of mind-boggling stupidity is awesome.
If one disaggregates PISA scores by demographics then the US educational system looks pretty good. US whites outscore all predominantly white countries other than Finland and New Zealand. Asian-Americans (who are not all East Asian) outscore Japan. US Hispanics outscore all Latin American countries. Even US blacks outscore most Middle Eastern countries.
True we probably over invest in education. The poor country of Slovakia doesn’t spend much more per capita than Mexico yet gets fairly decent results.
Jim, I was excessed to another school due to asking too many questions about how we were going to get all students including special ed to 100% by 2014. This principal stacked my class with every learning disability, emotional/behavior problem he could find the next year. Then he refused to let me get them help through child studies or even using the time out room. When I went to my Association for help, I received little. Many teachers were forced out of this school. Twelve complaints were filed–nothing happened to him–he went to another district. But, he hurt many teachers. Interesting side note–I did get one student the help she needed before he forbid it. She was full-blown autistic. Why no one had helped her before 4th grade, I don’t know. I also bonded with this very difficult class. But, one day at the end of the year after school, I cried my eyes out, because I felt like a failure, not being able to help these students and thinking what will become of them.
Thinkubus, I have absolutely NO problem with the idea and notion of reform. From your experience and my own ten years in a middle school classroom, we all know that something has to change if WE (and by this we I mean SOCIETY) truly want to see children succeed- I sometimes wonder if this a hope of everyone.
Let me start by giving you my own experience. I had 130 seventh graders this year. Of these 130 students, 24 could read on grade level or above. When I went to my supervisor with this information in October, I was told that I was not a reading teacher and I was to continue on with our boxed in curriculum. I was told, the kids would be successful because I could guide students through the readings that are in this curriculum in all different ways.
Now, with all of these mandates and these powerful people who have the answers about reform and getting our children to succeed, I have two questions: Why can’t my children read? What is the purpose of reform if results are not produced?
The idea of reform is amazing. What scares me is these powerful people, who just don’t seem to get it, are in charge of it all and have the most influence(are their intentions genuine?).
I don’t have the answers either, but I do know that children should be able to read, especially by the time they reach me (I know your frustration- I also served as an adjunct at my local community college for developmental writing because the population of students who need these services has skyrocketed). There is a problem, a solution is needed, but I don’t know if focusing on tenure and teacher pay is where the attention is needed.
Thinkubus, I sympathize with your willingness to deal with facts, whether pleasant or not. The host of this blog is also interested in dealing with facts, whether pleasant or not.
One unpleasant fact about social and psychological processes are that these are hard to measure. Also, the goals of successful education are very hard to define or understand well enough to be explained clearly. What we really want from schools is something like learning or ability to learn. But that’s hard to understand or define, let alone measure.
Another unpleasant fact is that measurements, such as standardized test scores, are only proxies for the real things we want to measure, namely learning and the ability or willingness to learn.
Yet another unpleasant fact is that statistical inferences from such measurements are muddied by the strong effects from confounders, especially parental income, which is the best predictor of achievement in every country. This is not a pleasant fact.
Yet another unpleasant fact is that even when research is done well, it will be reinterpreted by the political process for the advantage of political actors.
Yet another unpleasant fact is that social institutions, such as schools, are not easily engineered to produce results from humans, such as high test scores (let alone learning or ability to learn). So the test score cheating scandals in Atlanta and Houston should be no surprise.
Yet another unpleasant fact is that reforms of social institutions need not be carried out fairly in order to please their political sponsors.
One more unpleasantness.
You say, “Were tenure to disappear though, I would expect our salaries to rise in order to compensate for that lost intensive.”
Good luck with that.
There are new database products designed to allow university administrators to survey researcher productivity just like a stock portfolio.
Assessment is coming to higher education.
I hope you do well by your metrics. Your colleagues might not.
I am an academic librarian. Journals in some academic fields have consistently better metrics than journals in other fields. Mathematics, for example, always looks bad by comparison. Attempts to “normalize” the data are not really statistically sound.
Should I then cancel all mathematics journals, as one might fire the “bad” teachers, if it saves money? No.
Reading about the repute of Chetty and Kane within their field impresses me about as much as hearing that Ted Nugent got an Image Award from the NRA. It doesn’t improve my image of the reputees or their field one little bit. In both cases there is a radical disconnection from the realities of normal and optimal life.
But when it comes to expertise on what it takes to further the full development of growing human beings, I know I would trust the grossly average teacher and the field of education far and away above the theories of ivory-&-ivy econ artists.
If it is shocking what one bad teacher does, he would be electrified to know what administrative corruption, lawlessness and incompetence did to the LAUSD schools so that so many graduating seniors could barely read beyond second grade skills.
What was the evidence is the question. Essential questions lead the way to understanding and solutions… so here is one for you all, who are discussing this at great length while a teacher in LA who actually fought the battle is suing the union and the district. What happened before this judge re-wrote the law of due process for teachers?
It would help if you go to this link,
here is the link, but I am copying the essay below this commentary.
http://www.perdaily.com/2014/06/lausds-treacherous-road-from-reed-to-vergara–its-never-been-about-students-just-money.html
and read what that teacher, Lenny Isenberg said about the Reed case which PRECEEDED this fiasco. It would also help if you perused the per daily archive and pondered the real dilemma that Lenny put out there when he started this site. years ago… not long after he was led away in handcuffs, for blowing the whistle on the corrupt-to-the core-administration that had created a system where the ethnic population that was the majority in this system was being passed through the schools to graduate with 2nd grade literacy skills.
LAUSD’S TREACHEROUS ROAD FROM REED TO VERGARA- IT’S NEVER BEEN ABOUT STUDENTS, JUST MONEY by Lenny Isenberg
This is nothing new… it is just new here, because no one listened to this brilliant man, whose posts chronicle the entire corrupt process and call for TEACHERS WHO WILL STAND UP AND BE COUNTED.
The media knew the story, just like in NYC where media was in the pocket of the billionaire mayor.
http://www.opednews.com/Quicklink/HAVE-REPORTERS-BECOME-POLI-in-Best_Web_OpEds-Media_Media-Bias_Media-Blackout_Media-Corruption-140322-673.html and this one
http://www.opednews.com/Quicklink/HAVE-REPORTERS-BECOME-POLI-in-Best_Web_OpEds-Media_Media-Bias_Media-Blackout_Media-Corruption-140322-673.html
and while you are at it, do read this one, because Lenny could have taken the way out, too… but stayed the fight. Lenny is not perfect, and has his flaws like all determined brilliant people, but he knows that we all lose because we trust the unions and believe in justice… and that is what “THEY” count on, and why 100,000 veteran teachers have been hounded into obscurity and out of the PROFESSION they loved….it never was a job for the authentic teacher.
http://www.perdaily.com/2013/11/lausd-gives-me-a-chance-to-be-a-hero-for-student-teachers-and-families.html
Contrary to what was asserted in 2010 with the Reed vs. State of California case, and now with the Vergara vs. State of California case, cutting the budget by attacking tenured teachers at the top of the salary scale is the real motive of both the plaintiffs and the defendants in these put up cases that are anything but adversarial as actually required by law. The well-being of students and their constitutional right to an education has nothing to do with why teacher tenure and seniority are under attack. Corporate interests have gotten behind these two cases with collusive lawsuits against the State of California and the Los Angeles Unified School District as defendants, when these defendants actually share the plaintiffs desire to destroy a professional and fairly compensated teaching force solely for the motive of profit to more and more hedge fund run charters and the further dumbing down of our future electorate, so the average citizen will not have enough of an education to meaningfully comprehend just how badly they are being screwed.
If “teacher quality and effectiveness” as well as having the best education for students in “high-poverty and high-minority” areas that have not done well in the past was really the issue, insuring an environment of reasonable discipline, while finally eliminating destabilizing social promotion would have been implemented long ago. Most poor and minority students enter LAUSD behind and are never brought up to grade-level in a timely manner, which becomes more impossible as years go by. Inner-city predominantly poor and minority filled schools- LAUSD is 90% Latino and Black- are not just bad for the students, they are toxic for any serious teacher not willing to go along with the complete lack of rigor mindlessly enforced by entrenched LAUSD administration. No secondary single-subject credentialed teacher- whatever their level of seniority- can be expected to teach humiliated students that LAUSD administration continues to put in classes years beyond their objective ability. Clearly this is the recipe for the disaster that LAUSD continues to be, which has nothing to do with teacher seniority.
The reason that schools like Liechty, Gompers, and Markham Middle Schools, mentioned in the Reed case, were so adversely impacted when it came to loss of predominantly novice teaching staff, was because any teacher with enough seniority wouldn’t be caught dead in a school where there was no support for excellence in education that the plaintiffs in the aforementioned cases supposedly so desperately claim they want in their lawsuit. Any teacher who insists on excellence and has the teaching skill to do it has been systematically targeted over the last 5 years, brought up on fabricated charges, and removed or forced into early retirement.
Both Reed and Vergara purposefully ignore the context in which seniority-based reductions take place. No mention is made of excellent teachers being completely undermined in a system that values Average Daily Attendance (ADA) payments from the State more than whether the students are actually learning something of value in a timely manner. The fact that 55% of Roosevelt High School students have quit school before ever reaching the 12th grade and that only 30% of the graduating class has the A-G requirements necessary to get them into the University of California schools says it all, but is conveniently ignored in Vergara.
What is the financial and pedagogic cost of 50% of new teachers quitting LAUSD within their first 5 years of teaching? Why are they quitting? What’s the cost of having to constantly replace that level of attrition to a cash strapped public education system? If the students were really the concern in Reed and Vergara, wouldn’t some of those behind the financing of Reed and Vergara have found more fertile ground in trying to find out why so many initially dedicated teachers leave the profession like they are abandoning a sinking ship. But, alas, one would have to look at those running LAUSD into the ground, instead of scapegoating teachers, while their inept union United Teachers Los Angeles (UTLA) sits by and does nothing.
And finally, none of this scam would be possible without the complete complicity of the mainstream and public media, who are owned by some of the same players trying to privatize public education with charters. Supposed charitable trusts like Eli Broad, Bill Gates, and the Waltons control subsidized NPR to the point where NPR will report nothing to contradict what Professor Diane Ravitch calls “the dominant narrative of bad teachers and great charters. What is ultimately at stake is allowing 40% of the state supplied public school budget to go for “charter administration costs,” while allowing the further dumbing down of our future citizens. Check out the following very typical stenographic “news” report of Elex Michaelson of KABC that merely parrots Broad Academy party line:
Chetty’s evidence on which this extraordinary ruling is based hasn’t even appeared in any peer-reviewed journal – it was published as an NBER working paper. The charts he showed to the judge (see http://www.washingtonpost.com/blogs/wonkblog/wp/2014/06/10/a-california-judge-just-ruled-that-teacher-tenure-is-bad-for-students/) violate basic rules of scientific data presentation: the y-axes are all manipulated, the error bars and R-square values are missing. That a judge would accept such weak evidence to strike down statutes is disturbing and highlights the lack of scientific literacy of the judiciary. Maybe that too can be blamed on the teachers.
It is forthcoming in AER. See my other comment.
Is AER publishing the charts without error bars and r-square values and with manipulated y-axes? What is your opinion, teachingeconomist, of showing transparently manipulated statistical charts in a court room? Is that acceptable practice among economists?
You are free to download the data set and the SAS code if you want to look for signs of manipulation.
That is not what I asked. Are you one of the authors of the paper?
No, I am not.
I dug a bit deeper about the Chetty et al. study. Here’s what I found:
The study was first published as an NBER working paper at http://obs.rc.fas.harvard.edu/chetty/value_added.pdf in 2011. It was submitted to American Economic Review and accepted after being split in two papers. Preprints can be found at http://obs.rc.fas.harvard.edu/chetty/w19424.pdf and http://obs.rc.fas.harvard.edu/chetty/w19423.pdf.
All the charts in all versions of the papers violate scientific data presentation standards:
– all Y-axes are “super-extra-stretchy” (Bruce Baker), exaggerating the purported statistical effect by orders of magnitude;
– none of the charts have error bars or confidence intervals; if they were shown, the error bars would certainly exceed the purported statistical effect by far;
– no r-square values or correlation coefficients are given; the r-squares are likely to be very low;
– the sample sizes on which the regressions are based are not shown; although the study drew on millions of student records, most of the records were incomplete and the actual regressions are based on only a fraction of the records.
I dug a bit deeper about the Chetty et al. study. Here’s what I found:
The study was first published as an NBER working paper 2011. It was submitted to American Economic Review and accepted after being split in two papers. Preprints can be found online.
All the charts in all versions of the papers violate scientific data presentation standards:
– all Y-axes are “super-extra-stretchy” (Bruce Baker), exaggerating the purported statistical effect by orders of magnitude;
– none of the charts have error bars or confidence intervals; if they were shown, the error bars would certainly exceed the purported statistical effect by far;
– no r-square values or correlation coefficients are given; the r-squares are likely to be very low;
– the sample sizes on which the regressions are based are not shown; although the study drew on millions of student records, most of the records were incomplete and the actual regressions are based on only a fraction of the records.
Criticisms of the working paper by Bruce Baker (schoolfinance101.wordpress.com/2012/01/07/fire-first-ask-questions-later-comments-on-recent-teacher-effectiveness-studies/) and Moshe Adler (epaa.asu.edu/ojs/article/download/1264/1033) are worth reading. Adler points out that Chetty et al. report a small (albeit visually wildly exaggerated) but statistically significant effect of teacher VAM on student earnings at age 28 *but they found no effect at all on earnings at age 30*. Explains Adler:
“However, although the result they found for 30 year olds is not statistically significant, the words “not statistically significant” are nowhere to be found in their study. Instead the authors write, “The 95% confidence interval for the estimate is very wide. We therefore focus on earnings impacts up to age 28 for the remainder of our analysis.” … Furthermore, they didn’t just “focus” on earning impacts up to age 28; instead they proceeded as if the result for 28 year olds, an increase in income of .09%, was also the result that they found for 30 year olds, and made the assumption that this would have also been the result for any subsequent age group. Based on this assumption, which is contradicted by their own evidence, they calculated a life-time benefit of $25,000 from an increase of one standard deviation in teacher value-added. … After conducting a study its authors cannot ignore the results; science does not permit cherry picking. The result that teacher value-added does not have a statistically significant impact on earnings at age 30 must be part of any conclusion drawn from the Chetty et al. study.”
Damningly, the AER version of the paper omits even mentioning the negative finding for earnings at 30. The language from the working paper has simply been removed from the AER version! The editors and reviewers of AER must answer some hard questions about their scientific standards.
Here is Chetty’s response to Alder: http://obs.rc.fas.harvard.edu/chetty/Adler_response.pdf
I should add that Figure 1c, 2b, 4, and 7 all include the 95% confidence interval as a dashed line. The standard errors are all reported in all tables in the parentheses below the estimated coefficient.