A wonderful find by loyal reader KrazyTA, who has read the Vergara transcripts. The esteemed economist Raj Chetty of Harvard, a cheerleader for VAM, says he prefers large data sets (no humans) to anecdotes, then tells a theoretical anecdote about a coach who lays off rookie Michael Jordan. The fact that this never happened is of no consequence to the professor from Harvard.
KrazyTA writes:
A good intro to my last posting on the Vergara Decision.
What role do reason, consistency, logic, and facts play in the self-styled “education reform” movement?
From a posting on this blog of 3-23-2014, “Common Core for Commoners—Not My School!”—
“This is an unintentionally hilarious story about Common Core in Tennessee. Dr. Candace McQueen has been dean of Lipscomb College’s school of education and also the state’s’s chief cheerleader for Common Core. However, she was named headmistress of private Lipscomb Academy, and guess what? She will not have the school adopt the Common Core! Go figure.”
Link: https://dianeravitch.net/2014/03/23/common-core-for-commoners-not-my-school/
When you’re a supernova of the “new civil rights movement of our time” like Michelle Rhee you don’t have to get your numbers right—or even read the source you cite for your assertion!—when you make a claim.
“So the report Rhee herself cites contradicts her main point: standardized testing does, in fact, gobble up lots of classroom time. Her statement above, according to the source she herself cites, is just dead wrong.”
Link: http://jerseyjazzman.blogspot.com/2014/04/why-is-michelle-rhee-wrong-about.html
If you’re Bill Gates you can assert to Lyndsey Layton re the potential $tudent $ucce$$ synergy between Common Core, Pearson and Microsoft that:
“Yeah, we had the old Pearson stuff. I, it, it, there’s no connection, there’s no connection to Common Core and any Microsoft thing.”
But then there’s the joint statement by Pearson and Microsoft of 2/20/2014—
Link: http://www.digitaljournal.com/pr/1748922
But surely during the Vergara trial Dr. Raj Chetty was able to demonstrate the hard-nosed, data-based logic, consistency, reasons and facts behind VAMania and high-stakes standardized testing and the like, right?
Please refer to the following link for the references re Dr. Chetty’s testimony:
Link: http://www.vergaratrial.com/storage/documents/2014.01.30_Rough_am_session.txt
*Note that I will refer to the latest more refined version of the rough draft, which goes from pp. 508-595.*
Pp. 553-554, Dr. Raj Chetty:
“In my opinion there are two different approaches to analysis. You can look for anecdotes or you can look at large data sets. I prefer to look at large data sets because I think there is a psychological bias that any human being has to focus on outliers.”
Sounds reassuring until you realize that this is his way of “reasoning” away myriad accounts that VAM estimates fluctuate wildly or that Campbell’s Law is a real concern, not just a “conjecture.”
But here’s where the rubber meets the road.
When discussing LIFO (i.e., seniority) policies, he goes all in to make sure we of little understanding grasp why the least effective senior teachers should be laid off before the more effective novice teachers:
P. 576: “…[L]et me give an analogy which hopefully will resonate and be familiar with many people here.
Let’s say you are a manager of a basketball team and you have a new player, Michael Jordan in his rookie year who looks very promising, but in his rookie year he is not doing so well relative to the other players on the team. So you could take a short term perspective and say, well, this guy doesn’t seem to be doing so well this year, so I’m going to let him go and stick with the other players so that we do well in the next season, or could you take a longer term perspective and say Michael Jordan seems to have a lot of potential, he is going to be great in two years, he’s going to be one of the superstars, I’m going to keep him because I really care about my team in the longer run.
The LIFO policy is effectively saying let’s let Michael Jordan go, I wouldn’t want to have Michael Jordan on my team.”
There are so many things wrong with this brief example of “high-disorder” thinking that I hardly know where to start. But just some brief comments, and I won’t even bring up the whole mismangled approach to teamwork, cooperation and collaboration he brings up. *Michael Jordan could give Dr. Chetty some good advice: “There is no ‘i’ in team but there is in win.”*
So we lesser beings must zealously avoid the “psychological bias” we have in focusing on “outliers” and ground ourselves instead in “large data sets”? Is it too much to remind a numbers/stats person that the NBA has, maximum, less than 500 players [and not all can dress for any one game? And that these are exceptionally fine athletes, world class no less—literally, if you saw one of them playing in your local gym or playground, they would blow everyone else away. The bench warmers are outliers! They are ALL outliers!
And Michael Jordan is an outlier among outliers! Picking him as an example of what approach to use in assessing the effectiveness of millions of teachers via numerical rankings is betraying a hopelessly confused surrender to the “psychological bias” to focus on “outliers”!
Capiche?!?!?!?
And as for those millions of teachers: VAM already has moved the “highly effective” and “highly ineffective” and the “most effective” and “most ineffective” all over the rankings from year to year—how long could Dr. Chetty’s “Michael Jordans” survive such VAMboolzement before being kicked out of the classroom? [With all apologies to Dr. Audrey Amrein-Beardsley.]
I stop here. End of my Vergara Decision series.
Let’s pay heed to Joe Flood, author of THE FIRES:
“Initially, we use data as a way to think hard about difficult things, but then we over rely on data as a way to avoid thinking hard about difficult problems. We surrender our better judgment and leave it to the algorithm.”
[Jim Horn and Denise Wilburn, THE MISMEASURE OF EDUCATION, 2013, p. ]
😎
P.S. Even a very dead, very old and very Greek guy knew better:
“A good decision is based on knowledge and not on numbers.” [Plato]
Diane Ravitch's blog 
I can’t believe the crap now associated with the formerly revered name of Harvard. Maybe we should bring back the Puritan divines that founded the place.
LikeLike
Great comment, Harold.
LikeLike
Nice points, KrazyTA
IFor some reason, it brings to mind the three kinds of liars: liars, damned liars and outliars.
LikeLike
I’m getting to like your posts, Larry! TAGO!
You’ll have to join us folks down at the Pink Slip Bar one of these days.
LikeLike
Yes, how true. As I was taught in statistics and research design that research in the social sciences is based on ASSUMPTIONS that the researcher can live with when operationally defining the variables under study. All tests and studies of them are based on assumption. I think the eminent statistical wizards here can vouch for the veracity of that statement. Thanks for putting it in the right light Larry, and thanks Krazy for laying it all out so clearly.
LikeLike
All models are based on simplifications of reality. If I can again quote one of my favorite statisticians, Bill James:
Baseball statistics are simplifications of much more complex realities. It may be unnecessary to say this because, of course, all human understanding is based on simplifications of more complex realities. Economic theories are simplified images of how an economy works, replacing billions of complicated facts with a few broad generalizations. The same is true of psychological and sociological theories, it is true in medicine and astronomy. The search for understanding, wherever it roams, is a search for better simplifications-simplifications which explain more and distort less. Even the understanding gained from experience is, of course, a simplification of experience into the generalizations which are distilled from many experiences.
The New Bill James Historical Baseball Abstract
LikeLike
The more we simplify models, the less the models reflect reality. Add more parametes, and complexity increases nonlinearly. In other words, humans still rule over fools with statistical tools.
LikeLike
Actually humans tend to over generalize from very limited data.
If I can keep going with Bill James, have you read Moneyball: The Art of Winning an Unfair Game? Here is the Wikipedia entry if you do not have the time to spare: http://en.wikipedia.org/wiki/Moneyball
If you like small data sets (and live somewhere north of NYC you might appreciate this more), you might think about how many World Series victories they had before they hired the data guy Bill James as a consultant and how many they had after they hired Bill.
LikeLike
larry: I think you will find the crowd down at Pink Slip Bar & Grille a pretty congenial group. Linda and Ang are always a delight, except when talking about bee eating and masking tape and such… I follow Diane’s “Rules of the Road” for this blog so no turning the air blue here.
😳
And since Duane “TAGO” Swacker did the inviting, next time it’s on him! *Watch out, though, if he offers to pay with Mexican pesos instead of USA dollars and then leaves, sticking you with the difference. Quantity of money—not important, he says, just think of the quality time you spent with him talking about Noel Wilson. Yeah, sure…*
😜
Just a word to the wise: anything they officially serve is at least FDA approved but watch out for any, uh, edible outliers.
😕
I refer specifically to Socrates and what he calls his “hemlock special”—it’s a vial of something he carries under his ‘non-party’ toga when he’s feeling particularly aggravated, such as when putatively world-crass, sorry, world-class economists argue against using outliers in arguments and then use argumentative outliers.
Seems Socrates has a mental block against accepting anything that hypocrites say. Apparently he is a devotee of Homer—just about the only Greek guy older than him—
“”Hateful to me as are the gates of hell, Is he who, hiding one thing in his heart, Utters another.”
When I try to suggest that he might be blowing things out of proportion, he hits me with another Homer:
“Words empty as the wind are best left unsaid.”
Even my best efforts at lightening things up don’t always work. If I jokingly suggest to him to follow the advice of Charlie Chaplin [“Dr.” of Laughology]—
“A day without laughter is a day wasted”—
He sometimes responds with his argument-of-last-resort: “And just how old and how Greek is that guy?”
😡
As Dr. Raj Chetty would say, that’s the Michael Jordan of all “that’s the end of this discussion” retorts.
But other than that, he’s a pretty easy going kind of fella. And generous with the drachmas and the ouzo if you like to talk about truth and beauty and wisdom beginning in wonder and the importance of friendship and the like.
Be there or be square.
😎
P.S. “Outliars”—TAGO!
LikeLike
“”Hateful to me as are the
gGates of hell, Is he who, hiding one thing in his heart, Utters another.”There. Fixed.
LikeLike
KTA,
Como siempre has conectado con la pelota por un jonron que vuele más allá, que pasa sobre los asientos de los edificios de la avenida Waveland. (Ojalá que seas un miembro de la oposición y no de los Ositos)
LikeLike
mil perdones jonrón no jonron.
LikeLike
“Todos mis movimientos están fríamente calculados.” [El Chapulín Colorado]
Uh, yeah, “all my movements are coldly calculated”—just like that Mexican super-hero of yesteryear, The Red Grasshopper, said.
So, I knew what I was doing, ok? But don’t ask me to repeat it: time for a siestita [little nap]…
😎
LikeLike
LOL Duane!
LikeLike
I bet Michael Jordan would say that his great ability was not developed in a vacuum and that it was nurtured by playing with experienced and talented team players. If none of the experienced players were on his team when he was a rookie, how do you think he would fare? By the flawed analogy, Jordan would have been allowed to stay on while seasoned team players would be let go. Those seasoned team players might not make all the shots but they sure do the set up so that Jordans can develop and showcase their talents! There is another comment for the “capiche” folder! New teachers need seasoned teachers and any seasoned teacher will tell you how important this was when they were new. If “ed reformers” had their way, the majority of teachers in any one title one school would be TFA or other new teachers with very few seasoned ones. Where would a medical residency program be if only residents learned from residents????
LikeLike
Where would medical residency programs be without large data sets?
LikeLike
In the same buildings as they always have been!
LikeLike
If you had to choose, TE, would you really choose large data sets over experienced medical professionals to mentor residents? I prefer both but I want the data to inform professional judgement not determine it.
LikeLike
When you say an experienced physician, don’t you mean a physician with a large personal data set? I hope physicians are trained using and keep up with the latest results from large studies.
LikeLike
“I bet Michael Jordan would say that his great ability was not developed in a vacuum and that it was nurtured by playing with experienced and talented team players. If none of the experienced players were on his team when he was a rookie, how do you think he would fare?”
This!
LikeLike
Also Art, he was allowed to develop, how many years was he in the league before the bulls were a respectable team? He will also tell you he did not win a title on his own. Everyone on those championship teams had a defined role. The genius of Phil Jackson was to put players in roles that utilized their special and unique talents. He always said that the key was knowing what everyone could do well and putting them in situations where they could shine, not making them do things they could not do.. Jordan understood this too, quite often he would pass up a shot under pressure because one of his long range shooting team mates would be open at the right time. Jordan would have been Vammed out before reaching stardom.
LikeLike
old teacher.. definitely! But how could you write such wise words with a name like “old teacher” apparently according to teaching economist “data” should probably be entered into a computer by someone with a tag name like “TFAteacher” and the computer-generated after analysis from this newbie would tell all of us what to do.
LikeLike
I certainly seems to me that large data sets are to be preferred to anecdote. I have no idea why Dr. Ravitch put no humans in parenthesis. Humans are the source of the data.
LikeLike
Large data sets can be interpreted by humans in the way that is most beneficial to their cause. Just looking at cut scores on exams; who sets them? In my state, the math and science EOC had a cut score so low that 90% of most of the state passed, but in English it about 20 points higher, so the pass rate was lower by about 20-30 points per district. This caused a panic amongst “journalists” in our state, that English teachers were failing, but math teachers and science were succeeding; someone even stated that the “hard science” teachers were obviously “smarter”…
What I saw in these comments, was a lack of an informed public to ask the right questions such as 1) What were the cut scores on all the exams? 2) Why weren’t they the same? 3) Which grades are ones for which students need to pass an exam, in order to go to the next grade? ( these grades usually get more “buy in” from students, in terms of passing, and also, they get 3 tries to pass the test: this contributes to a higher pass rate overall eventually) 4) WHO sets the pass rate? 5) Has it changed in the years within which we have an election for Gov ( hint, it does…) 6)…I could go one.
Humans are indeed the source of data, but it is hardly unbiased. Like someone stated a few weeks ago: data can create whatever scenario you want it to create, depending on the interpreter. If people in this country really wanted to “fix” education, they would be looking at data related to poverty, education level of parents, support for quality teacher ed programs, support for wrap around services…
LikeLike
Title 1,
It seems that your concerns are not with anything related to Chetty’s work or the use of large data sets but with the public’s inability to interpret tests scores.
As for poverty, most of the research relating poverty to educational outcomes is based on large data sets. If people object to using large data sets for education, those conclusions must go as well, unless, of course, you are judging the quality of the research by the conclusions reached.
LikeLike
Okay. But the point is that Chetty uses an anecdote that never occurred which goes against his very argument.
And individual teachers don’t really have large data sets. An elementary teacher has a data set of 25-30 students per year. How many years before enough data compiles to make a suitably large data set?
Also, remember that Chetty never takes into account the quality of the data. I’m telling you, by Chetty’s measurements, I have no chance. I teach seniors and only seniors. They’ve flatlined on NAEP for two decades. They cease to care about school once they’ve set their post-high school plans.
On top of that, I have mostly AP seniors. They were near their ceilings before they got to me. How much growth am I going to get?
There are numerous variables for each individual circumstance that a large data set can’t and won’t capture.
Here’s an anecdote: Microsoft’s Lost Decade is largely traced back to its employee evaluation process of stack ranking. Microsoft dumped it. But teachers have to live by the same very flawed system with VAM.
LikeLike
I think it was a hypothetical, not an anecdote.
LikeLike
Steve
I was responding to TE.
LikeLike
Title one.. I was also replying to TE.
LikeLike
FLERP! Yes, a hypothetical. But still flawed nonetheless. There are exceptions and extenuating circumstances.
Under Chetty’s VAM methods, I have little chance to ever be rated properly.
He can function in his world of hypotheticals. The rest of us function in a world of realities. And since when are hypotheticals valued in a court of law?
LikeLike
“And since when are hypotheticals valued in a court of law?”
The point of expert testimony is to be helpful to the trier of fact, so, much like teachers, experts will use hypotheticals and analogies to the extent they help the trier of fact understand the expert testimony.
LikeLike
Steve K: a little experiment in reasoning a la charterites/privatizers and their educrat enablers and accountabully underlings…
John King, NY Commissioner of Education, thinks Montessori and Common Core are practically one and the same. How does he know? Because regardless of what Montessori teachers say and write, his kid goes to a Montessori school—that makes him an expert!—and to bring the hammer down—
“Montessori” and “Common Core” share the letters “m” and “o” and “n” and “r”—nobody can argue with that!
😏
And to turn the beans back on the bean counters: did you ever notice that “hypocritical” and “hypothetical” begin with the same four letters and end with the same five letters?
😱
No anecdotes here. Just data sets. So when it comes to Dr. Raj Chetty: I see you your Michael Jordan and raise you a Steve K and a Titleonetexasteacher.
Game. Set. Match. As the Bee Eater Herself would Rheespond if anyone disputes my attempt to stem the tide against the “psychological bias” to “focus on outliers”:
“I reject that mind-set.”
Who knew that something Michelle Rhee said would come in so handy?
😎
LikeLike
Would you go for an outlier with a psychological bias? Where is that bar?
LikeLike
Steve,
First a clarifying question. Some posters here like Harold at the top seem to be arguing that Chetty’s research is flawed, that is that either a) that differences across teachers in changes in student test scores over the school year do not reflect the differences in teacher’s performance teaching and/or b) that these differences in changes in student test scores do not have positive impacts on students later in their lives. Others, like yourself, seem to be arguing that the application of Chetty’s research to evaluate individual teachers is an error because, among other reasons, the sample size is too small and students have no incentive to do well on the exams (thought this is not consistent with the common claim on this blog that these exams create large amount of stress for the students). Do I understand your position correctly?
Your comment also brings up another important point. If you are teaching seniors who are already near the ceiling of what we expect high schools students to be able to know and do, we might well ask why those students are still in high school. Joe Nathan has posted extensively on co-enrollment in the Minnesota. When my middle son toped out on the math MAP exam we enrolled him as a special student in the local university for mathematics and eventually science classes. Perhaps part of the answer to the problem you bring up is to move those students who can not gain very much from the high school curriculum out of high school.
LikeLike
Ah so rather than make your own point, you challenge mine. Fine, I’ll play.
First, the Chetty study was over twenty years of test scores. How many years of test scores would somebody need before they could properly evaluate a teacher? Five? Ten? According to Chetty, one needs large sample sizes.
When evaluating me at the high school level, how many years? I have students for either half or a full year. (My AP students for a full year, my non-APs often for half the year.) With our schedule, I have them one-sixth of the time. How many years until there’s a suitable large sample size?
Because right now, if I have four years where I’m “effective” and then my fifth year is “minimally effective”, and layoffs happen in that fifth year, I can rightfully be laid off versus a second year teacher who received “ineffective” in year one and “effective” in year two. Even though I have a larger sample size and greater success.
Second, stress for the student depends on the age of the students and other factors. I’ve spoken with elementary colleagues and they say that comparatively, their students take formal tests rarely as compared to high school students. They are not old and experienced enough to know the test has little meaning for them but much meaning for the school, administrators and teachers. (A little like evaluating a dentist by how many cavities their patients have.) With Third Grade reading guarantee laws being passed, I suspect those little 8-year-olds will feel plenty of stress soon. But once kids get to high school, they know those tests are meaningless. They only care about ACT and AP tests. They laugh about the state tests. I’ve had more than a few AP 5’s get “not proficient” on the state social studies test. (Haven’t you noticed that the trends for student regarding stress are ALWAYS at the elementary school level? Spend any time around college-bound juniors in the week preceding ACT and you’ll see the same thing.)
As for your piggybacking on Joe Nathan, I don’t disagree and our school does have early college options in coordination with the community college (where I also teach evening classes). Also, as an AP teacher, that class is designed for potential college credit.
Applying Chetty’s research, whether one believes that it is flawed or not, to individual teachers is greatly misguided.
If I’m to believe your position, you don’t think so. Your so-called clarifying questions really had little to do with Chetty but rather were more aimed toward your own personal ideas on the matter. Especially that last part referencing Nathan.
I’d like to know how you’d take those short-ternm samples into account and how it connects to Chetty’s research. (Which by the way, I think is the silliest study that’s generated so much influence. I mean, on an individual level, students are making virtually no additional money over a lifetime. One sole high achiever can lift a class tremendously.)
LikeLike
Chetty’s research says that teachers matter. But his data set is so macro that it cannot possibly be used to identify a good teacher or a bad teacher.
LikeLike
Steve,
I think my clarifying question was very important. One can easily take the position that Chetty’s paper is a well done piece of research that shows how important good teachers are to the welfare of their students long after they leave that teachers class AND the position that it is poor public policy to use changes in test scores to evaluate individual teachers, especially when there are small sample sizes and the assignment of student to teacher is far from random.
As for stress, I have OFTEN made the distinction between younger students and older students when this is being discussed, and am generally hooted down for it despite quoting Dr. Ravitch in support of the difficulty of getting high school students to care at all about no stakes exams. I have no problem believing that young students can be made to be very concerned with the results of standardized exams even when they are no stakes exams for the students.
My local district has no early college options (though as I live in a college town, students can enroll at family expense in local university classes that do not count towards the number of units of work required for high school graduation) but does have some AP classes. Most of the high schools in my state are too small (the median size is a little under 250 students) to have any AP classes.
LikeLike
Hi teachingeconomist:
It seems to me you are very obsessive with data. However, it is obviously that you are not able to distinguish the differences between data and human beings. It is about quantity and quality. For instance, in Canadian Hockey, our famous Canadian Hockey player, number 99, Wayne Gretzky has shown his talent at very young age. Additionally, his father built their backyard as the ice arena for Wayne to practice passionately.
Now, please tell me exactly how do you form a big data of the human, a famous player in terms of his passion, effort, aspiration and talent?
However, I can see that you can plot his success based on a big data in terms of hours, days in practice, tournaments, his scores, his winnings when he is sick, sad, happy, or healthy…
In educational environment, the only data, and basic data is about students whose inner aspiration to learn, whose basic nurture at home emotionally and materially is the primary concern that affect the outcome of students’ behavior and learning.
Please take a look at my example of a famous hockey player. The greatest human being will need the proper nurture and guidance at home, then the support at school, finally the co-operation in the community and society where he live in.
I hope that my example illustrates the difference between data and human beings. Back2basic.
LikeLike
Why do they all seem to sound like Michelle Bachmann, with her blank look and lack of support for almost everything she says?? (Not a compliment in my book.)
LikeLike
I assume he chose Michael Jordan for this analogy because almost everybody’s heard of him and knows he was one of the greatest basketball players ever. I don’t care for the analogy, but that may be largely because it triggers my mansplaining instincts.
LikeLike
TY! This is GREAT!
LikeLike
NJ Teacher: you’re welcome to join us whenever you’ve got time if you don’t mind being around all sorts of unsavory characters…
¿? Look, Pink Slip Bar & Grille was probably the inspiration for the Marxist utterance:
“I refuse to judge any club that would have me as a member.” [Groucho, the famous one]
How do I know? On my first visit there, I almost walked out when I saw all the outliars [thanks, larry!].
For example, in a corner all to himself, is that Paul Vallas fella. He’s always begging the shady heavy hitters who come in for handouts [he calls them “jobs”], inspiring Duane Swacker to periodically, er, invite him to do some heavy reading. In this case, though, quantity [bound volumes of Noel Wilson] flung towards that corner in the hope that it might produce some quality thinking hasn’t produced even one observation that doesn’t betray the “psychological bias” to “focus on outliers.”
But hang with us. As Socrates likes to say when he’s in a good mood and has his party toga on:
“Light is the task where many share the toil.”
Meaning, of course, the “toil” of lifting our glasses high as we enjoy good conversation with genial companions.
Be there or be square. There’s always room for one more in favor of a “better education for all.”
😎
P.S. Diane gives me too much credit. I have not read the Vergara transcripts, at least in depth; just the one with Dr. Raj Chetty’s testimony. Truth be told, four [!] of them are LAUSD Superintendent John Deasy alone—and that would be a test of endurance more like a brutal hazing ritual than anything else. After all, my moniker is “KrazyTA” not “Glutton for Punishment.”
LikeLike
Quit Krazy TA, someone will think you are a genuine bona fide outliar, will you settle for exceptional? I think the next round is on me, or is that in me? I get confused these days.
LikeLike
@TE,
I think you completely missed my point.
My point was that data is used and manipulated politically to be “interpreted” in a manner that fits the needs of those in power.
LikeLike
Title 1,
Certainly data can be manipulated to some degree to fit the needs of any group, the question is if it has been manipulated in this case.
If anyone wants to see if Dr. Chetty et al’s conclusions are warranted from the data, he has given everyone open access to several drafts of their papers along with the complete data set and the econometrics programs they used to estimate the models. He has also publicly answered questions raised by some of his critics. This is far more transparency.
LikeLike
“Of Moles and Men”
Mountains outta mole-hills
Is Chetty’s claim to fame
Firings outta bubble-fills
On tests his lasting shame
If you don’t know what that’s about,read this
“Of course there are going to be mistakes — teachers who get fired who do not deserve to get fired.” — Raj Chetty
…but it’s worth it, because, you know, one bad teacher can mean a whopping $250 in lost in wages at age 28, which could mean no Planet Fitness membership…leading to obesity, chain smoking, unwanted pregnancy, jail time, heroin addiction (not necessarily in that order) and an overall downward spiral leading invariably to death at 29…
…but that fired teacher? They actually get a once in a lifetime opportunity to work at Walmart and make more money than they ever would have made teaching.
LikeLike
Larry.
Actually Chetty is a pretty accomplished guy. He is a MacArthur “Genius Grant” winner as well as John Bates Clark Medal winner (looked at as a baby Nobel prize given the number of Bates winners who have gone on to win the prize). Here is a link to his CV: http://obs.rc.fas.harvard.edu/chetty/cv.pdf
You do understand that there are all sorts of mistakes made in the labor markets all the time. People are not hired who should be hired, people who were hired who should not be hired, people who were fired who should not have been fired, people who were not fired who should have been fired. The only way to make absolutely sure that you don’t fire a good teacher is to never fire any teacher. Is that your suggestion?
LikeLike
A $250 differential in earnings at age 28 is a “molehill” whatever direction you look at from.
LikeLike
Larry,
My interpretation of the result is that by even using the crude measure of earnings as a way to evaluate the importance of good teaching on students we can see the impact that a long ago teacher had on a student.
It baffles me that the teachers here so vehemently argue that research showing the lasting importance of good teaching is incorrect.
LikeLike
larry: and if the economists help engineer another 2008 on us?
What if the students in that classroom don’t have jobs?
I think they’ll be worrying about a lot more than $250…
Keep commenting. I’ll keep reading.
😎
P.S. I first heard this from a self-sacrificing civil rights lawyer but it comes in handy for any occasion.
Q What do you call a thousand economists at the bottom of the sea?
A A good start.
😧
LikeLike
Chetty is brilliant but knows very little about schools ad teachers and students. He is the paradigm of Yale Professor James C. Scott’s book, “Seeing Like a State,” the big Thinker who sees the world from 20,000 feet up, designing big plans that fail to account for human will, frailties, dreams, capacity. As Scott shows, the big plans are excellent but consistently fail.
LikeLike
How involved is Dr. Chetty in designing state teaching evaluations? Did he design the systems in all 50 states or just some of them? Perhaps it is his coauthors who are doing that work.
LikeLike
Noise.
There seems to be a lot of it about these days — a veritable cacophony of congratulatory claims about VAMS.
What I find most curious is that people who should know better (folks at Harvard with stellar CV’s and “genius awards”) are drawing conclusions, making claims and supporting policies (firing teachers based on VAMS) which are simply not warranted based on the data and valid statistics.
Then again, one of the the most dangerous things about statistics (especially for researchers) is that you can use them incorrectly to fool yourself. (and others, unwittingly)
In their response to Chetty et al, The American Statistical Association specifically warned about reading too much into VAMS, particularly given the “noise”
The ASA was very clear about VAMS.
The following statements are from their report (with the bold and parts in brackets [] added by me)
The only way the ASA could have been any more clear about VAMS is if they had said in big bold letters at the top:
“WARNING! WARNING! DANGER! DANGER! VAMS SHOULD NOT BE USED AS A BASIS FOR FIRING TEACHERS”
How anyone who has a professorship at Harvard could read what the ASA said any differently is simply beyond my comprehension.
But then I never got an award for being a “genius” — or even ever had anyone call me one. 🙂
PS The idea that those of us who don’t approve of the use of VAMS to fire teachers somehow believe that “there are no poor teachers” or that “no teacher should ever be fired” or that “good and poor teachers have no impact on students’ later lives” is just silly.
LikeLike
Sorry, something happened above with the ASA quotes (with my comments in brackets [ ] after them) so here they are again (hopefully in more legible format).
“VAMs are complex statistical models, and high-level statistical expertise is needed to develop the models and interpret their results.” [******Warning!!: you can easily fool yourself and others if you don’t know what you are doing and/or are not careful******]
“VAMs are generally based on standardized test scores, and do not directly measure potential teacher contributions toward other student outcomes.” [Test scores don’t necessarily (and often don’t) translate to student success in later life and therefore neither do VAMS ]
“VAMs typically measure correlation, not causation: Effects – positive or negative – attributed to a teacher may actually be caused by other factors that are not captured in the model” [If you fire a teacher based on VAMs you may be firing them for something that is completely out of their control]
“The VAM scores themselves have large standard errors [due to “noise”] even when calculated using several years of data. These large standard errors make [teacher] rankings unstable, even under the best scenarios for modeling.” [VAMS are unreliable for ranking teachers]
“The majority of the variation in [student] test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.”
‘Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality”
LikeLike
Folks might be interested in Dr. Chetty’s response to the ASA statement:
Click to access ASA_discussion.pdf
LikeLike
Dr. Chetty cherrypicked the statement of the American Statistical Association. He chose to ignore this strong statement:
“VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.”
Does Dr. Chetty (and do you) have a reading problem? Or did he knowingly choose to ignore a resounding refutation of his flawed but famous “study”?
LikeLike
Dr. Ravitch,
I am a little confused by your comment, so I hope you will indulge me and allow me to post, in its entirety, Dr. Chetty’s response to the very same statement that you quote. I had posted the link earlier, but perhaps people did not click through or actually read it.
“ASA Point #7: VAMs should be viewed within the context of quality improvement, which
distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.
Discussion: The ASA is correct in noting that the majority of variation in student test scores is “attributable to factors outside of the teacher’s control,” and that this “is not saying that teachers have little effect on students.” These statements sum up some of the major findings from VAM research. First, comparing teachers based on raw student tests scores without a VAM approach would be biased against teachers serving students from disadvantaged backgrounds. A VAM approach helps to level the playing field, so that students’ knowledge and skills when they enter a classroom are not an impediment to a teacher receiving a positive evaluation of their performance. Second, while it is true that a single teacher is unlikely to turn a remedial student into an honors
student, our paper on teachers’ long-term impacts of shows that teachers do have meaningful effects on students. For example, we estimate that being assigned to a high-value added (top 5%) rather than an average teacher for a single grade raises a student’s lifetime earnings by more than $50,000. The fact that there is a lot of variance in student achievement due to numerous other factors – such as parents, neighborhoods, or health – does not take away from the important role that teachers can and do play in improving students’ outcomes.
The ASA appropriately warns that “ranking teachers by their VAM scores can have unintended consequences that reduce quality.” In particular, it is possible that teachers may feel pressured to teach to the test or even cheat if they are evaluated based on VAMs. The empirical magnitude of this problem – and potential solutions if it turns out to be a serious concern – can only be assessed by studying the behavior of teachers in districts that have started to use VAMs.
LikeLike
So if using VAM data can predict a $50,000 difference in lifetime earnings, I wonder what difference the factors accounting for the other 86-90% of variation would make?
LikeLike
I think you bring up a very important point. My state spends about half of its budget on education. Perhaps education could be better served by shifting some of those expenditures over to programs to support families with children.
LikeLike
I have no idea of what your state’s budget looks like, but I doubt the conclusions you draw for your own state would or should be the same for all states . Those Philadelphia teachers are functioning on fumes already. You know it’s bad if Arne Duncan feels the pressure to speak out. In Chicago, I think the community is tired of seeing their tax dollars go into the mayor’s slush fund (TIF) rather than into their schools, and the state of Illinois has a terrible record on school funding. I find it interesting that you always suggest that monies be taken from education budgets to fund programs to support family and children. Just because we expect education to solve all of society’s problems does not mean we have ever funded schooling with that premise in mind. It is only under those circumstances that your call for taking money from education to support such programs independently of schooling makes sense.
LikeLike
2old,
I am just following the suggestion that resources should be used where they are most effective. If student learning is largely independent of the skills of teachers in schools, it seems foolish to spend money to ensure highly skilled teachers in the classroom. That money would be better spent reducing poverty or improving healthcare.
LikeLike
If you accept that highly skilled teachers are best identified by student test scores, one could reach that conclusion.
LikeLike
I don’t necessarily accept that looking at changes in test scores is the best way to identify highly skilled teachers and interestingly neither does Chetty. I am simply thinking about the policy implications of claiming that teacher skill does not have a significant impact of student learning compared to other ways that we could use public resources.
LikeLike
Unfortunately, that is the way teachers are being judged. Funny how Chetty’s protests have become like auctioneer warnings on drug labels. The other part of my statement still holds true: the assumption that we ever have funded schools like they could compensate for social ills is the only way to justify taking education funding and shifting it to direct social services.
LikeLike
I think the point is that we should not try to fund the schools to make up for the deficits at home, but instead fund the homes directly.
LikeLike
“I think the point is that we should not try to fund the schools to make up for the deficits at home, but instead fund the homes directly.
Since schools most directly effected by poverty already tend to be the most poorly funded, it is highly unlikely that that they will become fully funded social service agencies as well.
LikeLike
I would not increase funding to the schools, but to the services that have a more direct impact on student learning like health and nutrition programs, parental education, etc.
LikeLike
So schools that no longer or never had libraries or librarians, music teachers, art teachers, or (gasp) gym teachers, counselors, nurses,… should not have their funding increased? How about schools with no toilet paper? What about those with paint hanging off the walls? Broken desks? Not enough desks? None or too few textbooks? You get the picture.
LikeLike
2old,
Given your point that the vast majority of student academic success does not involve the skill of the teacher, I would suggest that funds used to pay extra for teachers with graduate work be used to provide other, more important, resources to the students. No doubt toilet paper is among those more important resources.
LikeLike
Interesting that you should pick toilet paper as the one “resource” in short supply from my far from exhaustive list. I’ve got an idea! Heck, rather than eliminate all those teachers of non-essential stuff, why don’t we just get rid of special education all together? That would save lots of money. Those extra bright kids ought to be able to teach themselves, and those kids who don’t come up to snuff just need to rise to the occasion. Better yet, you are right, it would make sense to eliminate all compensation for advanced training. Why, we could even extend it into post secondary education! Who needs a Masters or a PhD to teach economics effectively? Get a few techie nerds to write some software and a pitchman type who can deliver online lectures. Man, just think of the savings in instructional costs!
“Given your point that the vast majority of student academic success does not involve the skill of the teacher,…” H-m-m-m, I don’t remember ever saying that, and a review of ALL the posts in this little back and forth would seem to support that contention. As we both indicated, VAM scores are probably not the best way to evaluate teachers, even though current policy is to do just that. Can a teacher have a significant impact on a student? Sure. Is his score on a standardized test going to be a valid method of determining that impact? Of course not! Can I translate that impact into a dollar amount in future earnings? Are you serious?
LikeLike
2old,
I picked one for brevity. Please remember you are the one that argues for teachers having a relatively small impact on student learning when compared to other factors like parental evolvement, student health, safety, etc. I am only pointing out the policy implications of your argument. It is difficult to argue that 1) teachers have a relatively small impact on student learning and 2) we should use resources to increase teacher credentials which we think will increase the effectiveness of teachers when those resources can be used more effectively in other ways.
LikeLike
I did not argue that. I argued that if suspect test data is going to be used as a critical measure of teacher effectiveness, then we ought to consider the amount of variability attributed to teacher effects by that measure.
LikeLike
Oh god: I was merely worrying about leaving a child “behind”, but now I need to worry that I will make them a chain smoking, overweight, heroin addict…
LikeLike
I would think a bigger risk is turning a student off from education entirely.
LikeLike
What is so strange is that I remember the NYTimes quoting Chetty as saying that his study suggests that poor teachers should be fired sooner rather than later. Which seems like an argument for getting rid of the young Michael Jordan, if we go along with this story in which the new player only has “potential” while the older players are known quantities. If the rookie is “not doing so well relative to the other players on the team,” doesn’t the cold-blooded, value-added, data-driven point of view advocate for cutting him from the team?
LikeLike
Dr. Chetty began his testimony in Vergera with a claim that 16 months is too short a time to evaluate teachers for their long term teaching potential. If it is true that poor teachers can always be fired from their position, the risk of the short probationary period is that great teachers are lost to schools because of the early decision.
LikeLike
I get it now. In my state, the probationary period is 3 years, which seems like plenty of time to figure out whether a new teacher is a keeper. And if one school/district lets the Michael Jordan teacher go within that time, he/she can probably get a job at another school and go on to be great; under a LIFO policy, people tend to assume that new teachers lose their jobs due to budget cuts, rather than incompetence, so they are still willing to hire them. It would be hard to imagine that the school system as a whole is losing potentially great long-term teachers. (Unless, of course, they can’t get jobs in the first place because all the slots are filled by 2-year TFAs…)
LikeLike
gloria41488: of course you are correct that when translating Dr. Raj Chetty’s advocacy research into practical terms, this is how the education establishment is going to handle, interpret it and use it. Then will come the “don’t blame me” version of the teflon defense by Dr. Chetty as he tries to salvage one or two shreds of his professional and personal integrity. *According to Bill Gates, there is a 98% chance of certainty that his attempt will be rated at nothing more than “satisfactory.”*
And the notion that his ideological constructs are logically consistent and mutually supporting—
VAMania all in my brain
lately reality just don’t seem so sane
makin’ it up but I know why
‘cause hey! me and Michael Jordan can touch the sky
😜
VAM ratings all around
Wildly go up, then wildly down
am I happy or in misery?
what me worry? ego and fame’s got a spell on me
😁
VAMania shining so bright,
keeps me up all day and night,
it’s got me blowing, blowing my mind
is it tomorrow or just the end of time?
😚
[heartfelt and profuse apologies to Jimi Hendrix, PURPLE HAZE]
Now that I’ve cleared that up…
😎
P.S. Deborah Meier also got a MacArthur Fellowship aka a “genius grant.” She is an ardent opponent of VAManiacal zealotry. The little finger of one of her hands knows more about teaching and learning and education than Dr. Raj Chetty’s entire body ever will.
Go figure…
😏
LikeLike