Moshe Adler, a professor at Columbia University, has emerged as one of the most incisive critics of the work of Raj Chetty, John Friedman, and Jonah Rockoff on Value-added measurement (VAM).
In the recent Vergara decision about tenure for teachers in California, the study by Raj Chetty and John Friedman of Harvard and Jonah Rockoff of Columbia played a prominent role. But according to the economist Moshe Adler the study is wrong and misleading. According to Adler, the authors suppressed a result that contradicts their main claim, they picked and chose which data sets to use, they used a misleading method of analysis that inflated their results and they misrepresented research that contradicts their claims as supporting them. These are just a few of the problems with the scientific integrity with the study. Adler wrote his review for the National Education Policy Center and it can found at: http://nepc.colorado.edu/newsletter/2014/06/adler-response-to-chetty)
A short time after the publication of his NEPC review, Adler received an email from Chetty that informed him that the study had been accepted for publication by the American Economic Review (AER). (Chetty also suggested that a Nobel Prize will likely follow!) Adler immediately wrote to the editors of the AER to alert them to the grave scientific problems with the study. The editor-in-charge did not evaluate Adler’s objections herself, nor did she send them to the referees to evaluate. Instead, she forwarded Adler’s letter to the authors who then replied to Adler’s NEPC review. The editor found this reply satisfactory, but as Adler explains in his response, Chetty’s et al.’s reply is without merit, and it only adds to the problems with the research. Chetty’s letter to Adler and Adler’s correspondence with the AER can be found at: http://www.columbia.edu/~ma820/Chetty%20Adler%20AER.htm
Diane Ravitch's blog 
Bravo to Prof. Adler for taking on these data-darlings of the right.
GREAT but it won’t make any difference to those who have already made up their minds. Don’t confuse them with facts.
those are pretty serious ethical issues for the journal and editor. very troubling.
“. . . very troubling.”
I noticed this term used by Adler.
To put it into the vernacular “very troubling” = bullshit!
Words fail!
The people of the United States have a lot more to worry about than the privatization of schools and it is this:
We are operating as an oligarchy right now with the super rich making decisions for all of us. I suspect that these people are dictating the results of research done by our most esteemed universities. This is likely why we’re getting papers with completely opposite conclusions from such places as Stanford and Harvard. Other previously independent institutions, such as the media, are also silencing any individuals who dare to oppose the 1%. Remember how the editors of the Washington Post protected school “reform” in DC, even in the face of obvious fraud and cheating? Remember how a Pulitzer Prize winning journalist of the New York Times was “reassigned” when he questioned the integrity of “reform?”
Students of history know that our present state of affairs is likely due to the effects of the Great Recession and will improve as the citizens wake up from the fog of economic despair; but still, it’s a frightening thing to see our great universities and newspapers compromise themselves. As the saying goes, “We’d all better pray.”
As for VAM, anyone with common sense knows that “data” that comes from invalid testing is invalid. The results of any whole-group standardized tests are invalid unless they are professionally handled and administered. And I’ll bet all university scholars know this.
“As for VAM, anyone with common sense knows that “data” that comes from invalid testing is invalid. The results of any whole-group standardized tests are invalid unless they are professionally handled and administered.”
No doubt about the results being INVALID or as Wilson puts it “vain and illusory” and it doesn’t make any difference whatsoever if they “are professionally handled and administered”. That has no bearing at all in regards to the COMPLETE INVALIDITY, ILLOGICALNESS AND UNETHICALNESS of using the results for any purpose.
Wow Duane. She made a comment about praying and you let it slip!
When appropriate, I will be pasting my music bit on the history of schooling in my state when you paste Wilson. Doesn’t fit here though.
Joanna,
I took the comment more as just a generic “Hey there’s this saying” and not as a proselytizing type or of the “true religion” that all should follow or they’re damned to hell.
Hope your non-contracted summertime is going well!
Actually, Linda did mention history and the context of our situation in it.
So I will paste my two cents here. Songs that put things in perspective are all I’ve ever had to give anyway, really.
And it is important that do look at all of this in the context of the economy and history so we can keep working to change it.
And for those who doubt that COMPLETE INVALIDITY (and I challenge all who do doubt to point out the flaws of the study, you know like Adler does to the Chetty, et. al study), read and comprehend what Wilson has proven in his never rebutted nor refuted “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
I won’t hold my breath while waiting for a cogent rebuttal/refutation!
Great stuff Duane! I was especially pleased to read the portion regarding epistemology.
Thanks, Joe N.
If you have not read Wilson’s work please do as my short summary doesn’t really do the study justice.
Another work on the validity aspect of standardized testing is found in his take down of the testing bible in “A Little Less than Valid: An Essay Review” found at: http://www.edrev.info/essays/v10n5.pdf
It is shorter-15 pages.
Thanks again Duane for the second PDF! I really enjoyed Koretz’s book “Measuring Up,” and I’m looking forward to reading this too.
The more people read and understand the complete invalidities involved in these educational malpractices the better off the students will be when we finally do get rid of those practices.
Keep spreading the word! And thanks!
Linda- you nailed it! Not all economists wear dollar-padded glasses. Steiglitz nails those economists in this great piece:
http://www.politico.com/magazine/story/2014/06/the-myth-of-americas-golden-age-108013.html
Steiglitz’ noble prize in economics is real not a figment of his inflated imagination.
About six years ago I wrote to a scholar from one of the nation’s premiere institutions. He and his colleagues had published an important paper about education. This paper was based on standardized tests given to students in a large city in the United States. The tests were administered by school personnel.
I said this to the researcher: “Don’t you realize that everyone is just drilling the children on the specific items of these tests?”
He responded, “Yes, we know that but they’re all we have.”
So a lot of the research on VAM is likely a perfect example of “Garbage in, garbage out.”
TE repetatively said that if someone doubted Chetty et al, all that person needed to do was look at the data. I guess Adler took that and ran with it.
teachingeconomist spends most of his time on this site asking willfully obtuse questions that seek to divert discussion from the issues at hand. We can probably expect much the same now.
I have said before that we should not feed the dragon; if we don’t, he’ll wither away and die . . . .
Then we can bury him and laugh.
Like all other vampires, he does not require food, only blood. You do know the tried and true way to dispose of Vampires?
yes, Yes, and YES! Ignore, don’t feed the beasts! (He is not alone.)
He is the worst among them all. He clearly has an emotional inter and intra-personality disorder. I wonder if he is dealing with it or even admitting it to himself, forget about his family.
He’s been too busy today claiming ownership of the page before this one to face or deal with his own issues. Must be an affinity with narcissist Gates.
I have asked teaching economist if he receives remuneration for one year now, with him refusing to comment on why he posts here. It’s been documented that certain right wing tax deduction outfits (think tanks) are paying employees to post obtuse nonsense on successful sites to set the argument. The internet is all we had as people. I will not let teaching economist ruin our last non-oligarch forum.
Nampa1,
For the eighth time I will answer your question: my income comes entirely from the good folks of my state that pay me to teach economics.
Repeating the same question over and over is a concern. You might talk to your physician about this.
I think TE does what he does because he saw his own child benefit from something different than a traditional neighborhood school and therefore does not feel like he can in good faith speak against the “new normal” of “market” ( even though we all know that is not really the term when it is tax money paying for it) reform lest someone call him out for the experience of his child.
TE. . . There was a balance of non-traditional neighborhood schools (like the ones we’ve discussed in NC) and states rather had that under control until reformists pushed for the flood gates to open. It’s ok to like that and still recognize when something is headed in a dangerous direction ( that is, the peril of public schools at all).
Which came first, the chicken or the egg? The study is so far out there and grasping, I mean, seriously.
Students who score high on tests have higher income at age 28. O.K. fine.
High student test scores are the result of high VAM scores?
OR,
High VAM scores are the result of high student test scores?
I don’t think that question was proven beyond the shadow of a doubt.
I don’t have to do research on this or write a paper. Just swap the teachers in my suburban district for a year with the teachers in a nearby urban district and I am fairly certain the scores in both districts would see little change. VAM is junk science. Everyone knows this so where are the unions?
What we have here is not a mere polite academic dispute to be argued in an academic journal(s). What we do have is junk, faux research being given the imprimatur of validity by a major journal in the face of significant disputation.
We have seen the consequences of acceptance of junk research: the consequences are not tit – for tat ever so polite journal replies, but, rather the education of public school students, the livelihood of teachers and finally, the fate of the public schools.
We continue to learn, often the hard way, that only way to fight injustice is by direct political action in the ballot box and in the streets.
Let’s not forget the upstanding but futile attempt by Dr. Moshe Adler to rectify shoddy, ideologically motivated research. He deserves our thanks.
Adler is not a “professor of economics” at Columbia, which is a good enough school that it wouldn’t tenure an economics professor who had never published any economics articles. He’s an adjunct in the urban planning school at Columbia.
He’s way out of his league tangling with the likes of Chetty and Rockoff.
Oh dear only an adjunct in the urban planning school at Columbia? Adler’s pedigree is certainly not worthy enough to allow him in the show ring with Chetty and Rockoff. Only they have the credentials to demand the highest fees for their stud services.
If I may correct your last sentence, Betsy:
“Only they have the credentials to demand the highest fees for their Pud services.”
double entendre intended.
I guess we had better not read anything WT writes unless he proves that he has published in peer-reviewed scholarly journals –which Adler has, BTW, as you will see if you do a search in a comprehensive database, such as a university library.
What a zoo this place has become. Do not feed the animals!
“He’s way out of his league tangling with the likes of Chetty and Rockoff.”
Only if you mean out of his league in confounding, obfuscating, fuddling and muddling or in more mundane terms bullshitting.
WT, since you attacked Adler’s credentials, please present your own. He signs his name to what he writes; why don’t you? He writes peer-reviewed articles; do you? Don’t throw stones if you can’t give your name.
Ms Ravitch,
How is this an attack? He is NOT a professor of economics at Columbia. You point out all the time those who falsify or exaggerate their CV. You should thank WT, and then make a correction to your post.
Moshe Adler
Adjunct Associate Professor of Architecture, Planning and Preservation
http://www.arch.columbia.edu/about/people/ma820columbiaedu
Moshe Adler
Faculty in the Harry Van Ardsdale Jr. Center
Mentor in the business, management and economics area of study
http://www.esc.edu/labor-studies-center/faculty-profiles/#d.en.41369
Moshe Adler
Visiting Associate Professor
Department: Grad Center for Planning
https://www.pratt.edu/faculty_and_staff/bio/?id=madler
I have no idea what an “Adjunct Associate Professor of Architecture, Planning and Preservation” is, but it certainly does not sound remotely close to your post describing him as… “Moshe Adler, professor of economics at Columbia University,”
—
Cynthia Weiss
C.V.
Mom of 3
No peer-reviewed articles or think tank reviews published.
Cynthia,
It was my mistake, not Professor Adler. You don’t have to be an economist to recognize that VAM is junk science. Sounds elegant on paper, but doesn’t work in real life. Did you read the American Statistical Association report? The joint statement of the National Academy of Education and AERA? The review of VAM by Stanford psychometrician Edward Haertel?
If VAM is so terrific, how come none of the elite private schools use it?
As for WT, he should sign his name. I preserve anonymity for teachers who fear that if their name is known, they may be fired.
WT is obviously NOT a teacher. He is a troll who enjoys baiting other commenters and entering snarky comments about others.he sneers at teachers. He should not be ashamed to post his name so we can judge HIS credentials, if he has any.
You sign your name. I sign mine. Why doesn’t WT?
Diane
Adler has a PhD in economics. A review of his courses, syllabi and publications (in the expansive database of a university library as someone suggested) indicates Adler is involved in urban planning, which encompasses urban and economic development. For example, Adler teaches a course on public/private partnerships, which are promoted so heavily by politicians today as if they are only positive, but I saw a historical and more balanced perspective in the syllabus for Adler’s course. Urban planners work with both public and private entities and funders, so economics courses for them make a lot of sense, especially with gentrification encroaching on low income neighborhoods and pushing out so many poor people from our big cities.
It looks to me like Adler has broken the mold for economics teaching in colleges and I have to applaud him for this. Traditionally, that has consisted of having two very different approaches to the teaching of economics in colleges, according to economist Richard Wolff. One approach is taken in economics departments, which tout theory and put a positive spin on the invisible hand of “free markets,” and a very different approach is taught in business school economics classes, where students learn practical skills, such as how to manipulate those “free” markets.
See Wolff’s discussion of this in “Taming Capitalism Run Wild” at Moyers and Company:
http://billmoyers.com/episode/encore-taming-capitalism-run-wild-2/
I’d like to point out that William Sanders, the creator of Value-Added, was an adjunct professor himself at the time that he sold the Tennessee legislature on TVAAS.
I suspect WT’s concern is that the original post describes Dr. Alder as a professor of economics rather than an adjunct associate professor of Architecture, Planning and Preservation. Inaccurate claims about academic qualifications have been seen as a concern in the past.
TE, I am sometimes tempted to ban you from this site because you hide your identity with no reason. I leave you alone because your contrariness is amusing. But you should not be ashamed of your name.
Dr. Ravitch,
I do not have the luxury of tenure or even “due process” protection that is so often discussed here. There is no need to ban me, if you wish me to stop posting I certainly will and this site can become the echo chamber that some of the posters here so fervently desire it to be.
It was Cynthia’s concern, TE.
Your “suspect” is rather suspect itself. WT obviously had other motives there. The “out of his league” comment clearly suggests that WT believes we should not believe Adler because he is not on as high a pedestal as Chetty et al is. If WT had a comment about Ravitch getting the CV right, then there was no reason to add in that little bit of nastiness at the end.
A full time college teacher that has been at the same school for decades (in a right-wing state), who makes nearly four times what I earn plus benefits (though I’ve had double the years of teacher experience & get no benefits), who supports education “reforms,” including stripping teachers of their hard-earned labor rights, and who promotes neoliberal economics policies, including privatization, all of which are consistent with the policies of both political parties, who is anonymous because HE fears losing his job! If this wasn’t so sick, then I might be amused. It sounds like sour grapes over his not having tenure.
I would much prefer the “echo chamber” of truth over his duplicitous black hole.
Folks,
It all seems quite easy to me. If one doesn’t want to have a conversation with TE then don’t respond to his posts. Sometimes I engage with him and other times, well, just don’t feel the need to add and at times even just skim over all the back and forths/tit for tats.
I don’t consider TE a troll and often times finds his questions pertinent and, no doubt persistent, because at times other posters will not directly answer his queries. Do I agree with him on many posts? Not at all, but many of his questions (which I realize that questions can be used as daggers-I employ them myself) add to the discussion many times but at other times seem to beat a dead horse (perhaps kind of like my posting of my summary of Wilson-read my comment above it and skip the summary if you’ve read it enough times to understand what is being said).
And while I would prefer that all feel free enough to use their names here I can understand why many don’t especially if, as with TE, they don’t have tenure. Fear can be an obnoxious tool of control. And social/peer, in this case the posters, pressure many times is used as one of those “obnoxious tools of control.”
Keep em coming, TE.
I would think that someone who plays the “I don’t have tenure card” and does not have academic freedom himself would see the importance of both and rally around teachers, not fight alongside powerful “reformers” who aim to strip teachers of such rights nationwide. His lack of compassion for those in poverty, except for foreign poor people, is very disconcerting as well.
I think TE is clearly a combative troll and l see no substance or value to his constant argumentative posturing. He’s very lucky that Diane has been so tolerant.
xxxxxxxxxxxxxxxxxxxxx I side with Duane on this one. I don’t find te’s comments troll-like. I for one find it helpful to refine my ideas by bouncing them off people with different ones. And I don’t use my own name for reasons of privacy. It is all too easy for a stranger to chase someone down & harass them on the internet.
I disagree with the encouragement of TE. I have gained nothing from his ongoing contentiousness, while I have learned a lot from others who have differing perspectives but are not intent on dominating the discourse and proselytizing. I think it’s pretentious when he comments on the relevance of other people’s posts and repeatedly questions and challenges people to present arguments to him. Contrary to what he may think, many of us are well educated and not subordinates who he has to make see the light. I am not his student and that approach to teaching rubs me the wrong way.
What I find disagreeable is the endless handwringing about what should be done with TE. The charges are: he is combative, he is disagreeable, he hides his identity for no reason, he does not rally around teachers, he appears to lack compassion, and above all he supports school choice. I like TE and I have no difficulty deciding when to read his comments and when not to. But this is up to all of you, not me. Should he be banned? Should he be ignored? Diane has the power to ban him, and everyone else has the power to ignore him. Get on with it already.
I don’t read TE’s comments anymore, and I skip certain others a lot too, including yours, Flerp. Too many devil’s advocates for my liking, which appear to be just for the sake of being contrary and fail at delivering insights.
Funny the way you select the experts based on your preference: economics comes first and it’s better than any other field. So, anyone who doesn’t belong to is already disqualified from refuting Chetty & Rockoff nest? Typical.
WT, if he is correct who cares? Maybe this proves the likes of Chetty and Rockoff are over rated. Krugman has also disagreed with them and I would say a Nobel Prize Winner is in their league. I also did not know that academic insight was the sole purview of the elite pedigreed. That assumption is in fact part of the problem.
What dianeravitch wrote just above.
Old Teacher: what you wrote.
It is a mystery to me why the defenders of VAMania and other numerical chimeras [e.g., the reliability and validity of standardized tests scores] stand by some Marxist principles when it seems to give them an advantage—
“The secret of life is honesty and fair dealing. If you can fake that, you’ve got it made.”
And yet deny the fundamental truth of others when they suspect it casts them in a bad light—
“I’ve got the brain of a four year old. I’ll bet he was glad to get rid of it.”
Groucho would be, like, so disappointed at their rigorously consistent inconsistency.
😱
At the very least they could acknowledge their deep intellectual debt to one of their most ingeniously disruptive thinkers re how to concoct the right kind of “research”:
“When I read about the evils of drinking, I gave up reading.”
Don’t be hatin’ on Henny Youngman.
😎
Now that Youngman quote is a quote to quote!!!
“…standing on Marxist principles” What drug has this character ingested?
Clear headed. Just under doctor’s orders:
“A day without laughter is a day wasted.” [“Dr.” Charlie Chaplin, “Doctor” of Laughology with a minor in Smile-esthenics]
My doctor would be so pleased that I took my daily dose.
😎
All research will contain exceptions to the authors’ findings and conclusions. In addition to limitations and delimitations, the exceptions or contrary findings need to be openly admitted by a study’s author(s). Suspicion is warranted if authors do not openly admit to and discuss these things, regardless of whether or not the study is published in a peer-reviewed journal. When I am reviewing submitted papers, this is one among many items that I look for, especially if it seems like an author is promoting his or her findings as the be all/end all.
Just for your information TE, Dr. Adler has a Ph.D in Economics from UCLA. He is not listed as a professor of Architecture
Old teacher,
Indeed he does have a PhD in economics from UCLA, but here at least (http://www.arch.columbia.edu/about/people/ma820columbiaedu) he is listed as Adjunct Associate Professor of Architecture, Planning and Preservation. Perhaps Colombia GSAPP website is out of date.
TE, I will say the same to you that I said to Cynthia Weiss. I misstated Professor Adler’s title. He did not. It is irrelevant. VAM is junk science. Or just plain junk.
I think it’s interesting to note that the creator of Value-Added, William Sanders, was an adjunct himself when he sold the Tennessee legislature on TVAAS.
How can we mere mortal innumerati [the mathematical equivalent of illiterates] possibly understand all this high falutin’ talk that involves numbers and stats and gosh who knows what other important subjects beyond the ken of mere mortals?
Click on the second link in the posting and see what Dr. Adler has to say about how how Dr. Chetty handled the 28-year-old and 30-year-old question in his ‘research.’
Now let’s take a look at the late Gerald Bracey’s READING EDUCATIONAL RESEARCH: HOW TO AVOID GETTING STATISTICALLY SNOOKERED (2006). More specifically at pp. 24-29, titled “Selective Use of Statistics” where he deals with the infamous A NATION AT RISK [ANAR] (1983) as a way of illustrating the following Principle of Data Interpretation: “Look for and beware of selectivity in the data.”
On p. 24 he quotes from ANAR: “There was a steady decline in science achievement scores of U.S. 17-year-olds as measured by national assessments in 1969, 1973, and 1977.” *Based on the National Assessment of Educational Progress aka NAEP.*
But then Bracey seems to remember Ionesco [I am bringing the old dead French guy into the discussion]:
“It is not the answer that enlightens, but the question.”
Also on Bracey 2006, 24: “…we should ask why the commissioners selected only science and why they selected only seventeen-year-olds to make their point. NAEP also tests nine- and thirteen-year-olds. NAEP also tests reading and mathematics at those three ages. So if the decline is widespread and awful, why weren’t the other ages and other subjects mention?”
Then Bracey employs what I consider the Voltaire card:
“No problem can withstand the assault of sustained thinking.” [I am bringing this quote in, not Bracey.]
The first paragraph on p. 26:
[start quote]
If we look at all nine trend lines (three subjects tested at three different ages) as show in Figure 2, we quickly see that the science trend for seventeen-year-olds is the only one that shows a “steady decline.” It is the only one that will support the report’s crisis rhetoric and it was the only one mentioned. (Terrel Bell, then secretary of education who commissioned A Nation At Risk, was quite candid in his memoir The Thirteenth Man about how he and heard many stories about the terrible state of public schools and had convened the commission to document the stories.)
[end quote]
Is Dr. Raj Chetty the new Terrel Bell?
😱
And would that make Terrel Bell a [posthumous] candidate for a Nobel Prize too?
😎
Chetty picking is as old as statistics (even older, in fact)
The only thing that has changed over the years (quite recently, in fact) is the spelling.
SomeDAM Poet: TAGO!
😏
Ponder this—what oh what did they do in days of olde when knights were bold and econometricians weren’t invented?
“In ancient times they had no statistics so they had to fall back on lies.” [Stephen Leacock]
Can I apply for the Nobel Prize by mail? And is the deadline 28 or 30 days? Only the 28 days count, as the latest unconfirmed rumors have it, but who these days pays attention to such scuttle butt?
😎
KrazyTA
I don’t think you can apply, but I’ll submit a nomination for you.
My credentials are impeckable (see below) and my recommendation carries a lot of weight (500* pages of “KrazyTA deserves to Win” which will be sent UPS-overnight)
Sincerely,
Some DAM Poet (PhD & Noble Prize in Econometrics, Degrees R U)
*PS I can make it 1000 pages, if you think that would be better.
Krazy, let me give you a better statistical likeness for Dr. Chetty, do you remember Dr. Lewis Terman at Stanford? He used and abused numbers too. As to the income effects of teachers, look at the current economy. What is the likely income of any college graduate at this time? Reality intrudes on assumption again, and all statistics are based on the assumption that IF a given condition prevails and is causal, THEN certain outcomes should be noticed…..If is still a very big word as Dr. Chetty’s number shaving would indicate. His effect disappears pretty quickly, and given changing conditions in our jobless and low pay job recovery may already be non existent.
Bob Shepard cx: on should be an.
Krazy or someone help me, I am throwing one more tid bit to the trolls. Cynthia Weiss, since you linked to the site, you should read a bit further. Dr. Adler is an economist, he teaches about the economic impacts of urban planning, something vitally important to the graduate school of architecture, which in the case of Columbia also includes the doctoral program in Urban planning. He also specializes and consults on the effects privatization of public utilities and services. Yes, he does teach a specialized aspect of economics at Columbia University. He has up close specialized knowledge that uniquely qualifies him to critique Dr. Chetty. He just isn’t in line with the dominant right wing point of view. You just need to dig a little deeper and seek a little more knowledge.
A typical student has how many teachers in K-12? 30 to 40? I would call it presumptuous to be able to attribute success to one teacher. Is the increased income based on the average score of 30 to 40 teachers? I want to give these guys the benefit of the doubt, but any way I look at this, there are far to many variables to isolate specific conditions and outcomes.
“I want to give these guys the benefit of the doubt. . . ”
Not me, I want to kick them in the mental petard to awaken them from their not so scientific opinionated stupor.
What’s the deal? One year they gave 100K students ‘the awesome teacher’ and 100K students got ‘the placebo teacher’, but both kinds looked identical with matching sportcoats?
Reblogged this on peakmemory and commented:
More on the Value Added controversy.
“For Whom Nobel Tolls”
A Nobel Prize in Hubris
Is what I do deserve
Although it might sound dubious
I’ve really got some nerve
I won’t let major sticking
Points get in my way
My trademarked Chetty-picking TM
Will surely win the day
“The Economics Prize has nestled itself in and is awarded as if it were a Nobel Prize. But it’s a PR coup by economists to improve their reputation,” — Peter Nobel, great great nephew of Alfred Nobel
I think I have pointed to Adler’s critique earlier on this site. It is quite devastating and Chetty’s response is truly troubling – he either really doesn’t understand the statistics he uses, or he is deliberately misleading. For example he claims that the statistical insignificance of one of his main results doesn’t matter because there weren’t enough observations. In fact there were tens of thousands of observations, and in any case it violates every principle of sound statistical inference to hide a negative result (i. e. supporting the null hypothesis).
When I saw the plots Chetty uses in his paper (which so famously impressed to the scientifically illiterate judge), I immediately noticed that the y-axis was stretched and the plots did not show either standard deviations or confidence intervals. The “data points” in these plots (so-called binned plots) do not represent actual data but are averaged over tens of thousands of observations without representing the extent of variation. Both of these tricks – stretching the axis and suppressing the variation – are strict no-nos in scientific data presentation. I was confident that no actual peer-reviewed scholarly journal would publish these plots, so transparently misleading. I was wrong (I give you that TeachingEconomist) – the AER editors did publish the paper, apparently in full awareness of the scientific (misleading plots etc.) and ethical (hiding of a negative statistical result that was reported in an earlier version of the same paper) issues. This is an extremely troubling occurrence.
I believe that the paper (both versions) should be examined by statistics experts with no stake in the dispute. They should clarify whether the statistical methods and data presentation in the paper conform with scientific standards.
Uarktransparency,
Unfortunately we will never know who the referees were that reviewed the article for AER (typically there are three), but I would suspect they were highly competent statisticians with no stakes in the dispute. The line between econometrics and statistics is extremely thin, and Ph.D. Statisticians are often hired in economics departments.
At one point in this debate, a faculty member at Arizona State was speaking about replicating the study. Does anyone know how much progress has been made on this effort?
Chetty’s comment (in response to Adler) to the lack of scatter plots revealing the variance in the data (“standard in papers that study large datasets” – absolutely wrong, it’s a clear violation of scientific standards) reminded me of another famous piece of cognitive research, The Bell Curve, about which Stephen Gould wrote (http://www.dartmouth.edu/~chance/course/topics/curveball.html):
“Their numerous graphs present only the form of the relationships; that is, they draw the regression curves of their variables against IQ and parental socioeconomic status. But, *in violation of all statistical norms that I’ve ever learned, they plot only the regression curve and do not show the scatter of variation around the curve*, so their graphs do not show anything about the strength of the relationships.” And concerning R-squares: “In Appendix 4, then, one discovers that the vast majority of the conventional measures of R2, excluded from the main body of the text, are less than 0.1. These very low values of R2 expose the true weakness, in any meaningful vernacular sense, of nearly all the relationships that form the meat of The Bell Curve.”
This is word for word true for Chetty et al., except that they don’t even reveal their R-squares (which are certainly very low).
Uarktransparency,
A couple million dots in a graph that must fit on a journal page would be a giant blob of ink. What information would you gain?
Why don’t you peruse a real scientific journal for once. Presenting large data sets graphically is not a new problem, and there are standard methods to achieve this. The way I would have presented these data would be by showing a box plot (http://en.wikipedia.org/wiki/Box_plot) or similar or a confidence interval (CI) for each of the dozen or so “binned” data points.
A box plot or CI would immediately reveal the following: how much variation is represented in each binned data point, and how much the bins differ from each other (a random example is here: http://connectmv.com/tutorials/r-tutorial/basic-plots-in-r/). If Chetty et al. had chosen this approach, their plots would show that the variation attributed to VAM scores is tiny compared to the overall variation (it is likely that the difference would hardly be visible to the naked eye). They also couldn’t have stretched the y-axis as shamelessly as they did because the box plots or CIs wouldn’t even fit on their scale (remember, they magnified the scale about 20-fold). Had the plots been properly done, I doubt very much that any judge or journalist would be impressed by them.
Chetty et al. chose to divide their data set of millions of observations in a dozen or so subsamples (“bins”), graphically report only the mean of each subsample, and run a regression against those means, reporting the slope of the regression line but not its R-square indicating the strength of the correlation. As a general rule, you NEVER EVER report a mean without also reporting variance/dispersal, and you NEVER EVER report a correlation without reporting the R-square. These are iron rules that you learn in statistics 101. There is no excuse for any author violating these long established scientific principles, and there is no excuse for any journal editors or peer reviewer to overlook such violations.
Now to your question. I can’t help asking, why don’t you peruse a real scientific journal for once. Presenting large data sets graphically is not a new problem, and there are standard methods to achieve this. The way I would have presented these data would be by showing a box plot or similar or a confidence interval (CI) for each of the dozen or so “binned” data points.
A box plot or CI would immediately reveal the following: how much variation is represented in each binned data point, and how much the bins differ from each other. If Chetty et al. had chosen this approach, their plots would show that the variation attributed to VAM scores is tiny compared to the overall variation (it is likely that the difference would hardly be visible to the naked eye). They also couldn’t have stretched the y-axis as shamelessly as they did because the box plots or CIs wouldn’t even fit on their scale (remember, they magnified the scale about 20-fold). Had the plots been properly done, I doubt very much that any judge or journalist would have been impressed by them.
Uarktransparency,
You can always write the editor of The American Economic Review about the sloppy research they are publishing.
Thanks for conceding the debate.
Btw Diane thanks for correcting the slight mistake in the title of this post.
It wasn’t really a mistake – Adler IS trained as an economist and author of a book about economics. He is not apparently the kind of economist to make career in academic economics departments, maybe because he has actual scientific standards as opposed to ideology.