Our friend and frequent commenter KrazyTA has analyzed the response of the VAM Gang (Chetty, Friedman, and Rockoff) to the American Statistical Association’s pithy demolition of their famous and much praised justification for VAM.
Here is his analysis:
I urge viewers of this blog to read the recent response by Raj Chetty (Harvard University), John Friedman (Harvard University) and Jonah Rockoff (Columbia University) to a statement by the American Statistical Association (ASA) [2014] on VAM.
A pdf file of same (less than five pages hard copy) can be accessed at—
Link: http://obs.rc.fas.harvard.edu/chetty/ASA_discussion.pdf
The last paragraph of their response to ASA’s point #7 (p. 4):
“The ASA appropriately warns that “ranking teachers by their VAM scores can have unintended consequences that reduce quality.” In particular, it is possible that teachers may feel pressured to teach to the test or even cheat if they are evaluated based on VAMs. The empirical magnitude of this problem—and potential solutions if it turns out to be a serious concern—can only be assessed by studying the behavior of teachers in districts that have started to use VAMs.”
Immediately followed by the last paragraph of their response, in full (p. 4):
“In summary, our view is that many of the important concerns about VAM raised by the ASA have been addressed in recent experimental and quasi-experimental studies. Nevertheless, we caution that there are still at least two important concerns that remain in using VAM for the purposes of teacher evaluation. First, using VAM for high-stakes evaluation could lead to unproductive responses such as teaching to the test or cheating; to date, there is insufficient evidence to assess the importance of this concern. Second, other measures of teacher performance, such as principal evaluations, student ratings, or classroom observations, may ultimately prove to be better predictors of teachers’ long-term impacts on students than VAMs. While we have learned much about VAM through statistical research, further work is needed to understand how VAM estimates should (or should not) be combined with other metrics to identify and retain effective teachers.”
My initial reaction.
While they don’t use the term “Campbell’s Law” — IMHO, they are deliberately avoiding it — notice how they take the import and sweep of Campbell’s astute observation and reduce it to “responses such as teaching to the test or cheating” with the added proviso that “there is insufficient evidence to assess the importance of this concern.” *Note that in his testimony during the Vergara trial, Dr. Chetty on p. 547 casually dismissed this challenge to his VAM-based beliefs as “Campbell’s Conjecture.”*
Link: http://www.vergaratrial.com/storage/documents/2014.01.30_Rough_am_session.txt
This is critical. First, they reduce Campbell’s Law to a statement about individual morality and ethics—of the employees no less!—rather than something that involves whole institutions [e.g., the recent VA scandal or the Potemkin Villages of the now-vanished Soviet Union] and is created/mandated/enforced from the top down. Second, by doing so they avoid having to address the destructive effects of Management by the Numbers/Management by Objective/Management by Results, i.e., the very management philosophy of those funding their “research” and leading the charterite/privatization charge. Third, they literally discard the already large amount of evidence proving the accuracy and trustworthiness of Campbell’s Law re VAM [and its fuel/food, standardized test scores] by referring to it as “insufficient” — while their pronouncements, of course, even though they need “further work,” is the current Gold Standard.
So it is hardly surprising that they are hot and heavy for heading off potential problems in data corruption by “studying the behavior of teachers in districts that have started to use VAMs” when what is needed is to independently study, monitor and regulate the behavior of folks like administrators, school boards, heads of CMOs and charter owners/operators, the DOE, and those who employ people like Chetty, Freidman and Rockoff—they’re the ones that set the numerical goals/straightjackets that drive data corruption!
*While W. Edward Deming would come in handy here, someone else thought along the same lines: “When a measure becomes a target, it ceases to be a good measure.” [Charles Goodheart]*
The next is a bit perplexing. Apparently they don’t know how to use google and Amazon to find (among many such works) Sharon L. Nichols and David C. Berliner, COLLATERAL DAMAGE: HOW HIGH-STAKES TESTING CORRUPTS AMERICA’S SCHOOLS (2010, third printing) or Phillip Harris, Bruce M. Smith and Joan Harris, THE MYTHS OF STANDARDIZED TESTING: WHY THEY DON’T TELL YOU WHAT YOU THINK THEY DO (2011). Perhaps they permit themselves no newspapers, internet, or television either, hence testing scandals such as those in Washington, DC and Houston, TX and Atlanta, GA (just to name a few) escaped their notice completely. Also, the above authors and many others, like Audrey Amrein-Beardsley (see her recent RETHINKING VALUE-ADDED MODELS IN EDUCATION: CRITICAL PERSPECTIVES ON TESTS AND ASSESSMENT-BASED ACOUNTABILITY, 2014) can be contacted by email. Is it too much to ask of those claiming to be researchers that they take the time and make the effort to, er, get the contact information they need to make sure their research is done properly?
In their response to ASA point #7 they quote the ASA to the effect that “Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality” (p. 3). The trio start off as best they can by stating that “The ASA is correct in noting that the majority of variation in student test scores is ‘attributable to factors outside of the teacher’s control,’ and that this ‘is not saying that teachers have little effect on students.’” Wait! You can read the rest for yourselves but a fly in the ointment—or the elephant in the room—when you’re in a debate is that when you concede the most critical point you lose the argument.
Since Chetty/Friedman/Rockoff didn’t dispute the 1% to 14% assertion then I would like to point out that I would be awfully interested in knowing why they’re ignoring the other 99% to 86%. Could it be that it’s poses intractable difficulties to their VAManiacal beliefs?
My very last point. Chetty/Friedman/Rockoff don’t understand that even under the most favorable circumstances, high-stakes standardized testing measures very little, is inherently imprecise, and is used for purposes so inappropriate to its few strengths that it needs to be junked. Take out of the Chetty/Friedman/Rockoff response those terms referring to “test scores” and the like and, well, the whole thing falls apart. Those “vain and illusory” [thank you, Duane Swacker!] numbers/stats are the glue that holds VAM together, the fuel that keeps VAM moving ahead, the food that sustains its very existence.
The Golem of VAM reverts to its inert form when you remove the magic of Testolatry.
Perhaps they should have taken that class in ancient Greece rather than Bean Counting For $tudent $ucce$$—
“I have often repented of speaking, but never of holding my tongue.” [Xenocrates]
Or if you prefer another very old, very dead and very Greek guy:
“Words empty as the wind are best left unsaid.” [Homer]
Take your pick. Odds are you won’t go wrong. [a numbers/stats joke…]
😎
P.S. I leave it to readers of this blog to read the triad’s response and make their own judgments and comments.
VAM is already producing ridiculous results in Ohio. Ohio seems to be where ed reform theory lands first and becomes ludicrous.
I read of an OH district that is in contract negotiations. They are offering the end of employer-provided health insurance (the teachers will be given a cash payment to purchase insurance) and “bonuses” of up to 30k (!) based 50% on test scores.
I’ll make a bold prediction. In this state,
the majority of teachers will lose as districts and the state use this measure
as justification to cut employee costs in school budgets across the board.
VAM is fundamentally flawed.
Assume the baseline test measures knowledge the students already possess.
Some students then score lower on the post test.
By the logic of VAM, the students lost knowledge I never taught them.
Senseless. The flaw questions the validity of the entire concept and its implementation.
You see it from a different perspective than I do. I see it as a method to measure compensation. I think it absolutely stinks as public policy, and public education is PUBLIC policy.
Math Vale: a couple of other points I didn’t bring up.
Last I heard, those “large data sets” from LAUSD that are critical [involving hundreds of thousands of human beings aka “data points”] to Chetty/Friedman/Rockoff weren’t/aren’t available to other researchers because, dontcha know, they’re confidential records. Unless that has changed in the last few nanoseconds— that would be made known almost instantly across the world wide web—then we just have to take partisan advocacy research done with the fervent blessing of LAUSD Superintendent John Deasy [contempt for genuine research is summed up in the “Dr.” that shamelessly fronts his name] at face value.
You know, take their research at face value, because proof by assertion is just what all the best “Drs.” like “Dr.” Steve Perry and “Dr.” Terence Carter and other self-styled leaders of the “new civil rights movement of our time” do. Campbelll’s Law is rheeally Campbell’s Conjecture because we say so. Nyah nyah nyah.
Let’s tamp down the heat a bit, throw a bountiful bucketful of cold water on these burning hot pontifications. Audrey Amrein-Beardsley, RETHINKING VALUE-ADDED MODELS IN EDUCATION (2014), chapter 2. That would be a good beginning—but no CCSS ‘closet’ reading. And for what a research-based book that actually surveys the, er, “field of research” it is contributing to looks like, take a walk on the wild side and look at this strange creature called “REFERENCES” on pp. 215-246.
Although I caution readers of this blog to review the specific psychometric definitions of the following terms, I urge them to buy Amrein-Beardsley’s book, read chapter 2, and ponder what she says about “reliability” and “validity. ” Consider as well “fairness” and “transparency” and “VAM-based use” and then ask yourselves—
How did Chetty/Friedman/Rockoff ever pass that pre-stats/number class where the teacher explains (with coloring books, videos, one-on-one tutoring, a term paper drowning in red ink pointing out all the areas for improvement) that “correlation is not necessarily causation.”
*Or maybe where they studied—It’s also been proven by some at certain points that the length of hair [males] and musical genius go together, plus hemlines and markets go up and down in tandem. Still, I insist: “Correlation is not necessarily causation.”*
As a charter [yes, a ‘closet’ reading joke!] member of the innumerati, even I get it.
Et tu, Chetty/Friedman/Rockoff? [yes, “tu” is in the Latin second person singular but it’s a joke, capciche? Doesn’t have the same punch if I write “Et vos, Chetty/Friedman/Rockoff?”]
😎
Please note that not once did I use Michael Jordan to buttress my argumentation, thereby avoiding [I am sure with Dr. Raj Chetty’s full approval] the “psychological bias that any human being has to want to focus on outliers.” See his Vergara testimony, p. 554.
KrazyTA doesn’t want to be one of those dreaded “outliars” [thanks, larry!].
😏
Krazy,
The data set used for Chetty et al is available to all. I am not sure that LA unified was the large urban school district that provided the data. Do you know that it was the school district?
The next time someone suggests that poverty causes poor educational outcomes I will be sure to ask your question about them having to learn the difference between causal relationships and corrolation.
“. . . ponder what she says about “reliability” and “validity”.”
Or ponder what Noel Wilson says about “unreliability” and “invalidity” in his take down of the testing bible–“American Educational Research Association; American Psychological Association; National Council on Measurement in Education. (2002). “Standards for Educational and Psychological Testing”. Washington, DC: American Educational Research Association”.
His essay review “A Little Less than Valid: An Essay Review” can be found at: http://www.edrev.info/essays/v10n5.pdf .
How sad to hear that the truth doesn’t matter in all this measurement hoax. Just reading that a teacher could earn a bonus makes it sound like something possible and realistic. This is about as awful as awarding needy people grants. It implies an awful lot and put all the focus on the person as if all the reasons they’re needy are all within their locus of control. So all this hocus pocus seems more like a way not to fund education and welfare without making it look that way.
Are Chetty/Friedman/Rockoff minions of the Axis of evil ignorance?
Lloyd Lofthouse: well put.
And I read all your comments on this blog. But your last one had an unintended consequence; see below.
Once again, I have been contacted by by an organization that claims to represent the interests of a group that has suffered from unfair—or at least exaggerated—negative stereotyping. And they insist I reply in some fashion…
Apparently right after you posted your one sentence I got a flood of emails from the MMA. Not, as you might suspect from the acronym, M[ixed]M[artial]A[rts], but rather M[aligned ]M[inions of ]A[merica]. Their old motto: “Thirty pieces here we come!” Their new motto: “It’s all for the kids!” [Hey, you can’t make this stuff up…]
Although all bear the MMA logo, each has a different name, some seemingly from individuals and others from the officers of the organization. I suspect, though, after going to their website, that like one of their favorite Minion-izations [as they like to call similar groups]—namely, ALEC—they were all sent by the same person. The only difference is that periods and commas are in slightly different places in each one, and some start with a more respectful “Mr/Ms/Mrs KrazyTA” and others with “😡KTA”—but otherwise all the exact same wording and talking points [though occasionally in slightly different order].
Anyhoo, to make a long story short, they feel that lumping Chetty/Friedman/Rockoff in with MMA-affilates and members and sympathizers is grossly unfair to highly effective minions everywhere who “do the dirty work that no one else will deign to do” [I cannot forget to include their footnote with this citation—“Alliteration is us!”—although I feel it is a bit boastful].
While they have long pondered the wisdom and “ethics” [they like to put this word in quotes all the time] of including edupreneurs [e.g., Bill Gates] and edufrauds [e.g., Arne Duncan] and edubullies [e.g., Michelle Rhee] among their ranks, they recently adopted a strong binding resolution against accountabully underlings/edubean counters, claiming that there is a 98% chance of satisfactory certainty [copyright Bill Gates] that the inclusion of those that massage, torture and pluck from the sky numbers&stats are giving everyone else a badbadbad name.
While I question their logic, consistency and facts, I am getting tired of deleting so much spam from my email in-box.
So please, the next time you are inclined to associate that trio with a “minion”-anything, have a heart and remember that my moniker is “KrazyTA” and not “Glutton For Punishment.”
But there is one nice touch to the MMA. They have a pretty eye-catching graphic with the words “Let a hundred minions bloom!” at the bottom of each email. *Perhaps you can help me out: in tiny print it says the artwork and wording are completely original and without parallel, but haven’t we heard something like before?*
Keep writing, I’ll keep reading.
😎
The MMA you describe has its counterpart in Communist China. The CCP has its own MMA where thousands of CCP members sit in underground bunkers—or in thousands of miles of tunnels deep under the Himalaya Mountains (I’m not kidding. These tunnels exist. They were chiseled out of bedrock by the PLA during the Mao era so the CCP could survive a nuclear war and eventually repopulate the earth with maybe Mao’s frozen sperm)—in front of computer screens flooding Chinese internet forums with comments that support the CCP’s MMA in an attempt to redirect or counter public option that criticizes the CCP’s economic agendas.
The United States has its one percent that seems to have learned form the CCP, and the CCP has 80 million members that represent 5.7 percent of China’s total population. To survive, the CCP must keep the other 94.3 percent of the population fooled and complacent.
I wonder where America’s MMA has its underground bunkers.
They are a competing law firm of Dewey, Cheatem & Howe!
Atlanta Public Schools, the epicenter of Harvard Based & Broad Foundation & Co. experimentation, held fancy rallies for hearts to showcase and financially reward schools/teachers of the high achieving schools. Schools with low performing scores were pressured, threatened, and received specially assigned principals and central office administrators. Years of this practice and over-the-top unachievable goals for students, teachers and schools resulted in the largest exposed Scandel, which Atlanta is still litigating and $M ATL taxpayers are paying for.
I dare Harvard and all other ‘non-institutions’ of Education to study the components and variables of squeezing the toothpaste tube to see what happens.
CorpEdReform is nothing but a witch hunt of traditionally trained teachers who have knowledge, skills and experience in educating children!
CorpEdReformers are determined to run this train completely off the tracks and then put their $M in other fall-out profit making endeavors. Absolutely sickening!
McCarthyism could not have done better.
“quasi-experimental studies”
Not quite good enough to be experimental studies, eh?!
Would that be like a….s, oops, I mean opinions???
“vain and illusory”
Not my words KrazyTA. I’ve quoted Wilson on that one because it is succinct. And no that quote doesn’t come from “Educational Standards and the Problem of Error” but his lesser known (and shorter, 15 or so pages) review/critique of the educational testing bible “Standards for Educational and Psychological testing” put out by the American Educational Research Association; American Psychological Association; National Council on Measurement in Education. (2002):
“A Little Less than Valid: An Essay Review” found at: http://www.edrev.info/essays/v10n5.pdf
Señor Swacker: but you use “vain” and “illusory” so well!
😉
You will never make it on the “A” [or “Z”] list of the education establishment’s email contacts because you are giving credit where credit is due, but I will always associate you with those succinct recaps of mathematical intimidation and obfuscation a la “education reform.”
“Men lie and women lie but numbers don’t.” [“Dr.” Steve Perry, channeling rapper Jay-Z]
Who can put a price on that? Can one go below 1₵?
Duane Swacker: “vain” and “illusory.” [channeling Noel Wilson]
Priceless! As in, no number, however large, could tell us its true value.
Keep writing. I’ll keep reading. My treat next time down at Pink Slip Bar & Grille…
😎
Lucid!
Thanks for helping revive interest in the classics. You prove that they’re useful.
A couple of points.
Cosmic claims that Chetty et all avoid talking about teacher and administrator cheating on exams in order ignore ” the very management philosophy of those funding their “research” and leading the charterite/privatization charge.”
Which funding agency is Cosmic referring to? Is it The National Science Foundation (the most frequent source of grants to Chetty) or the Smith Richardson Foundation (the second most frequent source of funding). Perhaps Cosmic is referring to the Social Security Administration grant Chetty received.
I think I can also answer Cosmic’s question in the following paragraph:
” Since Chetty/Friedman/Rockoff didn’t dispute the 1% to 14% assertion then I would like to point out that I would be awfully interested in knowing why they’re ignoring the other 99% to 86%. Could it be that it’s poses intractable difficulties to their VAManiacal beliefs?”
The answer is no, it poses no difficulty. The fact that teacher quality as measured by increases in their students standardized test scores does not explain all of the measured variation in outcome poses no difficulty for the claim that teacher quality is important for student outcomes, just as the fact that income level does not explain all of the variation in student outcomes posses no difficulty to the researcher that finds poverty to have an impact on student outcomes.
In other words, the models are fundementally flawed and confuse correlation with cause and effect. Chetty and the Reformers eagerly jump to the conclusion test score variations are caused predominantly by teaching practices. By dismissing the confounding effects in his models and the potential flaws in the measurement and generation of data on student learning, Chetty misapplies sound science to advance political ends. There so many contradictory examples to the entire VAM concept that it is a wonder we still even talk about it with a straight face. Chetty and his cohorts just need to venture out of the Ivory Tower to observe actual teaching. And I mean really observe.
MathVale,
You are incorrect when you say Chetty et al ” the confounding effects in his models and the potential flaws in the measurement and generation of data on student learning, Chetty misapplies sound science to advance political ends.”
Have you actually read the papers?
VAM works in theory but not in the real world.
Dr. Ravitch,
Chetty et al use real world data.
At one point I believe that you posted about a statistician at Arizona State was going to replicate Chetty et al’s results using their data set. Is there any news about how this project is progressing?
“The fact that teacher quality as measured by increases in their students standardized test scores. . . ”
Considering that those scores are COMPLETELY ILLOGICAL, INVALID and to use a score from a test that is not designed to assess the teacher UNETHICAL, all such supposed measurements/results are “vain and illusory”.
The educational malpractice of “raising test scores” as a proper and good goal in the teaching and learning process is LUDICROUS, RISIBLE, INANE, and COMPLETELY INSANE.
teachingeconomist (TE) says “The fact that teacher quality as measured by increases in their students standardized test scores does not explain all of the measured variation in outcome poses no difficulty for the claim that teacher quality is important for student outcomes”
Here’s what ASA said in their position paper
“Research on VAMs has been fairly consistent that aspects of educational effectiveness that are measurable and within teacher control represent a small part of the total variation in student test scores or growth; most estimates in the literature attribute between 1% and 14% of the total variability to teachers. This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.”
Technically speaking, I guess “1% to 14%” does mean “not all”, but somehow, I don’t think the way TE put it conveys “quite” the meaning that ASA intended.
And I don’t know what “outcomes” TE is referring to, but the ASA were referring specifically to “student test scores.” Nothing more.
In fact, the ASA state that “it needs to be recognized that, at best, most VAMs predict only performance on the test and not necessarily long-range learning outcomes.”
They also state that
“VAMs are generally based on standardized test scores and do not directly measure potential teacher contributions toward other student outcomes.”
Finally, when the ASA say that “Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores” (a statement that was not challenged by Chetty) they are making a statement about “correlation” between teacher VAM scores and student test scores, not about “causation.”
In ASA’s own words
“VAMs typically measure correlation, not causation: Effects – positive or negative – attributed to a teacher may actually be caused by other factors that are not captured in the model.”
Specifically, they are ***not*** stating that “Most VAM studies find that teachers cause (ie, are directly responsible for) about 1% to 14% of the variability in test scores” — to say nothing of stating that “Most VAM studies find that teachers cause about 1% to 14% of the variability in student “outcomes” other than test scores (future income, for example)”
“Correlation” is an area of statistics that is often misunderstood and often misrepresented (sometimes quite knowingly) in order to “prove” a causal relationship that in fact does not exist.
That “teachers account for about 1% to 14% of the variability in test scores” actually equates to a “correlation” between teacher VAM scores and student test scores (but not necessarily causal relationship) that is “moderate” (at best) to quite possibly “negligible” (effectively nonexistent)
To make matters worse, VAM scores are “noisy” (“The VAM scores themselves have large standard errors”,as ASA points it) which means a teacher’s ranking can change by a relatively large amount from one scoring to the next simply due to the “noise” (factors outside the teacher’s control).
Also, according to ASA “Under some conditions, VAM scores and rankings can change substantially when a different model or test is used”. In other words, one VAM score calculated with one model might lead to a teacher being deemed “poor”, another might imply they’re “acceptable”.
Reminiscent of Schroedinger’s cat Perhaps the poor/acceptable teacher should be kept on at half pay? Conflicting answers are usually an indication that there is something wrong with your model(s).
The upshot is that VAMs are unreliable, particularly for firing decisions..
But those who are peddling the use of VAMs for teacher firing talk as if there were actually a strong causal relationship between a VAM score and future student income (or some other student “outcome” related to future student “success”), when the reality is that any relationship that does exist is week, at best — and there may be little or no causal relationship at all.
Despite their “high standards” for teacher quality, some economists seem to have relatively low standards for “proof” when it comes to teacher ranking and even firing. ( but high (Olympic level) standards for mental gymnastics — eg, to twist ASA’s “about 1% to 14%” into “not all”.)
The overriding importance of “other factors” (“outside the teacher’s control”) is actually the primary point that ASA were making in their position paper (and that Chetty and some others have rather conveniently chosen to ignore):”the majority of opportunities for quality improvement are found in the system-level conditions.” [ ie, as ASA says “factors outside the teacher’s control…such as curriculum…”]
SDP,
The student outcomes I was referring to are the ones used in the Chetty et al paper. Have you read it? The measures of student outcomes used in the paper were not higher test scores for students, but measures from adulthood. This is from the abstract:
“Students assigned to high-VA teachers are more likely to attend college, attend higher- ranked colleges, earn higher salaries, live in higher SES neighborhoods, and save more for retirement. They are also less likely to have children as teenagers.”
‘ “Students assigned to high-VA teachers are more likely to attend college, attend higher- ranked colleges, earn higher salaries, live in higher SES neighborhoods, and save more for retirement. They are also less likely to have children as teenagers.” ‘
So let’s look at that statement from the other side. We know that VAM scores appear to be sensitive to SES factors: if a child lives in a high SES community, their test scores tend to be higher. They also are more likely to go to college and attend higher ranked colleges. People who attend higher ranked colleges tend to earn higher salaries. If you earn more money, you can save more for retirement. Are students who live in high SES communities less sexually active? Stringing all of these correlations together makes it sound like all one needs is a string of high VAM teachers in order to succeed in life.
2old,
You are correct that there is endogineity all over the place, which is why Chetty et al worked so hard to minimize their impact. You might want to read the papers to see how these issues were addressed.
It seems that TE may have been selective with a pile of cherry pits to suggest that all it takes is a teacher who waves a magic wand so their students score higher on standardized tests and off the students shall go from poverty to success after they attend Harvard, Stanford, Princeton, etc, and scads of success and money will follow them through life because they were good at filling in the right bubble.
What garbage!
This is the kind of crap handed out by Koch, Walton, Gates, Duncan and Obama, etc. designed to blame everything on teachers—which the critics of public education have been doing for decades—while providing excuses for dysfunctional parents and students who are not engaged in reading or education.
The evidence is overwhelming that high standardized test scores have nothing to do with success as an adult because of a few (so-called) superman teachers. Wasn’t that what the film funded by one of the robber barons of public education promised?
“Waiting for Superman”
To believe this insanity, you have to be either stupid, living in a make believe world created by Hollywood film or a corporate robber baron who is also a fake education reformer.
A study reported by the Economic Policy Institute says, “Low income hinders college attendance for even the highest achieving students”
“In 1988, the Department of Education began a longitudinal study of students then in the 8th grade, following them over the next 12 years as they progressed past high school and college and into the labor market. The study finds large, glaring disparities in high school and college completion rates according to their family’s socioeconomic status1 (SES) when grouped according to their 8th grade math test scores.”
Click on the link, scroll down and look at the chart for Figure A. If all it took was high test scores, then the columns for low income, middle income and high income would match.
http://www.epi.org/publication/webfeatures_snapshots_20051012/
Then there is also this:
Long-term Effects of Parents’ Education on Children’s Educational and Occupational Success: Mediation by Family Interactions, Child Aggression, and Teenage Aspirations
“In fact, research suggests that parental education is indeed an important and significant unique predictor of child achievement.”
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2853053/
Although children of high- and low-income families are born with similar abilities, high-income parents are increasingly investing more in their children. As a result, the gap between high- and low-income students in K–12 test scores, college attendance and completion, and graduation rates is growing.
http://www.hamiltonproject.org/papers/thirteen_economic_facts_social_mobility_education/
All of these studies—not the lies drenched in the fairytale world of Hollywood and Marvel Comic books from the robber barons behind the corporate education reform movement—indicate that children from poor families must have an early childhood education program available as young as even age two, which France has had for decades—a program that reduced poverty in France from 20 percent in the 1960s to less than 7 percent today.
Statistically speaking it is unlikely that having a good teacher in grades 4-8 has no impact on students in later life. Perhaps it is mere, correlation, but if so why spend the time and resources trying to have high quality teachers in the classroom?
So, you believe I am a mind reader and should have simply known specifically what you were referring when you said “student outcomes”?
Well I’m not, unfortunately. 🙂
And the very reason I asked you to specify what you meant by “outcomes” was that ASA actually cautioned that “it needs to be recognized that, at best, most VAMs predict only performance on the test and not necessarily long-range learning outcomes” and that “VAMs are generally based on standardized test scores and do not directly measure potential teacher contributions toward other student outcomes.”
But I already noted that above and you obviously chose to ignore it.
And speaking of “statistically speaking”, here is is what ASA actually said (and they are statisticians, after all)
“This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.”
If you (and Raj Chetty,for that matter) choose to ignore what the professional statisticians say about VAMs, there is not much that I (or anyone else) can do about that.
Some Dam Poet,
I certainly don’t think you can read my mind. I did have some hope that you had actually read the paper you were criticizing, but I now know that was foolishly optimistic.
Again, Chetty et al shows that teachers have an impact on student lifetime outcomes. Parental income, parental involvement, childhood nutrition, among other things probably have a larger impact, but the local school district can do nothing to change any of those things. What it can do is try to have the best possible faculty in the schools that it controls.
Like I have been saying again and again. VAM is not researched-based but politically based. It does not matter how much peer reviewed research shows the flaws in their statistical model, they will use the power of the media they control to try to force VAM on every public school teacher for just one simple purpose–to create a justification to privatize education in this country and set up a dual school system which will justify no longer educating the poor, the disabled, and English Language Learners.
Liberal,
Chetty et all have provided their entire data set, the SAS program that they used to analyze the data, and of course several versions of the paper(s) they wrote based on their analysis of the data. You are welcome to find any flaws in their work.
There’s an interesting article by Elizabeth Green in the Sunday Times on the “new math.” It’s true that teachers need to be better prepared. Her internet newsletter, Chalk Beat, is interesting, but has uncritical support of Success Academy and other charters. Evidently there is some movement toward including the community and making sure there is diversity in structuring the charters. The usual suspects like the Gates Foundation supports the newsletter. I’d be interested in your response.
Kathleen
Kathleen Migliore-Newton 362 Adelphi Street Brooklyn, NY 11238 Tel: (mobile) 1-917-804-2145 Paintings: http://urbanpaintings.com
Kathleen and Doug,
I like Elizabeth Greene but we disagree on lots of things. I dont think our schools are a disaster, except for the attacks on them, the constant claims of failure, the tracher-bashing, and the turmoil caused by NCLB and Race to the Top. Unfortunately, to keep a publication like Chalkbeat alive, one must go to the super-rich to raise money. When Chalkbeat was Gotham Schools, its main benefactor was a charter supporter.
How VAM will be used in practice in actual states and districts, and outside of think tanks and DC:
“The board also wants to change teacher pay. Gone would be automatic salary increases based on longevity and academic credentials, like a masters degree. Instead, the best teachers would get merit pay and performance bonuses.
“And it does this by creating some flexibility in how we pay teachers that’s partly based on performance and value that they contribute to our students,” says Moore.
The proposal includes a bonus of up to $30,000 for a top rated teacher. Teachers rated effective would be eligible for smaller bonuses based on their school’s performance on the state report card.”
They’re using it as leverage in contract negotiations, as a justification for a “bonus” system and paying teachers more based on the last ridiculous fad, which was “school report cards”.
Was this the intent or is this an unintended consequence?
The truth is, they have no earthly idea who this measure will be used. None. They also have absolutely no control over how it used in states and districts. It could raise average compensation or it could lower compensation. It could reduce turnover or increase turnover. It could put better teachers in poorer districts or chase them away.
Yet, we’ve decided to set national public education policy on it. That’s crazy.
The Obama Administration and a lot of very influential and powerful went “all in” on VAM.
Don’t expect any concessions to reality. If anything, they’ll double down.
There should be a formula for that. There should be a formula that describes what it takes to back off a bad policy when a huge group of famous and influential people have completely bought into it. You can bet Arne Duncan isn’t going to end up with egg on his face. Take that to the bank.
I don’t know why they always do this in DC. I don’t understand why it’s necessary to oversell these theories. If they’re taken with this theory (and they obviously are) just use some restraint and common sense and don’t portray it as rock-solid until they see if it has the intended result. It’s like they can’t help themselves. They have to trumpet it as “Truth!” when they have absolutely no idea how it will work in practice.
Propaganda or “research” in education
is consistently absurd, “ignoring the 99%-86%”.
The revenue generated by a typical teacher is $8,000 (amount the charters receive per student) x 30 students = $240,000. Left on the table, after paying a teacher $50,000, plus $5,000 in benefits, is $185,000. Business logic or rational economic man dictates focusing, first, on the area of greatest opportunity for savings.
Oh wait, the charter operators don’t want to focus on the area with greatest opportunity because it’s the pot they draw from, for their salaries, subsidized leases and mad money for advertising and campaign donations.
Linda,
What percentage of school performance is explained by household income and what portion is unexplained by household income? Does the fact that household income fails to explain all of the variation of academic performance mean that it should be ignored?
More disgrace for the economists of the 1% and Fox—“ALEC Scholar”, Stephen Moore. The Middle Class Political Economist, July, 26, 2014, “Heritage Foundation’s Chief economist) Can’t Even Get his Cherry-Picked Data Right.” A commenter added about Moore, “He looks smart with his glasses and sounds smart when he speaks. To the average Fox viewer, he has the attributes of an expert.” Of course, his degree is from george mason.
Linda,
Any chance that you might address the issue at hand?
From what I hear states have gone way overboard on the weight they have attached to VAM. Even if we ignore the fact that VAM is unreliable, How does assigning as much as 50% of a teacher’s evaluation to VAM make sense when only 1-14% of the variation in scores can be attributed to teacher effects?
The reason that a state or local school board might want to concentrate on teacher quality is that it is something they might have some influence over. No district or state could have any impact on the ability of my foster son to learn.
Do you suggest that a school district concentrate on things that they can not influence?
teachingeconomist says “The reason that a state or local school board might want to concentrate on teacher quality is that it is something they might have some influence over.Do you suggest that a school district concentrate on things that they can not influence?”
TE:
Ever heard of “curriculum”? (something that the state and district certainly have an influence on)
At a bare minimum, your failure to mention “curriculum” means that you did not read the ASA position paper on VAMs at all carefully because they specifically mention “curriculum” :
“The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.”
But actually, one need not read the ASA paper to appreciate that curriculum just might have some impact on student test scores.
Som Dam,
Indeed the board could work on a curriculum, but I don’t think there is one curriculum that is best for all students. Some students might do best in a Waldorf school, others do best in a school that concentrates on STEM disciplines. I doubt that the best way to match student to curriculum is by street address, so I prefer alternatives to traditional catchment schools.
What the school board cannot do is change what every one here acknowledges as the most important factors in education: the family circumstances that the student lives in and the history of those circumstances.
TE,
Again, you showcase your endless ignorance!
School boards don’t work on curriculum. Mandated curriculum comes from the state, the federal government and the courts through education codes and laws. Teachers only implement the curriculum and teaching methods are meaningless if the parents are not supportive and the students are not engaged.
In fact, there are a host of challenges for teachers to overcome. For instance, diet, lack of sleep, health problems, poverty, violence, etc. And these challenges are an epidemic that are not being dealt with as fools keep trashing about suggesting that we can fix them with bandages that have no ability to stick to the flesh and held heal the wound.
Children who are engaged and who have supportive parents learn regardless of the methodology used.
Take the same child from supportive, literate, educated parents who do not live in popover and the child would learn in any classroom no matter the methods the teacher used.
Do you have any valid, peer reviewed evidence from several larger and longer studies that reveals one teaching method works better than another when all socioeconomic variables for students are equal?
Come on, answer the question.
Lloyd,
Perhaps your insults are better aimed at Some DAMPoet.
It seems the truth hurts.
I suggest that instead of getting angry or pretending to be angry to avoid the question, answer it.
Lloyd,
Once again, why not hurl you insults at SomeDAM Poet who stated the following:
“TE, ever heard of “curriculum”? (something that both states and local school districts have control over)”
I was responding to that statement. Is SomeDAM Poet woefully ignorant?
TE,
The pattern that I see emerging from TE is that when someone calls TE’s comments or TE what they think the comments or TE is, TE responds as the aggrieved victim of an ad hominem attack and seldom if every answers any questions that TE is asked. Instead, TE usually asks another one of TE’s questions to avoid a topic TE can’t discuss for lack of evidence to support any personal agenda TE may be promoting.
ie, school choice
What do you think about school choice, TE?
What one person thinks is an ad hominem attack may be a fact to another person and is not an ad hominem attack.
Was this quote from Abraham Lincoln an ad hominem attack on all the fools he was referring to or a truism? “You can fool all the people some of the time, and some of the people all the time, but you cannot fool all the people all the time.”
When Lincoln said, “Some of the people all of the time” was that an ad hominem attack because he was saying that some people are always fools. I wonder what he meant by that.
Lloyd,
Any thoughts on SomeDAM Poets assertion that local school districts DO have some control of curriculum?
I’m not interested in your “SomeDAM Poets assertion that local school districts DO have some control of curriculum”.
But there is a more important issue that should be discusses. Have you ever asked any questions about a national early childhood education program and why there isn’t one?
In addition, what do you think about the importance of establishing a national early childhood education program that might start between the ages of 2 to 4?
Do you think this program should be transparent?
Do you think the public schools should run this program in partnership with public libraries or should the taxpayer give their money to opaque corporations who spend hundreds of millions of dollars to influence legislation that allows them to be opaque in just about everything they do—opening doors for fraud and abuse?
Do you think that the adults who work in this future early childhood education program should have training and go through an FBI check to make sure they were not criminals?
How much training should they have?
Should there be a state or federal watchdog agency keeping an eye on every early childhood education provider, private or public, to make sure they are doing the job that’s needed?
I think before we can discuss any other issue that focuses on the education of children in the United States, the first priority should be to a national early childhood education programs designed to foster a love of reading that would lead to the most important skill in education, a high level of literacy.
Why do you think the focus of the Obama administration on education started with the Machiavellian Common Core standards that implemented one of the worse management techniques used in the private sector instead of a national early childhood education program that Obama plans to ask Congress to support in 2015?
You may want to read this from Forbes first to educate yourself on this issue.
Pull quote: “Vanity Fair has an article in its August issue that tells the story of how Microsoft “since 2000 . . . has fallen flat in every area it entered: e-books, music, search, social networking, etc., etc.” According to a summary available online, the article finds a devastatingly destructive management technique at the heart of Microsoft’s problems.”
http://www.forbes.com/sites/frederickallen/2012/07/03/the-terrible-management-technique-that-cost-microsoft-its-creativity/
What I’m talking about is the “Rank and Yank” system that Bill Gates seems to love and can’t let go of, and it is arguable that President Obama is his partner.
Recently, Microsoft announced that are going to fire 18,000 employees this year but at the same time they joined a consortium of high tech companies asking Congress to more than double the number of H1 Visas to fill the vacancies after they fire American citizens with the same skills they are seeking overseas.
You might ask me what relevance that has to the education issue so here’s my answer:
The same “rank and yank” management system Microsoft is using is part of the Bill Gates funded Common Core that he bribed officials in almost every state to sign on the dotted line and accept before the program was even written.
Lloyd,
SomeDAM Poet was the poster I was responding to, not my assertion at all.
TE,
Thank you for clarifying. Then I’m not interesting in SomeDAM poet’s assertion.
How will this improve education?
What is more important as a topic that may lead to improving public education?
A. a national early childhood education program
B. the respect that teachers deserve
C. proper training for teachers—for instance, a national standardized year-long, paid residency program with master teachers and additional follow up support during the first year in the classroom after the residency.
D. An end to “rank and yank” policies through the Machiavellian Common Core standardized testing regime that would make a fascist dictator proud.
E. all of the above
Do you have any other suggestions to add to that list and what would they be?
Lloyd,
You seemed very interested in SomeDAM Poets assertion when I repeated it. Are you only interested in my posts? I would take that as a complement.
TE,
The use of the word Interested would be a stretch—a misnomer.
I’ve been reading about poverty and pollution and their impact on a child’s education. What do you think can be done to alleviate pollution’s impact on children living in poverty?
Lloyd,
The reason that I thought you were unusually interested in my post is that you expressed ABSOLUTELY NO interest in another poster who made the claim that districts have some control over the curriculum.
What is important in your mind, the claim or who makes the claim? The evidence is that it is who makes the claim, not the actual content of the claim.
TE,
When I open an e-mail that is a comment thread for one of Diane’s posts, I usually don’t have the time to read everything so I read the first comment and then search for my name to discover if someone replied to one of my comments. I don’t bother to read a string of comments. In fact, because I use search, I don’t even see most of them.
Sometimes I run into a comment that did not mention me and if I’m interested, I might leave a comment too.
I don’t search for your name or anyone else. Too many people to keep track of.
There is no way I could spend the hours it would take to read everything on Diane’s blog. I’d never get anything else done.
Lloyd,
I am well aware of the problems with using an email response. What I found interesting is that EVEN AFTER LEARNING THAT IT WAS ANOTHER POSTER THAT STATED THAT SCHOOL DISTRICTS DECIDED IN PART SCHOOL CURRICULUM, YOU EXPRESSED ABSOLUTELY NO INTEREST IN THAT POST, YET FOUND MY AGREEMENT WITH THE POST GROUNDS TO CALL ME IGNORANT. One might naturally think that the person who actually first stated the position would be the one that you might attack if in fact what you were interested in was attacking the position. If the position has little to do with your disagreement, well your posting do actually make sense.
When I first started teaching full time in 1978-79, there was no organized curriculum and not enough textbooks for the new teacher in the English department to have a set—that was me. With help and advice from veteran teachers in the English Department, I created all of the material I used with my students and spent a lot of time using a ditto machine to run off worksheets.
My primary goal was to improve literacy and the ability to write coherently.
Then as the years went by, especially after Reagan—it seems that a lot of what plagues the US today started with that fool—control from above dropped on the schools like a nuclear bomb and today it is still causing havoc.
Back before state and federal mandated curriculum and endless standardized tests, I had a lot more success with my students even without a textbook.
With a lack of material to read, I took my students to the library and had them check out books to read. We also did a lot of writing: poetry, short stories, essays. Then revised and shared through student oral presentations.
With all the engagement, I could actually measure student growth from the quizzes I created for my own use as the teacher to monitor if kids were learning what I was teaching.
“Do you suggest that a school district concentrate on things that they can not influence?”
Nice try, TE. Suggesting that reliance on VAM as a measure of teacher quality is flawed is not the same as saying school districts should not be concerned with the quality of their teaching staff.
2old,
School districts can not make parents care more, can not have their students live in higher income households, can not reverse the learning disabilities of their students. What they can do is hire and employ high quality teachers.
VAM is, as Chetty said, a crude measure of teacher quality and we might find better measures.
“VAM is, as Chetty said, a crude measure of teacher quality and we might find better measures.’
At the very least he owes teachers a public apology for allowing the misuse of his research.
2old,
Crude measures are used all the time. I tell my students (and my children) that a grade is at best a rough measure of understanding and effort in a class.
One would hope that the crude measure is at least one that the students received for work they did, not the one time test score of unknown peers.
VAM’s are not simply “crude.” In some cases, they are simply crap..
Here’s a sample of how inconsstent and unreliable they can be: The points on a plot of “Different grades, same year, same subject, (VAM score given to same teacher for) one grade vs other grade”
Here’s the link to article it comes from.The author(Gary Rubinstein) did a whole series on VAMs which is well worth reading.
In some cases, the same teacher who got a very high VAM score for one grade level received a very low VAM score for the other grade level they were simultaneously teaching, meaning that a a single value added model (which purports to measure “teacher value added”) can indicate that a teacher is both “bottom of the barrel” and “top of the barrel” at the very same time! (Like “Schroedinger’s Cat”, simultaneously dead and alive!)
That’s not just “crude”. That’s nonsense. It means that there is something seriously amiss with the model (to put it mildly). Models that give such inconsistent results are very unreliable (at best) and maybe even complete garbage. Certainly nowhere nearly reliable enough to base teacher firing, tenure and other important decisions on.
If the linked-to plot looks “random”, that is because it is, or close to it. The inconsistency of the VAM scores for the same teacher for one grade vs another is not just a bug (restricted to a few teachers), it’s actually a feature. There is little or no relationship between a teacher’s VAM score for one grade taught and that for the other. The correlation is very week (0.24)
Needless to say, if VAM is truly measuring “teacher value added” one would expect a single teacher to get roughly the same VAM score for the two different grades they were teaching (in the plot shown, for 6th and 7th grade math), particularly in the very same year.
One would expect the correlation between the two VAM scores to be “strong” if it is actually related to teacher value. In other words, what one would expect to see on the plot is basically an upward sloping line, with some scatter of points about that line. Certainly not points scattered all over the plot in random fashion like we actually see.
Unfortunately, when it comes to VAMs, these sorts of inconsistencies — indicated by low correlations (eg, between VAM scores and student test scores, or between VAM scores for a given teacher from one year to the next, or between VAM scores for the same teacher based on different value added models) — are not an anomaly, but actually the norm. (And as ASA notes, correlation does not even imply causation, a fact that is either completely lost on or willfully disregarded by those who are pushing VAMs)
In most (real) sciences, a correlation of 0.24 is considered “week” and would simply not be taken seriously, certainly not for making important policy decisions. In physics (the field I was educated in), this kind of stuff is actually considered a joke and anyone who claimed the existence of a significant relationship between two variables based on such low correlations would be laughed right out of the room.
But to some economists, such low correlations are regularly considered evidence for the validity of VAMs, sufficiently strong for making firing, salary raises and other high stakes decisions.
Chetty is certainly smart enough to understand that his support for the use of VAMs for teacher firing and other high-stakes purposes is unwarranted and very ill-advised (and he has had the problems with VAMs quite clearly laid out before him by American Statistical Association and others), but he persists nonetheless.
TE, ever heard of “curriculum”? (something that both states and local school districts have control over)
Had you actually read the ASA position paper on VAMs or the quotes I included above, I wouldn’t need to point this out again, but ASA specifically mentioned “curriculum”:
“The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.” (ASA)
I bolded curriculum this time, but perhaps ASA should have done so as well so you could see it. But of course, that only makes a difference if you actually read it.:)
Additional factors that are out of the teacher’s direct control but might nonetheless account for variation in student test scores and possibly other (longer term) outcomes as well: student attendance, school facilities, major classroom purchases that are outside the teacher’s usual budget (computers, major lab equipment etc), class size, class makeup, after school tutoring and enrichment programs, homework and time management skills classes and many other things as well (the list is actually very long). All of the latter can potentially be influenced by states and local school districts, notwithstanding what you may believe.
Rhetoric about teacher quality is a problem in itself. Context matters. A great kindergarten teacher working with 15 children is not likely to be as great with a class of 30 or 40. Add in the never mentioned resource factors, not discussed by the armchair pundits who presume that an algorithm will settle the matter of quality.