A reader explains why policymakers and the public are hoodwinked by numbers. Follow the links:
Diane-
I got to this article in the WSJ through the Naked Capitalism blog I read daily:
http://online.wsj.com/article/SB10001424127887323374504578219873933502726.html?mod=WSJ_hp_editorsPicks_4
The Naked Capitalism blog also included a link to this article from 2006 on the same topic:
http://www.auroraadvisors.com/articles/Webber-Metrics.pdf
I forward these because I think they explain why school boards and politicians are hoodwinked by VAM. I know enough about statistics to know that VAM is junk science… but most people (and especially businessmen) want to believe everything can be reduced to a mathematical model and so we find ourselves in VAM-land.
Diane Ravitch's blog 
Metrics Don’t Hoodwink People —
Hoodwinkers Hoodwink People ❢
LikeLike
Both articles are excellent. And I would highly encourage your readers to check out Naked Capitalism too.
One point that I always find key to understanding the power of statements that refer to mathematics is that while a correct mathematical statement will always be logically correct, we have to remember that the statement is meaningless if the definitions and assumptions regarding are unreasonable. In other words, mathematics doesn’t care what the symbols mean; it only cares about the logical consistency of the operations on the symbols. The meaning of the mathematics has to come from human beings, who interpret the symbols and their relationships; that means looking into the mathematical statements to understand how they describe the world.
When we get seduced by the mathematical statements themselves, we are falling for an appeal to power.
LikeLike
I disagree. I think businessmen, politicians and their bureaucrats understand perfectly well the foibles of statistics and mathematical models. In applying these kinds of simulations to real life the odds are clear that the “x factor” will come into play sooner or later–or maybe the whole game really is a crap shoot from square one.
These people are essentially gamblers–addicted to game of “stacking that cream” as my students are fond of saying. Only the game(s) they play are better than street games because they have it rigged where they keep the gains and let the taxpayer foot the losses.
Astonishing what sheep we all are.
LikeLike
Before supporting VAM or any other testing SPAM everyone should read the classic text, “How to Lie with Statistics” written by Darrell Huff in 1954.
Two quotes:
“There is terror in numbers.”
“The secret language of statistics, so appealing in a fact-minded culture, is employed to sensationalize, inflate, confuse, and oversimplify.”
LikeLike
David Greene: thank you for the Darrell Huff reference. Reminds me of the apt description of the intended result of “figures don’t lie but liars figure”: “mathematical intimidation.”
Edubullies are especially prone to substitute intimidation by numbers for actual facts, logic and statistical results based on hard data. Remember the Rheemeister herself who claimed that she took her students from the thirteenth percentile to the ninetieth? Strangely[?], turns out she has no hard data to prove it. How about school districts and charter networks that claim fantabulous results in raising test scores [‘no fudging by cheating here!’] and graduation rates [‘one third of the original cohort disappeared!’] while teaching just the same kinds of kids [‘percentages don’t lie!’] who go to public schools? And the icing on the cake? The charterites/privatizers can do it with so much cost effectiveness that they put those ‘government-monopoly’ public schools to shame! You know, like LEAP Academy in Camden and the American Indian Charter Schools in Oakland…
🙂
What is especially telling is how so few of the edubullies have even a minimal understanding of the numbers they use, so when a G F Brandenburg or a Bruce Baker or a Gary Rubinstein [to mention only a few] actually digs into their glowing self-descriptions and finds serious methodological [not to mention other, like GIGO] problems, the automatic response is “well, we have our studies and you have yours!”
I much prefer “Dueling Banjos” [with real musicians] to “Dueling Studies” [with real mathematicians versus think tank hacks producing whatever numbers support predetermined conclusions]. I’m very old fashioned that way.
🙂
LikeLike
The commercialization of public education means that ever-bigger slices of the public education dollar will be spent on advertising, hype, and PR, all of which grandiose advertising claims and agenda-driven studies require simple-minded measures with plausible deniability — auspicious sounding numbers that can be shouted from the media rooftops without anyone being able to get a media-word in edgewise that none of the numbers really make any sense.
LikeLike
Don’t understand why people, especially business people, continue to be so enamored with statistical/mathematical models to reliably measure success and ratings when the math geniuses on wall street with their “reliable” math models almost destroyed the world economy.
LikeLike
When my child’s school showed an Anti-Bullying video from the Anti-Defamation League that included excerpts of children using the names they had been called, including whore, lesbian, dyke, queer, faggot, I felt the need to find out why. According to the principal, research says it is important to “name the names” of those who get bullied. One source of data led me to learn that LGBT students get bullied more. However, the survey was put out by a group who stood to benefit from the data it reported.
It was put out by a gay-rights organization. My issue – 8000 children were selected to participate because they were “questioning” in middle and high school. As a result, every single public school in the United States was advised – through the Federal “Safe School Climate” initiatives that this particular group that gets bullied more, and we should “name the names” so that kids know. Yet, our school children had not even learned about heterosexual activity let alone the intricacies of various identities.
The statistics behind the scenes can be presented in any way to promote whatever policy those with a vested interest want to promote. Imagine what polices could result if the NRA only surveyed gun owners, and from that data, changed federal mandates to impact every child in a school!
LikeLike
I have to admit that the anti-bullying push seems to have made some kids use it as a weapon against fellow students. It’s almost as though it’s created more accusations than actual incidents.
LikeLike
I don’t know if I’d call it “junk science”, but probably imprecise science would be more accurate. The problem is the way it’s being instituted, which is VAM, bam, thankyou ma’am.
LikeLike
Alfie kohn has a good piece in september of education week titled schooling beyond measure. http://www.alfiekohn.org/teaching/edweek/sbm.htm
LikeLike
More people with backgrounds in philosophy rather than psychology could possibly help this epidemic. I think.
LikeLike
“The Signal and the Noise” by Nate Silver nails it. In an era of Big Data, belief molds and precedes understanding; not the other way around. Hopefully, one day Silver with aim his intellect at the mathematical dishonesty of the reformers.
LikeLike
“The commercialization of public education means that ever-bigger slices of the public education dollar will be spent on advertising, hype, and PR,” Jon Awbrey said above.
Exactly — it is the link between the potential for profit and the use of numbers that
induces terror.
I don’t believe, as Diane has said, that Value Added Measures are junk science — it has applications where it is quite useful, such as in the manufacture of ball bearings.
But kids are not ball bearings and the application of VAMs to teaching is wrong-headed
and usually wrong spirited.
Teaching has long-term outcomes that escape measurement. As Jerry Brown, who is now Governor of California again almost 40 years later, said, “In the right order of things, education—the early fashioning of character and the formation of conscience—comes before legislation.” Indeed, it is what makes up capable of being legislators,
The misuse, misapplication and misrepresentation of metrics is the bread and butter of those who would dismantle public education. The best example may be the ‘4 years with a top-quartile teacher will close the achievement gap’ between black and white students.
The study that is usually used to justify this, Gordon, Kane and Stiager (Brookings, 2006) is one study, quite limited in scope and in time. The Kane involved is Thomas Kane, the economist who heads up Bill Gate’s MET Project, but you see the quote everywhere — on the New Teacher Project website, from Rupert Murdoch-owned enterprises, in powerpoint presentations for districts across the country, in
Michelle Rhee’s statements — anywhere where people are working to
dismantle public education and create a void to make room for privateers and profiteers.
This includes, of course, our friends in the US DOE, who want to ‘transform teaching’ by using VAMs so that teachers will be paid like piece workers. It is one of my great regrets that Arne Duncan was not a better basketball player so that his Tony Robbins type pronouncements would be limited to the sporting world.
I spent a lot of time looking into this when i first heard the phrase ‘Respect for Teachers’ in President Obama’s 2011 State of the Union address. And, while this may be contradictory in a post that says ‘don’t believe the hype,’ I’m going to plug my book where I go into this in detail. RESPECT FOR TEACHERS . . . or . . . The Rhetoric Gap and How Research on Schools is Laying the Ground for New Business Models in Education (https://rowman.com/ISBN/9781475802078)
LikeLike
This is a lengthy example of the misuse and misrepresentation of research that is adapted from my book, Respect for Teachers. I hope that is okay.
Claims made by the researchers often end up being portrayed in such a way as to make the quality of teachers appear far worse than it actually is. Take Eric Hanushek’s argument that we need to get rid of the bottom rung of teachers. If you read press accounts, you are left to believe that he is advocating a Jack Welch type policy, replacing the bottom 5-10 percent of all workers every year . However, as Prof. Hanushek has stated explicitly, it is a one shot followed by replacing the bottom 5 to 10 percent of new teachers, something along the line of 0.35 to 0.7 percent of all teachers, in subsequent years. There are still questions and problems –
Can we identify the teachers who will not improve? Are VAMs both accurate and reliable?
Is a new special effort necessary if one-third of new teachers do not make it through three years in the first place? – but it is much different proposal than is usually portrayed.
So that is a misrepresentation. Another, closely related, is theargument on how a string of superior teachers can close the achievement gap in just a few years, for the most part it came from a 2006 Brookings paper by Gordon, Kane and Staiger. Though just one piece of research which might or might not be borne out over time, it has had multiple lives; it was extrapolated from and the extrapolations were being treated as scientific description of reality as it was cited in speeches, op ed pieces, foundation publications, powerpoint presentations, congressional testimony and all the rest.
Again and again, however, the evidence that does not fit with the prescription is discarded. As one reviewer commented, Mr. Brill’s book, which devotes a chapter to Kane and Staiger,
is filled with misleading discussions of complex education research, most notably a total elision of the fact that ‘nonschool’ factors—family income, nutrition, health, English-language proficiency and the like—affect children’s academic performance, no matter how great their teachers are. . . . The research consensus has been clear and unchanging for more than a decade: at most, teaching accounts for about 15 percent of student achievement outcomes, while socioeconomic factors account for about 60 percent.
(Dana Goldstein, later adds, “Most pernicious is Brill’s repeated claim that the effects of poverty can be not only mitigated but completely beaten back by good teachers.” Dana Goldstein, “Can Teachers Alone Overcome Poverty? Steven Brill Thinks So,” The Nation, August 29-September 5, 2011.)
But Mr. Brill’s polemic kept saying that teaching counted more than anything else, that teacher effectiveness could overcome the disadvantages of poverty and associated factors.
By attempting to create opinions that would be accepted and become commonplace in argument, people such as Steve Brill and Bill Gates (who also refers to Hanushek and whose foundation constantly quotes the 2006 Brookings paper) have used the finding to prepare the ground for restructuring of teacher pay scales. Mr. Gates suggested that if we had only top quartile teachers – obviously a mathematical impossibility, but we can assume he meant if we had a system in the future where all teachers were equivalent in quality to the top-quartile today– that our system would finally work and usher in an age of social equity and economic prosperity. The achievement gap would be closed and we’d climb to the top of international rankings.
Michelle Rhee came out with much the same argument, was eventually echoed by Oprah, and used it to push for pay-for-performance systems. So did Wendy Kopp’s Teach for America, along with the New Teacher Project (NTP), the group Ms. Rhee (a TFA alum) founded after she graduated from the Kennedy School in the late 1990s. The NTP quoted quite a few researchers, but they seemed particularly enamored of the Gordon, Kane and Staiger analysis of Los Angeles public school data that concluded
having a top-quartile teacher rather than a bottom-quartile teacher four years in a row may be enough to close the black-white test score gap
This is based on an extrapolation from one study that showed a one year gain and then multiplied it by four, an iffy proposition. It is like basing a baseball contract on someone’s career year. What if, “20-25% of teachers in the bottom groups one year are in the top groups the next – and vice versa,” how would that affect the analysis?1
Moreover, while the quote from the NTP is accurate, it is incomplete; in the first place, it was prefaced by the phrase, “ if the effects were to accumulate” – that is, if the one year gain were multiplied by four. There are problems enough with this – a student may gain more one year than another, there may be diminishing returns, etc. – but in addition, researchers stretched their research and drew unwarranted conclusions. I have included a lengthy quote because I don’t want to misrepresent them. According the their data,
the average student assigned to a teacher who was in the bottom quartile during his or her first two years lost on average 5 percentile points relative to students with similar baseline scores and demographics [and] the average student assigned to a top-quartile teacher gained 5 percentile points . . .. Therefore, the average difference between being assigned a top-quartile or a bottom-quartile teacher is 10 percentile points.
Moving up (or down) 10 percentile points in one year is a massive impact. For some perspective, the black-white achievement gap nationally is roughly 34 percentile points. Therefore, if the effects were to accumulate, having a top-quartile teacher rather than a bottom-quartile teacher four years in a row would be enough to close the black-white test score gap.2
As with much of the teacher quality rhetoric, the idea that “large enrollments of low-income and second language students make low test scores more likely”3 is ignored. Whether wishful thinking or willful denial, you see advocates saying things such as, “Effective teachers can close or eliminate the achievement gap.”
The final conclusion is only justified if, as they point out, the effects accumulate. But one other condition has to be met – the kids who are ahead would have to be assigned to those bottom-quartile teachers. After all, students did not move up or down 10 points, they moved up 5 points; the ten point difference is in comparison to someone who dropped back 5 points. So, if the achievement gap is 34 points, it would take 7 years to close it, but only if effects accumulate, only if those who had 5 point gains had not just had particularly good years, only if these results were repeated year after year.
Yet this four year quote is everywhere. A Google search of the entire quote shows it at the Denver school district; Congressional hearings; the Massachusetts DOE; Diane Ravitch’s 2010 book;5 the Recovery School District; Philanthro Media, ‘a dialogue for the discerning donor,” which calls it “an argument we business types can understand;”6 teacher training programs; the New York Times;7 The Houston Chronicle; the Breakthrough Initiative; The Root; and the NYC DOE, which put the quote in a slide show about how we need to change, if not tenure regulations, then tenure outcomes.
LikeLike
Two new studies I learned about yesterday:
Dear Readers,
Education Policy Analysis Archives has just published the introduction and
two articles of EPAA/AAPE’s Special Issue on Value-Added: What America’s
Policymakers Need to Know and Understand, Guest Edited by Dr. Audrey
Amrein-Beardsley and Assistant Editors Dr. Clarin Collins, Dr. Sarah
Polasky, and Ed Sloat Retrieved [date], from
http://epaa.asu.edu/ojs/article/view/1311
Baker, B.D., Oluwole, J., Green, P.C. III (2013) The legal consequences of
mandating high stakes decisions based on low quality information: Teacher
evaluation in the race-to-the-top era. Education Policy Analysis Archives,
21(5). Retrieved [date], from http://epaa.asu.edu/ojs/article/view/1298
Pullin, D. (2013). Legal issues in the use of student test scores and
value-added models (VAM) to determine educational quality. Education Policy
Analysis Archives, 21(6). Retrieved [date], from
http://epaa.asu.edu/ojs/article/view/1160.
LikeLike
It is true that the public is enchanted by metrics. it has been said that You can’t always measure what matters.
LikeLike