Thanks to Paul Thomas for the link to this impressive post by Kaiser Fung, a professional statistician.
Fung saw an article By Gates claiming that spending on education was rising but student achievement was flat.
Fung demolished this claim and said that Gates was promoting innumeracy.
The scales of his graph were wrong, the analysis was wrong, the arguments rested on fallacies. Gates, he said, compared apples and oranges, and he confused correlation with causation.
Fung writes: “Needless to say, test scores are a poor measure of the quality of education, especially in light of the frequent discovery of large-scale coordinated cheating by principals and teachers driven by perverse incentives of the high-stakes testing movement.” No one told Gates about that, apparently.
And he concludes:
” In the same article, Gates asserts that quality of teaching is the greatest decisive factor explaining student achievement. Which study proves that we are not told. How one can measure such an intangible quantity as “excellent teaching” we are not told. How student achievement is defined, well, you guessed it, we are not told.
“It’s great that the Gates Foundation supports investment in education. Apparently they need some statistical expertise so that they don’t waste more money on unproductive projects based on innumerate analyses.”
How refreshing to know that statisticians like Kaiser Fung are keeping an eye on what is called “reform,” but turns out to be the pet ideas or hobbies or whims of very wealthy people who know little or nothing about education.
Diane Ravitch's blog 
A common statistical mistake: equating one descriptor of a trait, like quality, with that descriptor. A test score, is one among a multitude of variables that might go into the description of what quality teaching is. The problem, of course, is that there are so many variables that mix into that definition, with the added problem of in some way saying which variable is most important, that in the end, we have no definition or practical application of the trait we call quality teaching —e.g. how to you quantify, for example, social-emotional relationships between teacher-student. For a man as bright as they say Bill Gates is, do not understand how he does not understand this Statistics 101 concept.
“A test score, is one among a multitude of variables that might go into the description of what quality teaching is.”
Unless that “test score” is from a test the teacher took (not gave) that was specifically designed to assess the quality of teaching of that teacher, and so far I’ve never seen or heard of such assessment, it is UNETHICAL to use the results of a test score for any other purpose than the one that the test makes has designated the assessment to be used for. In other words if the test is a third grade math assessment of the student, to use the results to assess the teacher is not only a LOGICAL FALLACY, but UNETHICAL.
Is that concept that hard too comprehend?
It sure seems so! But then again most non-teachers haven’t taken Testing and Measurement in Education 101. So there is no excuse for teachers and administrators to not understand this concept.
“. . . in the end, we have no definition or practical application of the trait we call quality teaching —e.g. how to you quantify, for example, social-emotional relationships between teacher-student?”
Well you can’t and the confusion lies in confusing quality concepts with quantity concepts. They are two separate logical categories. See #1 below. Noel Wilson has shown this and the many other epistemological and ontological errors and fallacies that are part and parcel of the educational standards and standardized testing regime in his never rebutted nor refuted “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
Mr. Gates:
“It’s the one thing we’ve been missing, and it can turn our schools around.”
They can’t fail, they can only BE failed, by other people.
I know it’s often the better approach to ascribe errors and foolishness to ignorance and misunderstanding than to malice, but I doubt that’s the case here. Gates’s column is a classic case of lying with statistics and pettifoggery. And I doubt Gates wrote himself. This is just another reformer hit piece from the reformers.
Agreed.Reformer hit piece = media buy. A commercial in other words. The inaccuracies are a feature, not a bug. A crafted, probably focused-group message.
Indeed, when the “errors” of these people so consistently
reflect their financial and/or ideological interests, we can assume greed, the will-to-power and malice towards those who stand in their way.
Attribution (malice or ignorance) is a difficult thing to nail down.
But one thing is clear. Gates is already heavily “invested” in education “reform” [sick] to the tune of hundreds of millions of dollars –and he has put all his eggs in one Common Core/testing/VAM basket.
I think that alone explains why he will continue on his current wayward path.
Even if he has doubts, he can’t air them because that would mean admitting he was wrong.
…and if there is one thing Bill Gates does not like to do, it is admit he is wrong. Just ask Paul Allen
What we are experiencing is the negation of research baed inquiry and its replacement by political and economic demands. We still hold onto our Age of Reason belief system, when, in fact, the education ‘reformers’ demands are, driven by the overwhelming desire to create a privatized, profit driven education system. It matters not a whit that Diane’s latest book or AERA research, or that we learn about malfeasance
Kr lack of ‘success’ of charter schools, or that charter schools discriminate against linguistic minority (English Language Learners) isr students with special education needs.
I do not argue that we lie, cheat and manipulate as do the ‘deformers’, only that we move forward aware of the dominant paradigm of a new non rational reality that resists research driven pedagogy and due process
Spending increases versus test scores over FOUR DECADES. Shouldn’t this graph be expected or normal. People aren’t getting smarter on average but inflation constantly exists.
Does the chart account for inflation-adjusted dollars? If not, then it’s worse than useless. It’s disingenuous.
Here, too, is the problem with statistics that can be massaged; in education dollars, isn’t it often failed to report how much money is spent on 1-to-1 aides, aides, safety officers, traffic safety guards, breakfast, lunch, after school snacks and remediation? Certainly the stats must include administrators and admin staff, but the reformers act as if every dollar per pupil is going into the pocket of a teacher.
Meanwhile, back at the charter schools, there are myriad layers of principals, sometimes a principal for each grade, and so many charters are NOT K-12, so that pie gets mighty divided. When they are bitching about the per pupil costs, are they including the duplicitous admin staffs of the charters, and some of the wacky positions that have been created, like “Dean of Continuity” and the like?
It really is high time that all of this nonsense is broken down, busted out, and lawsuits filed detailing the conspiracy that is edreform.
“Bill Gates should hire a statistical advisor”
As long as the name of that advisor is not “Raj Chetty”!
Chetty has also been peddling “questionable” statistics and conclusions based on those statistics (eg, in Vergara) and doesn’t seem to know the difference between correlation and causation.
Gates obviously dropped out of Harvard very prematurely, before he had taken “Intro to Stats”, “Intro to Graphing” and “Intro to Thinking”.
I’m not sure what Chetty’s “excuse” is, but I just hope he is not actually teaching “Intro to Stats” (for economists or anyone else) at Harvard.
The irony of “the statistically and educationally ignorant lecturing the real experts” is not lost on some of us.
There is actually a name for this phenomenon: The Dunning-Kruger effect.
Thanks for the Dunning-Kruger effect.
From Wiki (keeping KrazyTA in mind):
Although the Dunning–Kruger effect was put forward in 1999, Dunning and Kruger have noted similar historical observations from philosophers and scientists, including Confucius (“Real knowledge is to know the extent of one’s ignorance.”),[3] Bertrand Russell (“One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision”, see Wikiquote),[11] and Charles Darwin, whom they quoted in their original paper (“ignorance more frequently begets confidence than does knowledge”).[2]
Geraint Fuller, commenting on the paper, noted that Shakespeare expressed similar sentiment in As You Like It (“The Foole doth thinke he is wise, but the wiseman knowes himselfe to be a Foole.” (V.i)).
Or maybe Billy’s mom or dad dropped him on his head as a baby:
Anosognosia (/æˌnɒsɒɡˈnoʊziə/, /æˌnɒsɒɡˈnoʊʒə/; from Ancient Greek ἀ- a-, “without”, νόσος nosos, “disease” and γνῶσις gnōsis, “knowledge”) is viewed as a deficit of self-awareness, a condition in which a person who suffers certain disability seems unaware of the existence of his or her disability. It was first named by the neurologist Joseph Babinski in 1914.[1] Anosognosia results from physiological damage on brain structures, typically to the parietal lobe or a diffuse lesion on the fronto-temporal-parietal area in the right hemisphere.[2][3][4] Whilst this distinguishes the condition from denial, which is a psychological defense mechanism, attempts have been made at a unified explanation. (from wiki)
In addition to the errors noted, I think it is important to not simply accept K-12 spending being represented in inflation adjusted dollars only. I always try to include both a longitudinal inflation adjusted number and K-12 spending as a percentage of GDP. The United States is a wealthy economy and continues to grow over the long term (even if it doesn’t always seem that way, in part due to inequality). There are many sectors where we spend more today than 40 years ago in inflation adjusted terms.
K-12 spending as a percentage of GDP shows more clearly what percentage of our economic output we allocate to elementary and secondary education.
In reality, spending on elementary and secondary education in the United States is very close to the OECD average as represented here: http://nces.ed.gov/programs/digest/d13/tables/dt13_605.20.asp
For the endpoints shown on the Gates chart, elementary and secondary education expenditure as a percentage of GDP was 3.9% in 1975 and 3.9% in 2007.
In his hubris-riddled, condescending, I-can-save-the-planet interview with Rolling Stone a few months ago, Bill Gates discussed how very important it was to have the right experts and the right metrics. I think someone needs to revisit his programming. If this keeps up, it is headed for multiple crashes and will be prone to hacking and viruses . . . but Bill should be used to that. It’s amazing how a fat portfolio gives one the impression that he or she knows better than those of us with more average means. I suppose it is possible that when you drop out of society and academia and work with algorithms all day, it is possible to start treating other people’s jobs and children like mere abstractions, like variables to be played with; teachers, students, public education in America, the Common Core, high stakes testing — all just another software gambit.
This is a man, after all, who believes he is on the brink of curing malaria. Wonder how recently he checked those statistics?
Good post. And “the right experts and the right metrics” are defined by what exactly?
For most people, that would be defined by what one was pre-disposed to think already. (I am guilty of this as well.) Gates admits to cherry-picking according to his own “latest” idea.
And since Gates has such remarkable largesse, he can easily find any number of people who will simply feed his ideas so they can get in on the money. We see it all the time.
Many people promote particular ideas so that they can cash in later. The Detroit Free Press articles on charters demonstrated that. There were multiple university authorizers who left their posts for more lucrative charter chain posts. After providing those chains with lots of charter authorization.
Gates lives in an echo chamber like we do. He just has the money to enforce the voices in his own echo chamber.
Gates and those of his ilk need to step outside of their libertarian, neo-liberal bubble and start to accept that they don’t know what’s best for the rest of us just because they made billions–often at the expense of everyone else.
Interestingly, Gates probably owes his success to his parents. And if he had good teachers, that was because his parents provided them for him at above the median price, which is a case for spending more on public education.
Gates believes in research with a purpose, his purpose and as noted, he is never wrong, being so might just be fatal. I will be charitable and merely state that in my experiences with him he shows a narcistic personality type. As noted above, he would never admit error, in his mind he does not make mistakes and is not wrong.
Increased spending on tests, consultants, and bureaucracy does not increase achievement.
Prof. Norman Matloff, in an opinion piece at CNN, says, “Leaders in the tech industry are looking less heroic these days… In case after case, we see tech titans misbehaving or breaking the law… They push the boundary of acceptable or ethical behavior that most of us have to play by. They consider their products so beneficial to humanity that they act as boy kings… The latest case is colluding to halt labor competition-putting a break on engineer salaries.”
Matloff doesn’t single any one mogul out. He makes the indictment of them, all.