Jerry Z. Muller wrote a warning about how data-driven organizations can distort their own goals and purposes.
In education, we have known about the dangers of incentives for test scores for a long time. In 1976, sociologist Donald Campbell that “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social process it is intended to monitor.” When the tests assume too much importance, there will be cheating, gaming the system, narrowing the curriculum, and other unwanted consequences.
A dozen years ago, Richard Rothstein wrote an excellent paper called “Holding Accountability to Account,” showing how incentives can perversely affect and undermine the goal that are sought (it is free on the internet).
In 1990k Andrea A. Gabor wrote a book about W. Edwards Deming called The Man Who Discovered Quality, in which she explained Deming’s contempt for merit pay and bonuses, which cause employees to think about themselves and not about the organization and its larger purposes.
Muller wrote a recent article about “metric fixation” in which he reviewed the flaws of data-driven work:
More and more companies, government agencies, educational institutions and philanthropic organizations are today in the grip of a new phenomenon. I’ve termed it ‘metric fixation’. The key components of metric fixation are the belief that it is possible – and desirable – to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardized data (metrics); and that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance.
The rewards can be monetary, in the form of pay for performance, say, or reputational, in the form of college rankings, hospital ratings, surgical report cards and so on. But the most dramatic negative effect of metric fixation is its propensity to incentivize gaming: that is, encouraging professionals to maximize the metrics in ways that are at odds with the larger purpose of the organization. If the rate of major crimes in a district becomes the metric according to which police officers are promoted, then some officers will respond by simply not recording crimes or downgrading them from major offences to misdemeanours. Or take the case of surgeons. When the metrics of success and failure are made public – affecting their reputation and income – some surgeons will improve their metric scores by refusing to operate on patients with more complex problems, whose surgical outcomes are more likely to be negative. Who suffers? The patients who don’t get operated upon.
When reward is tied to measured performance, metric fixation invites just this sort of gaming. But metric fixation also leads to a variety of more subtle unintended negative consequences. These include goal displacement, which comes in many varieties: when performance is judged by a few measures, and the stakes are high (keeping one’s job, getting a pay rise or raising the stock price at the time that stock options are vested), people focus on satisfying those measures – often at the expense of other, more important organizational goals that are not measured. The best-known example is ‘teaching to the test’, a widespread phenomenon that has distorted primary and secondary education in the United States since the adoption of the No Child Left Behind Act of 2001.
How many times do we have to hear the same advice and ignoring it?
“metric fixation” will destroy the United States and speed up the race to the bottom of everything, even fighting wars.
I still get angry over the reliance on metrics to drive decision making. I was purportedly not rehired because my students did not perform to the standards promised by the program, nor did I rely on solely the data generated to make instructional decisions. No attention was paid to the fact that neither the students assigned to the program nor the resources provided met the criteria set by that program. I was criticized for not following the protocol as presented by the marketing team. My training by the creators of the program did not matter to those in charge who were not trained to teach special education or the subject I was teaching.
They are still talking about ‘teaching out there’ and of course Schoo is ALL about LEARNING
Data driven metrics, instead of allowing the professional to create the curricula… because an experienced , educated, talented professional educator, knows what learning looks like for each of those kids — who sits in that room for 10 months.
Thank you for this post. Numbers seem so concrete and certain that it’s easy (almost lazy) just to rally around the numbers – even if they are not measuring what’s most important or leading an organization in the right direction.
“narrowing the curriculum, and other unwanted consequences”…… yup. What is measured becomes the focus and other (socialization, group projects and play) get left behind.
This is an astonishing report on chronic absenteeism in our public schools. The interactive map allows users to pinpoint data for every school district in the country. And it does not take into account the time out of class due to counselling, suspensions, music lessons, and other interruptions. The correlation between test scores and attendance is willfully ignored by those who should know better. In high needs inner city schools, it is not uncommon to have students missing 20, 30, or even 40+ days of instruction. Secondary academic teachers are lucky to get 80 hours of instructional time prior to mandated federal testing in the spring, students who are chronically absent have face insurmountable deficits when it comes to academic success.
Chronic Absenteeism in the Nation’s Schools
https://www2.ed.gov/datastory/chronicabsenteeism.html
It goes without saying that my students who had weak attendance records also tended to be my weakest students.
The problem is, “that it goes without saying”
Someone once said that, 90% of success was just showing up.
“Someone once said that, 90% of success was just showing up.”
So true.
How long? Until the political forces shaped by monetary support are replaced by political forces shaped by informed vote.
If you read this blog, the conclusion you must reach is obvious: the forces shaping public education policy do not have the best interests of the children at heart. Rather, they seek to use the fact of education to line the pockets of themselves and their cronies, all the while accusing their philosophical opponents of their own sin.
“Data driven” means “Profit driven” and “educational standardization” means “facilitation of sales of products to schools”.
It’s Bill Gates and his electrical outlet analogy for schools.
By the way, I hope at least some of the former Gates worshipers were paying attention to his statement that vaccine formulas should not be shared with the developing countries ( who as it currently stands, are totally at the mercy of the vaccine companies when it comes to vaccine availability. “Please, Bill, Sir, can I have some more vaccine?”)
Gates stance on vaccines pretty much tells you his philosophy toward everything. Profit driven companies know best and yield the best outcomes in all cases.
Bill Gates needs to go away to his oasis community in the desert … forever and shut his mouth. .
Maybe his “not my friend, business associate or even social acquaintance” Epstein’s Island is up for sale
Paging Dr. Swacker. Is Dr. Swacker in the house?
The application of “metrics” and “measurement” to education is a sham.
It’s done by people who don’t even know what the words mean.
It’s obvious that this is the case because they regularly misuse the words.
A “metric”is a clearly defined standard (eg, a meter stick) that quantifies a clearly defined real world attribute (eg, the extension of an object in a given direction).
A so called “standardized” test is not a “metric” , not least if all because the attribute that it is supposedly “measuring” (“learning” or “achievement”) is not precisely defined. (Not even close. )
It is most certainly NOT legitimate to define the attribute in terms of the test, effectively claiming that “learning or achievement are what(ever) the particular test measures” — something that “standardized” test proponents do all be time, without even realizing how circular and ridiculous their entire “psycho-metrics” (with the emphasis on psycho) field is.
The attribute being gauged MUST be independent of the particular test being used. For example, there are many different instruments for measuring the extension/length of an object and provided they are all properly calibrated, they all give the same result within the measurement uncertainties of the various instruments (which are always present, since there is no such thing as a perfect instrument that gives perfectly accurate results).
The latter is simply not true of so-called “standardized” tests, which give results that are unique to each test.
It’s really this simple:
Intellectual development in young humans
is NOT LINEAR
and thankfully,
it CANNOT be MEASURED
Let’s not overstate this. Otherwise, we run the risk of sounding silly. Some intellectual achievements are clearly defined and can easily be measured. For example, I can pretty accurately test to see how many kanji you know. Some cannot. For example, I cannot validly test for whether you are capable of finding a main idea or making inferences from text–that’s just too broad and vague to be tested with a couple questions. BTW, there is no general “finding the main idea” skill. Only a total idiot would imagine that there was, even though this stupidity has been repeated ad infinitum et ad nauseam in K-12 education in the US for decades.
cx: Those things are just too broad
Our K-8 district has been obsessively testing reading and math for 17 years. We’ve doubled or tripled the time spent on these subjects. The result? Our kids are getting no better at reading and math, and know almost nothing of history, geography and science. Metrics fixation is not working for us, and yet our administrators show no sign of giving up the faith. They are mere followers; they do what the “authorities” in Sacramento and DC tell them to do.
Ditto on most of this. We were high performing before and are at about the same now (some yearly fluctuations to either side). All the extra tech and data collection has not mattered….. one…. bit.
But the powers at be cannot (do not care to) understand this – they have a one track mind with their own personal goals in mind.
The “authorities” don’t seem to understand that the last thing that any classroom teacher does is to, keep doing what isn’t working.
The result? Our kids are getting no better at reading and math, and know almost nothing of history, geography and science.
Nailed it, Ponderosa. But these “authorities” pay no attention to the “data” that show no improvement whatsoever in reading and math scores after an entire generation of this bs.
This is why Duane Swacker is obsessed with the nomenclature. Calling something measurement that is actually just descriptive stat grants it a level of legitimate consideration of which it is unworthy. Many took the measurements of Francis Galton to justify horrible acts against other human beings. Then we learned his data was driven by racism.
We have been here before. Time for a change.
We have been here so many times before that even the future is getting old.
Metric fixation is a serious problem. Fixation on standardized test scores is a far more serious problem because the data are completely flawed. If a student takes a test measuring “reading skills” on Monday, gets a score, takes the same test on Tuesday, and gets a different score, then both the scores are invalid. It is not scientific to accept the results of an experiment that is not repeatable. That is what Señor Swacker teaches. It is not possible to measure the skills of an individual engaged in a complex task. One of the trolls might say that if it’s impossible to measure skills, why give grades to students? Grades are not measurements; they are subjective evaluations. And it is not healthy or productive to fixate on grades, either. The only way for standardized tests to carry any validity is by making generalizations based on huge sample sizes, as does the NAEP, and even then, there is not enough validity to affix high stakes.
Fixating on data is stupid; fixating on false data from standardized tests is beyond stupid.
Is ” metric fixation” a euphemism for “fixation on unit size?”
What could go wrong with this* role model?
*Preach (testing) dissent,
BUT
Practice obedience (give tests).
There is no downside to the testing obsession. There can’t be a downside to data driven drivel because if there is a downside there must be an upside. There is no upside, unless you’re talking about data profiteering. There are no good data, just the bad and the ugly. All pain, no gain. Heads billionaires win, tails you lose.
Annual testing should have become known as pre-pandemic folly. Online learning should have become known as pre-pandemic folly. Our leaders work with blinders on. Blinder$.
A school system would be crazy to let data drive decisions and sane if it didn’t. If we close schools and fire people for test scores we are crazy and don’t get taken over, but if we don’t close schools and fire teachers we are sane and get taken over. The sane thing to do is fire ourselves.
“Orr would be crazy to fly more missions and sane if he didn’t, but if he were sane he had to fly them. If he flew them he was crazy and didn’t have to, but if he didn’t want to he was sane and had to. Yossarian was moved very deeply by the absolute simplicity of this clause of Catch-22 and let out a respectful whistle.”
Data will drive us to involution. From the NPE blog: https://educationalendeavors.substack.com/p/a-generation-at-risk
Data driven
When data are driven —
Along for the ride —
They never are given
A chance to decide
Gneiss to see you around these parts again.
Granite, it might just be that I overlooked your presence.
Diane asks: “How many times do we have to hear the same advice and ignoring it?” Today is the 50th anniversary of the publication of the Pentagon Papers. Consider the answer to her question in terms of American military and foreign policy–e.g., in places like Yemen.
Good point
Two thoughts:
If we had ALL the data, then perhaps we could use it to make decisions. But these measurements are only a part of the picture and a biased part at that.
Graduation rates are another example: Just make it easier for the students to pass and poof! magically, graduation rates go up!
This whole “data-driven education” thing reminds me of dumb sci-fi movies of the 1950s. I remember that in one of these, the console on the “spaceship” has a dial with a label that reads “Space Speed Speedometer.” The makers of this film figured that the audience was so stupid that anything even vaguely, remotely scientific-sounding would suffice to create the necessary suspension of disbelief in the illusion.
Likewise with “data-driven education.” Our school systems are now full of people so stupid that they think that just because a “test” comes up with a “score,” and that “score” is a number, then the “test” must be “scientific.”
The fact is that most of these state tests on the ELA “standards” don’t validly test those standards. They couldn’t do so because the standards are so vague, for the most part, as not to be concrete enough to be testable. But even if there were concrete enough that mastery of them or proficiency in them could be ascertained via one or more test questions that were the operational equivalent of such mastery or proficiency, there would be too few questions per standard (there are at most one or two questions per standard on each test) for valid testing of the standard to be taking place. And even if the “standards” were concrete enough to be testable and there were enough questions per standard for such testing of it, there would still be this MAJOR problem with the current ELA tests–that the questions are TYPICALLY so badly written that a) there is arguably more than one correct answer or b) there is arguably no correct answer GIVEN THE ACTUAL QUESTION STEM, SELECTION, AND ANSWERS on the test.
All this is so freaking obvious, or would be to anyone who actually studied one of these tests and thought about them at all carefully, that one must conclude that we have a LOT of education people right now, at the district, state, and federal levels, who are braindead or on autopilot or Vichy collaborators with the test-and-punish regime because they are being rewarded in some way for their collaboration.
Enough.
End the farce.
Or, rather, the scam. We need to end the federal standardized testing mandate. Then, we need a Truth and Reconciliation Commission to investigate this scam testing and to recover, from its perpetrators, some of the billions lost. There will be, however, no recovering the lost educations of the millions of kids subjected to scam “data-driven” test preppy substitutes for real educations.
Gates the monopolist created the Common Core and his Frankenstein monster, InBloom, in order to have a single set of questions to key computerized tests and education to, with him as the gatekeeper (via his control of what was planned to be the nation’s gradebook). Any bs would do. There would be enough people in education willing to sell out and buy in due to stupidity, venality, or a combination of the two.
The classic example of this was the :body count: metric of the US military during the Viet Nam War. It reached the absurdity that there should have been no people left in NVN if the counts were accurate.
I personally have worked in both k-12 and higher ed public systems and have seen the destruction these metrics can cause, not to mention the sheer wistfulness of the student an d teacher time spent. This is just one more example why we need to NOT run education “like a business” where profit is end goal, not service to the whole population.
Well, I score at the 99th percentile for awesomeness.
WTF?
Can’t argue with this. It’s data.
You totally just made that number up.
Bow down and worship my awesomeness.
You are seriously disturbed.
It’s science. It’s a number, so it can’t be wrong.
STUDENT: This question says, “If Johnny and Trevor each have 3 and a half pears, what is the square root of Tuesday?”
TEACHER: It is against the rules to talk about what is on the test. Your results can be invalidated.
STUDENT: But it doesn’t make any sense.
TEACHER: It’s a test. It’s infallible. Moses brought it down from the mountain. Hand in your paper and report to the Assistant Principal.
STUDENT: But you’re a teacher. You’re supposed to help me.
TEACHER: Haaaaaaa!
STUDENT Why are you laughing?
TEACHER: That’s so old school.
How to tell if your administrator is a total moron:
He or she takes seriously the results of state standardized ELA tests.
I was a teacher in North Carolina in 1993 when the state began high stakes testing through it’s ABCs legislation. Since NAEP began assessing educational progress in 1990 North Carolina has not improved reading or math results. 28 years of failure should indicate that what we are doing in this country in regard to the Standards Movement means high stakes testing does not work.
Same is true nationwide. A generation of utter failure.
One of my bugaboos is laws that promote short-term over long-term performance in publicly-traded companies– the phenomenon of all focus on the quarterly report. As in other fields (like education), the debate is current and heated, with a consensus by insiders that healthy long-term goals are getting sidelined. This brief analysis suggests the main culprit is the requirement for projections against which performance is reported. Part and parcel: how those projections are framed, and in whose interest. https://corpgov.law.harvard.edu/2019/07/08/do-securities-law-promote-short-termism/
Here’s another example where edumetrics warps education. USNews started ranking high schools in 2007. 30% of the rank is based on the proportion of students who took and earned a qualifying score on AP or IB exams. By 2012, the # of students taking AP courses was growing at 10 times the rate of the increase in # of hisch students.
The proliferation of AP courses in a school siphons off best teachers and students into small classes (leaving non-AP with bigger classes) which, unlike the old ‘honors’ classes, force-marches students through a nationally-scripted curriculum that leaves little room for creativity and in-depth study. Many teachers/ profs say the courses in no way reach the level of the introductory college courses which they claim to replace.
Low-income schools offer fewer AP courses and low-income students are way under-represented where they’re offered, making this just one more giant tracking/ segregation hurdle for their admission to college.
College Board [a “non-profit” that makes about 9% profit] gets 50% of its income from these courses, collects fees for taking—or not taking—the exams, while reporting nothing but a name and numerical score—which data is available for sale.
the “nonprofit” CB also pays its mind-blowingly incompetent president a fortune every year. A bunch of other CB execs consume excessively at this trough as well.
COLEMAN: Cynical Opportunist Leveraging Execrable Measurement and Assessment Nonsense
The only metric that really matters to most of these reformers is the calculation of the size of the checks being written to them, for supporting these utterly failed policies, by the slow learners at the Bill and Melinda Gates Foundation.
In other news, the geniuses in the Arizona legislature have advanced a bill to fine teachers $5K a whack for violating some incredibly vaguely stated rule not to advance a controversial opinion in the classroom.
We’ve been seeing this phenomenon in just about every walk of life for quite a while. (The writer mentions cops manipulating crime stats to please their bosses – the theme of 2004’s Season 3 of The Wire.)
This may just be a process that we’ll inevitably move beyond. Society has been digesting the digital revolution for about 45 years now. When I wore a biz hat [procurement for an engrg firm] in the ‘70’s-‘80’s, I watched “Management by Objectives” [MBO] infiltrate and warp the project mgt process just as soon as mainframe computers and their fan-fold printouts were cheap and fast enough to be used by large companies. It passed by the mid-‘90’s, a victim of global competition. You can only get away so long with loading down OH with a bunch of admin that does nothing for the construction schedule or the bottom line.
You can play the game a lot longer with performance that’s more difficult to measure, as long as the obfuscation is paying off for somebody with enough clout to keep running the clock. Eventually, the public notices somebody is picking their pocket while delivering zero results.
I wonder what the position of Director of Parent Obfuscation pays at the Fordham Institute for Securing Big Paychecks from Oligarchs for Officers of the Fordham Institute.
It’s sad that Education Deformers have so ruined the meanings of perfectly good words like “rigor” and “data.” But that’s nothing compared to the consequences for children. Those have been tragic.