Many of us on this blog criticize economists because it often seems that the only thing they value is scores on standardized tests. If they can’t measure a thing without precision, it doesn’t matter. They think they can measure teacher quality by student test scores, they can measure schools by test scores, they can measure students with test scores. As Daniel Koretz showed in his book The Testing Charade: Pretending to Improve Schools, the tests are misused and abused to make these judgments. They aren’t good enough to label students, teachers, or schools, and their misuse distorts the measures (Campbell’s Law).
Now a group of scholars seeks to rescue schools from the iron grip of standardized testing. (Among the authors is my favorite economist of education, Helen Ladd of Duke University.) They argue that test scores are not the only things that matter in education. They say that schools should be informed by evidence, not driven by it.
Decisions should be driven by what we value, what our goals are, not simply by test scores.
They write:
Although evidence clearly contributes to thoughtful policy-making, evidence cannot and should not drive policy decisions. When we make decisions, or policies, we are driven by a desire to achieve a set of goals. The role of data is to provide evidence on how our choices are likely to affect the realization of our goals. Evidence informs decisions so that, if the evidence is good and we interpret it well, the results of our decisions align better with what we value.
A challenge for many decision makers is to think clearly about the values they are seeking to realize. In education, decision makers are often motivated by the desire to improve student outcomes and increase educational equity. Yet both “student outcomes” and “equity” are vague terms. Which student outcomes, or combination of outcomes, are most valuable? Do we care about students’ understanding of trigonometry or their ability to run fast? Do we want to work towards all students having more equal cognitive skills or to increasing the skills of the least well off? Without more precise understandings of which outcomes we care about and which distributions of those outcomes are fair, decision makers lack orientation. Their decisions may end up relying on data about outcomes that happen to be available rather than about outcomes that align with their goals.
In a new book, Educational Goods: Values, Evidence, and Decision-Making (University of Chicago Press, 2018), we seek to spell out a set of educational values and distributive principles and to illustrate how they, along with a small number of non-education values, can be combined with the relevant evidence to improve education decision making. Two of us (Ladd and Loeb) are social scientists who bring a familiarity with the use of evidence, and two (Brighouse and Swift) are philosophers who operate in the realm of values.
This group of scholars is thinking differently. For example:
While educational goods and their distribution are central to education policy decisions, other values come into play as well. While it may at first seem like these additional values are too numerous and ill-defined to specify, in fact, only a small set of values—we identify five of them—typically come into play in education decision making. Think again of the possibility that equalizing educational goods would require extensive intervention in the family. Respect for parents’ interests limits the pursuit of distributive goals. Think of another independent value – what we term childhood goods. Childhood goods are the experiences that students have in childhood that contribute to their flourishing even if they do not build their capacities. These goods may include purposeless play, as well as the joys of learning or laughing. We may be unwilling to undertake an educational approach that develops students’ educational goods if it, in turn, makes the students miserable in the process. The other independent values—respect for the democratic process, freedom of residence and occupation, and other goods (e.g. heath care or housing)—may also put a brake on what should be done to pursue educational goods and their valuable distribution.
Diane Ravitch's blog 
Posted the Brookings article itself at :
https://www.opednews.com/Quicklink/Evidence-should-inform-but-in-Life_Arts-Decision-making_Educational-Crisis_Information_Information-Technology-180324-367.html#comment694348
with comments leading to posts here at your blog. The links are embedded at the address above.
“In this post, Valerie Strauss interviews Daniel Koretz of Harvard University about his new book The Testing Charade: Pretending to Make Schools Better.
Diane Ravitch says: “I just finished reading the book, which is a devastating critique of the current “reform” movement because of its reliance on standardized testing. Koretz is not anti-testing or even anti-standardized testing. He is upset by the misuse of standardized testing. He says that it was completely predictable that putting high stakes on tests would lead to score inflation, gaming the system, and cheating (I said the same things in The Death and Life of the Great American School System in chapter 8, about the false promise of accountability). He says that the so-called reform movement has been completely misled by its obsession with high stakes. Consequently, none of the gains that it claims can be trusted. He also lambastes the deeply flawed Common Core state standards, which presumes the value of having a single standard for all students regardless of their different ambitions, abilities, and interests. the current obsession with standardized testing is pernicious for other reasons. It reduces learning to multiple choice questions and answers. It rewards test-taking skills more than thinking skills. It punishes divergent thinking.”
“That Time of Year: Spring and Testing” by James D. Kirylo
“Testing has become a grueling rite of spring, he writes. “Whereas in 1950 those who completed high school took only approximately three standardized tests through their entire K-12 experience, and whereas in 1991 those who completed their K-12 experience took an average of 18-21 standardized tests, students today upon completion of their K-12 school experience can take anywhere between 60-100 standardized tests. In short, more than 100 million standardized tests are administered yearly across the U.S., annually costing the states approximately 1.7 billion dollars.”This intense focus on testing and its results have moved into the realm of obsession, so much so that we now refer to “high-stakes” testing simply because they are becoming the sole criteria on how we assess and evaluate our children, teachers, administrators, and school districts. In short, the “reform” movement provoked by A Nation at Risk can be characterized as one that is now controlled by the profit-making testing industrialized .
“Koretz is not anti-testing or even anti-standardized testing. He is upset by the misuse of standardized testing.”
He should be anti-standardized testing. Wilson has shown the myriad onto-epistemological errors and falsehoods, and the numerous psychological fudgings that are used in the making of standardized tests that render the usage of any of the results for any purpose to be COMPLETELY INVALID.
Yep, it’s really that simple-COMPLETELY INVALID!
“Reform” fails the evidence and equity thresholds. Charters get uneven results. The selective schools can get good results if they exclude populations that are difficult or expensive to teach. Charters create a system of winners and losers, and the big losers are the public schools that 90% of the students attend. Charters increase segregation which should offend our democratic values. Many charters also operate on a separate and unequal basis. Many minority students get a TFA teaching temp rather than a certified professional teacher. This is more separate and unequal treatment, and this treatment should offend our sense of justice. The same can be said about urban public schools that function with significantly less money than suburban districts.
Vouchers make no sense at all. There is no evidence that they have any academic merit. In fact, they get much worse results than public schools. Yet we are facing ever expanding vouchers because conservatives believe “choice” is of value. In states with conservative leadership, politicians are trying to impose vouchers on communities that do not want them. We should find this offensive to our democratic values. “Reform” is a top down anti-democratic movement.
“. . . because conservatives believe “choice” is of value.”
No, that is not what “conservatives” believe. Choice as a value is what regressive reactionaries, or reactionary regressives-take your pick, believe.
It’s worse than Orwell even imagined. Yes indeed, truth is stranger than fiction.
Sigh. Even the “good” economists don’t fully get it. Let’s take a step back and talk about what we mean by “evidence”. It appears that even these economists define “evidence” solely as test scores and other numeric “metrics”.
Of course evidence should not only inform, but drive everything. That’s what humanity is. We are driven to seek evidence and correct our worldview and adjust our actions to accommodate our new understanding. That’s what play is (read Teacher Tom’s blog). That’s what the scientific method is.
The problem with “evidence” is only when you boil it down to strict numerical “data”. Peter Greene has written many excellent pieces about how dry and stripped down the “data” is that can be tracked by computer or via spreadsheet. Actual evidence that constitutes real data is everything we see and feel and experience every day all around us. You can either give a narrative description, say, “Pat’s eyes lit up with enthusiasm as he appeared to grasp the concept of percentages for the first time, and then he proceeded to tackle and solve several more problems” or you can give “data”: “Pat scored an 80% on his test on percentages.” Which one gives you more data? Which one is better evidence for the appropriateness of this approach to percentages?
YEP!
Agreed. Evidence is a lot more than just data.
Most economists don’t even know what evidence is and would not be interested in it even if they did.
I have cited this before, but it nicely sums up the problem with economics.
“Economics could be science if more economists were scientists” (by William Black)
http://neweconomicperspectives.org/2013/10/economics-science-economists-scientists.html
Why are economists even weighing in on the teaching and learning aspect of education, which they know nothing about?
If they want to study the financial aspects of education, fine, but is problematic as soon as they start talk about “measuring” outcomes.
Where “Financial aspects ” is restricted to analyzing operating costs, NOT trying to make ridiculous predictions about lifetime earnings based on standardized test scores or (worse still) teacher evaluations based on test scores (a la Raj Chetty)
YEP and YEP!
I got hung up on how they were defining “evidence” as well. I also had trouble with the use of the term “goods.” I don’t they are speaking to the “common man,” whatever that means. Who was the target audience?
Other economists.
Thank you for the clarification.
“It appears that even these economists define “evidence” solely as test scores and other numeric “metrics”.
I didn’t read that in the article. There was no talk of stats. What I gleaned was that decision-makers have no substantive educational goals, just a couple of mealy-mouth platitudes which can be interpreted according to one’s agenda – & then proceed w/a widget-production agenda, allowing them to apply simple-minded, cheaply-obtained data, which drives their policy.
The authors show what meaningful ed goals might look like. “One policy choice may improve test scores but if it reduces student motivation or childhood goods [“the experiences that students have in childhood that contribute to their flourishing… may include purposeless play, as well as the joys of learning or laughing”] then it may be a poor choice, all things considered.”
Your last para perfectly illustrates this: if decisions are based on the test score only, the “eyes lit up w/enthusiasm” part, relating to ‘childhood goods’, is not part of the equation, which reflects lack of substantive ed goals.
Economists wake up … Teachers have the “real” evidence every moment of every school day s/he is with students.
Agree with the general content of the article, but must say, there is no such thing as purposeless play. All play is purposeful as all play requires sequencing, organization, processing, analyzing, and all play is crucial in the development of neural networks for the building of critical thinking skills….networks and skills developed at a very young age. Screw the high stakes testing and the use of electronics for young kids as well as the forcing of “academics” down the throats of 3,4 and 5 year olds. Adults do this for their own nefarious purposes. Children should be given maximum amount of time to play, create, dance, dress up, act, enjoy and participate in music, have recess AND physical education. Maybe then we’ll get kids who are socially/emotionally well adjusted and not anxious and disaffected. THEN it seems likely we’ll put a dent in classroom behavior issues, bullying and slow the horrendous number of shootings. Solve the psycho-social issues early enough and you won’t have the terribly egregious issues later on.
“. . . and slow the horrendous number of shootings.”
Until this country recognizes and realizes that it is the world’s largest supplier of death and destruction, and that its military is currently the largest purveyor of death and destruction around the world, we won’t slow any of “the horrendous number of shootings”.
Death and destruction as foreign policy will eventually “blowback” to those who foster the death and destruction attitude and life.
Thank you for highlighting that oxymoron, “purposeless play.” I t makes me sick to see the incursion of an academic curriculum in preschool and kindergarten that downplays the role of play in child development.
I don’t think that is a harsh or mean description. “Purposeless play” means play unguided by adult demands. The children choose their own purposes.
We obviously read that differently.
“Decisions should be driven by what we value, what our goals are, not simply by test scores.”
Let us not forget that eduquackers talk about upping test scores all the time not because that’s what their goal is but because they must hide their real goal which is that public education dollars end up in private hands.
In fact, this is why we hear ridiculous justifications for everything: to hide the real goal of collecting the money of the 99% for the 1%. The 1% doesn’t consists of idiots. They know very well that “More military spending will make us safer” and “We need to be able to buy AR 15s so that we can protect our families from terrorists” and “We need to up our test scores because our economy depends on it” are all bullcrap, but this is the best they could come up with to hide their agenda. And frankly, their BSing has been working prettty well.
“DAM* Econmanists”
Econmanists are cunning
Equating scores with “value”
Econmanists are conning
The public with devalue
*Devalue Added Model
And it is worth noting that “econmanists” contains the word “man”, which is not an accident, since a Conman is a man by definition.
Maybe if economics had more women it would have a more caring and less exploitative view of the world.
Professor Helen Ladd is an economist, a woman, and deeply understanding of the farce of school choice when kids are poor, hungry and ill.
That’s not an idle speculation given that women like Brooksley Born tried to warn about the dangers of derivatives but were bullied and silenced by Econmanists like Alan Greenspan, Robert Rubin and Larry Summers.
There is an old joke: “I wish all economists were one-armed. That way they could not say “on the other hand””.
“Zero-Footed Economist”
Enconomist
Without a foot
Would never get about
But also this:
Could never put
His foot inside his mouth
This is the first article I can remember reading that tackles what we so often decry here– the dearth of meaningful ed goals driving ed policy. The argument is well-framed: at the decision-making level, ed goals are vague platitudes that can support any agenda.
The distinction between evidence-informed & evidence-driven policy is useful to frame up the argument, but not quite right. Policy-makers do have ed goals, but they are dictated by whatever agenda is hidden by the platitudes. We could easily come up with a cluster of “ed goals” that today’s policy-makers hide under that cloak, which drives the type of supporting data they collect. Accountability to the taxpayer, anti-public ed/ public goods, anti-unionism, selling hw/sw stds/ assessment/ data-mining packages, mining student data for commercial sale, career/ corp/ mfr skills training, racial/ SES segregation, gentrification, privatization of public goods, deprofessionalization of teaching.
I like that the authors do not wade into the weeds of how to measure the social goals they find missing from policy decisions. In fact, the thrust is that currently, the hard decisions – identifying and prioritizing among equally-substantive goals – are sidestepped entirely. No mention of quack-personality-tests, student/ parent surveys et al cheap snake oil. Just a suggestion that ed-research could assist properly-identified goals: how refreshing – a return to professionalism. If pressed for specifics, the authors might even agree with Yvonne Siu-Runyan’s observation, “Economists wake up … Teachers have the “real” evidence every moment of every school day s/he is with students.”
The evidence that teachers have that Yvonne refers to is not in the form of numbers (and to a large degree can’t be represented as such), so most economists would simply not be interested in it.
They would not even know what to do with it, even if they did by some unlikely chance take an interest.
Mainstream economics (particularly econometrics) depends on numbers for input.
lamestream econ in a nutshell: Put the numbers in, turn the crank and out pops the result.
Of course, most economists never even ask whether the input means anything.
In order to look very sciency and mathematical, the economists have placed all their emphasis on the so called “analysis”, essentially ignoring whether the input data are valid. Bill Black, (in the piece I linked to above) talks about this when he points out the fact that if the input data are fraudulent (as they were in the case of the S&Ls), the outputs and conclusions taken from them are meaningless.
Though it may be a novel approach for economists (and certainly for politicians), the idea of defining the goal(s) before one develops and implements policy is not really novel.
Duane Swacker has often pointed out here the importance of going back to square one and defining the goal of public education.
Not incidentally, not considering the invalid/fraudulent data input was also what drove the housing bubble and the reason why so few economists saw the meltdown on the horizon.
In economics, ignoring the validity of the input is a feature, not a bug.
Incidentally, Duane Swacker has also pointed out (many many times ) the fact that if the input data are not valid, the conclusions simply CAN NOT be.
Duane thinks more like a scientist than many scientists do.
I think economy as a science is changing. I reread parts of Kahneman’s book “Thinking fast and slow” and Piketty’s “Capital in the 21st century” all the time. They use no jargon, little math, and their books are full of interesting and often striking observations, facts, conclusions.
Kahneman received the Nobel Prize in Economics for the very idea some of you are miss in economics.
Daniel Kahneman is an Israeli-American psychologist notable for his work on the psychology of judgment and decision-making, as well as behavioral economics, for which he was awarded the 2002 Nobel Memorial Prize in Economic Sciences. His empirical findings challenge the assumption of human rationality prevailing in modern economic theory.
The topic of “Educational Goods: Values, Evidence, and Decision-Making” is definitely promising, though the language and style may not yet be where Kahneman’s is.
It would be interesting to see research along the lines of Kahneman and show that teaching and learning are largely not rational—even when you learn “rational” facts in rational subjects.
For example, I always thought that the main problem with Common Core math is that it’s trying too hard to explain everything rationally and explicitly. This is the underlying reason why many says that CC is “not age appropriate”. It may be surprising to some when I claim that rigor (which presumes rationality) is not so important in teaching math (it’s even counterproductive for young kids), and if there was thorough research supporting this claim, it would end the relentless demand for rigor in education by the eduquackers.
Interesting comments, Máté, although perhaps the term “rational” needs to be defined since saying someone is not being rational is not exactly a compliment. I agree with what I think you are saying, though. Forcing an overanalysis of a procedure, topic,… can be counterproductive. We make a mistake by equating rigor with deconstruction of every operation down to its smallest parts and then investigating thoroughly how it might be put back together again, and I’m not just talking about math. I have a suspicion that the process can be applied in every discipline. There is a joy in learning that is easily killed by pushing too hard, too far, and too soon. I need to walk before I run.
“rational” needs to be defined since saying someone is not being rational is not exactly a compliment. ”
I am not talking about being rational but thinking in rational terms, providing logical explanations, understanding things at a rational level.
I dunno what’s the correct word. Perhaps unconscious and conscious are better than rational and not rational. The point is that kids can understand things unconsciously, intuitively, without being able to provide rational explanation.
If you tell them to run, they run, if you tell them to walk, they walk. They can tell, if somebody walks or runs. They understand perfectly what the difference is between walking and running. It doesn’t ,mean, they can (and should be able to) “rationally” explain what the difference between the two is.
All kids learn a language, and they understand it well. They know what the difference between an adjective and verb is before knowing grammar, way before being able to explain it.
Kids can understand the difference between 12 and 21 way before they can explain it.
Common Core, as far as I can tell, wants kids to provide rational (“rigorous”) explanations way before kids are ready for them.
Common Core wants rational explanations for everything because that’s what’s testable and can be scored even by computers while checking intuitive understanding of a material is easy only for humans.
Logical? Not quite right either.
There is a nonverbal element to learning that I don’t think we understand very well. We used to talk about automaticity as being a marker of having mastered/learned something. It didn’t require words for someone to understand that the child had mastered the task, skill,… when they could perform it without hesitation.
I always thought the main problem with Conman Core was that it was designed and imposed on the public by conmen: Bill Gates, David Co
lnman and Jason Zimbot. The Conman Core, as it were“The Conman Core”
The Conman Core
Is nothing more
Than slimy oil of snakes
The Conman Corp
Is who it’s for
A “con” for bidne$$ $ake$
“His empirical findings challenge the assumption of human rationality prevailing in modern economic ”
Humans do not necessarily behave rationally?
Who knew?
And they actually give out (fake) “Nobel” Prizes for that obvious “discovery”?
https://www.alternet.org/economy/there-no-nobel-prize-economics
The assumption of “rational behavior” is only one of many ridiculous “simplifying” assumptions which sever mainstream econ from reality. Others are “perfect information” and “no fraud” and “endless exponential growth”.
Einstein once said about good theory that it should be made as simple as possible but no simpler.
Most economic theory violates the second part.
If anyone ever needed proof that humans do not necessarily behave rationally, economists are it.
Read economist Dan Ariely’s book, “Predictably Irrational.”
Sounds like an interesting book, from the intro.
But I have to say that the author has a curious (and not entirely accurate) definition of the “rational economic model”
“In this book, when I mention the rational economic model, I refer to the basic assumption that most economists and many of us hold about human nature — the simple and compelling idea that we are capable of making the right decisions for ourselves”
Actually, the rational economic model does not only assume we are CAPABLE of making the “right” decisions. It assumes that we all DO.
From Investopedia:
What is the ‘Rational Choice Theory’
Rational choice theory is an economic principle that states that individuals always make prudent and logical decisions. These decisions provide people with the greatest benefit or satisfaction — given the choices available — and are also in their highest self-interest. Most mainstream academic assumptions and theories are based on rational choice theory.”
Read more: Rational Choice Theory https://www.investopedia.com/terms/r/rational-choice-theory.asp#ixzz5ArXcCerB
Not incidentally, it is not always even clear what the “right” decision for an individual is. “providing people with the greatest benefit or satisfaction” and ” in their highest self interest” are subjective.
Why don’t you read it? His chapter on education is excellent.
“Common Sense”
The most we need is common sense
To operate the schools
The last we need is conman cent$
And hard and software tools