Yong Zhao, the brilliant education analyst, writes here about the great PISA illusion. If you have not read any of Zhao’s books, do so now. If you have not heard him speak, google him or invite him to your next big conference. He is insightful, provocative, thoughtful, absolutely delightful! He is a master at making people think and debunking hoaxes. Please read the entire post to learn how we and the rest of the world have been hoaxed by promoters of fake ideas.
He writes:
PISA is a masterful magician. It has successfully created an illusion of education quality and marketed it to the world. In 2018, 79 countries took part in this magic show out of the belief that this triennial test accurately measures the quality of their education systems, the effectiveness of their teachers, the ability of their students, and the future prosperity of their society.
PISA’s magical power in the education universe stems from its bold claims and successful marketing. It starts by tapping into the universal anxiety about the future. Humans are naturally concerned about the future and have a strong desire to know if tomorrow is better than, or at least as good as, today. Parents want to know if their children will have a good life; politicians want to know if their nations have the people to build a more prosperous economy; the public wants to know if the young will become successful and contributing members of the society.
PISA brilliantly exploits the anxiety and desire of parents, politicians, and the public with three questions (OECD, 1999, p. 7):
- How well are young adults prepared to meet the challenges of the future?
- Are they able to analyse, reason and communicate their ideas effectively?
- Do they have the capacity to continue learning throughout life?
These words begin the document that introduced PISA to the world in 1999 and have been repeated in virtually all PISA reports ever since. The document then states the obvious: “Parents, students, the public and those who run education systems need to know” (OECD, 1999, p. 7). And as can be expected, PISA offers itself as the fortuneteller by claiming that:
PISA assesses the extent to which 15-year-old students, near the end of their compulsory education, have acquired key knowledge and skills that are essential for full participation in modern societies. … The assessment does not just ascertain whether students can reproduce knowledge; it also examines how well students can extrapolate from what they have learned and can apply that knowledge in unfamiliar settings, both in and outside of school. This approach reflects the fact that modern economies reward individuals not for what they know, but for what they can do with what they know. (OECD, 2016, p. 25).
This claim not only offers PISA as a tool to sooth anxiety but also, and perhaps more importantly, makes it the tool for such purpose because it helps to knock out its competitors. As an international education assessment, PISA came late. Prior to PISA, the International Association for the Evaluation of Educational Achievement (IEA) had already been operating international assessments since the 1960s, offering influential programs such as TIMSS and PIRLS. For a start-up to beat the establishment, it must offer something different and better. That’s exactly what PISA promised: a different and better assessment…
However, the claim, the foundation upon which PISA has built its success, has been seriously challenged. First, there is no evidence to justify, let alone prove, the claim that PISA indeed measures skills that are essential for life in modern economies. Second, the claim is an imposition of a monolithic and West-centric view of societies on the rest of the world. Third, the claim distorts the purpose of education.
Made-up Claim
The claim that PISA measures knowledge and skills essential for the modern society or the future world is not based on any empirical evidence. Professor Stefan Hopmann of the University of Vienna writes:
There is no research available that proves this assertion beyond the point that knowing something is always good and knowing more is better. There is not even research showing that PISA covers enough to be representative of the school subjects involved or the general knowledge-base. PISA items are based on the practical reasoning of its researchers and on pre-tests of what works in most or all settings — and not on systematic research on current or future knowledge structures and needs. (Hopmann, 2008, p. 438).
In other words, the claim was just a fantasy, an illusion, entirely made up by the PISA team. But PISA keeps repeating its assertion that measures skills needed for the future. The strategy worked. PISA successfully convinced people through repetition…
Although PISA claims that it does not assess according to national curricula or school knowledge, its results have been interpreted as a valid measure of the quality of educational systems. But the view of education promoted by PISA is a distorted and extremely narrow one (Berliner, 2011; Sjøberg, 2015; Uljens, 2007). PISA treats economic growth and competitiveness as the sole purpose of education. Thus it only assesses subjects — reading, math, science, financial literacy, and problem solving — that are generally viewed as important for boosting competitiveness in the global economy driven by science and technology. PISA shows little interest in other subjects that have occupied the curricula of many countries such as the humanities, arts and music, physical education, social sciences, world languages, history, and geography (Sjøberg, 2015).
While preparing children for economic participation is certainly part of the responsibility of educational institutions, it cannot and should not be the only responsibility (Labaree, 1997; Sjøberg, 2015; Zhao, 2014, 2016). The purpose of education in many countries includes a lot more than preparing economic beings. Citizenship, solidarity, equity, curiosity and engagement, compassion, empathy, curiosity, cultural values, physical and mental health, and many others are some of the frequently mentioned purposes in national education goal states. But these aspects of purpose of education “are often forgotten or ignored when discussions about the quality of the school is based on PISA scores and rankings” (Sjøberg, 2015, p. 113).
Zhao presents a devastating critique of the validity of PISA. It is a must read.
Diane Ravitch's blog 
Love Dr. Zhao’s work. He sure puts PISA in its place … a boondoggle for the few, and bondage for the rest.
Advice: Don’t take PISA seriously. PISA like all those tests promote FEAR based on falsities.
Response to PISA competition: The road to wellbeing (for all) will never be paved by beating someone else.
Clinging to the illusion of standardized tests make a difference is simply a distraction to avoid dealing with some real issues: climate change, advanced AI that will eliminate jobs and income inequality. Believing that standardized tests will “save” us is a neurotic response to all the things we cannot control.
“Nonstandard Deviation”
(versification of Yong Zhao – aka “The Zhao
of Education”)
Deviation from the norm’s
Anathema to school “reforms”
But variance is future’s seed
It’s NOT a thing that we should weed
“The Zhao of Education”
Graffiti on the wall
Says “Question one and all”
As Zhao of Ed
Has wisely said
“Without it, we will fall”
“The Zhao of Education”(2)
The Emperor told the Zhao
“The Test will boost the DOW”
The Zhao replied “You’re naked”
“The Test is simply fake ed”
Good one, SDP!
“The Leaning Tower of PISA”
Rigor is de rigueur
Testing is the norm
PISA is the figure
Of meaningless “reform”
Perfectly succinct: meaningless.
These tests are nothing but psychological warfare attacking children for marketing purposes….to sell a product. How do ad persons and marketing people know how to “tap” into the fears and anxieties of a society?….. by employing trained psychologists. Boy, it’s all starting to come together now. Abuse the children to scare the public, then use the abused children AND the high anxiety public to make money off of counseling (APA) and medication(Big Pharma). It’s a cottage industry! It’s a win/win for everyone except the damaged children….which in turn leads to a damaged society. Maybe I’m wearing my tin foil hat today?….
The American Psychological Association , the largest organization of psychologists in the United State already showed their true colors by aiding and abetting torture under the Bush/Cheney regime.
There is no reason for anyone to believe anything they say about standardized testing or anything else.
You are so right!
He nails it. PISA test is nothing more than a bubble-gum sheet. It’s tasty at first, but it then dies out before you knew it.
Please share Yong Zhao’s scintillating presentation at NPE’s 2015 Conference in Chicago: https://vimeo.com/126604445
The testing schemes are ludicrous. Here’s Ohio’s:
“Next year, the program meant to help students escape being stuck in failing schools will grow further, to more than 400 districts, which represents more than two-thirds of the districts in the state.
Even Solon, always at the top of state test score rankings, has a school considered failing and whose students are now eligible for vouchers. Next year, add a school in each of the high-scoring Brecksville-Broadview Heights and Mayfield districts.”
They changed the scheme to measure public schools in Ohio (again) and just designated 2/3’s of the public schools in the state as failing- which makes them eligible for vouchers to benefit private schools. The public school kids who took these tests? They got nothing out of two weeks of grim, mindless data entry that was then taken and used for the political purpose of expanding subsidies to private schools.
Once again ed reformers contribute absolutely nothing to 90% of the students in this state, and focus exclusively on the private and charter schools who meet their ideological requirements. Public school students get hurt, private school students benefit.
If your kids were mandated to take an Ohio state test in their public school be aware that test was designed NOT for the benefit of public school students, but instead was designed to achieve the ideological goals of the ed reform movement and expand vouchers. The public school kids who spent two weeks filling out forms? They got nothing out of this deal. It was ALL politics.
https://www.cleveland.com/news/2019/12/increase-in-private-school-tuition-vouchers-is-costing-districts-and-soon-you.html
And you know the tests are meaningless. You know this because ed reformers change the benchmarks constantly.
If Ohio can go from having 2/3’s of school “pass” the ed reform metric to 2/3’s of schools “fail” the metric with the stroke of a freaking pen than that’s not “science”- it’s politics.
They’re using your kids for their ideological agenda. The score is of no practical use to any regular person. How can it be? They pay themselves to change it every freaking year! It’s guaranteed contracting employment for the rest of their lives- no sooner does the last junk measurement system go in that they’re all concocting another one.
Endless. For 20 years.
The tests are a political tool. The tests based on the CCSS have never been subjected to the scrutiny of the norming process.. The cut scores can can go up or down like the waves in the ocean. If politicians want to fail a lot of students the cut score will be high. If they want to cast a wider net, the cut score will be lower. It’s politics!
and if the testing companies want more money….and that is always a sure thing….the scores are easy enough to manipulate before convincing the district/state that surely MORE TESTING will be required
BTW, I just read that MSNBC is hosting a forum on public education with six of the Democratic candidates on December 14th. I wonder if they will continue to avoid asking the questions we all want them to answer.https://www.nbcnews.com/politics/politics-news/msnbc-host-december-forum-education-issues-2020-democrats-n1095126
Zhao calls PISA “a magician” but magicians perpetrate their illusions on the public as fun entertainment.
PISA is dead serious which makes it not magic, but fraud.
Do people question the crazy swings in the numbers?
So 1/3 of Finland suddenly stopped learning to read? Was there some kind of public health catastrophe? I think what bothers me most is the absolute FAITH in these numbers. Half these people work for universities. Aren’t they supposed to be intellectually curious?
Universities have been co-opted by dollars.
So called “researchers” are funded by billionaires and corporations so it is no surprise when their results align with the claims of the latter and also no surprise when the so called “researchers” cherry pick data that supports their thesis and ignore any data that do not.
I term them so called because they are not real researchers, but frauds.
The WalMart heir-funded 74 has another hit piece on Elizabeth Warren:
https://www.the74million.org/article/74-interview-howard-fuller-on-schooling-elizabeth-warren-about-charters-african-american-families-school-choice-her-education-plan/
I don’t mind that they lobby exclusively for charters and vouchers and against public schools. That’s an ideological position and they’re allowed to hold it. Just please stop pretending this is about “public education”. It can’t be. They exclude 90% of students, families and schools from their advocacy.
This is a “charter and voucher” movement. Which is fine! It’s privately funded and the Waltons can do anything they want with their money (other than pay their employees a decent wage, I guess). But just be charter and voucher promoters and leave public school students out of it! You return no value to public school students. You’re either irrelevant or actively harmful to our kids.
After the forum for black women in Houston, many women were asked what candidate impressed them. They overwhelmingly mentioned Elizabeth Warren.
“However, the claim, the foundation upon which PISA has built its success, has been seriously challenged. First, there is no evidence to justify, let alone prove, the claim that PISA indeed measures skills that are essential for life in modern economies. Second, the claim is an imposition of a monolithic and West-centric view of societies on the rest of the world. Third, the claim distorts the purpose of education.”
Yes, and these three points apply to all of the standardized tests foisted on students in the United States.PISA and OECD forward atruncated view of education as little more than a national and global economic asset…or not. That view is also present in the United States.
Here is the current US Department of Education Mission Statement:
Our mission is to promote student achievement and preparation for global competitiveness by fostering educational excellence and equal access.”
Here is some little known, and fairly recent history bearing on this statement..
First, The very last section of the America Competes Act (2007) anticipates the impending meltdown of the global economy with the crisis of 2008 on the horizon. The “sense of the Senate” in the following section of the America Competes Act seems to be offered as if a roadmap for surviving the coming financial implosion.
SEC. 8007. SENSE OF THE SENATE REGARDING CAPITAL MARKETS. It is the sense of the Senate that—(1) Congress, the President, regulators, industry leaders, and other stakeholders should take the necessary steps to reclaim the preeminent position of the United States in the global financial services marketplace;
(2) the Federal and State financial regulatory agencies should, to the maximum extent possible—
(A) coordinate activities on significant policy matters, so as not to impose regulations that may have adverse unintended consequences on innovativeness with respect to financial products, instruments, and services, or that impose regulatory costs that are disproportionate to their benefits; and
(B) at the same time, ensure that the regulatory framework overseeing the United States capital markets continues to promote and protect the interests of investors in those markets; and
(3) given the complexity of the financial services market-place, Congress should exercise vigorous oversight over Federal regulatory and statutory requirements affecting the financial services industry and consumers, with the goal of eliminating excessive regulation and problematic implementation of existing laws and regulations, while ensuring that necessary investor protections are not compromised.
It is precisely on August 9, 2007, the day this bill was passed, that the financial meltdown of 2008 began. Members of the Congress, especially the Senate, knew the nation was at risk of a financial meltdown. Two days later, on 9 August 2007, BNP Paribas became the first major financial group to have a “liquidity “ problem—no money to cover debt from sliced and diced sub-prime financial instruments.
The America Competes Act was passed under the administration of George W. Bush, with Margaret Spellings Secretary of Education. Something else happened within one month after the America Competes Act passed. The Mission statement of the US Department of Education was formally changed.
Here is what I found via the “Wayback Machine:”
USDE Mission: September 10 2007. “Despite the growth of the Federal role in education, the Department never strayed far from what would become its official mission: to ensure equal access to education and to promote educational excellence throughout the Nation.”
only two days later
USDE Mission: September 12, 2007. “Despite the growth of the Federal role in education, the Department never strayed far from what would become its official mission: to promote student achievement and PREPARATION FOR GLOBAL COMPETITIVENESS by fostering educational excellence and ensuring equal access.”
The economy did not care about the rhetoric in this bill. It tanked.
Global Competition is the official mission of public education in the USA. Student achievement is pointless unless it is serves that end. This is a pitiful and narrowminded view of the purpose(s) of education and it is not radically different from OECD.
I would like to see what every presidential candidate would offer as a mission statementfor USDE.
Hahahaha. Love Yong Zhao. Hanushek uses 40 yrs of pre-PISA TIMSS/ PIRLS data to “prove” that raising PISA scores will bring untold riches to the world’s economies. And doesn’t even do that right! 😀
Noel Wilson debunked PISA even as it was being developed. Wilson’s never refuted nor rebutted 1997 dissertation destroys standardized testing, annihilates the onto-epistemological foundation of the standardized testing malpractice. It can be found at: https://epaa.asu.edu/ojs/article/viewFile/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other words all the logical errors involved in the process render any conclusions invalid.
The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self-evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.