When I learned that the latest PISA (Program on International Student Assessment had been released, I attended a webinar, where I learned once again that the scores of U.S. 15-year-old students were somewhat below the international average. The PISA tests in math, reading, and science have been offered since 2000, sponsored by the Organization for Economic Co-operation and Development.

My takeaway from the webinar was that we should try to be more like Singapore and Macau.

I have studied the results of international assessments such as PISA and TIMSS for years. Eventually, I began to wonder what the connection was—if any—between the test scores of 15-year-old students and the economic productivity of their nation 10, 15, 20 years later. We’ve been bemoaning our scores since the first international tests were given in the 1960s, even as our economy soars way beyond the nations with higher scores on the tests.

I invited Yong Zhao to share his reaction to the latest PISA scores. His response was as brilliant as I anticipated.

Yong Zhao is one our most accomplished scholars of education. Born in China to an impoverished family, he pursued his dreams, migrated to the United States, and has made his mark as a creative and innovative thinker. He is currently a Foundation Distinguished Professor of Education at the University of Kansas and holds an appointment as Professor of Educational Leadership at the University of Melbourne. His list of honors and publications is too long for me to recite here. But you can find it online.

Yong Zhao wrote:

It doesn’t make sense: Why Is the US Still Taking the PISA?

I have always wondered what America has got from participating the PISA every three years. Since 2000, the U.S. has been taking part in this nonsensical global academic horse race. Every time it took the test, American students stood at about the middle of the global league table. Every time the results were released, American media would point out how American students are not the best, but East Asian education systems such as China, Hong Kong, Chinese Taipei, South Korea, Japan, and Singapore are the best. And then U.S. authorities would invite PISA and other pundits to tell us how to improve American education.

The same story has been going on for more than two decades, but American education has not improved, at least according to the PISA scores. According to the most recent results (NCES, 2023), American students did much worse in math in 2022 than in 2003, with an 18-point decline from 483 to 465. Their reading and science scores, however, remained about the same without significant change over the past two decades. Although PISA experts largely blame the COVID pandemic as the reason for the decline in math, it does not make much sense because there is no decline in reading and science. Did COVID-19 only affect math, not science and reading? Of course, one can try to argue that reading and science are much less sensitive to COVID, but why? 

Basically, the international standing of the US and the test scores of its students have not changed much. Whatever the PISA data revealed and/or the lessons from other countries such as China, Japan, Singapore, or Finland have not helped improve America’s PISA scores. By the way, Finland, the country Americans view with the best education system because of its early stunning PISA performance, has seen a much more dramatic decline in its PISA scores: from 544 to 484—a 60-point decline in math, from 546 to 490—a 56-point decline in reading, and from 563 to 511—a 52-point decline over the past two decades. Not sure if America still views Finland as the best education country, but its scores have dropped to almost the same point as American students. 

In fact, other than Finland, the PISA league tables have not changed much either. East Asian education systems have consistently remained the top performers and the OECD countries’ average scores have been dropping. If PISA had any impact on the world’s education quality and equity, education should not be the same as 20 years ago.

PISA does not really have much to offer to anyone, except those who benefit from the test itself—the consultants, the test makers, the data processors, and possibly some education politicians.

In a review article (Zhao, 2020), I summarized the research about  PISA and found: 1) PISA markets itself as an assessment of abilities needed in the 21st Century, but it is the same as other international tests such as TIMSS, 2) PISA ignores the overall educational purposes of different countries by primarily assessing math, reading, and science, 3) PISA’s tests are not of high quality with numerous theoretical and technical problems, and 4) PISA’s sampling has been manipulated in different countries. My conclusion is that instead of bringing positive changes to the world, PISA wreaked havoc.

America has never excelled in international tests since the beginning of such assessment in the 1960s, but the low scores have not seemed to affect it much. In fact, a correlational analysis done in 2007 showed a negative correlation between international test scores and economic development (Baker, 2007). That is, countries with higher scores in the first international study did worse than countries with lower scores. If PISA or any other international tests truly measure what matters in education, America should no longer be a developed country. On the contrary, East Asian countries have always scored well in international assessments, but their economic development has been more related to economic, political, and international orders than their test scores.

What matters to economic development and prosperity is perhaps the non-cognitive factors that PISA does not typically emphasize. For example, in an analysis, I found that PISA scores are negatively correlated with entrepreneurship confidence across countries (Zhao, 2012b). American students, despite their lower scores, have always had more confidence than their peers in other countries. In fact, confidence has been found to have negative correlations with test scores (Zhao, 2012b, 2014, 2018b). High score education systems, except Finland, have always had a negative impact on students’ social and emotional wellbeing (Zhao, 2012a). Even PISA’s own data show that PISA scores are negatively correlated with life satisfaction of students (OECD, 2019).

Many education systems participate in PISA because they are fooled by its claim to measure global competitiveness. Somehow these educational systems are convinced that their PISA scores and rankings mean how competitive they are globally. But this is not true and cannot be true. In 2022, over 80 education systems took part in the PISA but these systems are hugely different. For example, the U.S. has three hundred million people and does not really have an education system (it has over 50 education systems based on the number of states and over 12,000 systems if we treat each school district as a system). How can it be compared with Macao, China, a tiny place with about 688,000 people and one education system? Likewise, how can the U.S., with a per capita GDP of over $70,000 be compared with Albania, whose per capita GDP is about $6,000.

Moreover, PISA has been operational for over 20 years. The first cohort of 15-year-old students took the test in 2000. If PISA truly has predictive power, it should have produced a longitudinal study to show how these students do in society. They are about 39 years old today. But we haven’t seen any such report except the wild guesses made by some scholars (Hanushek & Woessmann, 2010).

If PISA offers nothing, why does the U.S. spend the money and effort to join the game? For monitoring of basic education conditions, it already has the National Assessment of Education Progress (NAEP) or the national report card, which has been in existence since 1969. Why continue to participate in PISA?

Frankly, it’s inexplicable, for there is truly no reason the U.S. should continue to participate in PISA, let alone to pretend to learn from high performing countries. The lessons PISA offered have not been productive. For example, the lesson that high performing systems (e.g., Singapore, South Korea, and Finland) recruit high performing high school graduates to be teachers (Barber & Mourshed, 2007) is not based on real evidence and does not really produce better education outcomes (Gronqvist & Vlachos, 2008). The lesson that high performing systems have clear definitions of learning expectations, a good structure of different stages, and tough measures to ensure that students have met the expectations (Tucker, 2011) is intended largely to copy East Asian education systems; but, ironically, the East Asian countries have been working very hard to change these practices (Zhao, 2014). International learning may make sense sometimes, but there are great limitations (Zhao, 2018a). American education should focus on developing its own way to improve education instead of trying to catch up with others (Zhao, 2009). 

This is not to say that American education is perfect. Rather, it is to say the way forward is not to look at what others have been doing. The U.S. needs to solve its own problems and work on creating a better future. With the emergence of ChatGPT and other generative AI tools, the world has changed again. If ChatGPT had taken the 2022 PISA, it is highly likely that it would outscore all the students in the world. It would be the best education system accordingly. Today, many students use AI tools to do their schoolwork, and teachers use AI in their teaching. PISA has become even more irrelevant.

Since 2000, our scores on PISA have barely changed. While there’s much chatter about learning from other systems, it has not happened. There is no reason that the U.S. should continue its participation in PISA.

References:

Baker, K. (2007). Are International Tests Worth Anything? Phi Delta Kappan, 89(2), 101-104. 

Barber, M., & Mourshed, M. (2007). How the World’s Best-Performing School Systems Come out on Top. Retrieved from New York: https://www.mckinsey.com/industries/social-sector/our-insights/how-the-worlds-best-performing-school-systems-come-out-on-top

Gronqvist, E., & Vlachos, J. (2008). One size fits all? The effects of teacher cognitive and non-cognitive abilities on student achievement. Retrieved from Stockholm, Sweden: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1311222

Hanushek, E. A., & Woessmann, L. (2010). The High Cost of Low Educational Performance: The Long-run Economic Impact of Improving PISA Outcomes. Retrieved from Paris: http://books.google.com/books?id=k7AGPo0NvfYC&pg=PA33&lpg=PA33&dq=hanushek+pisa+gdp&source=bl&ots=2gCfzF-f1_&sig=wwe0XLL5EblVWK9e7RJfb5MyhIU&hl=en&sa=X&ei=MLPCUqaOD8-JogS6v4C4Bw&ved=0CGcQ6AEwBjgK#v=onepage&q=hanushek%20pisa%20gdp&f=false

NCES. (2023). Program for International Student Assessment (PISA). Retrieved from https://nces.ed.gov/surveys/pisa/index.asp

OECD. (2019). PISA 2018 Results (Volume III): What School Life Means for Students’ Lives. Retrieved from https://doi.org/10.1787/acd78851-en.

Tucker, M. (Ed.) (2011). Surpassing Shanghai: An Agenda for American Education Built on the World’s Leading Systems. Boston: Harvard Education Press.

Zhao, Y. (2009). Catching Up or Leading the Way: American Education in the Age of Globalization. Alexandria, VA: ASCD.

Zhao, Y. (2012a, December 11). Numbers Can Lie: What TIMSS and PISA Truly Tell Us, if Anything?  Retrieved from http://zhaolearning.com/2012/12/11/numbers-can-lie-what-timss-and-pisa-truly-tell-us-if-anything/

Zhao, Y. (2012b). World Class Learners: Educating Creative and Entrepreneurial Students. Thousand Oaks, CA: Corwin.

Zhao, Y. (2014). Who’s Afraid of the Big Bad Dragon: Why China has the Best (and Worst) Education System in the World. San Francisco: Jossey-Bass.

Zhao, Y. (2018a). Shifting the Education Paradigm: Why International Borrowing Is No Longer Sufficient for Improving Education in China. ECNU Review of Education, 1(1), 76-106. 

Zhao, Y. (2018b). What Works May Hurt: Side Effects in Education. New York: Teachers College Press.

Zhao, Y. (2020). Two decades of havoc: A synthesis of criticism against PISA. Journal of Educational Change, 1-22. doi:10.1007/s10833-019-09367-x