As I have written many times, the scores on international tests have no predictive value about our economy. If they did, we would be a third world country by now since our scores have always been mediocre to very low on those tests. To learn more, read my chapter on international scores in “Reign of Error.”
Now, surprisingly, an article appears in Forbes challenging the PISA methodology and asserting that Asian nations get to choose their best students.
This false information has been used by the media and “reformers” to bash public schools and promote privatization.
Shameful. A hoax.
Diane Ravitch's blog 
I do not understand: what’s hoax? The info in the Forbes article about Asian countries or the connection between test scores and economy?
LikeLike
I could not get to the Forbes article. The Hoax is saying that test scores of 15-year-olds predict or have a siginifance causal relationship with the economy.
PISA tests are given to 15-year-old students in mathematics, science and reading skills. The cycles of tests completed between 2000 and 2012 were completed on paper.
But, for the PISA 2015 tests, 58 of the 72 countries/economies administered the PISA test using computers.
There are known differences in how students respond to tests on computers and on paper http://theconversation.com/how-shift-to-computer-based-tests-could-shake-up-pisa-education-rankings-54869
PISA 2015 is a computer-based assessment. That means all questions appear on the computer screen and students respond by clicking, dragging and dropping or typing their answer.
Here are some PISA 2015 RELEASED FIELD TRIAL COGNITIVE ITEMS produced by one of the contractors for PISA tests, the US based ETS—Educational Testing Service.
Other contractors for the PISA tests are Weststat (US); Pearson (UK) and DIPF(The German Institute for International Educational Research). Remember that the tests are given to 15-year-olds.
Click to access PISA2015-Released-FT-Cognitive-Items.pdf
Here are some reading questions from 2009. Remember that the tests are given to 15-year-olds.
Click to access PISA2009-samplereadingquestions.pdf
May I suggest that every pundit who fixates on such tests as predictors of this nation’s economic fate do a couple of things.
Take the test and get it scored.
Publish your score.
Produce a chart that shows where that score lands you relative to the scores of 15-year-olds in the USA and in other nations.
Puvlish your economic assumptions and your assumptions about test scores.
LikeLike
Laura,
You said you could not get the Forbes article.
Here it is:
The results of the most recent Programme for International Student Assessment (PISA) are out, and again reveal students from East Asian countries dominate in maths and sciences. The assessment, run by the Paris-based Organisation for Economic Co-operation and Development (OECD) every three years, tested around 540,000 15-year-old students in 72 countries and economies on science, reading, maths and collaborative problem-solving. In 2015, students from Singapore performed the best, followed by Japan, Estonia, Chinese Taipei, Finland, Macao (China), Canada, Vietnam, Hong Kong (China) and B-S-J-G (China).
On September 8, 2016, a young girl writes Chinese characters at an elementary school in Zhujiajiao, on the outskirts of Shanghai. (FRED DUFOUR/AFP/Getty Images)
Western countries fared far worse, with concerns over declining educational standards seemingly confirmed. The U.S., the world’s favorite destination for international university students, saw its scores remain close to the OECD average on each category. The 5,700 15-year-olds who took the test scored 496 in science and 497 in reading, and 470 in math—close to the OECD averages of 493, 493 and 490 respectively.
Some argue this may herald a new era of creative thinking in Asia, and look to replicate an Asian model of learning. Others point to a culture in Asia which values education highly and is driven by “Tiger Moms” who push their children to excel in their studies. In Vietnam, some students are reportedly too busy with school and studying to take meals at home and eat while on motorbikes. Still others argue government investment in educational infrastructure, such as building more schools and attracting the best teachers, yields the greatest results. But before we accept the results at face value, we should delve more deeply into Pisa’s problematic methodology, which are two-fold.
First, as you may have gleaned from the top ten results, instead of choosing one consistent geographical entity, PISA selects a sample that “represents the full population of 15-year-old students in each participating country or education system.” What this means in practice is the ability of some education administrators choosing their top-performing students from smaller samples in cities or city-states such as Chinese Taipei, Macao, Hong Kong and Singapore. While most of the other results came from a sample of scores around nations, some countries such as Argentina and China were allowed to take their sample from their most educated cities or regions. This year, Chinese administrators chose their students from a group of cities and regions aptly named B-S-J-G, after Beijing, a province-level municipality, Jiangsu, a province on the eastern coast of the country, Guangdong, a southern coastal province, and Shanghai, a province-level municipality. Previously, Chinese authorities had chosen Shanghai as mainland China’s sole representative, whose students finished at the top of all three subject areas in both the previous two PISA studies in 2012 and 2009.
Now, if all countries were to take this approach, we would see London selected to be the sole representative of Britain, or Boston and its suburbs representing the U.S. This year, in fact, saw a separate score calculated for Massachusetts, which if taken as the nation’s results, would grab the top spot in reading with eight other nations, 2nd place in science with ten other nations, and 12th in math.
If we dig deeper into the sampling, we come across another potential problem with the PISA testing: that the sampling done on mainland China (Beijing, Jiangsu, Guangdong and Shanghai) and other cities was not taken from a wide variety of schools. Rather, the very best schools were chosen and the very best students were cherry-picked from those schools. Ong Kian Ming, a lawmaker in Malaysia recently raised this issue concerning Malaysia’s PISA results, claiming the education ministry attempted to rig the sample size in order to boost the scores. Ong added that the biased sample of schools in favor of high-performing schools can also be seen in Pisa 2015’s own data on Malaysia. Ong claims a concerted effort to take more samples from higher-performing fully residential schools: “Out of a total sample of 8,861 students, 2,661 or 30% were from fully residential schools. This is clearly an over sampling of students from fully residential schools since they comprise less than 3% of the 15-year-old cohort in 2015.”
Indeed, the sampling process has some flexibility, with each country or education system submitting a sampling frame to a research firm, which contains all age-eligible students for each of its schools. An independent research firm then draws a scientific random sample of a minimum of 150 schools with two potential replacements for each original school.
Since each country or education system is responsible for recruiting the sampled schools, if one of the randomly chosen schools refuses to participate, for any reason, the country or educational system can choose from up to two neighboring schools. Replacement schools can represent up to 35% of the sampling frame. Once the schools are chosen, each country or education system submits student listing forms. On test day, student participation must be at least 80%.
Which begs some questions, such as, did some of the weaker students not take the test? Could some of the weaker schools have refused to participate? Only an independent audit of the results could prove conclusively whether the very best fruit was picked from the tree, but the sampling process does not preclude the potential for manipulation.
At face value, the PISA results appear to be a huge propaganda victory for the educational systems of Asians and the Chinese. But the real danger in widely circulating the PISA results lies not in fooling thousands of headline readers around the world, but in the complicit cover-up of the huge disparities in education among Asian provinces. Almost two-thirds of all Chinese children live in rural areas, where school attendance rates can be as low as 40%. A survey by the China Association for Science and Technology showed only 6.2% of the Chinese people held basic science literacy in 2015.
By allowing countries to potentially rig the test, the OECD is failing in its mandate to help governments foster prosperity by providing information. In the case of the bizarre B-S-J-G grouping, PISA administrators have again made another exception for China—just as many foreign businesses have been forced to do—in the hope that the nation will eventually play by the rules. Perhaps the OECD’s intention in allowing cities and groups of cities/regions to compete is to coax China and others into eventually releasing nationwide results.
But in so doing, the OECD is merely kowtowing to Beijing, acquiescing in the samples submitted by other countries and sending a message to our children that bending the rules is acceptable.
LikeLike
Thanks for posting the Forbes article.
This narrative is unusual from Forbes. It is good to see how the cherrypicking of test takers operates at at the highest levels, enabled by the major contractors for PISA and OECD’s passion to rank the “economic performance” of countries based on these scores.
LikeLike
The international PISA tests only 15 year old in Shanghai. Even China’s leaders have publicly admitted that if all 15 year olds in China were tested, the average score would be much lower, and that doesn’t take into account that most children in China are out of the education system by the time they are 15. The ones who remain had to pass a rigid test to continue. The rest went to vocational schools or back to the farm often at or before the end of 6th grade.
In China, education is mandatory through the 9th grade but at the end of 6th grade there is little effort to hold on to children in the school through 9th grade.
To remain through 12th grade, children compete by taking tests at the end of 9th grade. Less than 10 percent of China’s youth go to college and they have to rank high on a test to make it. The college they are allowed to attend also depends on where they ranked on that test.
In the United States, education is mandatory through 12th grade, and every child is allowed to continue their education through public community colleges, and nearly 40 percent of working-age Americans hold a college degree putting the US among the top ten most educated countries in the world. In fact, the U.S. has about 3 college graduates competing every job that required a college degree.
LikeLike
And on another front…
It has come to my attention that in a recent test of the nation’s athletic pulse—with frightening implications for the future of all public school sports programs not to mention the very health of the entire US of A population—a random sampling of NBA players that participated in the recent playoffs, gathered into a single team with a bench 25 strong, beat every single public HS basketball team in the country put up against them. It was 100 – 0!
Every single one. Every single time. By BIGLY scores. Male and female. Even state champions and runners-up. Even those composed completely of seniors. Even those from the wealthiest neighborhoods and those whose entire roster is going to Harvard, Yale, Princeton, and MIT.
Rheeally! Symbolically if not literally!
🙄
And just where is the concern of all you defenders of the education status quo with your “better education for all” and desire for genuine learning and teaching?
Hah! Well, the solution is simple: charters, vouchers, privatization. And as quickly as possible. Before we’re buried by our BFFs in Russia and elsewhere with their pinpoint free throws and game-winning three-pointers.
But in a brief reminder of how things work in reality, here’s a Principle of Data Interpretation from the late Gerald Bracey (READING EDUCATIONAL RESEARCH: HOW TO AVOID GETTING STATISTICALLY SNOOKERED, 2006):
“4. When comparing groups, make sure the groups are comparable.”
Many thanks to the owner of this blog for giving us access to the entire piece and to the commenters for an excellent thread.
😎
LikeLike
I’m confused. Do you think we don’t want our children to be healthier and in better shape because we appear to focus only on the academic side of this issue?
Doesn’t it make sense that if stop using tests to rank and punish public school teachers and close public schools, and we get rid of the voucher movement, the corporate charter school industry, and get back to basics, that would mean more recess time and more time in physical education classes because the evidence shows this leads to better academic outcomes? A health physically fit child does better in academic classes, but the privatization movement ignores this fact because this cuts into their profits. Cut out physical education and you have less staff to pay.
LikeLike
Lloyd Lofthouse: all kidding aside, we are on the same page.
One of the casualties of corporate education reform aka “the new civil rights movement of our time” has been the health of millions of students.
Yes, curtail or cut out recess and PE and any number of other things and you guarantee that students won’t have “mens sana in corpore sano” [a sound mind in a sound body].
Thank you for the follow-up comment.
😎
LikeLike
Don’t give a damn about international standardized (sic) test scores.
Do “Give a Damn” a la Spanky and our Gang:
If you’d take the train with me
Uptown, thru the misery
Of ghetto streets in morning light,
It’s always night.
Take a window seat, put down your Times,
You can read between the lines,
Just meet the faces that you meet
Beyond the window’s pane.
And it might begin to teach you
How to give a damn about your fellow man.
And it might begin to teach you
How to give a damn about your fellow man.
Or put your girl to sleep sometime
With rats instead of nursery rhymes,
With hunger and your other children
By her side,
And wonder if you’ll share your bed
With something else which must be fed,
For fear may lie beside you
Or it may sleep down the hall.
[Chorus]
Come and see how well despair
Is seasoned by the stif’ling air,
See your ghetto in the good old
Sizzling summertime.
Suppose the streets were all on fire
The flames like tempers leaping higher
Suppose you’d lived there all your life,
D’you think that you would mind?
And it might begin to reach you
Why I give a damn about my fellow man;
And it might begin to teach you
How to give a damn about your fellow man
LikeLike
How did the generation of young people that grew up inspired by songs like this turn into the reining plutocrats that only give a dam about their stock portfolios? Eh?
LikeLike
Good question!
My conjecture would be that most people with those portfolios did not grow up with a questioning, skeptical mind as perhaps they already didn’t give a damn.
My mother tried to correct me of the “why” question. She tried to change my mind but I wouldn’t listen. Mom tried:
LikeLike
As many have point out here, all tests are invalid due to the problems that occur logically when. Anyone tries to see if someone knows something. We cannot really define the word, know, without twisting logic into mis-shapen enigma. The best we can do is to rely on competent teachers who see students every day for awhile. All testing does is make someone wealthy.
LikeLike
“The leaning tower of PISA”
PISA is leaning
The tower is falling
Test loses meaning
Cuz science is calling
LikeLike
Welcome back!
LikeLike
Whew! Such a flurry of posts Jan 9-12 I almost missed this one! I have bookmarked the article with glee. Soon to be posted on many an ed-article comment thread in response to the Henny-Penny int’l-ranking-misquoters.
LikeLike
Bethree
That is a VERY IMPORTANT ARTICLE
LikeLike