There has long been concern that the PISA international tests are “fixed.” Years ago, Tom Loveless of the Brookings Institution criticized PISA for ignoring the selection of Chinese students that are tested. Loveless pointed out that students in Shanghai are in no way representative of students in China.
In this article, Gary Sands reviews the problems of sampling from each country and how some countries can “rig” the samples, which invalidate the results.
Read the article for the details.
Sand concludes:
At face value, the PISA results appear to be a huge propaganda victory for the educational systems of Asians and the Chinese. But the real danger in widely circulating the PISA results lies not in fooling thousands of headline readers around the world, but in the complicit cover-up of the huge disparities in education among Asian provinces. Almost two-thirds of all Chinese children live in rural areas, where school attendance rates can be as low as 40%. A survey by the China Association for Science and Technology showed only 6.2% of the Chinese people held basic science literacy in 2015.
By allowing countries to potentially rig the test, the OECD is failing in its mandate to help governments foster prosperity by providing information. In the case of the bizarre B-S-J-G grouping, PISA administrators have again made another exception for China—just as many foreign businesses have been forced to do—in the hope that the nation will eventually play by the rules. Perhaps the OECD’s intention in allowing cities and groups of cities/regions to compete is to coax China and others into eventually releasing nationwide results.
But in so doing, the OECD is merely kowtowing to Beijing, acquiescing in the samples submitted by other countries and sending a message to our children that bending the rules is acceptable.
Diane Ravitch's blog 
You mean beyond the fact that all standardized tests are rigged in favor of the affluent?
LikeLike
Life is rigged in favor of the affluent.
LikeLike
Well, yes, there is that….
LikeLike
We can always count on China to lie, cheat, steal or game the system to make it appear as though they are superior. This provides a distorted picture to the rest of the world.
In the last post I thought it was interesting that our flat NAEP scores may be due to increasing poverty and the fact that high school graduation rates are higher. Therefore, more struggling students are taking the NAEP. Scores are generally lower when a larger number of students take a test.
LikeLike
Even China’s leaders said the same thing Tom Loveless of the Brookings Institution said about the PISA test administered in Shanghai. The clearly said that Shanghai does not represent China and most of China was way behind Shanghai’s public education system.
Shanghai is considered to have the best public education system in all of China. Back before NCLB and the high stakes rank and punish testing mania swept across the US like a terminal cancer, teams of educators from Shanghai came to the U.S. to see what our public schools were doing. They took what they learned back to China and implemented the methods they agreed with. Shanghai’s performance on the PISA represents those changes in Shanghai’s public education system.
In addition, mandatory education in China is for nine years and not thirteen like it is in the U.S. Students that come from families that want their children to have more than nine years of education must compete by taking a high stakes tests that ranks them. The top scorers are allowed to continue their education through 12th grade.
That means the 15-year-olds in Shanghai,China that took the PISA test were all highly motivated grade level students that passed that test taken before the eighth grade to determine who was allowed to continue on through grades 9 to 12 voluntarily since it is not mandatory after 8th grade to stay in school.
Many rural children in China drop out of public education at the end of 6th grade and go no farther. The rural population in China is about half of the country’s population now and rural Chinese were allowed to have two children a family vs one child a family in urban China.
Comparing Shanghai, China to any other country or city in the world is no different than comparing sweet, juicy naval oranges to the taste of stinky tofu.
LikeLike
I visited Shanghai in January. It is like going to Paris or London or New York. I was in Shanghai many years ago (around 1987), and there were many slums. The slums are gone, replaced by ritzy, glitzy high-rise apartment buildings. There are many Western luxury shops, and the streets are packed with Mercedes, Rolls-Royces, and other luxury cars. One day we hired a young woman as a guide and she told us she commutes 90 minutes into the city. The government decides where everyone is allowed to live, and her family was not allowed to live in Shanghai. The government has pushed poverty far away from Shanghai.
LikeLiked by 1 person
Shanghai like most if not all of China’s cities have extensive commuter rail and bus networks to move people from the suburbs into the city’s heart.
Between 1999 and 2008, I visited China nine or ten times with my former wife who was born in Shanghai and grew up there during Mao’s Cultural Revolution. Even though we are no longer married, we are still friends. If you want to read what it was like in that era, Anchee wrote a memoir called “Red Azalea” that was named a New York Times Notable Book of the Year and won the Carl Sandburg Award. It came out in the 1990s and is still in print. The book was translated into more than thirty languages and has sold millions of copies.
I’ve been all over China and all over Shanghai. During our marriage, Anchee had a three bedroom, two bath flat in one of the closer suburbs surrounding Shanghai. She sold it last year.
It is correct that China’s government attempts to control where people live but I’ve talked to enough people in China that my wife knew to know that where there is a will, there is a way to get around all the regulations. The Chinese people are not that easy to manage.
And that includes slipping past the internet censors until they catch you and blog your escape route.
LikeLiked by 1 person
I’ve either got that book on my shelf somewhere or I read it in decades past. Nice to know how to connect it to THIS world 🙂
LikeLike
the Shanghai model of how to change cities now much touted by rich investors and so-called “progressive” politicians
LikeLike
To answer the question of the post, albeit with a question:
Who cares?
Why care about COMPLETELY INVALID test results?
LikeLike
I have to disagree here in terms of degree. Allow a reference to Galton, who “proved” the superiority of Europeans by falsifying data on skull measurements. Here we have two sins against logic. Galton’s original premise was false, there is actually no relationship between head size and intelligence. But he went further and compared European skull size to female African skull size, not mentioning that he had done that. This second sin seems more egregious that the first to me.
In a similar way making test data of crap is bad enough, but then falsifying it conscious of the act seems more damnable.
LikeLike
And the solution?
Quite easy, Holmes, quit doing the tests.
LikeLike
The comparison of US PISA scores to China’s is like that of NYC tradl publicsch stdzd test scores to those of SA.
LikeLike