Assessment experts Judith Singer and Henry Braun wrote an article for Science warning about the risks of misusing the results of international tests. The article is behind a paywall, and I don’t have a subscription. Singer was interviewed by a writer for The 74, and she expressed her concerns. The bottom line is that the rankings distort more than they reveal.

“The rankings that are commonly used to report the results of [international tests] draw headlines, but they are often incredibly misleading,” she told The 74. “The countries aren’t sports teams to be ranked as winners and losers.” Indeed, she observed, the British press uses the same term to describe the hierarchy of international testing performance — “league tables” — as for soccer and rugby standings.”…

“Worse than the alarmism accompanying news stories, Singer says the rankings themselves are frequently arbitrary and mercurial. Positions change from year to year for reasons having little or nothing to do with student performance in a given country. And the rules of the tests allow for a certain amount of gamesmanship, as when Shanghai earned a top ranking for math in the 2012 PISA exam — only for the world to later discover that it had excluded 27 percent of its 15-year-olds from taking it.

”On the 2015 PISA, Japan improved on its fourth-place ranking for scientific literacy three years earlier, moving to second overall. But the jump wasn’t because of improved performance; scores actually went down, though not as much as other countries’.

“In a Japanese news item on the results, a graph shows scores and rankings over time. A line representing the country’s science ranking ascends from 2012 to 2015 — even though actual scores dropped by nine points.

“Deep-seated national differences also tend to skew our perceptions of who’s up and who’s down. It doesn’t really make sense, Singer remarked, to group countries with decentralized education sectors — like the United States, Canada, and Germany — alongside those with properly national school systems, such as France, that can mandate instructional and curricular choices at will across their entire student populations…

““Singapore has fewer schools than Massachusetts has school districts,” Singer said. “So when you look at the results of Singapore — which is a city-state, though it’s treated as a country — you’re talking about a very small jurisdiction. There are undoubtedly school districts in Massachusetts that far exceed the performance of Singapore.”

“Drawing apples-to-apples comparisons among disparate countries with wildly varying educational approaches leads to false narratives about what produces success, with low-performers looking to emulate the “special sauce” driving high achievement — whether it’s special curricula, smaller class sizes, or something else — in high-flying countries like Finland or Korea.”

“Rather than spending millions trying to ape the tactics of international competitors, Singer says that countries should use testing data to learn more about themselves.“