Professor Svend Kreiner, a prominent statistician and psychometrician at the University of Copenhagen in Denmark and Dr. Hugh Morrison of Queens University in Belfast have published studies blasting the reliability and validity of the PISA league tables. They describe PISA’s rankings as “useless,” “utterly wrong,” and “meaningless.”
According to TES (London),
“Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, said that an inappropriate model is used to calculate the Pisa rankings every three years. In a paper published this summer, he challenges their reliability and shows how they fluctuate significantly according to which test questions are used. He reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.
“Dr Hugh Morrison, from Queens University Belfast in Northern Ireland, goes further, saying that the model Pisa uses to calculate the rankings is, on its own terms, “utterly wrong” because it contains a “profound” conceptual error. For this reason, the mathematician claims, “Pisa will never work”.
“The academics’ papers have serious implications for politicians, including England’s education secretary Michael Gove, who justified his sweeping reforms by stating that England “plummeted” down the Pisa rankings between 2000 and 2009.
“The questions used for Pisa vary between countries and between students participating in the same assessment. In Pisa 2006, for example, half the students were not asked any reading questions but were allocated “plausible” reading scores to help calculate their countries’ rankings.
“To work out these “plausible” values, Pisa uses the Rasch model, a statistical way of “scaling” up the results it does have. But Professor Kreiner says this model can only work if the questions that Pisa uses are of the same level of difficulty for each of the participating countries. He believes his research proves that this is not the case, and therefore the comparisons that Pisa makes between countries are “useless”.
“When the academic first raised the issue in 2011, the OECD countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point. But Professor Kreiner’s new paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis.
“He is sceptical about the whole concept of Pisa. “It is meaningless to try to compare reading in Chinese with reading in Danish,” he said.
“Dr Morrison said that the Rasch model made the “impossible” claim of being able to measure ability independently of the questions that students answer. “I am certain this (problem) cannot be answered,” he told TES.”
To read Dr. Kreiner’s studies, google his name.
Diane Ravitch's blog 
Reblogged this on Transparent Christina.
Was looking at how Ohio ed reformers have skewed the playing field against public schools to exempt publicly-funded private schools from “accountability”.
Looks like Louisiana ed reformers are doing the same thing:
“According to Bulletin 133 Section 1305 (E)(3), if at any time LDOE determines that a school has “demonstrated gross or persistent lack of basic academic competence,” the school may incur penalties including ineligibility to participate in the Scholarship Program or ineligibility to accept new students. However, Bulletin 133 does not define “gross or persistent lack of basic academic competence” nor does it define the criteria by which to determine when a school should be deemed ineligible to participate in the program or to accept new students. Without specific criteria, LDOE cannot ensure it is holding schools accountable for their academic performance and treating schools consistently. … LDOE should develop internal procedures with more specific criteria for removing a participating school from the program based on academic performance. [Emphasis added.]”
No one could have predicted that ed reformers in government would set this up so public schools would “lose”, right? 🙂
What a joke of a system. The “winners” are preordained!
The public school kids shouldn’t participate in the OH and LA scoring systems. They’re the designated losers before they even enter the building.
readers should look at how the PISA tests are used to compare students in northern Italy with students in southern Italy. Cornoldi notes that it is differences in the test — response time/accuracy tradeoffs that seem to penalize the students in southern Italy. The use of these tests to compare across countries is totally wrong. It is pushed by this corporate competition and does not make sense with the purposes of education.
In my first statistics class in graduate school my professor began the class with the following statement: “anyone in this class who uses the term rankings will fail immediately.” I understood the meaning of that statement at the end of the course and which the contents of this post elaborate on.
Comparing scores across nations leaves too many uncontrolled variables to tell us anything about the reason . PISA-mania originates in U.S. politics because we like to frame our successes and failures in terms of competition. We want to keep score so that someday we see proof of our exceptionalism. Framing learning as a game you win or lose is destructive to our education system, which depends on cooperation between schools, teachers, parents and students. Europeans would be wise to disregard our pathological fixation on test scores and embrace models of collaboration and teacher autonomy, , not for their effect on PISA scores, but
because they are good educational practice.
Again I repeat myself. Tests are not an indicator of nacademic achievement. That also means that vwe, the public supporters can not use them when they favor us.. http://savingstudents-caplee.blogspot.com/2013/12/accountability-with-honor-and-yes-we.html
I wonder what European scholars think about our current American Healthcare system . . . .
Reblogged this on Middletown Voice.
It looks to me as if Professor Kreiner started from his conclusion and worked back from there. Fwiw, here is OECD’s full 2011 response to this paper. It’s clear that Prof. Kreiner’s work should not be taken at face value.
I think this frenzied attempt to discount the validity of PISA – China cheating, statistical errors, no proof that education systems affect a nation’s economic outcomes (!), etc. – is both unjustified and unnecessary. Alongside others opposing the privatizers, I have decried misinterpretation of the data (here, here, here and in many other places), but now I see it differently. I think that, although there are real limits to the value of testing, good tests (PISA included) are not useless. And we do not need to make the case that they are useless just because privatizers use them to advocate for their destructive solutions.
We can say that, yes, we have challenges to face in our education system. And it’s not only poverty, as central as that is. But high stakes testing used for teacher-bashing and school closing policies that feed charterizing and voucherizing privatization strategies…these are not the way to meet those challenges. We should be investing in our system of public education, not labeling it a failure and destroying it.
That’s a case that opponents of today’s dopy education reform policies can make. There’s no need to deny the realities around us to do it.
Bill Duncan, our scores on the first international scores were last in 1964. In 1983, we were told our bad scores made us “a nation at risk.” Do you still think those international scores predicted our decline?
To the viewers of this blog: doing an end-around a pertinent question is not the same as answering it. This is not a football blog; it’s a website to “discuss better education for all.”
“… our scores on the first international scores were last in 1964. In 1983, we were told our bad scores made us “a nation at risk.” Do you still think those international scores predicted our decline?”
So let me make this simple—
“Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.” [Albert Einstein]
The owner of this blog has done her part.
And the answer to her question is….
¿?
😎
Diane,
I don’t think a test given 50 years ago is relevant in today’s discussions.
And there are many reasons the muscular post-war economy of the country everyone wants to move to has stayed strong until recent times.
And, no matter what one thinks about the analysis or impact of “A Nation at Risk,” I don’t understand why OECD and the PISA project would be attacked as malevolent allies of the education reformers with dishonest intent trying to make us look bad. As far as I can see, OECD wouldn’t be bothered with that kind of agenda. They’re just trying to figure stuff out.
Maybe dig into the data. Maybe debate observations from the many detailed reports. But it doesn’t make sense to me to discount PISA (and all other testing, I guess, except maybe TIMSS, where we look better) because it might not predict our decline. Or because we don’t like the result. Or because in recent years the privatizers use it as well.
Bill Duncan, if you knew more about standardized testing, you would be less confident in what the scores mean.
I have no illusions about the evils of testing but I don’t see the point to a no testing position. Reformers have hijacked testing in the US for the moment, but we’re not about to enter an era of no testing.
As a former teacher/principal any form of standardized assessment offered in our state and the ACT/SAT always left me indifferent. What caught my interest and was my instructional focus was what was going on in classrooms. From my own teaching experience and then in a leadership position I observed too many classrooms,
including my own, that were too passive, too teacher directed, too worksheet oriented, and too often not as intellectually demanding as the should be. Much of the passivity and low cognitive demand of classrooms is driven by the institutional design and goals of schooling—accreditation, seven period days, bus schedules, etc (Goodlad’s: A Place called School, is still the best description of institutional schooling in action). Having said that, both in my own teaching and in the high school I led I did experience courses and programs that were able to redesign the curriculum and pedagogy to create classroom experiences that were intellectually engaging and resembled the “flow experiences” described by Csikszentnihalyi. The continual focus on numbers or a list of skills or standards diverts our attention from the hard work of creating classroom experiences that children talk about at dinner tables. When I thought about school reform in my building I always thought about Lillian Katz’s comment on school reform: have we ever considered making schools interesting places to attend. We will never discover direct links between teaching and learning, especially looking at computer screens, but we do have some idea of the types of questions/problems and activity structures that energize young people intellectually and emotionally. The problem we face, which Dewey formulated at the turn of the century, was how to transform a child’s social, emotional, and social proclivities into disciplined ways of understanding the world around them. I elaborate on how school leaders and teachers can redesign their schools to become more interesting places to attend in two books: Becoming A Strong Instructional Leader: Saying No to Business As Usual (Teachers College Press) and Teaching Matters Most: A School Leader’s Guide to Improving Classroom Instruction (Corwin Press).
http://www.amazon.com/Becoming-Strong-Instructional-Leader-Business/dp/0807753386/ref=sr_1_1?s=books&ie=UTF8&qid=1335271967&sr=1-1
http://www.amazon.com/Teaching-Matters-Most-Improving-Instruction/dp/1452205108/ref=sr_1_1?s=books&ie=UTF8&qid=1343052764&sr=1-1&keywords=Teaching+mATTERS+MOST
I’m glad someone is finally looking critically at the PISA tests. I suspect the Smarter Balanced and PARCC tests are modeled on the PISA tests. I hope some smart scholars will critique these tests too. What folly it would be to make education policy on conclusions drawn from bogus tests.