Since there is always a lot of chatter about what international tests scores mean, I invited David Berliner to share his views. Berliner is one of our nation’s pre-eminent scholars of education.
A few weeks ago you asked me a question about recent PISA test results and the role that is played by poverty in the scores of the USA and other countries. As I understand it PISA doesn’t compute the poverty-test score relationships in quite the same way we might in the USA, but the results they get are similar to what we get.
Investigations of the poverty-test score relationships in PISA 2012 (OECD, 2013) relied on two variables, each of which was a composite. First, they used a family social class measure that was supposed to capture the income and cultural resources of a family. They combined three factors to get one composite index of family social standing: the highest occupational level of either parent; the highest educational level of the parents; and home possessions (particularly, books in the home). This family index of social class standing is not income, and it is also not always a good measure of social class standing (for example, think of highly educated immigrants who hold low wage jobs). Nevertheless, it is this composite indicator of social standing that was used to examine the scores each nation attained in tests of mathematics, science, or reading. The relationships between social standing and achievement were quite similar on all three tests of subject matter.
When we ask what percent of the variance in US students’ PISA scores was accounted for by this composite of family social class variables, the answer is around 20% (OECD, 2013, Fig ll.2.3). Twenty percent explained variation in PISA scores that arise from differences in socioeconomic factors related to families is low enough to suggest that “poverty is no excuse,” or that, “demography is not destiny.” Such maximssound reasonable because it appears at first that about 80% of the variation in student test scores is still to be accounted for. Thus, if we just had great teachers and great school leaders every child would be successful. But this interpretation is completely misleading.
For example, PISA also informs us that the test score difference attributable to moving up or down one place on the social class measure is 39 points. That works out to nearly one year of schooling on the PISA scale. So, if in the recent great recession your family was hurt and you move down the social class scale one unit, the prediction from PISA data is that children of such families are likely, eventually, to be scoring one full year lower than they might have had their family just stayed at their more advantaged social level. So the “20% variance accounted for” estimate is not a trivial figure when we look at the score points that are involved in having only slightly different social class standing. The data convincingly suggests that social status variables are quite powerful and not quite as easily overcome as the maxims we hear that suggest otherwise.
This becomes even more apparent with some additional information collected by the PISA designers. The 2012 study used information obtained from school principals about the school attended by each child in the sample. Thus, schools were categorized on the basis of the wealth and the poverty of the student body, along with the housing patterns and values in the school catchment areas, the qualities of the teachers assigned to the schools the children attend, the funding of the schools, and a number of other school level variables that are correlated strongly with the incomes of students’ families. This is the second large composite variable used in understanding the relationship of poverty to PISA test scores.
The relevant data is given as the percent of the test score variance that is attributable to differences between schools because of the population they draw. Together the family and the school level variables related to social class account for 58% of the variance we see between schools. This is quite close to the data we usually cite in the USA, namely, that about 60% of the variance we see among schools is the result of outside-of-school factors, not inside-of-school factors. (It is generally agreed that in the USA we often have 20% of the variance in test scores accounted for by school variables, maybe half of which is a teacher effect. So, in the USA, the outside-of-school variables count for about 3 times the effect of the inside-of-school variables, and they count for about 6 times the effect of teachers on the aggregate scores of classes and schools.)
Thus the international data support the estimate of poverty’s effects on test scores that we have obtained from studying internal US test data. In fact, the 2012 PISA data provides a similar estimate to what was found in the Coleman report of the 1960s. The historical record, therefore, tells us that if we want to fix schools that are not now performing well on achievement tests, we might do well to work on the out-of-school factors that influence educational achievement, and not put all our efforts into trying to improve inside-of-the school factors, as the President and Secretary of Education continue to do. Our elected officials and numerous misguided individuals and corporations keep failing to interpret the extant data in a credible way.
To those who say “poverty is no excuse,” I would then ask how they account for poverty’s potency in explaining so much of the variance in achievement test scores in the USA and elsewhere? Indeed, poverty may not be an excuse for poor performance, but it sure is a quite reasonable hypothesis about the origins of student, school, and school district differences in achievement test scores. And, of course, it may not be poverty per se that is the causal factor in the low achievement seen on so many different tests. Rather, it may be poverty’s sequelae that is the culprit. That is, the wealth of families determines such things as housing, and it is housing that determines the types of neighbors one has, the mental health and crime rates in your neighborhood, the availability of role models for children, the number of moves a family makes while children are young, the stability of family relationships, low birth weight, teen pregnancy rates, Otitus Media rates in childhood, and so forth. Discussing “poverty” and “achievement” is a simple way of expressing the relationships we find between dozens of the sequelae associated with poverty and the many forms of achievement valued by our society.
PISA provides still more evidence that poverty is a strong factor in shaping students’ lives, supporting the contention that it is really quite common for demography to determine destiny. PISA looked at “resilient students,” those who are in the bottom quartile of the social class distribution, but in the top quartile in the achievement test distribution. These are 15-year-olds who seem to beak the shackles imposed by family and neighborhood poverty. In the USA, about 6% of the children do that. So 94% of youth born into or raised in that lower quartile of family culture and resources do not make it into the top quartile of school achievers. Admittedly, poverty is hard to overcome in most countries. But why is it that Belgium, Canada, Finland, Turkey, and Portugal, among many others, produce at least 40% more “resilient kids” than do we? Could it be because the class lines are more hardened here in the USA? Whatever the cause, given these data, the mantra that “Poverty is no Excuse” seems weak, and easily countered by the more rational statement that comes directly from the PISA data, namely, that family poverty and its sequelae severely limit the life chances of most children in the lower quartiles, quintiles, and deciles on measures of social class standing.
More evidence of this is also found in the PISA data. Housing patterns seemed to matter a lot in determining scores on the PISA 2012 assessments. There were striking performance differences observed between students in schools with socially advantaged students and those in schools with socially disadvantaged schools. Students attending socioeconomically advantaged schools in OECD countries outscore those in disadvantaged schools, on average, by more than 104 points in mathematics! This is of course quite a common finding in the USA where Jonathan Kozol once described our housing patterns as “Apartheid-Lite.” We should note, too, that a reanalysis of the Coleman report by Borman and Dowling (2010) broke out the variance in test scores attributable to individual background (like variable 1 in PISA) and the social composition of the schools (like variable 2 in PISA). Borman and Dowling say their reanalysis provides “very clear and compelling evidence that going to a high-poverty school or a highly segregated African American school has a profound effect on a student’s achievement outcomes, above and beyond the effect of his or her individual poverty or minority status. Specifically, both the racial/ethnic and social class composition of a student’s school are more than 1 3/4 times more important than a student’s individual race/ethnicity or social class for understanding educational outcomes. In dramatic contrast to previous analyses of the Coleman data, these findings reveal that school context effects dwarf the effects of family background.”
Many other nations have the same pattern of housing and schooling that we do: wages determine housing, and housing determines the characteristics of the student body and the quality of the school attended by children. This all suggests that there is a lot of support for the statement that demography, in too many instances, really does determine destiny.
The clearest case of this comes from analyses of other, earlier PISA data, by Doug Willms (2006). His analysis suggests that if children of average SES attended one of their own nations high performing schools, or instead attended one of their own nations’ low performing schools, the difference at age 15, the age of PISA testing, would be equivalent to about 4 grade levels. Thus a 10th grader of average SES who can attend a high performing school is likely to score at about the 12th grade level (a grade level approximation from PISA data). And if that same child were to attend a low performing school, he or she would score at about the 8th grade level. It’s the same hypothetical child we are talking about, but with two very different lives to be lead as a function of the makeup of the schools attended. It is not the quality of the teachers, the curriculum, the computers available, or any number of other variables that are often discussed when issues of school quality come up. Instead, the composition of the school seems to be the most powerful factor in changing the life course for this hypothetical, average child. PISA data from an earlier assessment in Australia documents the same phenomena (Perry and McConney, 2010). In science, the score of a low income student in a low income school averages 455. But the score of similar low income students at schools that serve upper income children is over half a standard deviation higher—512. And a high income student in a school serving low income students scores 555, but high income students enrolled in schools with high income peers score half a standard deviation higher—607. Note what is most impressive here: the low income student in a school with low income families scores 455, while a high income student in a school with high income families scores 607. That is about a standard deviation and a half apart! These are 15 year olds that are worlds apart in both housing patterns, school quality, and in measures of cognitive ability. In short, PISA data overwhelmingly supports the belief that demography and destiny are closely related, a terrible embarrassment for democratic countries that pay so much lip service to the principal of equality of opportunity. Apparently, the chant that “poverty is no excuse” can easily become a reason for doing nothing about poverty’s effects on many social variables that consistently, and cross nationally, affect both school outcomes and life chances. Horatio Alger may have never been fully believable, but a few decades ago it looks like Horatio simply died, mostly unnoticed.
References
Borman, G. D. & Dowling, M. (2010). Schools and Inequality: A Multilevel Analysis of Coleman’s Equality of Educational Opportunity Data. Teachers College Record, 112 (5), 1201–1246.
OECD (2013), PISA 2012 Results: Excellence Through Equity: Giving Every Student the Chance to Succeed (Volume II), PISA, OECD Publishing.
http://dx.doi.org/10.1787/9789264201132-en
Perry, L. B. & McConney, A. (2010). Does the SES of the school matter? An
examination of socioeconomic status and student achievement using PISA
2003. Teachers College Record 112 (4), 1137–1162.
Willms, J. D. (2006). Learning divides: Ten policy questions about the performance and equity of schools and schooling systems. Montreal, Canada: UNESCO Institute for Statistics.
Diane Ravitch's blog 
Excellent analysis!!
Lots of data to get your head around on this, but does provide statistical evidence of the link between achievement in schools that serve a majority of high and low socioeconomic populations. Unfortunately educators and the school environment only influence 40% of a students success in school. The loudest voices right now that are in opposition to current school reforms seem to be less concerned with the poverty divide and more with their own children becoming brainwashed and ill-prepared for college and life. The reality is that many of these families will have students successful regardless of whether there is NCLB, Common Core or teacher autonomy of curriculum and assessment. Unfortunately, at least on the conservative side, there is also the point of view that poverty is no excuse and that they should not need anymore assistance than my school or children receive.
Changing the culture of our mist impoverished neighborhoods is a lot harder than throwing more money and resources at their schools.
Reblogged this on From experience to meaning… and commented:
An interesting letter on the link between scores on PISA and poverty (focus is the US, but relevant for more countries).
Does this mean that we now believe the PISA results in general?
Bill Duncan,
I don’t believe the PISA results mean anything about the future of our economy. We have never had high scores on international tests for 50 years. Sometimes we were in the bottom quartile or dead last. Yet we surpassed all those with higher scores in economic growth and every other measure. How do you account for that?
That’s a very big question you ask, Diane. I won’t try to answer it in a comment, but there are a couple of layers to it, at least.
One is whether the PISA tests say anything useful about American education. The second is how direct a link there is between the kind of educational performance PISA measures and the economy. That second question is a long discussion. Maybe our relative position in the world after the WWII, our immigrants and entrepreneurs drove our economy and our schools produced the kinds of graduates needed by the economy at that stage. A long discussion, as I say.
It would be hard to make the case, though, that an education system that produced the kind of persistent problem solvers PISA tests for would not be a great economic resource.
But I thought you disagreed about the first level as well – whether PISA and the various analyses that spin off from it say anything useful about the quality and characteristics of the participating education systems, including ours. That’s why this post that takes that analysis seriously surprised me.
I do take the PISA analysis and observations seriously. I think PISA is right when it says, “Yes, poverty does drag US scores down, but we’re also not good at math.” Or when it says, “PISA results show that competition and choice don’t work.” Or “Teacher mentoring improves results.” Or “preschool works.” Or, “Successful countries allocate resources, especially teaching resources, to the schools poor kids go to.”
PISA provides exhaustive analysis to support all these propositions and more. So I do disagree with the effort to discount it. Yes, the privatizers use it to justify their mission as well. We need to counter that. But I don’t think it does any good to try to discount the PISA results.
Reblogged this on Reflections of a Second-career Math Teacher and commented:
“The historical record, therefore, tells us that if we want to fix schools that are not now performing well on achievement tests, we might do well to work on the out-of-school factors that influence educational achievement, and not put all our efforts into trying to improve inside-of-the school factors, as the President and Secretary of Education continue to do. Our elected officials and numerous misguided individuals and corporations keep failing to interpret the extant data in a credible way.”
Thanks to David Berliner for this great summary of the reletive contribution of outside of school factors to PISA scores. In response to Bill Duncan. What does it mean to believe PISA results? Do you mean how they measure the cognitive abilities or skill level for 15 year olds atrending schools in the nations or cities tested? Or do you mean the way aggregate scores are used to rank nations. The first approach is almost certainly a valid endeavor and is what Berliner addresses in this post. The second approach is deeply flawed on the basis of apples to oranges comparisons alone (i.e., a continental scale nation like the U.S. is nothing like Shanghai or Finland or Lichtenstein!, anyone with a freshmen level education in research methodology would know that such comparisons cannot be valid on their face … yet here we are with PISA horse race rankings for politicians and pundits!). The concern with PISA is that the former is all that politicians and pundits talk about, where are we in the horse race (as if children were horses to be bet on). Critics of PISA note that this is a complete waste of time, at best, and, a source lf state directed symbolic violence against children, at worst.
Actually, Ted, I did think after I finished that comment that I should have addressed the league tables. I agree that the league tables are the least useful PISA tables and that it’s the analysis that is valuable.
However, if you scan the anti-PISA posts here and elsewhere, the rejection of PISA is much broader than the league tables. It’s that rejection that I am responding to.
Thanks for the article. Americans need to hear more about the competitiveness of the American educational system. The media would have us believe that there is something wrong with our teachers and/or our children when the truth is that our politicians have been flooding the country with illiterate breeders for whom education is an afterthought. So now we are frantically trying to fix an educational system which isn’t broken. The result will be a system of education which didn’t work for anyone.