Since there is always a lot of chatter about what international tests scores mean, I invited David Berliner to share his views. Berliner is one of our nation’s pre-eminent scholars of education.
Dear Diane,
A few weeks ago you asked me a question about recent PISA test results and the role that is played by poverty in the scores of the USA and other countries. As I understand it PISA doesn’t compute the poverty-test score relationships in quite the same way we might in the USA, but the results they get are similar to what we get.
Investigations of the poverty-test score relationships in PISA 2012 (OECD, 2013) relied on two variables, each of which was a composite. First, they used a family social class measure that was supposed to capture the income and cultural resources of a family. They combined three factors to get one composite index of family social standing: the highest occupational level of either parent; the highest educational level of the parents; and home possessions (particularly, books in the home). This family index of social class standing is not income, and it is also not always a good measure of social class standing (for example, think of highly educated immigrants who hold low wage jobs). Nevertheless, it is this composite indicator of social standing that was used to examine the scores each nation attained in tests of mathematics, science, or reading. The relationships between social standing and achievement were quite similar on all three tests of subject matter.
When we ask what percent of the variance in US students’ PISA scores was accounted for by this composite of family social class variables, the answer is around 20% (OECD, 2013, Fig ll.2.3). Twenty percent explained variation in PISA scores that arise from differences in socioeconomic factors related to families is low enough to suggest that “poverty is no excuse,” or that, “demography is not destiny.” Such maximssound reasonable because it appears at first that about 80% of the variation in student test scores is still to be accounted for. Thus, if we just had great teachers and great school leaders every child would be successful. But this interpretation is completely misleading.
For example, PISA also informs us that the test score difference attributable to moving up or down one place on the social class measure is 39 points. That works out to nearly one year of schooling on the PISA scale. So, if in the recent great recession your family was hurt and you move down the social class scale one unit, the prediction from PISA data is that children of such families are likely, eventually, to be scoring one full year lower than they might have had their family just stayed at their more advantaged social level. So the “20% variance accounted for” estimate is not a trivial figure when we look at the score points that are involved in having only slightly different social class standing. The data convincingly suggests that social status variables are quite powerful and not quite as easily overcome as the maxims we hear that suggest otherwise.
This becomes even more apparent with some additional information collected by the PISA designers. The 2012 study used information obtained from school principals about the school attended by each child in the sample. Thus, schools were categorized on the basis of the wealth and the poverty of the student body, along with the housing patterns and values in the school catchment areas, the qualities of the teachers assigned to the schools the children attend, the funding of the schools, and a number of other school level variables that are correlated strongly with the incomes of students’ families. This is the second large composite variable used in understanding the relationship of poverty to PISA test scores.
The relevant data is given as the percent of the test score variance that is attributable to differences between schools because of the population they draw. Together the family and the school level variables related to social class account for 58% of the variance we see between schools. This is quite close to the data we usually cite in the USA, namely, that about 60% of the variance we see among schools is the result of outside-of-school factors, not inside-of-school factors. (It is generally agreed that in the USA we often have 20% of the variance in test scores accounted for by school variables, maybe half of which is a teacher effect. So, in the USA, the outside-of-school variables count for about 3 times the effect of the inside-of-school variables, and they count for about 6 times the effect of teachers on the aggregate scores of classes and schools.)
Thus the international data support the estimate of poverty’s effects on test scores that we have obtained from studying internal US test data. In fact, the 2012 PISA data provides a similar estimate to what was found in the Coleman report of the 1960s. The historical record, therefore, tells us that if we want to fix schools that are not now performing well on achievement tests, we might do well to work on the out-of-school factors that influence educational achievement, and not put all our efforts into trying to improve inside-of-the school factors, as the President and Secretary of Education continue to do. Our elected officials and numerous misguided individuals and corporations keep failing to interpret the extant data in a credible way.
To those who say “poverty is no excuse,” I would then ask how they account for poverty’s potency in explaining so much of the variance in achievement test scores in the USA and elsewhere? Indeed, poverty may not be an excuse for poor performance, but it sure is a quite reasonable hypothesis about the origins of student, school, and school district differences in achievement test scores. And, of course, it may not be poverty per se that is the causal factor in the low achievement seen on so many different tests. Rather, it may be poverty’s sequelae that is the culprit. That is, the wealth of families determines such things as housing, and it is housing that determines the types of neighbors one has, the mental health and crime rates in your neighborhood, the availability of role models for children, the number of moves a family makes while children are young, the stability of family relationships, low birth weight, teen pregnancy rates, Otitus Media rates in childhood, and so forth. Discussing “poverty” and “achievement” is a simple way of expressing the relationships we find between dozens of the sequelae associated with poverty and the many forms of achievement valued by our society.
PISA provides still more evidence that poverty is a strong factor in shaping students’ lives, supporting the contention that it is really quite common for demography to determine destiny. PISA looked at “resilient students,” those who are in the bottom quartile of the social class distribution, but in the top quartile in the achievement test distribution. These are 15-year-olds who seem to beak the shackles imposed by family and neighborhood poverty. In the USA, about 6% of the children do that. So 94% of youth born into or raised in that lower quartile of family culture and resources do not make it into the top quartile of school achievers. Admittedly, poverty is hard to overcome in most countries. But why is it that Belgium, Canada, Finland, Turkey, and Portugal, among many others, produce at least 40% more “resilient kids” than do we? Could it be because the class lines are more hardened here in the USA? Whatever the cause, given these data, the mantra that “Poverty is no Excuse” seems weak, and easily countered by the more rational statement that comes directly from the PISA data, namely, that family poverty and its sequelae severely limit the life chances of most children in the lower quartiles, quintiles, and deciles on measures of social class standing.
More evidence of this is also found in the PISA data. Housing patterns seemed to matter a lot in determining scores on the PISA 2012 assessments. There were striking performance differences observed between students in schools with socially advantaged students and those in schools with socially disadvantaged schools. Students attending socioeconomically advantaged schools in OECD countries outscore those in disadvantaged schools, on average, by more than 104 points in mathematics! This is of course quite a common finding in the USA where Jonathan Kozol once described our housing patterns as “Apartheid-Lite.” We should note, too, that a reanalysis of the Coleman report by Borman and Dowling (2010) broke out the variance in test scores attributable to individual background (like variable 1 in PISA) and the social composition of the schools (like variable 2 in PISA). Borman and Dowling say their reanalysis provides “very clear and compelling evidence that going to a high-poverty school or a highly segregated African American school has a profound effect on a student’s achievement outcomes, above and beyond the effect of his or her individual poverty or minority status. Specifically, both the racial/ethnic and social class composition of a student’s school are more than 1 3/4 times more important than a student’s individual race/ethnicity or social class for understanding educational outcomes. In dramatic contrast to previous analyses of the Coleman data, these findings reveal that school context effects dwarf the effects of family background.”
Many other nations have the same pattern of housing and schooling that we do: wages determine housing, and housing determines the characteristics of the student body and the quality of the school attended by children. This all suggests that there is a lot of support for the statement that demography, in too many instances, really does determine destiny.
The clearest case of this comes from analyses of other, earlier PISA data, by Doug Willms (2006). His analysis suggests that if children of average SES attended one of their own nations high performing schools, or instead attended one of their own nations’ low performing schools, the difference at age 15, the age of PISA testing, would be equivalent to about 4 grade levels. Thus a 10th grader of average SES who can attend a high performing school is likely to score at about the 12th grade level (a grade level approximation from PISA data). And if that same child were to attend a low performing school, he or she would score at about the 8th grade level. It’s the same hypothetical child we are talking about, but with two very different lives to be lead as a function of the makeup of the schools attended. It is not the quality of the teachers, the curriculum, the computers available, or any number of other variables that are often discussed when issues of school quality come up. Instead, the composition of the school seems to be the most powerful factor in changing the life course for this hypothetical, average child. PISA data from an earlier assessment in Australia documents the same phenomena (Perry and McConney, 2010). In science, the score of a low income student in a low income school averages 455. But the score of similar low income students at schools that serve upper income children is over half a standard deviation higher—512. And a high income student in a school serving low income students scores 555, but high income students enrolled in schools with high income peers score half a standard deviation higher—607. Note what is most impressive here: the low income student in a school with low income families scores 455, while a high income student in a school with high income families scores 607. That is about a standard deviation and a half apart! These are 15 year olds that are worlds apart in both housing patterns, school quality, and in measures of cognitive ability. In short, PISA data overwhelmingly supports the belief that demography and destiny are closely related, a terrible embarrassment for democratic countries that pay so much lip service to the principal of equality of opportunity. Apparently, the chant that “poverty is no excuse” can easily become a reason for doing nothing about poverty’s effects on many social variables that consistently, and cross nationally, affect both school outcomes and life chances. Horatio Alger may have never been fully believable, but a few decades ago it looks like Horatio simply died, mostly unnoticed.
References
Borman, G. D. & Dowling, M. (2010). Schools and Inequality: A Multilevel Analysis of Coleman’s Equality of Educational Opportunity Data. Teachers College Record, 112 (5), 1201–1246.
OECD (2013), PISA 2012 Results: Excellence Through Equity: Giving Every Student the Chance to Succeed (Volume II), PISA, OECD Publishing.
http://dx.doi.org/10.1787/9789264201132-en
Perry, L. B. & McConney, A. (2010). Does the SES of the school matter? An
examination of socioeconomic status and student achievement using PISA
2003. Teachers College Record 112 (4), 1137–1162.
Willms, J. D. (2006). Learning divides: Ten policy questions about the performance and equity of schools and schooling systems. Montreal, Canada: UNESCO Institute for Statistics.
Diane Ravitch's blog 