Dear Diane,

A few weeks ago, I was heartened by your column about the return of the public schools to the citizens of Philadelphia. Since then, I’ve been mulling over four things that I wish I could communicate to them. Perhaps you can do so if you think it appropriate. I don’t know the folks there.

First, it will be difficult for teachers to show that they can turn Philadelphia’s schools into higher-achieving institutions. Teachers may help their students become stronger and more engaged learners, but they probably won’t be able to demonstrate student learning in the way that most people understand it, namely, through higher standardized achievement test scores.

The education research community clearly knows what politicians and the media don’t fully grasp: teachers simply don’t account for much of the variance in standardized tests scores. A reasonable estimate is that teachers account for about 10 percent of the variance in standardized achievement test scores. Research also suggests that outside-of-school factors account for 6 times more of that variance! We even have a Philadelphia based study corroborating these estimates.

In a 2014 Educational Researcher article by Fattuzo, LeBoeuf, & Rouse, 10,000 achievement test scores from Philadelphia were examined. The researchers used two sets of variables as predictors of students’ standardized test scores. The first set were school-level demographic variables such as race, gender, and degree of economic disadvantage. These are the kind of variables that the Coleman Report first revealed as strong influences on standardized achievement test scores.

My own research, and that of many others, has repeatedly confirmed this truth. In Fattuzo et al. these kinds of variables predicted 63 percent of the between school variance, quite close to the usual estimate of 60 percent of variance accounted for by demographic variables in students’ achievement test scores. Their data, then, are similar to what was found across nations in PISA and PIRLs, and similar to what other researchers find when school-level demographic variables are put into regression equations to predict variance accounted for.

But what Fattuzo et al. also did was add student-level variables to their equations. For each student, they knew whether the child was pre-term or low birth weight, had inadequate prenatal care, had a mother who was a teen, had high lead exposure, had a report record of being maltreated, had ever been homeless, and had a mother with less than a high school degree. It wasn’t surprising that each of these variables was a negative predictor of achievement test scores, and similarly, it wasn’t surprising that all but one variable was a statistically significant predictor of the standardized achievement test scores.

When the conventionally used school-level demographic variables were combined with student-level variables into the same equation, something quite different was revealed. The between school variance in reading test scores increased from 63 percent to 77 percent of the variance accounted for in the students’ standardized test scores. This leaves us with the task of trying to estimate what accounts for the remaining 23 percent of the variance in these standardized achievement test scores. We can separate that remaining variance into variance accounted for by error in the measurement system (a reasonable estimate might be about 10 percent) and school effects that are independent of teacher effects (which may also reasonably be estimated to be about 10 percent). Now we can account for about 97% of the variance. So, what percent of the variance in student test scores remains for teachers to affect? The answer is clearly almost nothing!

This all suggests that the good citizens of Philadelphia will probably not find whatever important things teachers might be accomplishing in their classrooms reflected in the standardized test scores routinely used in Pennsylvania. Instead, they should now think about other credible ways to judge teacher effectiveness.

Second, it won’t be Philadelphia’s newly taken-over schools that demonstrate how to get greater achievement from the students they serve. Schools, like teachers, may also be doing great things. But as noted, they usually affect about 10 percent of the variance on standardized achievement tests. Given Philadelphia’s high poverty rates, the variables associated with poverty may well account for most of the variance in citywide test scores, leaving little variance for the schools to effect, similar to the teacher effects just discussed.

The third point addresses the obvious question, what then should we do to better understand how teachers and schools effect local student outcomes? Usually we measure this via a standardized achievement test score, but as noted, that is quite likely not to be adequate for those purposes.

Instead, let’s try something different, and use the funds ordinarily paid to a test publisher to train selected parents who, alongside school principals or teacher leaders, could routinely observe classrooms and assist in monitoring the quality of instruction. Parents, principals, and teacher leaders can better learn to evaluate the artifacts of teaching, among them teachers’ tests and students’ answers. Those classroom tests will show both teachers’ understanding of and instructional alignment with the desired curriculum (as evidenced in the test’s items). And those tests will also show the quality of teaching that curriculum (as evidenced in the students answers to the items).

These assessments can be conducted as informal observations, or can use some of the of the more systematic and frequently used observation scales, such as those found in Danielson’s and Pianta’s systems. Scriven’s duties-based evaluation approach is certainly worth trying. And Meier’s and Knoester’s recent book offers a half a dozen other ways to assess students and classroom practices that don’t rely on standardized achievement tests. The point is this: assessing the quality of teachers and schools can be done without using standardized achievement tests that are known to be highly insensitive to what teachers and schools accomplish.

My fourth bit of advice is directed at those who serve on the newly constituted school board. It’s to remember that education outcomes are the result of much more than education polices. The school board will not achieve their improvement targets until other city and state policies better address the needs of Philadelphia’s schoolchildren. Housing policies must be strengthened to eliminate segregation by race and income, and affordable housing is needed to minimize residential mobility, which impedes school achievement. Physical and behavioral health policies must be strengthened, and nurses, counselors, and social workers must be sufficiently resourced to ensure that students attend school daily and have the supports needed to fully engage in learning. Policies to assure food security are needed so students aren’t preoccupied with hunger, and fair wage policies can provide income security for working families overwhelmed by living in poverty. And so on.

Getting their schools back is good for democracy in the city that played center stage in the founding of our nation’s democracy. Getting those schools to function well enough so its students can take on the role of stewards of our democracy is a whole other matter. I hope this advice helps them to do just that.