This post was written by Charles J. Morris, Emeritus Professor of Psychology at Denison University, who lives in Indianapolis.

Does the ISTEP Measure School Quality and Teacher Effectiveness?

Charles J. Morris1

While there appears to be general agreement that teachers can make a big difference in the lives of students, there is little evidence that performance on standardized tests provides a valid assessment of teacher effectiveness. Nonetheless, at the national, state, and local levels, we are seeing increasing use of test scores to evaluate both schools and teachers, to award merit pay, and even sanction low performing schools and corporations.

This growing trend toward using test scores to evaluate schools and teachers fails to recognize the evidence that factors beyond the control of schools account for most of the variation we see in test scores among school districts throughout a given state. Matthew Di Carlo of the Shanker Institute sums it up this way: “…roughly 60 percent of achievement outcomes is explained by student and family background characteristics…schooling factors explain roughly 20 percent, most of this (10-15 percent) being teacher effects.”2 (The remaining variation is unexplained and considered error variance.) What this basically means is that schools and teachers are being judged to a substantial degree on the basis of factors over which they have little control.

Is the above conclusion also true for the ISTEP, Indiana’s test for measuring student performance and evaluating school quality and teacher effectiveness? The purpose of this short piece is to briefly summarize some evidence which indicates that the same conclusion holds for the ISTEP: Out-of-school factors, namely the socioeconomic profile (SES) of a school district, explain most of the variation we see in test performance from one district to the next.

Consider, for example, the following chart which shows the percent of students who passed both the ELA (English/Language Arts) and Math portions of the 2013 ISTEP as a function of the percentage of students in the corporation (Indiana’s districts) who qualify for free- or reduced-price lunches (FRPL, a commonly used measure of SES):

istep1

These data are based on the 56 corporations that have at least 5000 students in the district. As can be seen, there is a very strong correlation between the two variables: The higher the percentage of kids who qualify for FRPL, the lower the passing percentage. Another way of putting it is, if we know the socioeconomic profile of a corporation we can make a very good prediction of where that corporation stands compared to other corporations on the ISTEP. This should not be a surprise to those familiar with the research literature. The same relationship has been found for the various standardized tests used throughout the country.

The above results are based on the performance of all students in each corporation. The following charts show the results separately for 3rd and 8th graders:

istep2istep3

Again, we see the same pattern for both grade levels, basically unchanged after 5 years of schooling in a high-scoring or low-scoring corporation. The SES influence is quite strong independently of the schools and teachers in a particular corporation. In fact, if anything, the SES impact appears to become slightly stronger as students progress from the 3rd to 8th grade.

So what are we to make of this obvious association between ISTEP scores of SES? The seemingly inescapable conclusion is that corporations and teachers deserve neither praise nor criticism for how their student compare to other corporations and teachers. Clearly, the socioeconomic profile (SES) of the corporation plays a decisive role. So I ask a simple question: Does anyone seriously believe that if Carmel and Gary (a high and low-performing corporation, respectively) exchanged teachers, the ISTEP scores would suddenly reverse themselves? I don’t think so.

The challenge thus becomes how to respond to the fact that poorer kids are not performing well in our schools. Is there less parental involvement in these communities? Are expectations lower? Do these parents need additional help in becoming more effective mentors? Are after-school tutoring programs a possible solution? What about summer programs? Or pre-school programs? Perhaps all of the above, along with addressing the well-documented and devastating effects that poverty has on the health and well-being of poor children long before they even enter school3.

But one thing seems clear: Judging school and teacher quality on the basis of test scores offers little in the way of a solution. We need to look beyond our schools and teachers if we are going to better prepare all kids for the world they will face in the days ahead.

1Charles J. Morris is an Emeritus Professor of Psychology from Denison University. He resides in Indianapolis.

2Matthew Di Carlo, Shanker Institute (see http://shankerblog.org/?p=74#more-74)

3Diane Ravitch, Reign of Error (New York: Knopf, 2013, pp. 91-98).