Statisticians Mark Palko and Andrew Gelman explain why a relentless obsession with test scores ruins the value of the scores. As their prime example, they refer to Eva Moskowitz’s Success Academies, where children and teachers live for higher scores. Not only are the children’s names and ranking posted, so are the teachers’.
You remember Campbell’s Law? That’s the axiom that says when you attach consequences to a measure, the measure loses its validity.
“When a school uses selection and attrition policies that effectively filter out many of the extremely poor, students speaking English as a second language, and the learning disabled, that clearly calls into question test score advantages that such a school might have over an ordinary public school.
“But the problems run even deeper than most critics realize: A look at the data combined with some basic principles of social science suggests that the practices of no-excuses charters are undermining the very foundation of data-based education reform.
“As statisticians with experience teaching at the high school and college level, we recognize a familiar problem: A test that overshadows the ultimate outcomes it is intended to measure turns into an invalid test.
“Back in the old Soviet Union, factories would produce masses of unusable products as a result of competition to meet unrealistic production quotas. Analogously, many charter schools, under pressure to deliver unrealistic gains in test scores, are contorting themselves to get the numbers they’ve promised. They’re being rewarded for doing so. But that monomaniacal focus on test scores undermines the correlation between test scores and academic accomplishment that originally existed.”
They note that Success Academy has astonishingly high test scores, yet for two years in a row, not a single one of their eighth grade students won admission to one of the city’s elite high schools. In the third year, some did (11% of those who took the test from SA).
In a comment on this post, Gary Rubinstein (a blogger who teaches at Stuyvesant High School, an exam school) writes:
“One thing to note, the 11% specialized HS acceptance rate–6 out of 54–is inflated since there were 200 kids who feasibly could have sat for that test but only 54 did.” Of 200 students at Success Academy who were eligible to take the test, 54 did, and 6 gained admission.
It is better to have high scores than low scores, but they should never be the measure of teacher quality or school quality. Making them too important ruins their value.