Archives for category: Campbell’s Law

Teacher Aaron Pribble wrote a critique of high-stakes testing for “Edutopia.”

He explains how high-stakes testing warps teaching and distorts the educational process.

When teachers get rewards and punishments tied to test scores it ruins education.

He has a simple idea: Raise the standard for entry into teaching. Then give teachers the freedom to teach. Treat them as the experts they are.

Ken Bernstein, retired teacher, posted this comment:

“There are other issues that need to be included, even if there are no problems with erasures. States report scaled scores. They can and do change the conversion rate from raw to scaled in order to show “improvement” year to year. The year before I taught in Virginia the school in which I taught had a 58% pass rate on the Middle School American History test. The year I was there the pass rate was 81%, my own was 89%. Sounds great, right? Except they had lowered the cut score and changed the conversion. If the previous year’s raw scores had been converted using the same matrix, the score would have been 71% or so. Thus we showed “improvement” but not that much.”

A reader in Florida saw that the school grades were phony, as noted in an earlier post. But no matter how many times the grades were shown to be meaningless, everyone accepted them, organized their schools to get them, changed their instruction to raise those grades. worked to get the bonuses, worked to avoid the sanctions, on and on.

And still the grades tell us which schools enroll poor kids and which kids don’t.

There is a lesson here. Something about adjusting to absurd demands and therefore making them seem reasonable.

But if you step back, they are still absurd.

Giving a school–a complex organization with many moving parts–a single letter grade is insulting, demeaning, and stupid. It is the product of people who know how to count but don’t care what they are counting.

Having spoken out against Florida’s school grades since they were detected by my ears so many years ago, I took my concern further and created non statistician produced data about their failures and circulated it. Sadly, these failures were buried just as professor’s findings that indicated the system was a bomb. I listened to comments such as well, you may not like it but Tallahassee does or appreciation of the money it brings to Florida’s poorly funded schools. Very few persons of power had much to say and those that did, seemed to embarassingly acknowledge that my concerns were valid.Occasionally, I would hear a reference from a board member which alluded to my concerns. I watched the paper for years try to educate the public as to the bogus nature of these farcical indicators. Mostly, I heard the empty boasts of nothngness ring on and on. I wondered why and I hypothesized that Donald Campbell’s Law was in action. After all, the state’s school grades are high stakes. Districts scramble to earn meaningless boasts and a bonus money flow. Districts scramble to avoid unfair sanctions. Gaming the system becomes a solution. Looking good became the goal rather than doing good. Children are pawns and parents provided information of a poor nature, This is called an accountability system. Although such a system is to be fair, valid, and reliable, Florida’s system is not and thus seems to be unfit for the terms of both A+ Plan and accountability system. We see a lack of reliability in the most interesting jump from 80 to 30 % in proficiency rates in the three grade levels tested being changed in the course of a phone call to be no longer 30% but near the 80% level after all.This change did not necessitate a change in answers thus the results had remained the same and the outcome oh so much for reliability. (Certainly somethng seems amiss when the state allowed comparison of two different tests with variations in administration, weighting factors, scoring, and cutoff scores.) Skewing by SES reveals an unflattering picture on fairness and interferes with validity as well since instructional quality is not the indicator being measured.
Florida’s schools and students have always deserved better.

I am reposting this because I forgot to include a link to the study, its title and the names and affiliations of the authors in the first posting. Pretty awful oversight. Actually inexcusable on my part. I apologize to my readers and to the authors of the study.

We have known for some years that the scoring of state tests is easily gamed. In fact, proficiency rates don’t tell us much, because state officials may raise or lower the passing score for political reasons. It happened in New York for years, when the proportion of students passing the state tests went up and up until it collapsed in 2010 as a result of an independent investigation. The state officials enjoyed their annual press conferences where they announced annual too-good-to-be-true gains. And they were too good to be true. They were fake. When the fraud was revealed, there was no accountability. No one admitted having done the dirty deeds. No heads rolled. Accountability is for “the little people,” as real estate queen Leona Helmsley once said about paying taxes. In education, the little people are teachers and principals. At the top–at state departments of education–heads don’t roll. They crown themselves and use their exalted position to blame those who are far, far below them. Think “Yertle, the Turtle.”

An important new study  by Professors Adam Maltese of Indiana University and Craig Hochbein of the University of Louisville sheds new light on the validity of state scores. This study found that rising scores on the state tests did not correlate with improved performance on the ACT. In fact, students at “declining” schools did just as well and sometimes better than students where the scores were going up. The study was published in the Journal of Research in Science Teaching. Its title is “”The Consequences of ‘School Improvement’: Examining the Association Between Two Standardized Assessments Measuring School Improvement and Student Science Achievement.”

Consider the ACT an audit exam.

Consider the state tests an invalid way of measuring student achievement and an invalid way of judging students, teachers, and schools. Consider them an invalid way of closing schools and awarding bonuses and firing people.

When students are prepped and prepped and prepped to pass the state tests, they aren’t necessarily better educated, just prepared to take a specific test. Too much prepping distorts the value of the test.

When your measure is invalid, don’t use it for rewards and punishments.

Perhaps if we used these exams appropriately, just for information, they might begin to have some value. As high-stakes, their validity is corrupted, as Campbell’s Law predicts.

Read this article, which documents how data-driven policing has caused police to report statistics wrongly, classify crimes as more or less serious depending on the quota needed to fill, and has created constructs of “productivity” that warp the goals of policing.

What is the primary goal of policing? To keep our communities safe and crime-free. What is the primary goal of education? To assure that the younger generation is prepared in mind, character and body to assume the responsibilities of citizenship in our society. But what are the goals of education in a data-driven environment? To raise test scores, by whatever means necessary. This is akin to setting a quota for felony arrests for police or directing them that the crime statistics must go down.

Here we see a restatement of Campbell’s Law. When the stakes are high, people will not only forget the goals of their activity but the measure itself becomes corrupted. Thus, the data that are generated–whether by police or teachers–become meaningless because of the pressure applied to get them. In effect, we are paying people bonuses to generate good news that is not true. The good news is not true, the data are not trustworthy, the measures are no longer useful, and we are not achieving the purposes of policing or teaching. It’s what you might call a lose-lose.

But it does have certain benefits. It creates new industries for those who love counting and measuring and reporting. It creates new work for the consultants who will tell you how to reach your targets. It provides a rationale for endless workshops and professional development and study groups, all of which divert even more time from the original goals. It creates new work for the experts who will opine about better ways to reach the targets. And it gives bragging rights to the politicians who think they accomplished something.

Everyone talks about high school graduation rates, but no one-including me–has any idea what they mean and what they really are.

We operate from the assumption that 100% of students “should” graduate from high school and excoriate the schools when the numbers are anything less. The assumption–which is wrong–is that we used to have high graduation rates but now we don’t. This is simply wrong. Over the course of the 20th century, graduation rates started from a very low point–less than 10% of young Americans finished high school at the beginning of the 20th century–and the rate rose steadily until it reached 50% in 1940. By 1970, it was 70%, and since then it has inched up.

Today, it is difficult to know what the graduation rate is because there are so many different ways of counting. If you count only those who graduate in four years, then it is about 75%. If you include those who graduate in August, after four years, it goes up. If you add those who took five or six years, it goes up more. If you add those who received a GED or some other alternate degree, it is up to 90%. (Aficionados of the issue can have fun poring over the latest federal data here).

These days, politicians play with the graduation rate to make themselves look successful (never mind the students). They lament the “crisis” in dropouts when they enter office, then crow at every uptick once they are in office to demonstrate “their” success.

Unfortunately, the pressure to raise the numbers typically overwhelms the standards required for attaining a high school diploma. When teachers and principals are sternly warned that their school will close unless they raise their graduation rate, they usually manage to raise their graduation rate without regard to standards. The usual gambit these days is called “credit recovery,” a phenomenon that was unheard of twenty years ago.

Credit recovery means simply that students can earn credits for courses they failed by completing an assignment or attending a course for a few days or weeks or re-taking the course online. As I wrote this week in Education Week, online credit recovery is typically a sham, a cheap and easy way of getting a diploma that was not earned. Students sit down in front of a computer, watch videos, then take a test that consists of multiple-choice questions, true-false questions, and machine-graded written answers. If they miss a question, they answer again until they get it right. Students can”recover” their lost credits in a matters of days, even hours. I wrote about online credit recovery as academic fraud in my EdWeek blog this week. Students realize quickly that if they fail, it doesn’t matter because they can get the credits in a few days with minimal effort. In this way, the diploma becomes meaningless, and students are cheated while the grown-ups fool themselves into thinking that they succeeded in raising the rates.

In this way, Campbell’s Law applies. When the pressure is raised high to reach a goal, the measures of the goal become corrupted.

The same number may be used either to bemoan a lack of progress or to claim victory. For example, the recent “Blueprint” created by a business strategy group for the school district of Philadelphia lamented that “only” 61% of its students attained a high school diploma in four years. At the same time, Mayor Bloomberg in New York City was delighted to report that the graduation rate was up to 65.5%, a figure that included summer school plus a heaping of credit recovery. The state of New York, which did not include summer graduates, put the actual figure at 61%, no different from the rate in Philadelphia. The state says that only 21% of students are “college-ready,” and the City University of New York–where most of the city’s graduates enroll–reports that nearly 80% require remediation.

So what is the real high school graduation rate? I don’t know.

Everyone interested in understanding how the ceaseless pressure to raise test scores can corrupt the tests should be familiar with Campbell’s Law.

This is an adage written by social scientist Donald T. Campbell in a 1976 paper. It says:

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” (You can google the paper, or find it linked on Wikipedia: Campbell, Donald T., Assessing the Impact of Planned Social Change The Public Affairs Center, Dartmouth College, Hanover New Hampshire, USA. December, 1976.)

Campbell’s Law explains why high-stakes testing promotes cheating, narrowing the curriculum, teaching to the test, and other negative behaviors..

In his 1976 paper, Campbell also wrote that  “achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways. (Similar biases of course surround the use of objective tests in courses or as entrance examinations.)”

Campbell’s Law helps us understand why No Child Left Behind and Race to the Top are harmful to education. They put pressure on teachers and principals and school districts and states to get higher and higher test scores. As we saw in Atlanta and in Washington, D.C., this kind of pressure may cause educators to betray their ethical duty by changing test scores. As we saw in New York state, this kind of pressure may cause a state education department to lower the passing mark on state tests so as to boost proficiency rates.

As high-stakes testing has become the main driver of our nation’s education policy, we will see more cheating, more narrowing of the curriculum, more gaming of the system. None of this produces better education. And even the test scores–on which so much public policy now firmly rests–will be corrupted, by making them so important.