I confess I never heard about Seth Godin. Then I read these interesting reflections and concluded I have to learn more about him. He is an author and a high-tech entrepreneur. He wrote the following wise thoughts, which are a direct hit on our current obsession with test scores. The scores are a proxy for good education. Our policymakers are satisfied to get the scores, even if the students don’t get a good education.
Godin writes:
Avoiding the false proxy trap.
Sometimes, we can’t measure what we need to measure, so we invent a proxy, something that’s much easier to measure and stands in as an approximation.
TV advertisers, for example, could never tell which viewers would be impacted by an ad, so instead, they measured how many people saw it. Or a model might not be able to measure beauty, but a bathroom scale is a handy stand in.
A business person might choose cash in the bank as a measure of his success, and a book publisher, unable to easily figure out if the right people are engaging with the book, might rely instead on a rank on a single bookseller list.
One last example: the non-profit organisation that uses money raised as a proxy for difference made.
You’ve already guessed the problem. Once you find the simple proxy and decide to make it go up, there are lots of available tactics that have nothing at all to do with improving the thing you set out to achieve in the first place. When we fall in love with a proxy, we spend our time improving the proxy instead of focusing on our original (more important) goal instead. {Why do I keep thinking 5×25?}
Gaming the system is never the goal. The goal is the goal.
Diane Ravitch's blog 
Also known as the Phrenology Paradigm.
Seth Godin is great. I saw him speak at a TEDx event hosted by the Brooklyn Free School…check it out—
I have heard that the great post WWII British sociologist Michael Young made this point decades ago–whatever you measure will move in the direction that is rewarded. Love to find the actual quotation. Young also coined the term “meritocracy,” which he did not see as a positive development. The linguistic connection to aristocracy is the clue to what he thought would go wrong with it.
I’m not sure what to think of him. He is intriguing. On the one hand, his comments on testing as false proxies gives me a new way to look at the testing mania. However, on the youtube video, he sees the educational system as broken and proposes flipped learning as the solution. He presents a rather extreme example of poor teaching with the hammer class. I’m not sure how flipped learning would have improved that teacher’s practice. If you accept his assertion that education is to train people to be obedient, it was a rather glaring example of that aim. His focus really appears to be on education beyond the elementary grades although he still tends to use hyperbolic examples. I like the feeling that education should be messy. I could agree with the sentiment that the more controlling we become the less learning occurs.
I obviously do not know what I am talking about. I am playing with ideas.
I like that – a false proxy. My friend Ted Chittenden of ETS used to remind us that at best tests are indirect measurements while we have in our classrooms live children from whom we can get direct evidence. Same point.
Deb
I made the connection to all the money raised by TFA, Students First, and Stand.org. People think they make a difference for children, but all they do is make money for entrepreneurs and influence elections.
The inherent problem with all of Mr. Godin’s analogies is that they are all measures of inputs or outputs, but not outcomes, which is unlike standardized assessments (except the bank/business analogy which I’ll address in a bit). In all of this examples, there are clear alternatives that would be available to measures outcomes, which would be more accurate to use in the analogy with test scores.
TV ads: roll out the ad in one market and notice different sales patterns from a “control” market
Book publisher: focus group or survey
Nonprofit: outcome measurement of program goals (e.g., number of GEDs attained by clients in adult ed program).
So, right away Godin’s examples aren’t parallel with standardized assessments, because standardized assessments measure outcomes, not inputs or outputs.
The clear difference is in the business example in which cash was used as a measure of business success. While cash in bank might not measure everything (e.g., there might be other assets, there might be a lot of income-producing potential in the form of a new business that is cash poor), certainly cash in bank (or money earned) would be at least one indicator of success of a business person.
To be sure, standardized assessments do not measure all expected outcomes, just as cash doesn’t measure all areas of success of a business person, but they do measure some. Reasonable folks who support using standardized assessments do not support using them exclusively, but as a part of the whole.
To address the second part – the gaming of the system: that argument holds true with any measurement, whether that be standardized tests, standardized teacher evaluations, etc. So, as long as we are going to measure educational outcomes, we will have some that attempt to game the system because they have a lot at stake. The real question is if this is really a problem with the assessment, or the construct of using assessments, or the folks misusing the system? As an analogy, sometimes the media is misused to push propaganda – does this mean that all forms of journalism are bad, or that the media should be squelched because it occasionally misuses its power? Or, should we hold those accountable that misuse that system in order to protect the true benefit in journalism?
Is seniority a proxy measure for quality?
Yes, but a true one.
For example, in academe, all associate professors automatically become full professors on their 39th birthday.
It’s called the Jack Benny promotion.
The unique proxy measure that is immune to this concern.
Alas, not all associate professors get that promotion.