Our policymakers claim that their decisions are based on data.
What our policymakers seldom admit is that numbers by themselves are not reality. They are representations of reality. To draw conclusions from numbers, you must be awfully sure that you are measuring what matters and that your measurements are accurate.
In this post, the Red Queen in Los Angeles warns us that some of our leaders are guilty of making a “common, insidious mistake of believing that just because some concept is quantified it holds inherent meaning.”
Not so.
“Without knowing what a number represents – what it measures, how and why – without that backward tie to reality, any mathematical modeling has no practical interpretation, no meaning.”
Diane Ravitch's blog 
New Jersey is experiencing that with the new school performance reports, which are “confusing, inaccurate, create useless competition, encourage gaming of the system rather than high quality decision making, drive a false narrative of failure…dishearten students and teachers, and cause unnecessary alarm.”
Quantifying junk data doesn’t improve the quality of that data.
Diane,
Here’s what’s happening in my state:
http://triggercancount.blogspot.com
The man crushes on Jeb Bush coming out of Raleigh seem to be really big ones. They have it bad for him, apparently.
I think their grandfathers did not take them fishing in the hidden waters of coastal Carolina’s creeks enough. . .or teach them how tobacco and farming relied on hard workers. I think they consider the taste of down East barbecue with coal slaw on bun to be a God-given right, not something that results from hard work. I don’t think they have read Ephesians 4:3 enough. I don’t think they have driven on the Blue Ridge Parkway enough, or stopped to look at the waterfalls or had a picnic with pie from a mom and pop bakery at a public park along the way. I don’t think they have had school children’s eyes light up when they see them (black, white, Asian, Hispanic), who share with them a connection they made outside to something they learned in class. I don’t think their grandmother read them 1 Timothy 6:6 or sang them Swing Low to sleep with the windows open on a hot summer Carolina night. I don’t think they have slid down the sand dunes at a North Carolina beach or gone oystering in a small wooden boat. I think they have made idols of one another and they are projecting that onto our state. I think they will not rest on these decisions.
I think that not only do they dislike public school, I think they dislike humanity because humanity cannot be measured. As for me and my house, we are going to keep on trying to make public school the best it can be no matter what is going on around us. We love North Carolina and the people in it. I don’t think they realize NC has RTP because of how we have supported public education in this state for so long. I don’t think they know how embarrassing it is to have them doing this to public education in North Carolina.
Well stated. That’s exactly what I’ve said. You can use student test scores to assign a teacher a number, but it does not measure the quality of the teaching.
A great post. The meaning behind the numbers must be disclosed and discussed. It takes both writer and reader investment to understand what numbers are and what they are not. One should also critically consider what the “numbers” (including sample) should have been in order to truly contribute to some knowledge base. Here is a prime example:
Here’s another caution: Do not let reputation replace the documentation in a research study. I have an established reputation as a thorough researcher, but if the day comes when I present numbers and offer as my justification, “I have an established reputation,” then do not trust my numbers, and do not trust me.
M. Schneider: using caps is often counterproductive in online writing [it is like screaming] but I hope that EVERYONE who reads this posting sears your last paragraph into their memories.
EVERYONE!
To touch on your particular expertise: an ethical number/stats person doesn’t have one strict ‘gotcha’ standard for those with whom they disagree and a lax “all is always forgiven” one for ‘their side.’ While I am sure that someone will point out some exceptions, IMHO, in general the only principles that count are those that sooner or later will hurt. Why? Because those with high ethical standards apply standards of excellence and proper conduct to themselves [perhaps more] as much as they do to everyone else.
The edubullies flail in every direction with the most outlandishly contrived numbers & stats all the time, subjecting us all to the most “vicious” and “Dante-esque” pummeling [aw, in all honesty, thanks Rick Hess for the vocab!] they can manage in order to beat us into silence. 13th percentile becomes 90th percentile. A state grading of 76.7 becomes 93.7. 97 sixth graders in 2006 become 62 graduates in 2012 = 100% graduation rate.
The edubullies use numbers & stats “like a drunken man uses lamp posts — for support rather than for illumination.” [Andrew Lang]
Permit me as an outside observer to make a simple statement of fact: when KrazyMathlady or Gary R or Bruce Baker or GF Brandenburg [just to use good examples] use numbers and stats, readers of their blogs and postings [like me] can easily detect a painful restraint on their parts to hold back from saying more than what the data allows.
But they do exercise restraint. In the ed debates they fight hard but fair and lead by example. Any wonder, then, that the edufrauds and their accountabully underlings loath even acknowledging their existence?
Let me hazard a guess why: the transparent, ethical and professional handling of data does not support the leading charterites/privatizers. They only thing they can bank on to support their drive for $tudent $ucce$$ is doctored stats, truncated and visually misleading graphs, and fictitious [I am being kind here] numbers.
Thank y’all again for using your powers for good. You are making a difference.
🙂
I recently read a report about the new, “better” testing to replace old testing in Oregon state schools to which there was this reply: “Gaston [Oregon] superintendent David Beasley says testing can go too far. ‘In eastern Oregon, we have a saying that cattle get bigger because you feed them, not because you weigh them.’”
I think there are a lot of policy makers with man crushes on Jeb Bush.
the face that launched 1,000 tests
It is of course a misnomer to call this data-driven decision-making when these data-driveling cargo cultists violate every standard of practice in the valid use of measurement.
But I suppose it will take the entire field of qualified data professionals rising up in unified protest to prevent these carpet-bagging bunko artists from giving their profession the bad name it is getting today.
In the mean time, Phrenology will have its revival, and the Great Dumbing Down will continue …
Yes, phrenology. and eugenics, from which high stakes testing sprang.
Yep, psychometrics = phrenology = blood letting = eugenics
I certainly agree that knowing how any measure is constructed is essential. I have made this very point when discussing measures of poverty, though my posts were not as well received.
If we proposed to judge schools and fire teachers based on the kids’ time running a mile, everyone would laugh and know it was a ridiculous way to rank schools.
All the more reason to consider using a Data Shield: http://ahlness.wordpress.com/2013/04/18/the-data-shield/
I agree with Schneider and KrazyTA as I also tell people to trust no one, including me until it is all proven out as to what, where, when and how. Just remember Jack Webb. I give people the instructions to the data base so that they can check as Reagan said “Trust, yet Verify.” Good enough for the Soviet Union with nuclear warheads should be good enough for our youth and the future of the country concerning education. You must have credibility. The way we blow their credibility is when they have to put up either in public information requests or in a public venue. You cannot do anything without the data in a proper form so that no one can blow you out. You trap them in the box. I always use their data on them. They think everyone is so stupid and such a short attention span that they can do anything they want because no one checks. The only way I find the rainbow is by that tedious checking I call comparative analysis. It has to come out the same no matter which way you run it or who else runs it. Now ideologues will argue insanity is clairvoyant and that is to be expected by those like Gates, Broad, Walton, Emmanuel, Villaraigosa, Rhee and friends in the warped world.
In reality their arguments continue to not hold water despite the media pushing these failed ideological concepts for their masters. Thank Bill Clinton for the 1996 Telecommunications Act which wiped out the “Free Press” as it is not much of one anymore. We, the people, are now what it depends on.
Even the most precise data will still remain as an approximation of the unseen, the unknown, the psychic, the soul, the humanism, and the sentiment of mankind.
Cognitive processes are mainly seen from an exterior point of view, and the statistics that explain them are far less precise and subjective than the cognitive neuro-process itself. If only we could peer into the working brain all the time and with precision. . . . It’s not going to happen in the near future. It will take a few centuries of development in technology. Our portable MRI’s now don’t work well when a person is ambulating.
Tying test scores to teacher acumen is therefore statistically insignificant compared to holistically measurement.
Don’t you love it when a policy maker opens up a missive with “Research shows that . . . . “, and without even digging too deeply, you find out that their brand of experimental design had about as much sophistication and depth as a hostess cupcake.
Robert Rendo: I was with you there until the very last two words.
Do you know how much shame and humiliation you have inflicted on the hostess cupcake community by implying that they have “about as much sophistication and depth” as the Holy Metrics of EduExcellent $tudent $ucce$$?
You do so much good in the ed debates—don’t spoil it [as the kids I worked with would put it] “Hatin’ on the hostess cupcakes.”
Even a twinkie deserves more respect than any VAManiacal perniciousness unleashed on public education by the edubullies and their accountabully underlings. Let’s fight fair, ok?
🙂
I apologize to aficionados of RIng-DIngs, Twinkies, Ho-Ho’s, Snowballs, Choco-diles, and all those other haute cuisine must-haves.
Come to think of it, Twinkies are far more profound than Mr. Duncan and company.
RR,
“holistically measurement.” by this do you mean “holistic evaluation” or “holistic assessment”?
It seems to me that “holistically measurement” might be an oxymoron in that any measurement by definition has error in it. Now the margin of error might be quite small, let’s say 1/1,000 of an inch, but it is always there. For a measurement to be holistic wouldn’t it have to take into account that error?
Yes, it was typo. Sorry.
Margins of error only are adverse if they are big enough to trigger a standard you are setting. But one has to accept that the standard will always involve some degree of subjectivity. Therein lies the problem for me.
http://quoteinvestigator.com/2010/05/26/everything-counts-einstein/
Not Everything That Counts Can Be Counted