This is one of the best posts I have read in a long while. I have been thinking quite a lot about Big Data and trying to understand why so many Big Thinkers are in love with Big Data.
This post by John Kuhn helps me figure out how this happened and what it is wrong.
He refers to Campbell’s Law when he describes the principle that the more a measure counts, the more it distorts the very process it was intended to measure. That means that an audit should have no stakes attached to it. If testing is high-stakes, then people start acting as if the test score is the same as education instead of a measure.
This is where John Kuhn starts, and it gets better and better with each paragraph, written in the fierce and direct style of this great education thinker of our day:
While data (plural) should inform, each datum wants to rule alone. In an America that is uncomfortable with nuance, we have two dominant political parties, two dominant soda brands, and so on. We like to reduce things to manageable-if-extremely-imprecise chunks. As such, a single datum such as a Standardized Test Score–like the ring in The Lord of the Rings–invariably wants to take over. Data-informed quickly gives way to data-driven, and then data-driven gives way to datum-blinded. And that is, in my opinion at least, where we live today.
Another excellent point:
Data, like fire and shotguns, is neither intrinsically good nor bad. In fact, like fire and shotguns, it can be a life-saver when used properly in the right circumstances, and it can be deadly when used improperly in the wrong circumstances. Teachers and parents who get labeled “anti-testing” (because, again, nuance is hard) are often not at all against testing. The vast majority of the so-called “anti-testing” teachers give tests in their classrooms. So it isn’t the test that motivates much of the opposition to reform. And it isn’t the data, either. It’s the fact that many, many stakeholders don’t trust the people hoovering up the data to use–they presume, because of their experience with the school reform movement as it has unfolded–against students, teachers, and schools.
Truth is, I could quote almost every line in this excellent post, because every line has the wisdom of experience, the wisdom of an educator who cares about teachers and students, and who doesn’t want to see them abused by Big Data.

This makes sense to me: “Data should indeed inform decisions. And, indeed, educators need data at their fingertips. But data should come with some serious warnings, and we’ve seen just how bad things can get in education.”
But I don’t agree that data demands all your time, or that “big data drives out little data.”
Moreover, there is important data beyond standardized test scores and graduation rates.
LikeLike
It’s easy for you to say data doesn’t demand all your time Joe. You are NOT on the front lines, in the classroom, forced to give never ending standardized tests, rank and sort, use data to defend every decision. Maybe you should leave this one to those who are affected: children and teachers. You’re not an expert at everything related to life under the RTT charade. You are merely an observer. Stop preaching to the real educators.
LikeLike
Many fine educators see a great value to using forms of data. Like anyone else, you can decide what to read and what to believe.
I am on the front lines of relationships among educators, and among educators, families and community groups. That’s where I spend a lot of time each week.
LikeLike
And many of us are in the classroom with children every day. You are not.
LikeLike
Diane, Reminds me of something from the movie “Race to Nowhere” where a leader in education said that high school is no longer preparation for college, but preparation for the application process for college. What a crazy, crazy world of education we are living in! Things are upside down and backwards!
LikeLike
Here is an instance where I profoundly disagree with John Kuhn. John writes,
“The best-case scenarios–individualized highly-effective remediation, personalized educational experiences, de-tracking and de-grading students, a great flourishing in American schools–are dizzying in their hopeful promise.”
The process of “personalization” in Big Data is an inherently flawed application of statistical reasoning. It consists of extracting information about a student through test scores or a panel of coded qualitative judgments. That information is used to zoom out and create a data landscape of all the students in a population, like a multi-dimensional Pointillist painting. Algorithms can then extrapolate to other likely associations in other areas within the vast data fields, to project and prescribe specific conjectural outcomes for any one child, with a specific statistical confidence interval.
The next step is to zoom back in again, and use the algorithms of the data landscape to make and enforce intimate, personal decisions about that individual child. We are already there; this is what is already happening under RTI and accountability-driven program placements of individual children.
That’s just wrong, brothers and sisters. Data masters might argue that they only need more and more detailed data to achieve greater resolution, but the coding process itself precludes the exercise of judgment, human insight, and free will on the part of the human coders who are actually in the room with the child, within reach of her individual voice.
LikeLike
“Data masters might argue that they only need more and more detailed data to achieve greater resolution, but the coding process itself precludes the exercise of judgment, human insight, and free will on the part of the human coders who are actually in the room with the child, within reach of her individual voice.”
Well said, chmtchr!
However, I would include the STUDENTS as having the right to “exercise judgement, human insight and free will” in that “coding process”.
LikeLike
I would agree, Duane, but she has her voice, and/or other immediate instruments for her own authentic expression, and we absolutely must have the determination to hear her, and respond to her.
Raising human children is a responsibility we’re all born to, and all our empathy and analog judgment evolved to meet it. It’s not perfect, but certainly no marketing plan for proprietary algorithms has earned the authority to regulate it.
LikeLike
This!
LikeLike
Let me see if I understand chemtch because I think I agree. If we want all humans to become good swimmers we dissect swimming into skills that can be measured and listed as data points. We note that most profient swimmers move 2 legs in a certain way that propels them quickly through the water. This becomes a data point and is listed as a standard common among Olympic Ready Swimmers. Curriculum is written to teach all swimmers to move their 2 legs in a certain way. Some people show up at swimming school with only one leg. Because the data point is the movement of 2 legs and the data shows that one leg is missing the swimming teachers focus on the “skill” of growing another leg so the one legged swimmers can pass that part of the test and improve the data. A one legged swimmer spends years being remediated because of a missing leg and only learns to swim when she goes to a swimming school that stops testing (and failing) her and teaches her how to move from one end of the pool to the other using her one good leg. The teachers in the new school understand swimming so well that they know it can be done with one leg. They, therefore, ignore the data the (harmful) 2 leg data and design a program for swimmers with one leg even if it means their scores on the VAM go down and that results in a decrease of their hourly wage.
LikeLike
Thank you, chemtchr, well said.
LikeLike
chemtchr & Duane Swacker: y’all are two of the reasons I visit this website. Thank you for your insights. They got me to thinking…
I am in the middle of reading a collection of the writings and speeches of W. Edward Deming [please google] aka “the father of quality.” Turns out he was very active in statistics and sampling theory—note his work in Japan starting with the 1945-1952 occupation. He eagerly applied heavy duty “data analysis” to a wide variety of human endeavors. Yet based on what I have read so far he strongly agreed with Henry Clay that:
“Statistics are no substitute for judgment.”
For example, at the beginning of chapter 6, “There is No Substitute for Knowledge,” the editor notes that “Everywhere he went, Deming saw tables of data, computer print-outs, and information of all types, but little knowledge. … Deming would point at tables of data and say, ‘Ton of figures—no knowledge.’”
This reminded me of Deborah Meier’s useful contrast between “data informed” schooling versus “data driven” schooling. She favors the former while the latter characterizes the charterite/privatizer movement.
Link: http://blogs.edweek.org/edweek/Bridging-Differences/2009/03/dear_diane_sometime_i_imagine_1.html
Another point the Deming made is that those using statistical methods and techniques should always be acutely aware of their limitations and ignore—at their peril—the content-specific knowledge and experiences of those expert in other fields. In other words, that his contributions lay in complementing and extending, not substituting for, other types of expertise.
Now think of the charterites/privatizers who shamelessly use numerical data as a form of mathematical obfuscation and intimidation to beat down public education and educators; in the vast majority of cases, they clearly don’t understand even the most elementary principles of such critical features of their belief system as VAM and standardized tests. Yet they forge recklessly ahead, confident that their lack of educational experience and expertise, combined with tortured numbers and massaged facts, will yield positive results.
Or at least, for them, $tudent $ucce$$. After all, wasn’t it one of those old dead Greek guy that gave them license:
“Profit is sweet, even when it comes from deception.”
Forgetting, of course, that he also said:
“Things gained through unjust fraud are never secure.”
My comment is already too long. Perhaps on future comments I will add a few snippets of Deming’s observations re education that even Quixotic Questers will find gratifyingly supportive.
😎
LikeLike
KTA,
I first came across Deming in the early to mid eighties, first in changing over a hospital pharmaceutical stocking, purchasing and distribution system integrating J.I.T. concepts (just in time inventory control) to lower overall purchasing and stocking costs to insure fewer “outages” of drugs. And then at a metal building manufacturer production scheduler and briefly materials manager when I took a course in Deming’s work on continuous improvement, just in time manufacturing and production scheduling.
You are on the right track in delving into his work.
One of the problems I see is that most folks in public education really don’t understand (mainly for lack of experiences) how “process” (the practices and policies) can really have an effect (both positively but many more times negatively) on how we can improve the teaching and learning processes. Most focus on the wrong things-test scores, attendance has to be entered into the computer the first five minutes of class, mis-usage of the intercom, and many many more things. The ability to step back and say “what is the fundamental purpose of what we are trying to do with this particular practice/policy and how does that effect the overall teaching and learning process?” is sorely lacking. Top says jump, middle says jump, and bottom gets jumped upon. Certainly not Deming’s thinking of how continuous improvement of processes should be carried out/implemented.
LikeLike
Anyone who references Lord of the Rings and refers to Asimov’s Three Rules of Robotics, definitely gets it. I love the way he says that the people pushing back are not Chicken Little, but chickens who refuse to get into the pot. Definitely worth reading.
LikeLike
Ha, beat me to the punch, Ellen!!!
LikeLike
“These people [referencing those who are against Big Data] aren’t Chicken Littles. They’re Chickens Who Won’t Get in the Pot.”
Excellent line, John!
LikeLike
In systems-analysis, ‘data’ and ‘information’ are very different. Data that informs is ‘information’. Data
that does not inform is …. just ‘data’. (Information technology v. Data processing) So, in a sense,
if you are a corporate reformer, there’s a little irony in the phrase ” Data-driven decision-making”.
LikeLike
For me, a big issue is this: Data about what? I challenge those telling me to test test test my students to give me good data on these things:
• Which of my students said they didn’t like reading at the beginning of the year but now read constantly having gotten “hooked on a book”?
• How many of my students have figured out that students with special needs are just like everyone else – individuals with unique strengths and challenges?
• Who are the students coming into school tired because they stayed up too late or skipped breakfast?
• Which student has become a “big sister” by engaging in a little gentle nagging to encourage others to get to class on time?
• Who deserves praise for writing a genuinely moving piece about a difficult time in her life?
• Who has stopped acting like he doesn’t care and started asking questions?
• Who has excellent reading comprehension which might be missed because of poor spelling and handwriting?
I actually have “data” on each of these issues. I know the answers to these questions and hundreds more. Perusing my students’ results on the latest Pearson product hasn’t given me anything nearly as useful.
LikeLike
Middle school teacher, those are great examples. One thing fascinates me is the the National Board for Professional Teaching Standards has hired Pearson to work for them. Given the venom that’s regularly displayed here toward Pearson, I’m surprised this happened. But it did.
LikeLike
Joe – we on this blog are interested in the issues and informed. There are many in the general public who are clueless about what is happening. The only reason parents have gotten involved is the testing results and the impact assignments have had on their daily lives via homework and/or a child’s sudden dislike towards school (at least in NYS). To most, Pearson is an unknown quantity.
LikeLike
I agree that many in the general public are not as involved as people who post here. But given all the criticism Pearson has received on this blog, I am surprised that people who are so critical of Pearson have not challenged the National Board’s use of Pearson.
LikeLike
Middle school teacher points out,
“Perusing my students’ results on the latest Pearson product hasn’t given me anything nearly as useful” as actual first-hand knowledge of students.” The Big Data industry would like you to upload a few observations of student behavior. Susan OHanian points out that Pearson has a product for that, AIMSweb® Behavior.
Here are her excerpts from the Pearson brochure:
“RTI for behavior:
“At Tier I,the module screens for behavioral
and emotional problems, prosocial behavior problems, and problems with motivation to learn. At Tiers 2 and 3, it helps you target these problems, provide effective and proven strategies for improving behavior, and create progress monitor forms.
“Comprehensive behavior management system in the classroom: As the accountability requirements for school programming increase, there are fewer tools to help schools meet these requirements. AIMSweb Behavior fills this gap. . . .
“•Universal screening
•Research-based interventions
•Progress monitoring
•Data management
“Presents a complete program to screen, intervene, and monitor students’ behavioral performance that you can tailor to meet
the structure of your behavior management program. K-12 . . .
“•Early identification:
Documents more than just Office Discipline Referrals (ODRs). AIMSweb Behavior identifies
students before behavior problems result in ODRs and/or poor academic performance
“•Evidence-based interventions from leading behavior and social skills experts are provided within the system. . . .
“With AIMSweb® Behavior, you screen all students—ideally up to three times per
year—to establish each student’s level of behavioral and social skill functioning. . . .
“A teacher form that rates students K–12. With
27 items, the assessment takes only a few minutes to complete
•A student form for grades 3–12 that:
–Identifies problems that are not easily observed. . . (emphasis added)
“Scoring results in one of three risk levels—
normal, elevated, and extremely elevated.,, ”
Click to access AIMSweb-Behavior-Brochure.pdf
LikeLike
OHanian’s Data Command Force
http://susanohanian.org/data.php?id=535
LikeLike
The problem is that we have reduced everything to a numerical value. We’ve taken the humanity out of education – true learning is an experience, and testing should be an evidence of this, not the means to an end. To borrow an old cliche – it’s about the journey, not the destination. I’m afraid we’re too far down the road to turn back. Thanks for the post!
LikeLike
I don’t do scantron tests. Why would I need a machine to tell me something that I glean from going over student work and tests?
LikeLike
Reblogged this on David R. Taylor-Thoughts on Texas Education.
LikeLike