The Shanker blog published a badly needed critique of our policymakers’ insatiable appetite for data by sociologist Esther Quintero.
Quintero makes the sage observation that you need to know why you are collecting data before you acclaim its value. You need to have a theory that you are testing. Without knowing which idea you seek to validate, the data prove nothing other than your ability to accumulate lots of it.
She writes:
“At a basic level, it [collecting data] seems to signal a general orientation toward making decisions based on the best information that we have, which is a very good thing. But there are two problems here. First, we tend to have an extremely narrow view of the information that counts – that is, data that can be quantified easily. Second, we seem to operate under the illusion that data, in and of themselves, can tell stories and reveal truth.”
But the data we can get may not answer the most important questions. First, we must define the questions.
She concludes:
“My colleague recently wrote that NCLB “has helped to institutionalize the improper interpretation of testing data.” True. But I would go even further: NCLB has helped to institutionalize not just how we handle data, but also, and more importantly, what counts as data. The law requires schools to rely on scientifically-based research but, as it turns out, case studies, ethnographies, interviews, and other forms of qualitative research seem to fall outside this definition – and, thus, are deemed unacceptable as a basis for making decisions.
“Since when are qualitative data unacceptable in social and behavioral science research and as a guide in policy-relevant decision-making?
“Our blind faith in numbers has ultimately caused impoverishment in how (and what) information is used to help address real world problems. We now apparently believe that numbers are not just necessary, but sufficient, for making research-based decisions.
“The irony, of course, is that this notion is actually contrary to the scientific process. Being data-driven is only useful if you have a strong theory by which to navigate; anything else can leave you heading blindly toward a cliff.”
Diane Ravitch's blog 
This is a very insightful piece. I agree with the overall gist but would suggest two important qualifications.
1. Yes, it is true that those who believe in the power of “data” fail to interpret it correctly–and that this is because they fail to begin with a theory or question. But the problem goes deeper than Quintero suggests. She writes, “Excessive faith in data crunching as a tool for making decisions has interfered with the important task of asking the fundamental questions in education, such as whether we are looking for answers in the right places, and not just where it is easy (e.g., standardized test data).” But before asking such a question, one should ask, “what are we hoping to find out here, and what does each of our key terms mean?” If our main question pertains to ‘achievement’; one should ask, “how am I defining ‘achievement’ in this context, and why?” Give up or ignore such a question, and everything else gets muddled.
2. Instead of expanding the definition of “data” to include enthnographies, interviews, and so forth, I would narrow it but acknowledge that data do not provide all the important information. This is a technicality but a key one. Data, in their traditional sense, are comparable bits of measurable information that, when collected in large numbers and interpreted in relation to a research question, can provide information. Student essays do not count as data, because you cannot treat them as comparable bits or break them down into comparable bits (rubrics aside). Some scholars would disagree with me and say that the term “data” has loosened over time to include things that can’t be directly compared with one another. I see problems with such loosening: in particular, methodological confusion and corruption. Calling something “data” often sets the stage for breaking it down into comparable and supposedly measurable bits, when it should not be broken down in such a manner. On the other hand, I see many reasons to consider things that do not fall under the stricter definition of “data.” Just call them what they are: essays, interviews, or what have you.
I wrote about this here: http://dianasenechal.wordpress.com/2012/06/30/the-misuse-of-data-the-word/ (non-satirical piece) and here: http://dianasenechal.wordpress.com/2012/11/10/student-shows-23-percent-growth-in-finding-central-idea/ (satirical piece).
Also, for an alarming example of data-worship in a second-grade classroom, see this video (thanks to Robert Pondiscio for originally pointing it out and to James O’Keeffe for reminding me of it):
http://blog.coreknowledge.org/2010/12/09/data-is-fabulous/
I am going to cross-post this comment at the Shanker Blog.
LikeLike
Reblogged this on Transparent Christina and commented:
Perfect analysis of how our lame duck Governor is hurting Delaware schools with his policies.
LikeLike
The business/data model is informed by the notion that we can obtain more productivity for the same inputs by simply demanding or coercing it. The theory holds that if we create conditions of higher expectations but not increase resources, the outcomes will leads to working harder or smarter. The problem is that there is a kernel of truth to this theory as we can all see in our own lives when we are not working to full capacity.
But the notion that we can will our way out of a problem ignores the reality of context. If we funded the NASA lunar landing program sufficient to get us to the moon but not back, Neil Armstrong would still be on the moon–and not in very happy. Government leaders recognize the limits to “high expectations” budgeting: Every congressperson buys a round trip ticket to return home during congressional breaks. They don’t rely on
human will to make two trips on the cost of one ticket.
We know the real costs of quality urban education and we know that for the most part the nation does not budget based on projecting those costs–even with all the precision that data can provide. Instead of actually costing-out education in advance based on desired outcomes, we use a “residual funding” formulas: whatever is left over after funding courts, prisons, police, roads, and government and social services, gets tossed into to the leftovers education budget. No one pretends that level of revenue is sufficient to accomplish the goal.
Indeed, most government and corporate education reform leaders spend at least $20,000 a year to send their children to private schools. They know there are limits to working harder and smarter–even for the rich and privileged– and the outcomes of squeezing the turnip of public education funding are only a few drops of improvement.
The lesson in this is we should not let locker room platitudes become our singular theory of change. There is always room for improvement, but the education problems did not develop by human will alone and the will never be solved by human will alone.
LikeLike
I am curious about the source of the statement “Indeed, most government and corporate education reform leaders spend at least $20,000 a year to send their children to private schools”. Is there a data base that collects this data?
LikeLike
I wrote a comment, but it seems to have gotten stuck in the moderation queue.
LikeLike
Data is not a perfect synonym for information. When Bill Gates directed OVER A BIILLION outside DOLLARS to promote small schools a decade ago, he did so based on data. He believed the FACT that small schools were over represented in the ” most effective rankings” meant something meaningful. Districts across the country broke up larger schools, hired more administrators and such , and many are still doing so. Eventually Gates ABANDONED the initiative and went after teacher effectiveness.
Gates was a cheerleader for a unjustified but ” data driven” disaster. Why? Because his genius did not extend to statistical analysis. If it did he would have seen that the least effective schools in the same data he used also showed that the lowest performing schools were also over representively Small!
He did not understand how to turn the data he had into meaningful information. One researcher concluded, “we have done a terrible disservice to children” .
Currently, there is a near perfect negative correlation between use of the term ” data driven” and understanding of education.
Beware, those most vocal about being “driven by data” are those least likely to get the school bus where it needs to go.
LikeLike
Re: Currently, there is a near perfect negative correlation between use of the term “data driven” and understanding of education.
Currently, there is a near perfect negative correlation between use of the term “data driven” and understanding of data.
LikeLike
Well put.
LikeLike
It’s called the Phrenology Paradigm.
LikeLike
Reblogged this on Thinking in the Deep End.
LikeLike
“Data driven”= Driven to collect more data.
LikeLike
The problems begin when the people analyzing the “data” do not have even a basic understanding of what public education actually involves. When people who attended private schools and send their children to private schools are allowed to drive the education agenda, then there is a skewed vision of “”what works” for other people’s children. When the decision makers are allowed to make policy based on outdated or uniformed information, or worse yet, to ignore the truth about the relationship between poverty and education, then education policy is doomed to fail ( i.e. NCLB /RTTT) . Until one actually spends real time in a high poverty school, it is impossible to understand the myriad of social problems that these children bring with them to school. That is where “context” becomes a critical component of the data that is often ignored by the education “deformers”.
What is truly amazing to me is the sudden onslaught of “experts” on educational policy who have little or no experience in educating children nor any substantial knowledge or academic background in educational theory and practice. Somehow our president seems to think his buddies from Harvard and other elite institutions know what is best for children in poverty. Their elite education may be of some value, but until they can back it up with real life experience with real life students, then their decision making will be based on false premises and uninformed by real life applications. Two years experience is laughable in terms of education.
Until I spent time with students and families, I too thought I knew what was needed to fix education in my high poverty school district. As an administrator, I now understand that data has its place in our decision making, but it can’t take the place of individual student/family information in making important decisions that affect our students. There are no easy fixes to difficult problems. Gates and Duncan and Rhee and all of the others haven’t a clue. It is time for the real education academics to come forward and have a seat back at the table. Public education should never be put on the hands of “for profit” entities.
LikeLike
Perhaps it would be helpful to point out the people who do “have a clue” and what they advocate.
LikeLike
Scientists continue trying to impose their idea on Nature, rather than observing Nature and letting her present her evidence to us. We don’t observe suppression. We are blind to the facts. The facts are that our society is a society of suppression. It begins with ideology, regardless whether this is religious or atheist. Then follows education, which is meant to make you a memory bank, engaged in classification, enumeration and calculation. We have effectively been reduced to bookkeepers, if we succumb to the demand for numbers as a sign of proof.
LikeLike
Please don’t put this on scientists, or any community of genuine inquiry. What we have here is a failure to care about the facts, driven by ideology and implemented by bean-counters.
LikeLike
Scientifically-based research!? Bunk…not the research I read or is reviewed in research universities. Lawmakers are not known to be scholars…just play one on TV. They need to take the basic high school tests and publish their results. Are they even smarter than a 5th grader? It is embarrassing to have our ignorance spread all over the world, via Internet and CNN, as we beat up on tea hers and children in the name of education.
LikeLike
The so called research on what works in education has one great flaw. The human element in play when the research is applied in schools with adults and students who interact with each other in unique situations. My experience has been that each student and adult bring to the situation a myriad of prior knowledge and experiences that have a huge effect on the outcomes actually realized when the research based practiced are implemented. Also, the fidelity with which it is implemented varies greatly from situation to situation. These are things that are difficult to measure purely using a quantitative approach to data collection and analysis. Thus the importance of qualitative data.
LikeLike
Numbers are easy. Numbers are cost effective. You can chart and graph them. Make ’em all preeeety. Anything else requires work, analysis and reflection. This is a crisis, they would argue. We don’t have time to study the problem and write ‘papers’. Those silly academicians . Neolibs, DFER, billionaires, Arne, Michelle, Joel and others have told us the solution. Now, just get the data to prove it. If not, create or selectively choose.
LikeLike
One way to combat this sickening obsession with data – the good old Data Shield: http://www.flickr.com/photos/ahlness/5516514694/
LikeLike
It’s a symptom of the globalism sickness: http://temkblog.blogspot.ca/2012/11/death-of-vision.html
visionaries and morally directed leadership: “panicky by-products of a lack of control in an era of false computational certainty” – we love to think like our technology acts – in the 18th and 19th centuries we were clockwork people and mechanical creatures… nowadays we define ourselves by the machines we surround ourselves with.
LikeLike
What is missing here is a voice of someone who has successfully used data in a real school and grade level team context.
The confusion seems to come from the misconception that data blindly drives instruction. In a well developed system, which evidently is not in place in the districts/institutions from which the comments have emerged, theory (based on the convergence of multiple sources of data/information) drives the decisions about instruction and data confirms whether the theory or hypothesis was on the right track. Based on the progress monitoring data, either a new hypothesis is developed or the existing one is maintained. It is action research that is designed and executed by teachers. It empowers teachers to meet the needs of all of their students.
And lets not forget that the theory is created to address a problem or need. Progress monitoring data is directly connected to changes in that problem.
A well organized system with clear guidelines for data use is needed to pull this off. It can be successfully implemented at the district, school, grade, classroom or individual student data level.
It is silly to debate the use of data and associating the word “ignorance” with it is inflammatory.
LikeLike
This is true. Qualitative research in social and behavioral sciences is valid, unless you think that all human activity can be quantified, enumerated, and manipulated by a computer algorithm. Teachers have little experience with actual science and actual research methodology and how to interpret research studies. They get intimidated when they hear or see the term “data.”
LikeLike
This is such a critical point–and it deserves to be at the center of a good science education. deb
LikeLike