A reader wonders about whether the data can be trusted when there are so many reasons to inflate the numbers.
She writes:
“Merrow finally points out the most explosive point in this memo, one that’s being passed over in commentaries. Fay noted that specific erasure patterns occurred across many classrooms. That would indicate that systematic cheating occurred at the building level, and point toward the principals.
“Massachusetts has never requested erasure analyses, so far as I know. I wish they would. My own district is in an odd bind, because it was an education reform superstar, ranked at level I (the top 20%), in 2009. We’ve been data-driven for a decade. This year, we are flagged as level 3, the bottom 20%. I’m concerned our new principal will be blamed for testing a higher percentage of our actual students, but maybe there’s something more.
“I’m walking on eggs here. The state assistance team that came to our building quoted an improved graduation rate, and I want so much to believe it, but it didn’t make statistical sense. They told me they couldn’t divulge the denominator they used, because of confidentiality involved in tracking students after they left the building. The gap between the number of kids in the testing rooms and the number of scores reported makes actual data analysis impossible. I have no idea what any “data” anywhere means, even if the tests did measure anything worthwhile.”
Diane Ravitch's blog 
There are other issues that need to be included, even if there are no problems with erasures. States report scaled scores. They can and do change the conversion rate from raw to scaled in order to show “improvement” year to year. The year before I taught in Virginia the school in which I taught had a 58% pass rate on the Middle School American History test. The year I was there the pass rate was 81%, my own was 89%. Sounds great, right? Except they had lowered the cut score and changed the conversion. If the previous year’s raw scores had been converted using the same matrix, the score would have been 71% or so. Thus we showed “improvement” but not that much.
TK,
As you state pass rates, conversion rates, improvement all a bunch of Grade AA USDA Choice Bovine Excrement. Much ado about nothing. Have you read Wilson’s study I mention below?
If so, your thoughts please.
Thanks,
Duane
have not, but have it in a window to read later
cannot get to before this afternoon at the earliest
It’ll probably take more than an afternoon. See below for another of his writings on the subject.
I simply said I wouldn’t get to it before this afternoon. I have taken a glance. FWIW, I read over 1000 WPM and absorb info very quickly. I will get to it when I get to it.
I’m a slow reader. Good luck and happy reading!
Until we abandon the use of grade-level standardized tests and move toward individual mastery tests we will continue the kind of shenanigans teacherken describes… We have the wherewithal to move away from the factory model… we only need the willingness to do so
wgersen,
Have you read any of the Wilson studies I have linked to? If not I highly suggest that you do.
Duane
I think the DEFORMERS get a kick out of blaming teachers for their own egos. Plus they make money too. I won’t name them, but they are everywhere.
From the reader “I have no idea what any “data” anywhere means, even if the tests did measure anything worthwhile”
The data (in this case educational data) means whatever the individual who looks at it determines it means, nothing more and nothing less. In other words speculation, opinion, nothing, everything, falsehoods, maybe by sheer chance some semblance of truth, etc. . . .
The fact (yes it’s a fact, an irrefutable truth) remains all “data” gleaned from the standardized testing process is, as Noel Wilson states, “vain and illusory”. Wilson has pointed out thirteen errors, any one of which destroys the validity of the making, using and disseminating of the results of educational standards and the supposedly “measuring” of them though standardized testing, that render the whole process completely invalid (and I added the “completely” as emphasis because once a process is proven to be invalid it can never be valid no matter what the psychometricians may assert).
To understand why the above “fact” is true please read Wilson’s “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700 .
Also, if you’re not so inclined to read this study (as I’ve found to be the case with almost everyone, mainly supposed educators, I have given a hard copy to or implored to read on many an education blog) then read his “A Little Less than Valid: An Essay Review” found at: http://www.edrev.info/essays/v10n5.pdf
Oh! Oh! Oh! Oprah! Come to Jesus, Honey! This is gonna get hotter than the hinges on the GATES of Hell!!!
Maybe the OW can have a “Rhee Repents” show.
I think of all those children who literally got cheated out of the chance at an education. How do you make that up to them? The buck stops with Gates, et al.
ML,
One can’t. And you hit at the heart of the problem. So many children for so long have been cheated out of a proper education due to the rephormers’ methods. In the olden days they (the reformers) would have been drawn and quartered, or maybe skinned alive, not that I am suggesting that now. My comment was meant to be facetious as I don’t watch TV as I refuse to watch such refuse (especially were it to involve the rheeject.
Duane
Duane Swacker: as such a big Oprah fan, I am surprised you missed her special where she distanced herself from her formerly ‘Recommended Reading List’ number#1 pick, the moving story of rhadical rheephorm called A MILLION LITTLE PIECES OF MASKING TAPE (Michelle Rhee, ghostwritten by James Frey).
I know I was particularly struck by her rhadical rheedefinitions of such old-school terms as “cut scores” as she movingly described how she cut effective classroom teachers out of high-needs schools as well as cut the poverty rate among data-drivel [not a typo] principals with impossibly high WTR erasure rates by shoveling bonuses at them.
It is such a brilliant primer about $tudent $ucce$$ that I have it on good authority [hush, Arne!] that Adell Cothorne displays them prominently for sale at her cupcake store.
Honest. You can’t make this stuff up.
“Laughter is poison to tyrants.”
🙂
With the emphasis on data, I have spent considerable thinking about how we use data. I have a close friend who is majoring in Informatics which involves massive amounts of data collection, mining and dissemination. We conversed about the idea of meaningful data. And this is the problem.
There is no agreement on meaningful data. As someone noted in another post, it depends on who is interpreting the data and what their bias and / or purpose might be. For example, some local schools have seen declines in test scores but increases in graduation rates. What does this mean? Depends on who gets asked.
I also see us collecting data for the sake of collecting data. Pearson has a program that we have access to called “Pulse.” We can see student test scores throughout their academic career. As a high school teacher (11th/ 12th grade), 3rd grade reading scores are of no real use to me. Data is simply being used as a weapon rather than a tool by certain groups.
And it’s difficult to say if the data means anything at all. High school kids don’t try on non-college admissions tests. The juniors blow off our state assessments and admit it. If that’s the case, what are we measuring. The state changes cutscores every few years. Gaming the system is easy. Charters often don’t replace those who disappear or get counseled out. As students leave,there percentage of proficient students increases. Sure, they shed the worst and kept the best. But again it depends on who is interpreting.
By relying so heavily on data, much of it unreliable or meaningless, we’ve managed to make getting the data we seek into a game rather than a sincere endeavor. When people get bonus or pink slips based on this data, it really becomes a game. Isn’t that right Ms. Hall and Ms. Rhee?
An excellent post.
I have been confused when the term “high stakes” is applied to tests which are actually no stakes exams for the students being tested. The students in my household have always been far more stressed about teacher written exams that count towards grades than state standard tests which do not count toward a grade or college admission.
The conundrum we face is that high school grades are data as well, and without knowing the context in which each student was awarded each grade, the grade is meaningless as well. How do we compare grades across different teachers, different history classes, different buildings, or different school buildings?
I don’t think of the end of year standardized tests as “high stakes” for the students. In my state, at least, the students will still advance to the next grade.
Those tests ARE, however, high stakes: for the buildings, teachers, principals and districts. That’s one of the MANY problems with using these tests to “grade” schools. Kids don’t take them seriously. And why should they? And I try not to remind the students that our school depends on their scores. That’s unfair pressure to put on a young teenager. But that’s really what’s happening.
Jennifer Baker, you make an important point. High-stakes tests puts the power in the hands of students to determine the reputation, career, and livelihood of their teachers and principal. How good is that in making children masters and destroying adult authority?
but perhaps the entire notion of “authority” is what is skewing the process of education. I did not view my role with respect to my students as one of authority, either as to content knowledge (although I certainly knew more than almost all of my students, who were largely 10th graders while by the end of my career I was in my mid 60s) or with respect to the operation of the classroom. If there is not some degree of shared power how can there be shared responsibility? The students have to have some degree of ownership over their own learning, with me – and their peers – available to assist them.
When I had a circumstance of an Advanced Placement class that arrived with almost no one having done the preparatory reading, I walked out of the room after telling them I cannot do my part of the learning process if they have not done theirs. Within ten minutes they had organized themselves and began to go through the material in a systemic fashion. The discussion might not have been as focused as when I would be leading it, but since most of the time I made students lead the discussions anyhow with my role being to be sure they did not miss key points, they had some preparation for what they were doing.
Given that our schools should be preparing our young people to be participants in a liberal democracy (and that is the correct political science term and implies nothing about either party affiliation or ideology) it is surprising how undemocratic the experience of schooling really is.
One last point – one can by force or threat obtain sullen compliance, but that is not a positive learning environment in which students will be willing to take the intellectual risks necessary for truly deep learning to occur.
One of the many problems cited with these exams is that they create excessive stress on students. I think we agree that this is not the case, at least for high school students.
TE,
Who has determined that we must “compare grades across different teachers, different history classes, different buildings, or different school buildings?” Who is that almighty authority?
I find it amazing that this country managed to survive and prosper before these comparisons were being made. It must have been one hell of a lucky streak that we survived into the late 1900s without comparing this sort of data. Amazed, truly amazed I am!!!
Duane
Duane,
One issue is increased mobility of students. They may go to school on one side of the country, work on the other side.
In my Adamowski influenced district in CT there were many students who were moved to the non testing (MAS) group. Also, a principal told students to leave the answer blank if they didn’t know it thereby eliminating wrong to right erasures. Then he sent out an email telling teachers not to take attendance for one week. Absent students and students who refused to attempt the test scored at the proficient level. Adamowski threw a big party for the unbelievable increase in test scores.
LIES, DAMN LIES AND STATISTICS.Yes the data can and is manipulated. Many districts know this put it’s all about the numbers supporting your fake reform efforts. Not all schools are cheating but there’s a hell of a lot more than what is being presented. Seems to me that when schools, in one year, take a 50-100 pt jump, bells and whistles should sound and investigators summoned. This is not the case, especially in LAUSD.
The bells and whistles that should sound should be those that all these data points are pure bovine excrement and nothing else. Unfortunately they can’t even be used as fertilizer unless one composts the paper on which the results are written and even then I might be afraid to use it due to the sheer toxicity of the crap that went into the compost.
Money can make any statistic look the opposite of what it is. The purpose of a free press is to keep the politicians honest.
In New York, the tests are sent off to be scored. The children that are over- achievers always ask how they did on the tests. We do not get their scores back until August or September. We have to TRUST that they have been scored correctly and TRUST that the data that has been collected is accurate. How can this test taking procedure even remotely be considered to be a learning experience for children? They never get to see the questions again and learn how they actually performed. How can TRUST be established that there are no computer error/glitches made when determining a teacher is EFFECTIVE or INEFFECTIVE and possibly be terminated when the scoring/ranking formulas are confidential?? Confidential and executed by computers and total strangers. People who have probably never left Albany. Really?!
The credibility of this system is declining rapidly. We have reached an all time low.
Marge
The same concern could be voiced for end of the year exams designed by teachers in classes. They are not learning experiences, e students must trust that they are evaluated correctly and final scores are correctly calculated.
TE,
Agree, grades are crap anyway, almost all involved know it but still play the game (myself included because I love teaching).
Duane
The data is simply a tool for achieving the ends of the reformers and will mean whatever they say it means in a particular instance. Here in Florida, the birthplace of educational deform, legislative handmaiden to ALEC’s educational law perversion arm, the goal posts are moved so frequently that no one can really say what the goal is anymore, if they ever really could. The most obvious goal was to enhance Jeb Bush’s political resume and personal fortune while smacking down the FEA for daring to question some of more idiotic ideas and that, in itself, was an embarrassment of bullying in this right to work state where the union has no power whatsoever and actions taken by the union are illegal and levied huge fines and jail time, leaving them a toothless milquetoast of a figurehead.
FCAT scores are used over and over again to “prove” that Jeb Bush’s reforms worked (the so-called “Florida Miracle” so as to sell it to other states), Rick Scott’s rule has been golden, and at the same time the same scores are used to “prove” that public school teachers are public enemy number one and we must have more vouchers, unregulated charters by the thousands, and fly-by-night online education profit schemes mandated by the state.
If the results of the FCAT come out poorly (like they did last year, by state design — more “rigor” you know) then the state simply drops the cut score so they can say that they were right and their ridiculous programs are working, pay not attention to the man behind the curtain changing the results. Repeatedly. That puts them in a bind though because they need to use that same data to close public schools and fire teachers and open up the tax dollar profit pipeline to private corporations so they are in a real dilemma.
How do you keep voting parents from storming Tallahassee with pitchforks and torches while dismantling the public school structure to the tune of Jeb Bush’s fever dreams? Say the data shows whatever you want it to mean, tailored to the particular audience you are speaking to at the moment.
And contrary to the “teachingeconomist” these tests do produce high anxiety in children here and everywhere they are used for high-stakes decisions, something Jeb Bush is peddling all across the nation and is successfully selling to Republican and Tea Party legislatures everywhere. If you don’t “pass” the 3rd grade FCAT tests you are retained. Of course what counts as passing changes every year and sometimes more than once in a year (is it an election year?). If you don’t pass the high school FCAT exams (5 now required) then you don’t graduate from high school. There is talk of offering a tiered set of certificates of completion in lieu of diplomas but no word on how that will work with college applications.
Florida’s educational system has become such a looking glass world of up is downism and opposite world that I defy any thinking. educated person to make sense of anything that we are doing here in our public schools.
Since we all were poorly trained in schlock education schools, it would appear to be time to clear the decks. Who wants a bunch of stupid, cheating whiners teaching their children? Why waste time? I know! Why not require all business and pre-law graduates from prestigious school to teach for three years before they are allowed to attend graduate school?! To catch the slick ones who try to by-pass service to their country by majoring in music, make attendance at a prestigious graduate school dependent on three years of teaching before entry. Entry into the job market from certain schools in certain professions even at the undergraduate level should immediately trigger the service requirement. I’m sure we all want the highest quality professionals teaching the next generation.
To get a real dropout rate and close to the graduation rate take this years enrollment and compare it to the class 4 years before in the 9th grade as that is the same group. Even LAUSD best high school, Grenada Hills with an over 800 API has a 34% dropout rate. If you dropout you certainly cannot graduate.
I just read this article on NPR and noticed the parallels to education. I hope it copies, since I couldn’t find a link:
That constellation of information known as Big Data can be a sight to behold.
Adam Frank of NPR’s 13.7 blog explains Big Data as “the ability to understand (and control) a seemingly chaotic world on levels never before imagined.”
Big Data is like gathering digital dust, says New Yorker tech blogger Gary Marcus. “It’s a very valuable tool,” he says, “but it’s rarely the whole solution by itself.”
This new information era is brought to you by a few different factors, says Chris Barnatt, who teaches computing at Nottingham University in London and runs ExplainingComputers.com.
“We’ve got more powerful processors. The cost of memory gets less. The cost of storage gets less. And also, a lot more data is being created,” he tells NPR’s Jacki Lyden on weekends on All Things Considered. “Everything from social media sites, to buying things online with e-commerce, to simulations going on in companies’ medical facilities. Everyone is creating more and more information.”
Marcus says “Big Data” versus just “data” is really a matter of magnitude. But he says that’s important because quantity can actually make a difference in the story the data tells.
Here’s the catch: While Big Data can uncover correlations between data points, it doesn’t reveal causation. Sometimes, that doesn’t really matter, but other times, it might — in ways we’re not always aware of.
For instance, the city of Boston developed a smartphone app called Street Bump to track potholes. “It passively collects GPS data and accelerometer data so it can report when you drive over potholes,” Kate Crawford, a researcher for Microsoft Research and visiting professor at MIT, tells Lyden.
The thing is, not everyone bouncing around on bad roads has a smartphone — or the app, for that matter — to help gather the data.
“And indeed, this maps very closely around how rich a particular neighborhood is and what the ages of the people who live in that neighborhood [are],” Crawford says.
She says the ethics get tricky with that sort of skewed data. The city is trying to address the discrepancy by working with academics. But Boston isn’t the only city taking on such projects. The Wall Street Journal reports on an initiative in New Jersey to manage traffic.
To summarize the tension around Big Data, New York Times reporter Steve Lohr quotes Albert Einstein: “Not everything that counts can be counted, and not everything that can be counted counts.”
Lohr, who writes about Big Data and privacy, tells Lyden he’s more concerned about what the data gets wrong than how much is revealed about us.
“The real danger for most of us is discrimination by statistical inference,” he says.
Remember when you searched for “deep fryer” online for your cooking class? Well that action could now be associated with unhealthy behavior, and the nuance gets lost.
It’s not just online tracking consumers should be wary of, Lohr says. Credit cards, for example, store a lot of information that you might not think of. His advice:
“Put the health club membership on the credit card, but your visits to the liquor store should be in cash because those things will follow you in ways you don’t know now.” [Copyright 2013 NPR]
The quote by Albert Einstein is another one of those bumper stickers we need to make!