A frequent commentator, Bob Shepherd, with many years in curriculum development, education publishing, and assessment, offers sage advice:
“The tests are infallible. They are objective measures. And we know that because they produce data. And not just any old data. Data with numbers and stuff. Very rigorously determined raw-to-scaled-score conversions and cut scores and proficiencies. Super-dooper, charterific, infallible data. Lots and lots of it. I mean lots. Tons. You wouldn’t believe the data!!! Data for days. Rivers of data. Big, big data.
“If the new tests show that 70 percent of students are failures, that’s because 70 percent of students are failures. And if the tests show that 70 percent of our students are failures, that’s because 70 percent of our teachers are failures too.
“You see? The data show that those shiftless, ungritful kids and teachers just can’t measure up to “higher standards” produced by folks with VAST experience as educators. Folks like David Coleman.
“And that’s why teachers need to be replaced with educational technology.
“And that’s why the public schools need to be closed down and replaced with private schools and charter schools.
“And that’s why the country needs to spend about 50 billion dollars making the transition to the Common Core and Big Data.
“Because the Common Core data show a 70 percent failure rate!!!
“Because numbers in a report, however they got there, are never wrong!
“Why are they never wrong? Because they are data!
“data data data data data
“You see?
“It couldn’t POSSIBLY BE that the tests are poorly conceived and written. It couldn’t possibly be that the standards are likewise poorly conceived and written. It couldn’t possibly be that what’s being called data-driven decision making is a variety of NUMEROLOGY.
“Because the masters who designed these tests and these standards are infallible. They are the best makers of tests and standards (well, if you use those terms very, very loosely) that a plutocrat’s money can buy, that is, if the plutocrat is in a hurry, and if he doesn’t really give the matter much thought. You know, if he does this in the way that ordinary, nonplutocratic folks might, say, order up a pizza.
“Glad I could straighten that out for you.
“Just remember: The DATA show that everybody failed and needs to be fired and that everything needs to be privatized.
“Oh, and lots and lots of new software and data systems need to be bought. I mean, billions of dollars worth. Billions and billions.
“You’re welcome.”
Diane Ravitch's blog 
What Bob said!
enlighten me please: New York students were exposed to common core standards for how many months years before taking the tests? Were these tests common core tests (since I understood that two entities are just now developing tests aligned to the common core and are going to be field testing them soon)? Did New York develop their own common core tests? How did they align them and test them? Is it the test that is poorly designed or aligned, or are the standards not what they tout them to be? Or are the standards not appropriate for the grade level at which they are given? Or is it a failure of teaching methods and materials using the new standards? Indiana is rushing to rewrite their new standards which are uncommonly high and not common core with no discussion of aligning materials, professional development/pedagogy, or developing, field testing tests, and then setting passing scores. Just rushing to get it done darn quick and high!! Lay out a timeline please?
sounds like a reasonable process
1. IDK
2. maybe
3. probably not
4. they didn’t
5. yes & yes
6. ditto
7. the answers are n the back of “Reign of Error, teacher’s copy…”
The CCSS aligned, paper and pencil tests administered in NY last year and this year were/are written by Pearson. Pearson has been writing math and ELA tests for NY for many years under NCLB. They are also one of two consultant groups writing the computer based, online PARCC tests scheduled for next year.
Reasons for the 70% failure rate on last spring’s CCSS aligned Pearson math and ELA tests (grades 3 to 8):
1) Rushed implementation
2) Crappy standards. Generally content free (esp ELA)
Very abstract, subjective, skills based learning targets
Skills required by standards not exxplicitly teachable
3) Test battery length: 6 test days over 11 school days
Crowded schedule created test taking fatigue that was palpable
4) Testing time insufficient; many unfinished tests. NYSED refused
to make adjustments for this serious issue
5) Difficult and boring reading selection
6) Tricky and confusing reading comprehension items
7) Subjective reading comprehension items in MC format
Frequent demand for students to identify the “best” or
“most likely”answers
8) Reading comprehension items with more than on possible
right answer; some with no apparent right answer
9) Tests riddled with culturally biased items
10) Ridiculously high cut scores set intentionally to achieve
a 30% passing rate
11) Tests were norm referenced instead of being criteron based
12) Standards and tests so bad that they were essentially
test-prep-proof
13) NO-stakes tests for students, and they knew it. Teachers
had no academic leverage to motivate students.
They really could’ve cared less.
NOTE: We are currently administering the 2014 CCSS Pearson exams. Scores will GO UP because the political pressure has gotten to great. NYSED will not permit another academic bloodbath. To ensure this they will make sure tests are easier and/or adjust tests scores to “prove” that the CC is “working”
Smoke and mirrors at a snake oil festival.
“enlighten me please”
Okay my singular locust, Akla, you name hints at your lineage Acrididae!
It need not matter what the common garden variety educational standard or standardized test to which you refer. All educational standards and standardized testing contain so many epistemological and ontological errors in the making of, giving tests of and interpreting the results of that render the whole process completely invalid.
The timeline is 1997 when Noel Wilson proved to the world (I know of no legitimate rebuttal nor refutation) the insanities that are the educational malpractices known as educational standards and standardized testing in his seminal work “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Download and read it for sheer educational enlightenment-guaranteed, money back of course!
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A quality cannot be quantified. Quantity is a sub-category of quality. It is illogical to judge/assess a whole category by only a part (sub-category) of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as one dimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing we are lacking much information about said interactions.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any
result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it measures “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
Witty, and a perfect illustration of ed “reformers” circular logic.
April Fool!
Is this a joke? It sure sounds like one
Not joke. Ohio currently has 3,203 standards on the books, including 1,600 CCSS (counting parts a-e). That’s about 267 per grade level. The effective instructional year in our state has dropped to 167 days, from 180 days. I will let you do the math on how many standards a day ( on average) must be addressed.
We are drowning in standards created by discipline-centered groups who have written standards in the hope that grade-level (not grade span) standards will command more curriculum time.
I do not want suggest that emphasizing the so-called “21st century skills” is an alternative because that is just a meme conjured by PR expert Ken Kay, who put some “human relations” and corporate language into circulation with considerable support from tech companies. The standardization of education along disciplinary lines is, by design, intended to kill off any remnant of the progressive impulse to frame education around problems or broad themes.
It is instructive to take a retrospective look at the standards written under the Goals 2000 Educate America Act (H.R. 1804, 1994), At that time, K-12 standards were written in 14 domains of study, 24 subjects, then parsed into 259 standards, and 4100 grade-level benchmarks.
A dispute over the status of history versus social studies ended in no “approved” standards for the latter, but 87 standards and 1,281 grade level standards for history. In those history standards, facts are supposed to matter. Even so, students were (falsely) expected to know that Mary Cassatt was a famous American Regionalist painter. (Wrong. The artist she lived most of her life in Paris, is best known as an Impressionist). That is just one of many other questionable standards, never subjected to independent review. Source: Content Knowledge: A Compendium of Standards and Benchmarks for K-12 Education, “Process” Midcontinent Research for Education and Learning, (2011), http://www.mcrel.org/standards-benchmarks/docs/process.asp
If Ohio’s current standards are typical, there has been no crosschecking of the sets of standards for duplications, synergies, contradictory expectations, feasibility, developmental coherence, or simply dead wrong content.
These standards are surrounded with all of the mandatory rhetoric of the day. They are strictly academic. They are rigorous. Students must master them on time, grade-by-grade with no regard for incremental understandings that may later produce insight and understanding. Our spreadsheets and checklists for real tests of perfect mastery must be complete or we, and our students have failed
One day someone will write a musical about the education reform movement and the hit sing lyrics will come straight from this post.
Glad I don’t teach school and live in Ohio any more. The state is run by folks who are wolves in sheep clothing.
BOB – Does this mean the US treasury will be redesigning the backside of our currency?
In Data We Trust
Diane, This is in response to the funny post on data, and is also a commentary on the SATs.
I am a beneficiary of SAT scores and related data, such as Achievement Test scores. But I also know how capricious SAT scores can be from my personal experience.
When I was in the second half of my junior year in high school, I took the SATs and scored 680 math and 670 verbal. This was pretty good, but I thought I’d score higher if I took the SATs again in my senior year. Boy, was I wrong! My senior scores were 650 math and 624 verbal.
Did I get much dumber in the intervening sic months? Were my senior year teachers that much worse than my junior year teachers? Did I just get lucky as a junior?
I made it into an Ivy League college. But I think my high school grades and the fact that I was yearbook editor and a member of the basketball team counted for more than my SAT scores.
It burns me whenever I hear about test scores being used for purposes for which they were not designed. It is child abuse and teacher abuse to do so. But the rich bastards and their flunkies (Michelle Rhee comes to mind, although she is gaming the system in her own despicable way) don’t care about children and teachers. They care about themselves.
Rob Broderick
Gosh! I feel like such a fool for fighting this. I am truly a data convert – a Shepherdite as it were.
The Sheperd is thy lord and thou shall follow?
(to the pastures where the only data that matters is green?)
For that, Comrade Boland will be receiving one of our free secret data decoder rings!
I love this! It is so true. Data can be manipulated. It NY “officials” are expecting this year’s scores to improve. I guess they made the tests easier,or they plan to score them in a more favorable manner. Children are not data! Tests should be used to inform instruction, not label children ,and their teachers. It reminds me of what we used to hear about the old USSR- communists -LOL.
Sent from my iPad
>
Just for fun, try graphing the raw scores over the scaled scores from the scaled score conversion charts produced for the New York ELA and math assessments–the charts used to produce the “data” that the state reported to the Feds for years under NCLB.
Those raw-to-scaled-score conversions are supposed to be linear.
But what you get if you graph the numbers from New York’s conversion charts are lines that jump around like gerbils on methamphetamine–strange zig-zags, huge leaps up or down.
That’s one of the beauties of this standardized testing “data.” You can manipulate it to make it mean whatever you want it to mean, and no one will look at the stuff closely enough to figure out what you did.
The ed deform numerologists have the same approach to “data” that Humpty Dumpty has to vocabulary:
“I don’t know what you mean by ‘glory,’ ” Alice said.
Humpty Dumpty smiled contemptuously. “Of course you don’t—till I tell you. I meant ‘there’s a nice knock-down argument for you!’ ”
“But ‘glory’ doesn’t mean ‘a nice knock-down argument’,” Alice objected.
“When I use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just what I choose it to mean—neither more nor less.”
“The question is,” said Alice, “whether you can make words mean so many different things.”
“The question is,” said Humpty Dumpty, “which is to be master—that’s all.”
Thank you Bob Shepherd, finally someone has given clarity to this well, B.S. (sorry, I couldn’t resist.) And to think that I thought data was a four letter word!
Well done, Comrade Cartwheel!
For the rest of you, a Data Check:
O’Brien: The Individual mind can make mistakes, but the mind of the Party is immortal. . . . You must humble yourself before you can become sane. How many fingers am I holding up, Winston?
Winston: Four.
O’Brien: The Party says that it is not four, but five. . . . How many are there?
This message brought to you by the Common Core Curriculum Commissariat and Ministry of Truth.
CCCC MiniTru. A wholly owned subsidiary of the Bill and Melinda Gates Foundation.
“He loved Big Data.”