An earlier post predicted that the faux reforms of the day will collapse like a house of cards when the public realizes the damage done to children and the quality of education. This reader says that the tests that are the foundation for all of the current education reforms–like merit pay and evaluation by scores–are fundamentally and irreparably flawed.
Queue the Kafka indeed! I’ve worked in the educational publishing industry for years, and I have had occasion to read hundreds of state tests. Almost every test that I have read has been RIDDLED with errors–was so full of errors that it looked like some sort of rough draft. Often, the errors on the state tests are such that the officially correct answer is actually incorrect. Here’s an example:
“There are 8 apples on a table. If you take away 2 apples, how many apples do you have?”
The answer is supposed to be “six,” but, of course, the answer to the question that was actually asked is “two.”
I have a standing bet that I can take any one of these state standardized tests and find at least ten errors in it. It’s a bet I’ve never lost. These tests are really sloppily prepared, and as the experience of the teacher in this video indicates, there is little accountability for their quality.
However, the problem runs much deeper than the editorial vetting of the exams. The biggest problem with them is that the supposed research adduced by the testing companies to support the validity and reliability of their tests is a lot of smoke and mirrors. A test of reading ability is like a test on “ability to make one’s way in the world.” What’s being tested is extraordinarily vague, broad, and complex. Suppose that one tested driving ability by giving people an exam that looked at their ability to identify car parts–to distinguish, say, a hub cap from a windshield wiper. Such a test would be very like the state exams that we give to test reading ability.
Diane Ravitch's blog 
Robert,
“I have had occasion to read hundreds of state tests. Almost every test that I have read has been RIDDLED with errors–was so full of errors that it looked like some sort of rough draft”.
Yes, there is a reason why the manufacturers of these tests don’t want the test givers to read them, exactly what you have pointed out about them in your post. As an ethical proposal all teachers should refuse to give a any test that they have not read. Teachers are the unwitting (and generally should be unwilling) dupes in the standardized testing con game.
Ah, you’ve been in the belly of the beast and survived, much like Noel Wilson in Australia. We need more people like you who have been in the belly to stand up and expose all the errors and frauds involved in the process. I invite you to join in the discussion of his 1997 dissertation “Educational Standards and the Problem of Error” at my blog “Promoting Just Education for All” @ revivingwilson.org to learn and understand the thirteen sources of logical errors involved in educational standards and standardized testing (and grading students) that render the whole process invalid and harmful to students.
Duane
Bingo, Duane! I believe I have some old sample tests (there was no direction to destroy them &/or not re-use them, so we’d oftentimes re-use them for a couple of years)–I will dig them out at the earliest opportunity. I KNOW–for a FACT–that the writing samples are RIDICULOUSLY incorrectly scored. (My colleagues & I were supposed to use them as “models” for test prep, & we would just shake our heads in bewilderment at the samples that were graded 2 as opposed to ones we thought WORSE that received the highest score!)
Does anyone out there also have old sample test booklets from their state? Let’s all shake them out a take a look, reporting back to this blog as to your findings! This should be a real learning experience for us all!
I recommend “The Tyranny of Testing” by Banesh Hoffmann. Hoffman has many examples of the sort you showed in this post–The ambiguities that leave students struggling to guess what the testers want for answers rather than demonstrate their own understading of the subject matter.
I also note another aspect of testing that I think Hoffmann doesn’t address in great detail–The use of tests to normalize student social behavior. The fact that tests inherently reward those students who can either intuit or infer the testers’ own biases in questions and answers–and punish those students who are truly independent thinkers–creates a huge force that selects for intellectual and moral conformity among the students. I don’t find it hard to see a very Orwellian side to this method of “testing as indoctrination”.
Indeed, I think many of the failures we now suffer at all levels of government and business reflect the inherent weakness of our academic “meritocracy” that took off with the Conant Commission and the reforms in the ’60s after Sputnik. By obsessively promoting “the best and brightest” as defined by standardized test scores and class rank, we have created a ruling class that cannot lead, has little imagination, and absolutely no moral courage. Instead the ruling political classes demonstrate a complete dependence on external measures of achievement, usually wealth and titled power, and cultivate a sense of self-justification that borders on the nihilistic. Lacking any intellectual curiosity or honesty these elites enlist on academic “experts”, who are usually paid to reinforce the views of the financially dominant classes or chant some theoretical dogma, to support and give cover to their schemes. We rarely see any politician stand up to do the right thing anymore or challenge the conventional wisdom of their particular constituency. A great parable of this thinking can be found in the book “The Rise of the Meritocracy” by Michael Young.
Again M&S excellent analysis, especially the second paragraph!
I think this is very true – the unconscious reward for conformity is very very strong in these tests.
I’ve met plenty of people who can conform for a great test score and then think and tweak outside the box in their real life (and I consider myself one 🙂 ), but I think a lot of very creative and very bright people get lower scores than they deserve because of this.
I’ve always thought of “test-taking skills” as “Knowing how to think like a test writer.”
“The fact that tests inherently reward those students who can either intuit or infer the testers’ own biases in questions and answers–”
Also one of the reasons why tests are biased against minority groups…
Amen, Moosensquirrels!
What is especially scary about this reader’s comment is that success with VAM can be influenced by as little as one more “right” answer!
Let’s push the driving test analogy. It fits perfectly. But, the fact is we care more abut having safe drivers on our highways than well-educated ones.
deb
My husband was helping our child with a reading comprehension worksheet this week. It was set up with line numbers on the short story and questions right next to it. After noticing her difficulty, he realized that she had decided that she didn’t need to read the whole story, but was just trying to pick out the answer from the apparent relevant lines. What’s especially frustrating about this is that I know all of her teachers have been strong and enthusiastic readers and all about academic rigor and would never coach her to do this. He felt it was the structure of the exercise itself that made that an obvious shortcut for her… and explains why she doesn’t do them well.
I have subbed for longer than I had my own classroom. She is using a strategy that children typically use. It is more “effective” with textbook assignments since they typically ask questions that are looking for cut and paste answers. Someone may mistakenly believe the line numbers are scaffolding the assignment for those who need extra support. Your husband is right that the structure of the assignment encouraged a strategy that would not support improvement of comprehension.
As per my usual, I must, again, recommend Todd Farley’s 2009 book “Making the Grades: My Misadventures in the Standardized Testing Industry.” As hilarious as it is heartbreaking, and a quick read at 242 pages. PoliPointPress, $11.95 online, $16.95 in bookstores, & I originally found it at my local library. Read it, & pay it forward!
When my child was doing test prep for homework and got stumped on a question that didn’t have clearly correct answer, I would tell her to try and think like the test writer, who probably doesn’t believe that 6th graders are really all that smart.
Seemed to work.