Fred Smith, a testing expert who worked for the NYC Board of Education for many years, poses an interesting question: why were three test questions quietly removed from the Pearson tests?
He writes:
“One wonders why SED [New York State Education Department] might have killed the item. Might there be no answer? Could there be more than one correct answer? Perhaps, a higher percentage of students selected one or two confusing distractors than chose the answer SED deemed to be right? Maybe the item is biased against a certain group of students. Any of the above would give it a failing grade.”
And he raises other questions:
“So, students, what do we draw from these revelations?
A) Clearly items that Pearson claimed were vetted by review panels and experts were unrefined and no better than field test items that somehow passed muster only to flop in prime time.
B) SED’s dirty secret is out of the bag: Its performance-defining cutoff scores are set after tests are given—in this case, after the raw score distribution had been studied and truncated.
C) SED plays fast and loose with data at its disposal, withholding information from the public that paid for it.
D) Efforts to classify students and evaluate teachers that rest on such shaky grounds are indefensible and unsustainable.
E) All of the above.
“E” certainly seems like the smart choice. But we can’t know for sure until an outside investigation is conducted into how SED and Pearson have run the testing program. Parents should hold their children out of all statewide tests until SED comes clean by providing complete and timely item analysis data and is able to demonstrate that the test results are relevant to the purposes they are being bent to serve—in other words, until there are meaningful alternative assessment programs in place.
“Transparency in all matters concerning educational testing is a moral imperative. We must demand passage of revised Truth-in-Testing legislation, opening the testing process to sunshine and scrutiny, restoring its balance and something immeasurable—a level of trust in educational leadership that’s been missing too long.”
* * * *
* * * *
Fred Smith, a testing specialist and consultant, was an administrative analyst for the New York City public schools. He’s a member of Change the Stakes, a parent advocacy group.
Diane Ravitch's blog 
Think of the scenarios in other areas: medicine, safety standards, sports,mwhatever, if measurement was so fast and loose. But kids, teachers, and schools apparently aren’t all that important.
“No better than field test items that somehow passed muster only to flop in prime time.”
The students taking these tests are all being treated as guinea pigs. The tests have been done on the cheap and minimally field-tested. Why? Field tests cost a lot of money and they produce a risk of dropped security for items that might be in a pool for later use.
Setting up cut scores after-the-fact is a clear case of insufficient thinking on the front end of the test–a well-developed set of constructs, aims, a thoroughly vetted set of reasons for the test, and discussion of the NEED for cut scores, including their use and educational value.
Although I agree with the idea of Truth-in-Testing legislation, that should not be necessary if the test designers followed the published, peer-reviewed, and updated standards for the development and use of such tests. These standards can be found here:
Standards for Educational & Psychological Testing (2014 Edition) http://www.aera.net/…/StandardsforEducationalPsychologicalTesting(2014Edition)/ …/Default.aspx
“Molding Children”
Testing is a mold
That can’t withstand the sun
And if the truth be told
It spoils all the fun
No matter how the state determines the cutoff scores or what questions they add or eliminate, one thing is for sure: The testing companies are capitalist cronies of the federal and state government.
I wish I had a dollar for every sucker that has bought into nearly all of this reform movement.
And the testing companies wish they had about $8.00 (and they have been granted their wish!) for every child that gets tested.
Cha-ching, baby!
I will say that I like your art, Robert. I can’t put my finger on what it reminds me of. I want to say some combination of Byzantine art and Soviet agitprop. It also reminds me of a cartoonist named Terry Colon.
Do you like my politics, FLERP?
I care more about that . . . .
Eh, I guess they’re ok.
Stark rendering! Or should I say Rendo-ing. The only thing I don’t see is the children getting ground up in the exchange of tests for cash. They are the treasure that is being taken from us.
When I did renderings of upscale home on the north shore of Long Island for a realty firm known as “North of 25A”, ones that were published in the NY Times (back then the Times published hand drawn renderings in the real estate section), I did have a company, while still in architecture school, known as “Rendo’s Renderings”.
Beat you to it, Fred.
Ah-ha! Kids getting ground up like sausage meat? Such perfect, tender little vittles for the reformer$! Another idea for a cartoon. You will share the credit, Mr. former-test-maker-turned-anti-reformer.
Kidding aside, tests are great! It’s the purpose they are used for that is messing everyone up and turning our democracy into an evil fascist country with smiley faces, lies, and poverty.
If I had a kid, I’d tell him or her to immigrate to Western or Northern Europe, meet someone they really like, and get married to live there. Europe’s perfection and weaknesses are America’s dreams still to be realized . . . .
Okay, Swacker, let’s have the Wilson rant!
(And my own–any city w/a Pear$on campu$ need$ to hold a huge protest outside–point your fingers, tell the public how much $$$ their state is absolutely WA$TED on their lou$y product$, & cut the head off the mon$ter, once & for all, becau$e…yes, WE did, yes, WE can & yes, WE WILL!!)
What has competition done for schools? Cheating at unprecedented levels, deception and working hard to assure our kids are better than yours. At what point do we ask “Whose children do you want to see fail?”
Competition destroys collaboration works. Take those great ideas and pass them to every educator who can use them. Because it’s about kids!
What has competition done?
Well, if there is an upside, it’s that finally many, many, many PARENTS are waking up and getting informed and protesting what’s been happening to our public schools and our kids because of the testing beast, funding cuts, and assorted outrages in the name of “reform”.
Competition has made many wealthy, at the expense of the children and the USA taxpayers, who had no say in where our monies got spent, and who siphoned them out of the education pool.
“Every Astronaut for Himself”
Before the decade’s out
We’ll put a man up there
But as for turnabout
He’ll have to take the stair
Maybe the tests were created with the latest programming fad, agile development. In agile, vendors go to market very fast and let the customer do the QA work. Vendors then respond, or iterate, and fix the bugs. That was probably the goal of the PARCC practice runs, to get in an iteration before the real test. Agile development would not be a good model for test software.
“Democracies die behind closed doors,” to quote Judge Damon J. Keith.
The full quote can be found at: http://www.nytimes.com/2002/08/27/us/traces-of-terror-immigration-a-court-backs-open-hearings-on-deportation.html
It’s obvious that the Andrew Cuomo, Merryl Tisch, the NYS Education Department et al have created a testing fiasco. What an embarrassment! No wonder they want to hide it.