This article appeared in the business section of the New York Times.
Author James B. Stewart decided to try to answer a sample question that is supposedly representative of the admissions test that students in New York City take to get into a handful of elite high schools. He found the question confusing. He got the wrong answer. He sent it to a legendary editor at the New Yorker magazine. She found it confusing. She got the wrong answer. I tried the question. I got the wrong answer. Pearson said it was a sample question, and no one actually had to see it on an exam. They revised the question. It was as confusing as the original.
This is the question.
“In the passage below, which of these is the most precise revision for the words “talked to some people who did the best in the contest?”
“During a nightly news-segment about a cooking contest, a reporter talked to some people who did the best in the contest.”
A. Conversed with some of the people who won the contest.
B. Spoke to the three contestants who did well.
C. Discussed the contest with some of the winners.
D. Interviewed the top three contestants.”
Stewart writes:
“The question didn’t say how many people the reporter interviewed, and a reader has no way of knowing. So an accurate revision would need to be equally vague. Any revision that specified “three contestants” is not an accurate reproduction of the original, but an embellishment. That eliminated answers B and D.
“Answer C refers to “some of the winners,” but doesn’t say winners of what. The original is explicit: “the contest.” And C embellishes “talked”: “discussed the contest.” The original doesn’t say what the reporter talked to the winners about. So C failed on two counts.
“That left A, which is both vague and explicit in the same way the original is, and thus the most “precise revision.” I chose it and pushed the “submit” button and got an immediate response.
“Wrong!“
So he tried the question on a language expert:
“So I sent the question to Mary Norris, author of “Between You and Me: Confessions of a Comma Queen” and a legendary copy editor at The New Yorker. If anyone understands revisions of English prose, it’s she. I didn’t tell her anything about my experience and asked her to answer the question and tell me what she thought.”
She thought the question was confusing.
“She said she was stumped immediately by the reference to “people” who “did the best in the contest.” Can multiple people be the “best?” Can there be more than one “winner”? What kind of “contest” would that be? “To say there are three people adds information that isn’t in the original,” she said. “And we have no way of knowing if that’s accurate.”
“C was tempting. “It’s nice and vague, and in this context, vague equals precise,” she said. Nonetheless, she picked answer B. “At least it doesn’t say ‘winner,’” she reasoned.
“Wrong again!
“The “correct” answer, according to the New York City Department of Education, is D. “The top three” in that answer is more specific than “some people who did the best” in the original.
“I would never have picked D,” Ms. Norris said.”
Back to the Education Department and Pearson:
“Will Mantell, a spokesman for the New York City Department of Education, said Pearson investigates “any items with problematic or unusual results.”
“If an error is found,” he added, “the item is not scored.”
“The risk of erroneous answers is reduced if students can take a test multiple times, as they can with the standard college admissions test. But students can take the SHSAT only once, except in unusual circumstances.”
Stewart says that this ambiguous and confusing question shows the risk of using one test score to determine admission.
It’s worse than that.
Pearson and its stupid and confusing questions and answers have turned me into a skeptic of standardized tests. There is nothing “standardized” about the question, and nothing “standardized” about the answer. They are both subject to human error amd completely subjective.
No student’s destiny should be determined by such a flawed instrument.
