This article may explain why the big testing corporations want machines to score essays.
How do you think they score them now?
How do you think they have been scored for many years?
By temps. By people earning $10-12 an hour with no particular skills and minimal training.
Not long ago, I posted a story that said that Pearson was advertising on CraigsList in Texas for people to score essays.
Just think.
Children’s lives; teachers’ careers and reputations; and the fate of the school hinge on snap judgment made by people reading these essays, scoring as many as possible in an hour.
It is a multibillion dollar business, and the name of the game is cutting costs.
So having the essays scored by machine if your goal is to maximize profit.
Education? Don’t start talking about that. That’s an abstraction.
Diane Ravitch's blog 
This is something I knew all along. You know…it’s hard not to cuss up a storm after reading this. I’m surprised that more teachers and parents aren’t outraged by this. It would certainly be worth looking into to see if this (scoring fiasco) could be the grounds or basis for a lawsuit. I’m glad I took the day off. That cussing thing. I’m starting to feel the urge.
LikeLike
Ethics, they hardly exist anymore. Honesty, what is that? Proper grading, what for all we want is the money who cares about what really is happening.
LikeLike
“Just think.”
Exactly, but not thinking about the base cause of all this edudeformer nonsense is the problem. One can talk of the crappy scorers, talk of machine scoring, of the Rheeject of White, of the Rahminator, etc. . . but until one gets to the root of the problem all else is for naught. The root problem is the concept that the teaching and learning process can be quantified. It can’t. Until we dig out the root of the dandelion reforms they will continue resprout!!
LikeLike
But the temps hired by Pearson to score the edTPA, the ones doing the work that teacher educators already do, those are not an issue? Pearson has taken up the edTPA and is charging $300 per student, and paying temps $75 to score each submission (which they estimate to take 2 1/2 hours to score) out of the goodness of their hearts? Curious.
LikeLike
But the temps hired by Pearson to score the edTPA are not a problem? Not even when they are doing the work, determining certification readiness, that teacher educators already do? Pearson is charging student teachers $300 and paying temps $75 to score each edTPA (at an estimated 2 1/2 hours of work for each submission) out of the goodness of their hearts? This time they put profit aside? Curious.
LikeLike
I noticed an ad from ETS looking for scorers on Amazon Mechanical Turk.
LikeLike
Yes, it is absolutely true. They hire these people for such a low salary, because, after all, the bottom line is “efficiency” … saving money by paying people as little as possible. There is no way that this is fair or accurate. In fact, within a school, teachers don’t even agree on scoring essays of their own groups of students. Sometimes, actually knowing the student helps to understand why they write as they do. Sure, they can look for technical errors, etc. But content? Not easy to grade at any time. This is one major reason that I think the tests are so ridiculous!!! We’ve know this ever since Ohio, in the 19902, has used the Proficiency, then OAT, then OAA and graduation tests.
LikeLike
Correction, 1995.
LikeLike
Someone I know scored elementary school-level standardized essay tests as a temp in California. She got fired for not working fast enough, and for repeatedly arguing with her supervisors about the illogical grading rubric. For example, per the scoring rubric for this exam, students who used very simple vocabulary but made no spelling mistakes were scored much higher than students who used very sophisticated vocabulary but made even one spelling mistake.
LikeLike
Is this for real?
I can not believe that some outsider comes in and grades the essay of a student they do not even know!!
So what!! If I were the teacher, I would make sure the student got the final grade they deserve from the teacher!!!…Sure would!!
The grading is a bunch of diddlysquat and means nothing in the real world!!
LikeLike
Year after year, decade after decade,the edufrauds and their accountabully underlings keep promising to tweak & fix & improve their Sacred Metrics, the most notable part known to us mere mortals being the scores derived from high-stakes standardized tests.
Proof by bloviation works on Planet Rheeality, also known as the Errant Orb of Ever-Changing Excuses. There the New Math[ematical Intimidation] reigns supreme. 13th percentile becomes 90th percentile. 76.7 becomes 93.7. 97 sixth graders in 2006 become 62 graduates in 2012 = 100% graduation rate.
Gloryosky!
On Planet Reality, we find ourselves stuck in a rut. We just can’t seem to get away from those old stick-in-the-mud ways of arguing on the basis of facts and logic, and unlike Houdini we can’t undo ourselves from the ethical ties that bind us hand and foot—like being transparent and honest in our use of data [a respectful nod in the direction of KrazyMathLady and Gary R and Bruce Baker and GF Brandenburg et al.].
So for all of you less fortunates who don’t make up the impatient cagebusting EduExcellencies who have the motto “Facts are stubborn things, but statistics are more pliable” [Mark Twain] tattooed on, er, non-public areas of their bodies, a blast from the past that amply anticipates everything in this well done but recent article:
Todd Farley, MAKING THE GRADES: MY MISADVENTURES IN THE STANDARDIZED TESTING INDUSTRY (2009).
For a shorter read: “Dan DiMaggio” are the first two words in the article you access by clicking on the link provided by Diane. A 2010 piece by him on the same topic: http://monthlyreview.org/2010/12/01/the-loneliness-of-the-long-distance-test-scorer
Even on something as minor [?!?!?] as the Sacred Metrics being misapplied [?!?!?] and leading to scoring errors with painful real world consequences, recent events demonstrate that there is nothing really new under the sun, as evidenced by this 2001 NYTimes article: http://www.nytimes.com/2001/05/20/business/20EXAM.html
Reflecting on the above kinda makes me lose confidence in VAManiacal teacher evals too…
🙂
LikeLike
So basically Shane Brainerd is scoring Shaun Brumder’s essay. Rest easy all you hard working, high achievers. You’re essays are being graded by the kids who were ditching high school and getting baked and playing video games all day.
LikeLike
Wait, what’s the problem with temps grading the exams? With the Common Corporate Standards, VAM and TFA metastasizing, soon temps will be teaching the kids and giving the exams.
LikeLike
Nobody teaches in these testing states..
They are TESTERS!!
LikeLike
Here was a scary comment posted on the website from the article:
“The guy you quote at the beginning of the article must something serious going on. Even if you want to score the creative but off topic essays higher, the number of those are really limited – like maybe 5% or less. So even if you score those drastically different than anyone else, that is only 5% of the tests, you can still screw up another 15% before anyone cares about your “agreement rate.” ”
Apparently messing up 5-20% doesn’t matter to the people who score the tests, but ut sure as @#$% matters to the individual whose test got mis-scored. Apparently the fact that these are actual humans and that these tests carry HUGE consequences gets lost in the race for efficiency
LikeLike
Yup, temps are grading high-stakes tests. I saw an ad on Craig’s List for graders. This is wrong! This can be taken to court! It’s insane and horrid.
LikeLike
AGREE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1
LikeLike
Just one insane outcome from the constructionist practice of divorcing instruction from assessment. A fine example of how the enterprising and commoditization of education creates a conflict with the ethical goals of the educators, and education values that have been in practice since the Enlightenment.
LikeLike
New York City’s controversial new “common core” test is apparently being graded by classroom teachers. Every school must contribute a certain number of teachers–based on the school’s size–to grade tests. I wonder what’s worse: temps grading the tests or pulling full-time teachers out of the classroom to mark tests–and hiring subs?
LikeLike
CLASSICAL EDUCATION. ONLY CHOICE. AND GOOD LAWYERS.
LikeLike
What a Joke!!!
I might grade the tests completely different depending on my likes and dislikes.
Biggest bunch of CR*P to date!
LikeLike
Pearson also has a “score from home” option. Now temps can juggle FB, Twitter and our kids scores while watching TV.
LikeLike
http://www.flexiblescoring.pearson.com/index.cfm?a=cat&cid=1413 Link to score from home.
LikeLike
Until we get out of the sorting and separating mode and get into individualized assessment conducted by the teacher in conjunction with the student (and parents depending upon the age level) these peripheral arguments will dominate. But individualizing the education of all would require massive amounts of monies with having no more than 15 students per class at the elementary level and having at least an aide and perhaps a SpEd teacher also. Max 20 students at the secondary level again with an aide and SpEd teacher as needed (and teachers teaching a max load of five classes per day.
I can continue to dream my Quixotic dream and hope, someday. . . .
LikeLike
This already exists, Duane, in private education. Roughly $40,000 per kid per year in the fifty best schools Diane provided a link to. It could be done for less in public education.
LikeLike
This is not news. My brother worked in human resources for a big call center in Utah in the 1990s and he told me then of his struggles finding English-speaking temps he could hire for grading essays on standardized tests that his company had contracted to score.
The good news is that algorithms are now nearly perfected that will allow computers to score these essays with high correlation to human readers. And they can read thousands of essays per second without bathroom breaks. Everyone wins!
LikeLike
Whenever you create a client/business situation, of course the business will tell the client it’s demands are being met. What business would honestly discuss its short comings? I’ve been a scorer for many years and I know when a state demands the scorers have a 90% accuracy rate the business will make sure their scorers qualify, even if it means giving the scorers the correct answers on their qualifying tests. I’m serious. Everyone qualifies.
But I don’t blame the testing companies as much as the states. Their “ready, fire, aim” approach to formulating questions often means students become guinea pigs. Badly constructed question that students have difficulty understanding are apparently not tested by many states before being issued. Often times the rubrics provided by the states are woefully inadequate and confusing, and testing businesses are forced to modifying them during the scoring process. This means the poor kids who are scored first are graded low..
I’ve asked and been told there is no real oversight in this process. The test scoring businesses provide a service and are loath to complain about to their customers (the states) about the quality of their test questions. This is a competitive business. The states might go to another scoring business that doesn’t complain about their questions. And the states are happy to believe the businesses are providing them with accurate results. It’s a win win situation for states and for scoring businesses. Unfortunately, as a scorer I read answers that don’t address the prompt: “I don’t understand. I’m going to fail.” That’s a zero. “Why can’t we use the grades my teacher gave in me for my work in class?” that’s a zero.
LikeLike
Whenever you create a client/business situation, of course the business will tell the client it’s demands are being met. What business would honestly discuss its short comings? I’ve been a scorer for many years and I know when a state demands the scorers have a 90% accuracy rate the scoring business will make sure their scorers qualify, even if it means giving the scorers the correct answers on their qualifying tests. I’m serious. Everyone qualifies.
But I don’t blame the testing companies as much as the states. Their “ready, fire, aim” approach to formulating questions often means students become guinea pigs. Badly constructed questions that students have difficulty understanding are apparently not tested by many states before being issued. Often times the rubrics provided by the states are woefully inadequate and confusing, and testing businesses are forced to modifying them during the scoring process. This means the poor kids who are scored first are graded low..
I’ve asked and been told there is no real oversight in this process. This is the free enterprise system. The test scoring businesses provide a service and are loath to complain to their customers (the states) about the quality of their test questions. This is a competitive business. The states might go to another scoring business that doesn’t question their questions. And the states are happy to believe the businesses are providing them with accurate results. It’s a win win situation for states and for scoring businesses. Unfortunately, as a scorer, I read answers that don’t address the prompt:
“I don’t understand. I’m going to fail.” That’s a zero.
“Why can’t we use the grades my teacher gave in me for my work in class?” that’s a zero.
“IDK” that’s a zero.
LikeLike
Few, if any of the( blogger/whiners )on this site understand this “scoring” process! The essays or student responses are ” scored ” according to the individual STATE Board of Education rubric. The STATE decides what the student response should be “scored” NOT GRADED! The college educated scorers are trained by the STATE ( IE*OHIO) to recognize the comprehension level of the student response to the prompt or question. The STATE provides a rubric or guideline for recognizing the SCORE, not the GRADE! The ” class-room teacher” merely hands out the tests. In many case, that teacher can sway the result by giving advice! These “scorers” are quite qualified and do a great service to your State Board of Education. Most have a TEACHING background, not that teachers are ALL qualified or able?
LikeLike