Did you know that big testing corporations hire low-wage temps to score students’ written answers to their tests? The scorers need to have a bachelor’s degree in any subject but no teaching eexperience is necessary. On their snap judgment hang the test scores that will determine the future of students and the jobs of their teachers and the fate of their schools.
I first learned about this shocking arrangement when I read Todd Farley’ “Making the Grades: My Misadventures in the Standardized Testing Industry,” which is the most important insider exposé of the industry that sorts, ranks, and labels our nation’s children. And if you want to read another stunning insight into the industry, read this brilliant article by Dan DiMaggio in 2010.
The beat goes on.
Just a few days ago, someone tweeted this ad he found on Craig’sList in Indianapolis:
“Test Evaluators Needed (to score K-12 standardized tests) (Indianapolis, IN)
“Compensation: $11.05/hr contract job
“If you have a Bachelor’s degree or higher, we need your help to evaluate student assessment tests. Come apply!
“For more information and to schedule an appointment visit our website http://www.kellyservices.us/ctb or call us at 1-877-535-5981
“Please be prepared to spend two hours going to through the application / orientation process. Please bring two forms of identification to complete and I-9 and bring proof of your degree.
“These are project based positions. Monday – Friday, 8:30am – 4:30pm
“Position Requirements:
-Must hold a completed Bachelor’s degree or higher
-Ability to sit and at a computer station for full work day
-Basic computer knowledge
-Knowledge of standard writing conventions and mechanics
-Availability to work Monday through Friday for the entire duration of a project
-Demonstrate flexibility while working on various projects”
Nothing new. Just remember. These crucial tests–make or break for students and teachers–are being scored by hourly temps.
Diane Ravitch's blog 
As a side note, I have scored the free response questions on the AP US Government and Politics exam. We got our transportation, housing and meals for free. I believe we got around $1800 (as of 2011) for 7 days of scoring. But we either had to have taught the course for three years or be teaching/having taught it at a college level to qualify to do the work. And we were monitored, and if someone was not scoring accurately they could be dismissed and sent home.
In California, for two summers, I worked for one week as a trained essay scorer. Everyone who was trained was a full time high school English teacher in the public schools. After a full day of training, we were organized into teams and used a rubric as our guide. If the rubric scores of an entire team did not agree on even one essay, then a roving judge read the essay without looking at the team’s rubric scores. The judge’s rubric score would break the tie.
A few years later, the state decided it didn’t have enough money to continue this program so they dropped it and resorted to the same method this post describes.
You were much better prepared than most graders. My friends, math, Spanish and special ed teachers, were trained only briefly, paid poorly and often told to keep moving forward if they graded too hard or easy. No surprise, when we English teachers received our students’ graded papers, students who should have had A’s received B’s and students who should have had C’s received B’s. Yet this system was supposed to yield unbiased or “standardized” results. Hardly.
$1680, teacherken, not $1800, and the hours have gone up and the quality has gone down. When I began scoring AP English, we worked 8:45-4:30; now it’s 8:00 to 5:30 and they’ve cut out the stretch break in the middle of the three-hour stint. Last time I scored, we normed for an hour in the morning; when I began, we didn’t begin “live grading” until after lunch and re-normed at the beginning of every session. The table leaders have been told that they can’t leave until the tables are clear, which cuts down considerably on the “monitoring.” Even when things were better, I never heard of anyone being dismissed for inaccurate scoring. Not invited back, sure, but the only thing I heard anyone being dismissed for is posting derogatory comments on FB about the exams.
Yeah, I still do it. I always wonder why–so when I say that AP English isn’t worth anything, I can prove I know what I’m talking about.
Test evaluators being paid contractually means no benefits and they’ll be responsible for double payroll taxes. This is in effect a minimum wage job and it is probably illegal to pay them as contractors, because they will go to work every day for an employer!
Sent from my iPhone
>
For all the money we’re paying government contractors one would think they could “create” some actual jobs in the communities that are funding these giant contracts, instead of race to the bottom temp work.
Why doesn’t it surprise me that our money (and it’s ALL public money) all goes to the top tier managers and execs instead of the front line local people doing the scoring?
My wife graded the written portion of MCAS tests, on the same basis, for whatever vendor had the Massachusetts contract, a decade ago.
This is as old as the ‘accountability’ movement.
““Please be prepared to spend two hours going *to through* the application / orientation process. Please bring two forms of identification to complete *and* I-9 and bring proof of your degree.” [Emphasis added]
Guess they’re not looking for a detail-oriented person.
And soon they’ll be recruiting teachers the same way …
They already are
It’s called “Wendyslist” (also known as TFA)
Please note that the original article cited here by Diane was published in Monthly Review. MR is a publication which, like substancenews.net (which I edit), is in need of money. I have been an MR subscriber for most of the past 40 years, and it is well worth the investment. You may not always agree with MR’s analysis of economics, history, and events, but they are more honest than many so-called “progressives” — they say that they are an “Independent Socialist Magazine” right on their cover as they have been saying since I began reading them…
But below is the notice that goes with the 2010 article cited here by Diane. Please share this for people who want to help out by going to monthlyreview.org and subscribing to their print edition (which still comes out monthly) or in some other ways helping…
“Dear Reader,
“We place these articles at no charge on our website to serve all the people who cannot afford Monthly Review, or who cannot get access to it where they live. Many of our most devoted readers are outside of the United States. If you read our articles online and you can afford a subscription to our print edition, we would very much appreciate it if you would consider purchasing one. Please visit the MR store for subscription options. Thank you very much. —Eds.”
This is another first-hand account:
“Without getting into too many of the details, the explanation score often ended up being a total judgment call, as there was often no clear, objective score. In a subject like math, where objectivity should be a priority, this is a problem. During my time at Pearson, the grading criteria for the explanation section changed three times. Supervisors were nearly always in a state of confusion about the criteria and often appeared to know less than the scorers, which was likely not their fault, but a fault of their bosses, who create the rubrics and don’t even work at the test scoring sites.”
It sort of perfectly describes the larger ed reform movement to me, the chaos, the change for the sake of change and especially this: ‘the bosses who create the rubrics don’t even work at the test scoring sites” 🙂
” the grading criteria for the explanation section changed three times.”
just three times, you got lucky then. Must of been three Pearson reorganizations. That place is like a constant game of musical chairs. They’re doing a major revamp of the US business right now. They hired a new hotshot from a Management Consulting firm to run the place, should make for an interesting story.
The Uber-ization of education.
Craigslist has come under fire recently for murders associated with ads that have appeared on the site. Some municipalities have addressed the danger by setting up “safe zones” for transactions/property exchanges under the watchful eye of the police, etc. Yet someone thinks trolling on Craigslist for test scorers is a good idea? Gimme a break!
Yes, I fear this may be a scam. In terms of just needing a B.A., however, when there is a shortage of teachers in California, a district can hire subs who have only a B.A.
A) There is a difference between a sub and a full time teacher.
B) There is no teacher shortage in California or anywhere else in the nation right now.
Are you sure about the teacher shortage Dienne?
The Gulan chain of charters is forced to bring import teachers from Turkey because they can’t find enough American teachers (that they can intimidate, browbeat, manipulate, and impoverish) to teach in their charters.
Just sayin’
In California, certain subject areas have shortages, specifically special ed, math, and science. There is also a smaller pool of people going into teaching in California.
In Newark, Dienne, we have 400 Educators Without Placement Site. Many classes are staffed by TFA, long term subs and revolving door subs. Go figure!
And yet another reason I fear the upcoming end of year assessment. God help us all.
BB
This is just the overture: the main act of this farce, still under development, is to hire teachers as poverty-wage temps.
Coming soon to a school district near you…
For $11.05 an hour who is going to be able to summon up the willpower to fairly, accurately, and objectively follow the levels of discrimination in a rater’s guide like this one? You MUST open this link to believe it. ELA teachers I know that scored the Pearson exams with Questar said that it was an inter-rater reliability nightmare.
https://www.engageny.org/resource/sample-student-work-from-the-2013-nys-ela-common-core-test
Try comparing a Grade 7 “3-point” score with a Grade 8 “3-point” score and you will immediately realize just how bogus these tests are.
I have to laugh (sarcastically). In the past years, Pear$on had been placing pretty large ads (I had asked someone in the know what these would go for, & he answered, “Oh, $2K-$5K, easily,” especially the last few years, because they were in full color!?)
in the newspapers. So, perhaps they noticed what we noticed (“Oh, so THAT’s where our taxpayer education funds are going–NOT to school nurses, counselors, libraries &/or the arts!”) & are keeping it low-key–ya think?
And–if you haven’t–read Todd Farley’s 2009 book, Making the Grades–My Misadventures in the Standardized Testing Industry. While it’s a hoot, it’s a sad commentary–& today, rings true more than ever.
BTW–Todd Farley, won’t you please come to Chicago for the N.P.E. in April?
(EVERYONE needs to hear this brilliant guy!)
…and here is a “market analysis” of paying someone $11/hr to score tests.
So, Pearson, or some corp, is paying $11/hr to score tests, which is a debit. How does Pearson make more, additional, revenue to pay for all these scorers? Will it be by market monopolization and certainty of having an endless stream of future tests to grade? If so, all this new revenue is coming out of districts that must pay Pearson; so the districts have additional debits. How do the districts earn additional revenue to pay for their debits of paying Pearson? Where will this money come from, additional tax revenues to the Fed, to be given to the states for more testing???? Someone has to pay for all this testing, who?
Also, fundamentally testing is a service that only makes economic sense (only is financially logical and sustainable) if it makes its consumer (the student) more productive; the student after testing is now a better producer toward increasing GDP. So, I guess all of our students will produce and make $11/hr more because of their increased skills (earning potential?) from all the additional testing they now must endure???? If not, then testing is not economically feasible, logical or sustainable. It will inevitably lead to some entity becoming bankrupt (hopefully Pearson, not the districts, State or Fed, or certainly not the taxpayers!!)
So, whoever dreamed up more testing has never really though through the economic debits/losses caused by it (for I don’t believe there will be any actual economic credits/gains created by more testing, now or ever)
Unfortunately, I think the same ad will run if Governor Cuomo manages to ram through the proposal to have 35% of teacher evaluations done by ‘independent’ state evaluators. Why should a person have any experience, education or skills to evaluate students or teachers?
Since nearly everyone has experienced 13 years of public schooling, they should be well trained to evaluate teachers, right?
Overqualified in fact!
I cannot recommend highly enough the book by Todd Farley and the article by Dan DiMaggio. They are both highly informative and highly readable.
From the last four paragraphs of Todd Farley’s MAKING THE GRADES (pp. 241-242):
[start quote]
What does it really mean to entrust decisions about his country’s students, teachers, and schools to the massive standardized testing industry? In my opinion, it means trusting an industry that is unashamedly in the business of making money instead of listening to the many people who went into education for the more altruistic desire to do good. It means giving credence to the thoughts of mobs of temporary employees who only dabble in assessment while ignoring the opinions of the men and women who dedicate themselves daily to the world of teaching and learning. …
I understand people may disagree with me, but none of that sounds like too good an idea to me. It seems to me entrusting the education of this country’s children to “professional scorers” in far distant states instead of the mean and women who stand in front of their classrooms each day is about as smart an idea as entrusting your health not to the doctor holding a stethoscope to your heart but to some accountant crunching numbers in Omaha. Personally, I’d trust the guy who’s looking me in the eye.
If I had to take any standardized test today that was important to my future and would be assessed by the scoring processes I have long been a part of, I promise you I would protest; I would fight; I would sue; I would go on a hunger strike or march on Washington. I might even punch someone in the nose; but I would never allow that massive and ridiculous business to have any say in my future without battling it to the bitter, bitter end.
Do what you want, America, but at least you have been warned.
[end quote]
“Knowledge makes a man unfit to be a slave.” [Frederick Douglass]
😎
This made me laugh, because it’s so CLEARLY a purely political response:
“After years of calls by educators and parents for a reduction in standardized tests, Florida Gov. Rick Scott said Wednesday that yes, in fact, students in public schools in the state are being over-tested (such as in Duval County, where kids in K-5 take 14 assessments), and he issued an executive order eliminating one statewide 11th grade standardized assessment for English Language Arts.”
It’s just pandering. If any of them thought it was a legit problem they would have done something about it before they had a revolt on their hands. Instead they spent YEARS telling people they were “coddled” and “hysterical”. How could they have been so clueless for so many years about what was going on in schools? They are full-time, paid ed reformers and there are thousands of them. This is their JOB 🙂
Pandering in Ohio as well. State Superintended of Public Instruction has a garbled set of requests to cut testing time heading to the Governor and legislators. His recommendation would eliminate tests for SLOs in favor of a “shared attribution score, meaning teachers who do not have scores from a test from a state approved vendor test and/or VAM ( still in place) will get an “attributed score” for their evaluation, based on student scores in grades/subjects they do not teach. State did some surveys and interviews and discovered the SLO testing was taking more time overall than other required tests.
https://education.ohio.gov/getattachment/Topics/Testing/Ohio-s-State-Tests/Testing-Report-and-Recommendations-2015.pdf.aspx
This debacle has been going on in Florida for almost two decades and several colleagues of mine have actually accepted these positions at one time or another. I feel it demeans the profession of teaching to accept these jobs at such pathetic wages. On one hand these thieves tell you how important these tests are for our children and then when its time to grade these so called end all be all tests they pay the person whom is scoring them a Burger King wage. Classic!
Geez, this is like ETS hiring writing assessment and test evaluators for TOEFL/SAT/GRE/GMAT at $9.50 or less. How desperate are these testing companies?
So, the cost of paying someone to score tests (a service) must be offset by an additional source of revenue, ex. a new good produced. Otherwise, continued new costs of scoring tests will bankrupt the company. So, are we to imagine that our students from all this “advanced preparation” of excessive testing will now have a greater potential/capacity to create new goods, which will increase the GDP, give companies new revenues, which can become new sources of taxation, that can eventually flow back to the Fed and States, in order to offset the increased costs of scoring tests? If not, then the entire testing equation is financial suicide and unsustainable. Somewhere additional revenue must be created by the tested-students, or else where is the money going to come from. We pay for services (testing) in order to create more goods (product); otherwise, what good is the service?
Who placed the ad with Kelly Services in Indianapolis? Was it Pearson? McGraw Hill? A school corporation? A state agency? That is a significant piece of information. I need to know. I may call them and ask.
Yes, website and phone call identify the ad as being for McGraw Hill. I am retired and this is the hourly wage I earn when I sub. I may consider looking into this. However, I really sub as much for the experience of being with kids and teachers as for the income. But scoring ISTEP would be an interesting bucket list addition. Hmmmm……
Let’s inundate them with applications.
Good idea. Retired teachers who volunteer many hours in schools might want to get in on the act!
I make less than that, and I work IN the schools. There is something wrong with that…
As bad as it was when Todd Farley graded for these companies, he got far more training than a two-hour online tutorial. He was also supervised (or did the supervising) in the same room so that supervisor-rather discussions could occur.
Fast. Cheap. Good.
Pick two.
It’s not just for computer science any more.
Not least among the many advantages of hiring temps for scorers is that you can blame them and “fire” them if there is eventually a problem with scoring — even if it is a result of the scoring method itself.
As we know, CEO’s and other officials of big corporations like Pearson rarely (if ever) take responsibility for their own failures.
Testing company Measurement Incorporated only pays $10.70 an hour for people to read tests. Craigslist seems to be their primary way of recruiting people.
http://tampa.craigslist.org/hil/etc/4880121514.html
http://charlotte.craigslist.org/wrg/4884617026.html
http://fortmyers.craigslist.org/col/etc/4880220404.html
http://detroit.craigslist.org/wyn/edu/4890272315.html
From a Google search it looks like Tennessee is their client.
sbac and harming our students
Hyung Nam
Thu 2/19/2015 10:05 AM
Inbox; Sent Items
To:
wilson-staff-list (Mailman-List);
Action Items
I am working with Rob Manning at OPB who is doing a series on SBAC. I gave him this info below.
Also take a look at this at the failure of testocracy and how it has been counter to reducing inequality and actually harmful to students of color and the disadvantaged. “More significantly, testing will continue to feed, not eradicate the real great civil rights issue of our time; the growing school to prison pipeline, which like a malignant cancer, continues to eat away at the fabric of many inner cities by robbing students of their future. http://www.huffingtonpost.com/yohuru-williams/by-any-dreams-necessary-m_b_6648002.html
SBAC CCSS testing is invalid as is PARCC tesing. Both tests included essays and constructed responses even for math questions and are supposed to be superior to older tests that were strictly multiple choice. (These new tests still do have much multiple choice in them too.) I just confirmed that “human raters” for the #SBAC group of #CommonCore tests hire low paid, unqualified temp workersmuch like Pearsons and actually seems to pay less than many of Pearsons workers who are mostly paid $12 an hour (confirmed by both craigslist ads and on Pearsons job center website). SBAC contracts with AIR, the American Institute for Research, but then subcontracts to “Measurement Incorporated.” See link to their jobs. https://www.appone.com/MainInfoReq.asp?R_ID=933769&B_ID=5&fid=1&Adid&SearchScreenID=1172&CountryID=3&LanguageID=2
I have screenshots of all of these craigslist ads for the same jobs. Notice that all the ads link to this http://www.measurementinc.com/Employment
https://wilmington.craigslist.org/edu/4839819781.html
http://detroit.craigslist.org/wyn/edu/4890272315.html
http://tampa.craigslist.org/hil/etc/4880121514.html
They have also been developing computer algorithmic essay scoring. http://www.measurementinc.com/news/can-computers-score-essays
This is all strange because AIR is doing this directly in Florida already and Pearsons does not seem to subcontract with a middleman. Perhaps that’s why Pearsons pays $12 instead of $10.70. http://stateimpact.npr.org/florida/2014/03/21/new-florida-writing-test-will-use-computers-to-grade-student-essays/
Should students, parents and qualified teachers rely on random low paid temp workers to evaluate them?
Reblogged this on We Are More.
I have several retired friends that have been grading papers and tests for Pearson and they did not know it. Eleven dollars an hour, just a little above the minimum wage!
A lot above the min wage…4 bucks
Gates and Hewlett funded ASG’s report that said machine graded essays would be cost effective for “state consortia”.
The required hardware and software, a boon for Silicon Valley. $10 per hour for graders, unnecessary.
Something like Common Core, standardized testing and curriculum, would be required to make it work. I wonder who thought of it?
I posted this comment on another post but I think it’s relevant here, too:
NPR finally gets something right on education. Diane Rehm show, February 18, 2015:
http://thedianerehmshow.org/shows/2015-02-18/rethinking_standardized_testing
The program listing . . .
“Thirteen years ago, Congress passed No Child Left Behind. It mandated annual standardized testing as a way to ensure that students did not fall through the cracks. Over the years, programs like Race to the Top ratcheted up the stakes, increasing pressure on teachers, schools and districts to perform. Now No Child Left Behind is up for reauthorization and some lawmakers are calling for a removal of the mandate. This would allow states more wiggle room to design their own assessment methods. Meanwhile, parents and educators are increasingly questioning the number of tests students take and what the focus on testing is doing to our school system. An update on the debate over the way we use standardized testing in America’s schools.
Guests
Anya Kamenetz education reporter for NPR and author of “The Test: Why Our Schools Are Obsessed with Standardized Testing – But You Don’t Have to Be”
Elaine Weiss national coordinator of the Broader Bolder Approach to Education
Matthew Chingos senior fellow at the Brookings Institution and research director of its Brown Center on Education Policy
Chanelle Hardy senior vice president for policy and executive director of the National Urban League Washington Bureau”
The two speakers who made the least sense were the Chingos and Hardy, both entrenched in policy positions at “important” institutions. Neither of these officials questioned the fallacy that standardized tests give a true picture of “how our students are doing” (the same fallacy that Amanda Ripley’s book is based on).
Chingos, in particular, came across as an out-of-touch technocrat with remarkably lame arguments to justify his view that test scores equate with school and teacher performance, and therefore students need to be tested annually. He pointedly defended the use of inexpensive tests to measure “what students have learned.” He didn’t show any concern that bad tests might yield bad data.
I’m not sure if Kamenetz’s book is worth buying. I don’t know what she thinks of the “common core standards” or the federally funded tests designed for them. But she talked pointedly about the failure of federally mandated tests to do anything worthwhile and the (supposedly) unintended consequences that have made it a disaster. I can only hope she’s urging parents to opt their kids out of the tests.
I also hope Kamanetz’s and Weiss’s strong showing helped Diane Rehm and her staff realize that “ed reform” is bankrupt. A big thanks to whoever booked those two!
Where do I begin. I’m going to keep my opinions of the ISTEP test itself in check for a moment.
In response to your “commentary”, Yes Kelly Services does hire scorers for CTB McGraw Hill, and Craigslist is one of many sites they use to find qualified scorers. All scorers hired by Kelly must undergo a background test, drug screening, and rigorous training. They must be college educated with a minimum of a bachelor’s degree. I’ll agree, the pay isn’t great, but there are many retired professionals out there, and plenty of out of work professionals who could use the money.
I’ve been on both sides of the fence. As an educator, I have administered numerous ISTEP tests over the years. I also had the opportunity to score for CTB McGraw Hill last year as a test evaluator. We scored for Indiana’s ISTEP test and several other states. We were given a rubric with many examples of correct and incorrect responses. We only graded the open response sections on these tests. So your argument finding the “qualifications for scoring such a high stakes test discouraging and questionable” is flawed!
Further, your commentary comes across as though you are biased toward Craigslist in general, as well as people who work temp jobs or jobs that don’t pay well! Do you believe they’re simply hiring people at random off the street? I met many educated, caring people while working as a test evaluator for CTB McGraw Hill, who were all genuinely concerned with doing a good job, and with making sure they provided an accurate assessment of Suzie and Johnny’s test responses.
As an educator, I want to know every one of my students’ abilities and shortcomings, but as an evaluator, I have to put all that aside and simply score a test based on a rubric. Isn’t that the whole point?
If I feel that a response is questionable one way or the other, I call a supervisor and together we choose the best course of action, based on the student response and the rubric. On several occasions, I had two or three people looking at a test question and a response to determine whether a point should be given or taken away.
I believe the state of Indiana has much work to do concerning ISTEP and all of the problems associated with it, but You are barking up the wrong tree here!
Andrew Bales, have you read Todd Farley’s “Making the Grades,” his account of 15 years as a test scorer? Or Dan DiMaggio’s “Confessions of a Long Distance Test Scorer ?”
http://monthlyreview.org/2010/12/01/the-loneliness-of-the-long-distance-test-scorer/
You make test scoring sound like a thoughtful process. DiMaggio describes unemployed college graduates scoring as quickly as possible, with no time for discussion. With thousands and hundreds of thousands of tests to score, speed matters.