An Interview with Todd Farley

This was sent by a reader of the blog. Todd Farley wrote a terrific book about the testing industry called “Making the Grades,” based on his many years on the inside of that industry. He knows the tricks of the trade. If you haven’t read his book, you should.

Interview with Todd Farley by Rebecca Rubenstein.

She writes:

In 2009, I had just graduated college and was part of a class fumbling its way into one of the worst economic climates in U.S. history. Our prospects were, to say the least, uncertain, though my class’s guest commencement speaker, former Chief of Staff and current Mayor of Chicago Rahm Emanuel, told us not to worry—employers were looking for young minds peddling liberal arts degrees. Whether or not that hope rang true was yet to be seen, but one thing was certain: my days of worrying about standardized tests were over. I’d side-stepped taking the GRE by applying—and gaining acceptance—to a university overseas, and I was on my way to a year of graduate school before being forced to dip my toes in choppy job-hunting waters. Shirking the GRE felt like a source of pride; after years of battling severe testing anxiety in middle and high school, I no longer had to prove my intellectual worth through multiple-choice questions. I had triumphed, simply by being a diligent student (a few professor-penned recommendations helped, too), and found myself only three months away from attending one of the most prestigious universities in Europe.

Halfway across the country, a writer by the name of Todd Farley was about to publish a book that would give just cause to my—and many others students’—celebrations. Making The Grades: My Misadventures in the Standardized Testing Industry is more than just a tell-all about the corrupt nature of the testing industry; it is a searing critique of how the American education system has allowed itself to be had by these for-profit companies. For fifteen years, Farley played witness to the most grotesque of money-making schemes: while promising our government a platform with which to evaluate educational performance, these testing companies, instead, provide an environment in which unengaged, underpaid workers—many of whom are also unqualified, as Farley’s first-hand account shows—are asked to assess test answers using arbitrary rubrics, and, at times, their own subjective viewpoints. Over and over, Farley provides scenarios that read like Onion article set-ups: re-scoring tests for the sake of meeting psychometrician standards; giving the same score to an essay with pedestrian word choice like “beforehand” and “succinctly,” as an essay with outstanding word choice like “alacrity” and “perspicacious”; the hiring of poor English speakers to score English-based comprehension tests. After reading Farley’s book, it’s a wonder anyone makes it to college—let alone past middle school—in the U.S.

Though published three years ago, Making The Grades still finds its relevance today: American public education remains sorely underfunded and at the mercy of the standardized testing industry, which continues to threaten everything from teachers’ jobs, to their livelihood, to the schools themselves. (It’s no secret that poor test performance has led to the closure of many a charter school, and to continued state budget cuts—money otherwise used to assist struggling public school systems.) And I can assure you that every minute of every school day, some unfortunate teenager, who is probably in several honors classes but can’t buck up when it comes to taking a standardized test, is worrying his or her head off about getting into his or her dream college because his or her test scores just won’t do. (Been there, done that. Never again.)

Now, in 2012, right before an election that will—among many things—determine the direction education in this country is headed, I catch up with Farley to see how, if at all, his opinions have changed. Farley joins the ranks of Diane Ravitch, Jonathan Kozol, and Alfie Kohn, all of who have expressed written grievance with the path American education continues to gallivant on. In addition to Making The Grades, Farley has written extensively for The Huffington Post, and has also published pieces in The New York Times, The Christian Science Monitor, Education Week, Rethinking Schools, and for the organization Edutopia. Though stationed in Iowa in 2009, he has since left the standardized testing industry, and now resides in New York City. Our conversation below took place over e-mail.

***

Rebecca Rubenstein: Since your book was published in 2009, has the “standardized” testing industry improved?

Todd Farley: Not the slightest bit. There was a story in The New York Times in 2001 about how test-scoring was a wildly out-of-control industry, which quotes various employees—not me!—as saying that they faced “too little time, too much to do, not enough people.” It implies the industry was doing a terribly suspect job. Since then, the industry is about a hundred times bigger, but those problems mentioned in the Times article or in my book have never been addressed. The industry has simply grown exponentially, and there are hundreds of millions of dollars to be earned by companies that are completely unregulated—to repeat, completely unregulated, so whatever Pearson et. al. tell us, we’re supposed to say “thank you very much” and just write them a staggeringly large check—but of course things haven’t gotten any better.

In my time in test-scoring, we never had enough temporary employees to do the work; we always had too much to do and too little time to do it; and there were always financial punishments looming over our heads if we didn’t get things done. We cut whatever corners we could to get it done (I’m sorry to say). Today the work load is a hundred times bigger and the money to be made is a hundred times bigger, but the system didn’t work to begin with and of course it doesn’t work now.

The same is true in the test development business. When I worked for one publisher as a test developer, it was always a madcap race to get tests written on time, and we faced absurd deadlines and pressure to do so. The reality is that quality was always secondary to the bottom line when developing tests, and then when the Common Core standards were introduced, and tests and products needed to be written for them, our deadlines became laughably absurd; I was once involved in the development of 200 tests in two months, which I think is literally more tests than ETS has produced in its entire existence. With the Common Core standards released, all the companies knew all the other companies were racing to finish their tests and products first, so quality became even worse than secondary. It became tertiary, or “fourthiary,” or whatever. Subcontractors who had been fired for poor work were rehired; item writers were hired off Craigslist; test developers with neither teaching experience nor test development experience were given full-time jobs. It’s important to remember that at the end of the day, companies like Pearson are for-profit enterprises. They want to make money. They want to make money, so of course they do a crappy job, because the quality of the work is never anywhere near as important as their desire to make a profit, and there’s always too much work and too little time to do it.

Rubenstein: Do you continue to talk to former colleagues who are still employed by Pearson and/or other test companies? If so, what have you heard from the inside? (Are those still employed for whom English is a second language? Are they still hiring and retaining unqualified people? Do erasures and rubric juking continue? Have state administrators come in to check the scoring, and, if so, have any demanded that rubrics and/or scores be changed?)

Farley: I have many friends still in the testing industry and still speak to many of them. Most are sheepish, because they know the job they do is suspect. I interviewed many and all say the same thing: they trust their kids’ education to teachers more than to the tests they write or score, because the deficiencies that exist in that process are so obvious to anyone in the industry. One person told me the “testing industry is worse than porn, and I feel dirty,” but she still works in it because she has a family and bills to pay. Others shrug and say, look, this will all be forgotten in twenty years, because they know standardized testing is a house of cards destined to fail. Most of these people, however, have bills to pay and need a job, so they keep at it.

I will tell you, I worry about that myself. I couldn’t be more convinced of the folly of entrusting decisions about America’s students, teachers, and schools to an industry entirely invested in making a profit, but I have a family, too. If I’d not written my book—which I both wanted to write for my own selfish reasons (to get published) and because I thought it was “the right thing”—I’d be in a way better position than I am financially. Now that I have a son, there are times I wish I still worked in testing, because I could use a nice, fat steady paycheck.

Specifically, I hear from people in test-scoring all the time who write to me and tell me they loved my book because it perfectly epitomizes the foolishness that they participate in every day (most recently I got a bunch of those e-mails last spring, which was the last busy testing season). These people (who I usually don’t know) tell me that, yes, Pearson (or DRC or whoever) is still cutting corners, changing rules, hiring anyone, firing no one. The thing is, of course they are, because they have too many tests to score and not enough time to do it, and the paltry wages they offer mean they never have enough people, never have enough time. People misconstrue about my book that I was saying the only problem is that there are bad people hired as scorers, but that was only one part of it; there’s also the fundamental problem with getting twenty people to score a hundred thousand tests in any standardized way, especially in some two-week time frame. It just can’t happen. The students’ responses are too varied, unusual, unique. Meanwhile, state administrators and oversight companies always did come visit our score sites when I worked—and they continue to do so—but they stay for about four hours. Whatever genius expertise they think they bring is ignored as soon as they leave, and most decisions are made by supervisory staff that are also temporary employees. The system don’t work! That’s all there is to it.

Rubenstein: If any, what new information have you uncovered about the “standardized” testing industry?

Farley: There’s not much new about the testing industry to be learned. Test-scoring has always been a dubious proposition, and as it gets exponentially bigger, it gets more unwieldy and more untrustworthy. Meanwhile, there’s not so much money to be earned in test development, that everyone and their brother is getting contracts to write tests, and again, quality diminishes. What is new—and laughable—to me, is how everyone is saying the Common Core standards are going to change everything, which seem, at best, to be slightly better standards that used to exist. But what’s so interesting about this is that now there are getting to be all kinds of Common Core products and practice tests available, but they’re all being written by the same companies and literally the same people who used to write the standards and tests before. The idea these new standards are going to change everything when the tests, et. al. are written by the same people, with the assistance of the same teachers and under the same cloud of deadlines/money to be made as before, is just ridiculous.

An example: I recently saw someone explaining the phenomenal difference between a state standard that asked students to “determine a central idea” and a CC standard that “upped that” by asking to determine the central idea and “explain it through specific details.” One can pretend that’s a big difference, but on a standardized test, they’re both just a multiple choice test that a kid barely looks at—not exactly groundbreaking stuff.

Rubenstein: Can you speak to and elaborate on your most recent Huffington Post article (about computers scoring written portions on tests – link: http://www.huffingtonpost.com/todd-farley/lies-damn-lies-and-statis_b_1574711.html)?

Farley: To me, the fact open-ended questions on standardized tests may be scored by automated systems that have no idea what a student has written pretty much sums up the fact that no one is really in this ed. reform game for anything but the money. Seriously, in my Huff Post article, I make the point that the study’s author concedes that most people agree that standardized test scoring of essay or open-ended questions done by temporary employees for Pearson, et. al. is a terribly inexact science—and then the author of the automated essay-scoring study said, “Look, we have computer program that do just as well as humans!!!!!!” So, his selling point is that computers can do as bad a job scoring essays as can temporary employees, and all kinds of people are yelling “hallelujah.” Anyone with any common sense, or any concern about writing or students, has to see this is a stupid idea. Stupid, stupid, stupid. I say that as a writer and parent, because I’m never going to have the slightest interest in what a computer that doesn’t know what my son has written thinks about what my son has written.

Rubenstein: How many copies of your book have sold, to date? What are your thoughts as to why—when the book came out in 2009—it wasn’t exactly a best-seller?

Farley: Well, to put that in perspective, when I got my most recent royalty statement for Making the Grades, my wife and I had a long laugh. A bitter and sad laugh it was, but a long one, too. Every single line on the royality statement was a 0. 0, 0, 0. It seemed in the last six months I’d sold 0 books, meaning 0 for the year, 0 for the last two years, etc. etc. etc. I think after it came out, I sold a couple thousand, which is nothing to be ashamed of. I got some decent reviews and some decent publicity, but it just didn’t sell. In part, that has to be because I wrote a first-person account and a memoir about my time in standardized testing, which was surely a mistake; many people are very riled up about education these days, so my attempt to be glib and humorous was surely the wrong tone. I also feel like I didn’t get much help in selling it, I have to say. My agent was a fellow first-timer (like me), and my publisher was small and is now out-of-business. But the week my book came out, I was on the op-ed page of The New York Times. Somehow, no one parlayed that into anything at all for the book, and it died on the vine, as they say. I’m still annoyed when I hear people say, “I didn’t know such a book existed.” I’d just feel less dissatisfied about the book’s sales if I knew people knew about it and didn’t want to read it; instead, I feel like no one knew about it.

In the end, I’m glad I wrote it, because it was the right thing to do, but if I did it again I would not be so personal or so glib, and I’d incorporate the stories of others in the industry who feel the same way I do. I certainly erred in writing the book as a memoir, but I just think with so many people out there against standardized testing, my little insider’s view of all the industry’s foibles could have sold so much more.

Rubenstein: Well, it’s never too late! (You can purchase Todd’s book here – link: http://www.amazon.com/Making-Grades-Misadventures-Standardized-Industry/dp/098170915X/ref=sr_1_1?ie=UTF8&qid=1351399495&sr=8-1&keywords=making+the+grades) If you had to pick the worst testing company (in terms of short-cuts, unqualified employees, cheating, poor criteria, etc.) you worked for, which would you choose, and why?

Farley: This is a trick question, right? The easy answer is Pearson, which is not only the evil empire of testing, but a British company. Seriously, it’s impossible not to see them as the East India company of modern times, with our government trying to force this country’s citizens to purchase that company’s products whether we want to or not. They have a history of failures in test-scoring and test-delivery but because they’re so big, they keep getting more and more million-dollar contracts. Even audits of their test-scoring done by the U.S. government have shown that they cut corners, cheat, hire unqualified people, etc., but they dominate the industry. It’s insane.

Rubenstein: At this point in time, do you know how much money is being made in the industry? Can you break that down by companies (which one is making the most, and why)?

Farley: Seriously, I’m too depressed to look this up. It’s easy to find on the Internet, however.

Rubenstein: At the end of the book is a chilling statement: “Do what you want, America, but at least you have been warned.” Just now people are crying, “Foul!” Groups such as United Opt Out and Fair Test have been involved in petition-signing and outreach. However, the dent has not been made thus far. What can we—the public—do to stop this testing frenzy and waste of billions of taxpayer dollars that would, of course, be better spent enriching the education of our public school children?

Farley: Speaking of the East India company, I love the story of the Boston Tea Party. When the British government—and its business cronies—tried to force laws down the throats of the citizens of the Massachusetts colony, those citizens argued against the laws they didn’t like, they boycotted various companies, and they made political entreaties. When the government ignored them and tried to do whatever they wanted, the citizens of Massachusetts ended it by donning war paint on their faces and arming themselves with hatchets…cue the Boston Tea Party. While I’m not exactly arguing for armed insurrection against the U.S. Department of Education and/or Pearson Education and friends, people need to protest the current ed. reform movement—they need to argue to change the laws, and they need to ask honorable politicians for help. If that doesn’t work, I’ll say this: the first Pearson rep that starts moving towards my preschool son is going to be met by a dude (me) wearing war paint. And I’ll be holding a tomahawk in my hand.

Rubenstein: What an awesome image. Maybe we should all stock up on tomahawks? To get back to our serious tone: I graduated high school in 2005, and I distinctly remember standardized tests playing a large part of the anxiety I suffered as a teenager (as a pretty good English student, I could write one hell of an essay, but found myself shaking with lack of confidence when it came to standardized tests). I received the same score both times I took the ACT—a 26, which, I was told, just wouldn’t cut it when it came to applying to most of the colleges I wanted to go to. Due to my testing anxiety, I didn’t even take the SAT (which, fortunately, my state didn’t require). Fortunately, the college I applied Early Admission to—and eventually went to—wasn’t even accepting standardized test scores at the time; these tests, the school argued, did not present a clear case that the student would thrive within the college’s academic environment, which was based on Oxford University’s research-heavy model. Can you speak to this disconnect: how is it that a student, who holds honors in certain subjects, might be deemed “unqualified” for a certain college or university just based on the merit of their standardized test scores? Isn’t this just another form of corruption—that a bad test-taker might be the best student in the world, but is set up for failure because our higher educational system puts so much weight into these seemingly arbitrary tests?

Farley: I don’t know that I’m the best person to answer this question. The fact is, all I was was someone “on the slaughterhouse floor,” and I saw that the product we in the test industry were producing was rancid, rotten, useless. I wrote my book to warn people about trusting that industry to make real decisions about students, teachers, and schools. I mean, if a student takes a standardized test today, it gets chopped into pieces (literally) and sent around the country. The multiple choice questions (which most people believe are theoretically flawed) get scored in one state one day, while all the open-ended questions get scored out of order, on different days, by different people, in different states, with everyone under various stressful edits and deadlines. Then those numbers are jammed together and a score is produced that is supposed to represent an actual human being? It’s like anti-holistic education, when no one can see a whole person but all anyone can see is minutiae. Putting any faith in the industry I worked for is a laughable idea as far as I’m concerned—although, I should say that I was involved largely in K-12 work, not the SAT or ACT.

Rubenstein: During the recent Chicago Public Schools strike, teacher evaluations based on state testing requirements were cited as one of the reasons for the strike itself. If these tests are still being scored in the same way as conveyed in your book, it’s inconceivable that anyone would be able to argue with the teachers’ grievances. And yet, their grievances continue to go on ignored. Can you speak a little about education reform, and what the next presidential administration needs to do in order to keep more teachers in the classroom? Do you think there needs to be a huge overhaul of the way our government funds public schools, and that the importance of standardized testing needs to be reassessed in order for there to be equity within the education sector?

Farley: Again, this isn’t my area of expertise, but what ed. reform clearly looks like is that a lot of people wielding a ton of power want to make money on schools, and they are literally willing to do anything to make it happen. Evaluating teachers based on test scores is an idea that has literally never worked anywhere, and is literally unproven scientifically. Pretending it can help is to simply ignore facts, which it seems many people are willing to do to get at the billions that can be made in our public schools. How else would anyone entrust Pearson and their ilk to be so important in this process when Pearson and their ilk have a history of mistakes and failures. How can anyone entrust Pearson when I (and others, like Dan Dimaggio) have sworn based on our personal experiences that they can’t be trusted. Whatever problems that exist in Chicago’s schools, Pearson or more standardized tests aren’t the answer.

Rubenstein: Mother Jones recently published an incredible investigative piece on one of the “failing” high schools in America: Mission High in San Francisco. (Link here: http://www.motherjones.com/media/2012/08/mission-high-false-low-performing-school) Failing, and yet, here is a portrait of a high school that openly embraces an immigrant population, and turns non-English speakers into valedictorians. Can you address why our government—and, in some respect, our culture—would rather assess a school by its inability to rank within an acceptable percentile when it comes to state standardized tests, than its ability to create change through effective lesson plans, extracurricular activities, and strong teachers? Especially when considering the standardized testing industry is so scandalized in its practices.

Farley: Right! Is helping students grow and mature and become good people and citizens important to us, or is it the ability to fill in the right bubble on a flawed test written by a for-profit company? I always thought of ours as a great country. To be American always meant to be adventurous and big-thinking and brave. It meant believing in the power and possibility of the individual. To fixate on standardized testing seems to me to be inherently un-American, as I’m not interested in people thinking the same things, doing the same things, being one of a cookie-cutter many. If I think of my own son, I dream of what kind of world he can create, what things can he discover; I wonder what kind of world he is going to make for himself, and I hope it’s so much grander than anything that currently exists. I have no interest in him getting in line and being like every other kid in New York City or the United States, so a standard curriculum or a standardized test are going to tell me nothing about him. I am interested in a school that helps my son become his best self, and I think that happens through people, through teachers and coaches and administrators and other students. The only way I am ever going to judge a school is based on its humanity, not its test scores. Why anyone would do anything else is beyond my understanding…especially considering a standardized testing industry that “is so scandalized in its practices.” It makes no sense to me.