The central feature of the Obama administration’s $5 billion “Race to the Top” program was sharply deconstructed and refuted last week by the American Statistical Association, one of the nation’s leading scholarly organizations. Spurred on by the administration’s combination of federal cash and mandates, most states are now using student test scores to rank and evaluate teachers. This method of evaluating teachers by test scores is called value-added measurement, or VAM. Teachers’ compensation, their tenure, bonuses, and other rewards and sanctions are tied directly to the rise or fall of their student test scores, which the Obama administration considers a good measure of teacher quality.
Secretary Arne Duncan believes so strongly in VAM that he has threatened to punish Washington state for refusing to adopt this method of evaluating teachers and principals. In New York, a state court fined New York City $150 million for failing to agree on a VAM plan.
The ASA issued a short but stinging statement that strongly warned against the misuse of VAM. The organization neither condemns nor promotes the use of VAM, but its warnings about the limitations of this methodology clearly demonstrate that the Obama administration has committed the nation’s public schools to a policy fraught with error. ASA warns that VAMs are “complex statistical models” that require “high-level statistical expertise” and awareness of their “assumptions and possible limitations,” especially when they are used for high-stakes purposes as is now common. Few, if any, state education departments have the statistical expertise to use VAM models appropriately. In some states, like Florida, teachers have been rated based on the scores of students they never taught.
The ASA points out that VAMs are based on standardized tests and “do not directly measure potential teacher contributions toward other student outcomes.” They typically measure correlation, not causation. That means that the rise or fall of student test scores attributed to the teacher might actually be caused by other factors outside the classroom, not under the teacher’s control. The VAM rating of teachers is so unstable that it may change if the same students are given a different test.
The ASA’s most damning indictment of the policy promoted so vigorously by Secretary of Education Arne Duncan is:
“Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.” The ASA points out: “This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.”
As many education researchers have explained–including a joint statement by the American Educational Research Association and the National Academy of Education– the VAM ratings of those who teach children with disabilities and English language learners will be low, because these children have greater learning challenges than their peers, as will the ratings of those who teach gifted students, because the latter group has already reached a ceiling. Those two groups, like the ASA agreed that test scores are affected by many factors besides the teacher, not only the family, but the school’s leadership, its resources, class size, curriculum, as well as the student’s motivation, attendance, and health. Yet the Obama administration and most of our states are holding teachers alone accountable for student test scores.
The ASA warns that the current heavy reliance on VAMs for high-stakes testing and their simplistic interpretation may have negative effects on the quality of education. There will surely be unintended consequences, such as a diminishment in the number of people willing to become teachers in an environment where “quality” is so crudely measured. There will assuredly be more teaching to the test.. With the Obama administration’s demand for VAM, “more classroom time might be spent on test preparation and on specific content from the test at the exclusion of content that may lead to better long-term learning gains or motivation for students. Certain schools may be hard to staff if there is a perception that it is harder for teachers to achieve good VAM scores when working in them. Over-reliance on VAM scores may foster a competitive environment, discouraging collaboration and efforts to improve the educational system as a whole.”
For five years, the Obama administration has been warned by scholars and researchers that its demand for value-added assessment is having harmful effects on teachers and students, on the morale of teachers, on the recruitment of new teachers, and on the quality of education, which has been reduced to nothing more than standardized testing. Secretary Duncan has brushed aside all objections and pushed full steam ahead with his disastrous policies, like Captain Ahab in pursuit of the great white whale, heedless to all warnings.
Based on the complementary statements of our nation’s most eminent scholarly associations, any teacher who is wrongfully terminated by Duncan’s favorite but deeply flawed methodology should sue for wrongful termination. What is not so clear is how the nation can protect our children and our public schools from this administration’s obsessive reliance on standardized tests to rank and rate students, teachers, principals, and schools.
I just started this at Petition2Congress. It is very easy to sign, copies are automatically sent to President Obama, and your own senators and your representatives. Please take the time to read and the petition entitled: STOP COMMON CORE TESTING. Thank you.
http://www.petition2congress.com/15080/stop-common-core-testing/?m=5265435
TOGETHER WE CAN STOP THE MADNESS.
THE TIME IS NOW!
LET THE POLITICIANS IN DC KNOW THAT WE ARE MAD AS HELL AND UNWILLING TO SIT BACK AND WATCH THE STATUS QUO CORPORATE REFORMERS DISMANTLE OUR PUBLIC SCHOOLS.
My apologies for screaming
Thanks NYTeacher for the link. My emails are off to my Senators and Representative and Obama.
Thank you. Together, we can make this happen.
Thank you NY teacher…from our family.
Shameful, Mr Obama…a national disgrace you have supported.
Do the right thing and stand up for the America’s children now…do it as the president, do it as a father and human being.
Thank you Steve for taking the time. Together we can stop the madness and take back our public schools. Re-make them into places where students go to learn, and grow, and develop, and explore. No longer will we allow our schools to become data centers for Gates and Co.
Thanks NY Teacher…I signed yesterday.
Signed and spread!
It doesn’t matter.
“Stay the course!” “Don’t back down!”
Too many ambitious and influential people backed this, and poured money into it. They’re rather pull out their own fingernails than revisit their misinterpretation publicly.
There should be a formula for that. How many politicians + pundits + CEO’s does it take before “truth” is set in stone and impossible to revise?
We’d be able to find that sweet spot where we could still knock them off course before they march off a cliff 🙂
Chiara Duggan: with all respect, I think your last sentence might have been missing a few words…
“We’d be able to find that sweet spot where we could still knock them off course before they march THE MAJORITY OF US off a cliff[.]”
Never forgetting that they ensure an education for THEIR OWN CHILDREN that is radically different from, and superior to, what they mandate for OTHER PEOPLE’S CHILDREN.
When was the last time we heard of VAM being used at Harpeth Hall [Michelle Rhee] or Sidwell Friends [Barack Obama] or U of Chicago Lab Schools [Rahm Emanuel] or Lakeside School [Bill Gates] or Delbarton School [Chris Christie] and the like?
Thank you for your comments.
😎
Hah! I guess I refuse to consider myself marching with them 🙂
What/where is “Harpeth Hall”? I thought Rhee went to Maumee Valley Country Day outside Toledo? I’m slightly familiar with it because I once lived close to there when my (grown) children were small. We went to some sort of childrens event there- balloons and face-painting or something.
Harpeth Hall is an elite private school in Nashville
The Rhee children attend Harpeth Hall, a tony prep school in TN.
KTA,
Thank you for continuing to remind us that not ALL children will be subjected to inexperienced teachers, data mining, endless test prep, draconian discipline, common core close readings, dilapidated buildings, constant disruption and etc.
Only some people’s children.
Also, this unrelated, but you mentioned Christie and Chris Christie’s twitter feed is laugh out loud funny.
It is ALL “me, me, me” down to some phony soul-searching on whether being governor is “worth it”. How people ever were taken in by this corrupt fraud is beyond me.
I thought it was a parody account, but it’s not. It’s that bad.
Or Woodland Hills Montessori (King)?
Fitting in here is this link from 2009 which was sent to me today by Rene Diedrich. It is all happening just as Obama planned it early in his first term. So much for hope and change…at least any that is positive.
http://www.globalresearch.ca/the-democrats-vs-public-education/14545
Ang: you’re most welcome.
dianervatich & Alan: thank you for adding that info.
Chiara Duggan: I made the painful transition from “they know not what they do” to “they know they done wrong” some time ago when I realized that the self-styled “education reform” leaders of the “new civil rights movement of our time” were mandating for OTHER PEOPLE’S CHILDREN something radically different than what they were ensuring for THEIR OWN CHILDREN.
How could I be so sure? Because they are so careful in making excellent choices for THEIR OWN CHILDREN that it is impossible to imagine they don’t know what a genuine enriched education is all about. I include some links with the names of the edufrauds that send THEIR OWN CHILDREN to genuine institutions of teaching and learning. This is but a minuscule such listing.
Link: http://www.harpethhall.org [Michelle Rhee]
Link: http://www.delbarton.org [Chris Christie]
Link: http://www.sidwell.edu [Barack Obama]
Link: http://www.spenceschool.org [Michael Bloomberg]
Link: http://www.ucls.uchicago.edu [Rahm Emanuel]
And I save the best [?] for last. Bill Gates went to this school himself and now his children go there—
Link: http://www.lakesideschool.org
And just what does Bill Gates think about Lakeside School?
No need to guess about such issues as class size when it comes to THEIR OWN CHILDREN and themselves—
Link: http://seattleducation2010.wordpress.com/2012/06/18/bill-gates-tells-us-why-his-high-school-was-a-great-learning-environment/
And another link to follow up: http://seattletimes.com/html/dannywestneat/2014437975_danny09.html
Sometimes more painful than enlightening, if you look at these websites you will see not only what is provided for the “worthy” few but what is being denied the “unworthy” many.
And yet they talk about giving poor parents the same choices as rich parents and switch between lauding standardized testing and decrying it [while profiting from it] and try to rebrand CCSS in order keep the product launch going and so on. Such hypocrisy, clinging ever so desperately to their rigid Marxist ideology:
“Those are my principles, and if you don’t like them… well, I have others.”
¿? The famous one of course: Groucho.
But don’t expect a change of mind or heart anytime soon.
“You can’t teach an old dogma new tricks.” [Dorothy Parker]
😎
P.S. I would invite you all down for a round at Pink Slip Bar & Grille but now’s not a good time. The atmosphere has gotten a bit dour. Seems that Socrates was given a VAManiacal rating for his work in the teaching of philosophical principles based on 3rd grade reading scores in some small rural Greek village and he was rated ‘highly ineffective’! For some strange reason his oft-quoted statement that “I know that I am intelligent, because I know that I know nothing” dealt a fatal blow to his numerical standing. Socrates took this all pretty hard. I am not sure why this has happened, but that Paul V loner fella has been badgering him lately, plying him with unusually large amounts of ouzo and filling his head with nonsense about “you can’t manage what you can’t measure” and such like.
What particularly hurt his pride was that Plato was rated ‘in need of improvement’ but just on the cusp of making ‘effective.’ If Aristotle did even better I don’t know what we can do to save him besides making sure he doesn’t start sucking on the hemlock again…
😉
Arne only believes in the fear and intimidation that VAM brings with it. The only way to FORCE teachers to teach to a test and standards that they would not otherwise support is by threatening careers and reputations. Since this is the foundation upon which the status quo reformers constructed the CCSS – its collapse is imminent. Tear down the walls.
Ultimately, this is what all VAM methods are to be. I work in a completely unrelated field and it’s metrics are mostly counter productive to what they purport to do, increase profits. What they do is make it easy to fire people by assigning blame. Ultimately, as the system is stretched, there is a loss of institutional expertise and constant breakdowns.
I will admit that it does make it possible to speed up production by firing workers and assigning blame to the remainder when a variable appears that precludes meeting metrics. This is possible with unlimited amounts of labor, even in a regulated worker industry.
In the words of the late Madeline Hunter: “research articulates the obvious.”
Always to hear the words of Madeline Hunter, Alan…did you train with her?
Here is an insisive article from 2009 sent to me by Rene Diedrich today. And it came to pass….
http://www.globalresearch.ca/the-democrats-vs-public-education/14545
Left our the operant word…GLAD to hear about Madeline.
too many typos late at night…phooey.
In Los Angeles Lenny Isenberg is suing the union and the law firm that represents them. This is one of several posts on his wonderful site that chronicles the debacle in LAUSD and the war on teachers that has removed 17,000 veteran teachers, including Lenny.
http://www.perdaily.com/2014/03/has-utla-rank-and-file-been-told-that-im-suing-utla-why-not.html
We’ve projected your performance teaching 3rd grade PE, based on the 5th graders math scores.
VAM is hocus pocus garbage through and through.
Long article, but all I can say is WOW!!!!
Reblogged this on Professor Olsen @ Large and commented:
This should be another nail in the coffin of “education reform”, also known as privatization.
“quality crudely measured.”
well put.
It’s about time. The ASA’s code of ethics reads, that members must:
Support sound statistical analysis and expose incompetent or corrupt statistical practice. In cases of conflict, statistical practitioners and those employing them are encouraged to resolve issues of ethical practice privately. If private resolution is not possible, recognize that statistical practitioners have an ethical obligation to expose incompetent or corrupt practice before it can cause harm to research subjects or society at large.
How the people at Mathematica or AIR can blithely go about developing VAMs for states and districts without questioning how they will be used is negligent, at best.
Diane, don’t confuse education deformers with the facts.
Reblogged this on Reflections of a Second-career Math Teacher and commented:
“Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.” The ASA points out: “This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.” American Statistical Society, April 8th, 2014.
FOLLOW THE MONEY AND IT ALL MAKES PERFECT SENSE
Teaching is the only occupation I know where people who have never taught think they know what constitutes “effective” teaching. It is also the only occupation I know where these same know-nothings want to tie pay, job security, etc. on WHAT OTHERS DO, NOT on what the teachers do. Teachers have NO control over students’ learning–it is strictly up to the desire or the ability of those students to learn.
Yes, you can become Sec. of Ed. and have no experience whatsoever!! It’s time to end the VAM SHAM!
I think everyone would agree that teachers have no control over students’ learning, but surely you must believe they have influence of student learning. If not, why do we worry about teacher qualifications?
200 letters/emails sent to President Obama and Congress in 7 hours.
It is very easy to sign, copies are automatically sent to President Obama, and your own senators and your representatives. Please take the time to read and the petition entitled: STOP COMMON CORE TESTING. Thank you.
http://www.petition2congress.com/15080/stop-common-core-testing/?m=5265435
Join this grass roots Resistance and tell the powers that be that their power is being threatened by the outraged parents, teachers, and administrators of America. Stay fierce and fearless.
Last year I was rated ineffective on NC standard 6 based on VAM models of students I didn’t even teach! I taught kindergarten and every child in my class was at or above grade level yet I’m ranked ineffective based on hocus pocus that can’t even be explained by the district statistition in charge of VAM for our county. Junk science
Sorry, auto correct going nuts.
Statistician
Step #1: Sign the petition to STOP COMMON CORE TESTING
Step #2: GET A LAWYER
I don’t think the ASA report is nearly as “stinging” as Diane makes it out to be. It struck me as a pretty gentle caution. It’s full of hypotheticals about how VAMs could cause this or that problem, and says that they should be used with caution and only by sophisticated statistical experts, but it does not say that VAMs in general aren’t being used properly and it doesn’t cite any specific instances of abuse. To an Arne-ite the obvious response is “thanks for the heads up. We’ll make sure we continue to do a great job.” This report is not going to change anyone’s mind.
John
Forget the ASA report. Did you read Amber’s post from NC?
Teachers being rated using the scores of students that they don’t teach. This is a common practice because Arne dug a hole before checking for gas lines. CCSS only covers math and ELA in grades 3 to 8 and eventually HS. The vast majority of teachers do not teach to the CC standards yet needed to be judged on student test scores nonetheless. Different states took different approaches in regards to the way VAM would be implemented. Some teachers get to write their own tests, the scores of which are used to evaluate them. Gas line!
Now John go write a test for your students, and don’t forget, if they don’t do well, you might get fired. This should promote some real rigor, no? The alternative is what happened to Amber and countless other teachers like her. All caught in Arne Duncan’s gas line explosion.
Teachers here in Buffalo are being evaluated on kids they don’t teach or teach for one period a week. Last year the principal of the state run OMH psych unit where I teach, who is responsible for 2 sites, was judged on the scores of a recent Somali refugee, a cutter who was hospitalized for half the year, a schizophrenic girl who hears voices hourly and a school a phobic kid who missed approx 80% of the school year. The kids are in a middle school class 6 miles from the building where he has his office. Someone in NY thinks this is a good way to evaluate administrators.
I’m not defending VAMs in teacher rating systems. I’ve read many accounts of problems with them and am convinced that they shouldn’t be used for this purpose. I’m just saying that the ASA report doesn’t strike me as particularly scathing. I would love to see a scathing demolition of the the overall VAM picture in the US, but this ain’t it
.
I think the report is trying not to alienate either side so gently puts the criticism in very neutral terms. As a statistician, I think it pretty much nails why a statistician shouldn’t go near these models for use in schools.
I’m going to disagree with you on how to interpret what the ASA report says. They basically say (or imply) using VMA for teacher effectiveness is not “best practices” and is better used to evaluate instructional quality given, such as comparing one type of instruction to another, versus the quality of the teacher.
ASA was just being professional in their wording. This really is a stinging rebuke of VMA. Especially when they say teachers only account for 1-14% of test variation while Duncan and other supports of VMA are implying teachers are the sole or dominant cause. This is a huge rebuke of VMA.
Breaking news for people like us. Since when does reality, math, science or logic play any role in the blind money grab of ed reformers. This will amount to nothing.
My last copy of the Columbia Journalism Review says it all. The cover story: “Who Cares if it is true”? The media castigates teachers and American education but is in the hands of 5 or 6 major corporations which control 80% of the “news” which shapes public opinion. When these corporations set their sights on the millions of dollars in education, look out.
Just one more area: They scream for more scientists but do not listen to the 97% of the world’s best.
Black holes exist not only in space.
“Secretary Duncan has brushed aside all objections and pushed full steam ahead with his disastrous policies, like Captain Ahab in pursuit of the great white whale, heedless to all warnings.”
And we all know the results of Captain Ahab’s obsession. Who will survive to utter those famous words: “Call me Ishmael.” when Secretary Duncan’s Pequod sinks?
I teach Language Arts in a magnet program at a high school in Florida. My VAM score was 3.49. The social studies teacher who had the exact same students received a 1.5. Ditto for the science teacher. The foreign language teacher–who had most of the same students–received a 2.49. We went to the District and they couldn’t give us a straight answer as to why our scores varied when three of us had the same students and the fourth had most of the same students. After we repeatedly tried to get them to explain the “formula” with no coherent answer, they finally raised everyone’s score to 3.49.
outofthecave: your whole comment is a “money quote.”
They can’t explain it—a polite way of saying they don’t understand it themselves. And they adjust the scores just to get you out of their hair, meaning, an arbitrary measurement that can be changed willy nilly for political and human resources reasons.
I hear the death knell of even the pretense of the mantra of “hard data-driven decision making”—so nobody should object in the future when I write comments describing VAM-related nonsense as being a part of “data-drivel decision making.”
Again, outstanding!
😎
The ASA took a decent step to try and preserve the integrity of statistics.
Obama, Race to the Top will be your Vietnam, your Watergate. (Let’s call it Gatesgate) With it, you enabled the underfunding and destruction of public education, memorialized presidential overreach, collaborated in the slander of an entire profession and undermined that profession as a viable and worthwhile middle class career path. You have been a disaster as a president. So, now as you wind down your tenure… you make big broad brush statements to cement your place in history… taking safe stands… like equal pay for equal work and advocating liberal causes like maintaining unemployment benefits in a stagnant economy. For whom? For those teachers half way through a dedicated career whose jobs stood in the path of privatization? That’s like your advocacy for returning funds to the food stamps program… funds that you took in order to pay for your wife’s pet project. Your legacy is this: You make the wound and then promote the bandaid.
Who are you with as you rush to hand over public education to billionaires and publishing companies? Your constituency are the privileged who dabble in social justice to increase their personal cache…. who advocate closing schools as a means for dumping teachers and unwanted children, who call their entrepreneurial k12 solutions the “civil rights issue of our time.” You stand idly by while your governors fire whole districts full of career teachers to fund a bottom line profit motive. You consort with a cynical TFA youth movement that conflates ambition and idealism and whose members singular contribution is their ability to quickly replace middle aged teachers who are fired without cause and be the front lines for the erosion of work place protections. You support using charters to covertly abandon the most troubled youth while blaming the beleaguered teachers who serve them. You support a test publisher bonanza even though it is manifestly clear that the testing culture is detrimental and one you would never allow for your own privileged children. You sanction the continued rise of disreputable figureheads like Michelle Rhee whose career history is speckled with dishonor, self promotion and corruption. And, you piously advocate a culture of fear of job loss, community collapse, school takeover and loss of placement as the primary basis for reform. We are not with you, Mr. President because you are not with us.
Powerful writing. I can’t stop reading this.
Where were you when I needed you audhilly?
Please take the time to read and the petition entitled: STOP COMMON CORE TESTING. Thank you.
http://www.petition2congress.com/15080/stop-common-core-testing/?m=5265435
“He enabled the underfunding and destruction of public education, memorialized presidential overreach,collaborated in the slander of an entire profession and undermined that profession as a viable and worthwhile middle class career path.”
This could be his epitaph.
Thank you audhilly, for your powerful words. Please consider letting your post be a narrative for a You Tube video which we could help go viral. Perhaps that might puncture the arrogant veneer of Obama, Duncan, Rhee/Rahm & Co. He’s a relatively young man for the office he holds, and after all the damage he has inflicted, I wouldn’t
mind seeing Obama, post-2016, become the quasi-pariah that GWBush has become. With all the pain and destruction Obama’s education policies have create,Obama has earned his ineffective rating.
The only time Mr. President was with us was when he wanted our money and our endorsement.
Eleanor, a great idea…a narrative on Youtube. Many of my friends, have only a minimal grasp on what is happening in our country. The layers are many and when I share my concerns with them they look at me aghast and move quickly on to the next topic; something a little more digestible I suppose. The word is not getting out.
Reblogged this on peakmemory and commented:
Another caution about the use of value added measures of teaching, this time from the American Statistical Association.
Reblogged this on Matthew Karram and commented:
Really scary stuff. It is amazing how the people in charge can so vehemently ignore or blow off expert research and real world results.
Interesting
Means something. I bet they will give another message–I don’t know when, though– if the things get worse.
Plain and simple: If you do not agree with VAM then please propose an alternative way to assess teacher performance. All teachers are not created equal. How can we put students first if we are not willing to assess teachers and make those assessment count. Students have benchmarks and are assessed so should teachers. And I am NOT for high stakes testing, but I think we need something. We can’t just look the other way and say because you somehow got hired you should be allowed to be terrible. The system has to change, but we need accountability on all ends. If assessment is based on student gains it is about as fair as it gets. If you can’t get your kids to grow and actually learn something over the course of the year, then why do you have a job? The job is teacher not babysitter. Motivating a classroom to learn is part of basic classroom management. That’s step 1. Anyone who tries to blame the child over the adult for lack of learning is really missing the mark about what teaching is.
Kathryn
Read the research. VAM is junk science. It is not professional. No other nation uses it. Professional judgment works best, by peers and administrators.
Have you read but learned nothing Kathryn? Or are you new to the conversations here? If you are new, please continue to come back often and read. You will find the answer to your questions when you put aside your preconceived notions and listen to the expert witness of the professionals and the parents here who are fighting the status quo of corporate education reform.
Reblogged this on David R. Taylor-Thoughts on Texas Education.
Reblogged this on rightfulwriter.
Sorry about the length of this note.
Education Deformers typically operate with arrogance, at a distance, and in breathtaking ignorance.
They seem to think, for example, a) that one can draw valid conclusions about reading and writing ability in general from scores on standardized tests like PARCC (spell that backward) and b) that one can draw valid conclusions about teacher quality by comparing a teacher’s students’ scores at the beginning and end of a year.
Both propositions are false. Let’s consider each in turn. I am going to try to keep these arguments simple enough that even a plutocrat or politician can follow them. And I am not going to treat these at all exhaustively.
The Invalidity of the Standardized Tests in Reading and Writing
In general, if you want to find out whether someone can do something, you have them do that for you.
If you want to find out whether knows how to sew on a button, you ask him or her to sew on a button. If you want to find out whether someone knows his or her times tables from 1 x 1 to 12 x 12, you ask him or her do random multiplications: 4 x 6, 11 x 9, 3 x 8.
If you want to find out whether someone can read and write, you have to ask him or her to read and write.
Here’s the problem: What students do in these unnatural, inauthentic testing situations on these crude, narrow standardized summative assessments of reading and writing bears little relation to actual reading and writing. So, since the tests do not engage in anything like real reading and writing, the tests are, ipso facto, not tests of real reading and writing.
Let’s consider reading of literary texts. There are many thousands of ways in which people in the real world interact with literary texts. The most common category of interactions with texts is personal reading for pleasure and, secondarily, for edification. People read because they love to read, and then they engage in public discourse about that reading—they talk to their friends about books they’ve read, they discuss those books in reading groups, they write and read reviews, they make comments on others’ online reviews, they join fan fiction sites and write their own spin-offs, they allude to the texts in other speech and writing that they do, they create parodies and other adaptations, and so on. This is one set of things that “reading literature” means. It means all of that. And none of that “reading literature” is done on these tests. Let me repeat: This is the largest category of “reading literature,” and none of those types of interactions, those “readings,” are tested by these assessments.
The second most common category of interactions with literary texts is the scholarly, and this set of textual interactions is also extraordinarily varied. Scholars have developed many, many approaches to texts, including, but not limited to explication de texte, source study, Textual Criticism, philological analysis, Romantic or Expressionist Criticism, Formalist analysis, Freudian Criticism, Jungian and other forms of Archetypal Criticism, New Criticism, Mimetic Criticism, Historicism, Structuralist analysis, Intentionalism, genre study, reader-response, Marxist critique, feminist critique, Deconstruction, critique of instantiated power relations and interpellations, stylistics, discourse analysis, New Historicism, frame analysis, writing process/revision analysis, Biographical Criticism, Didactic Criticism, Pragmatic or Rhetorical Criticism–one could go on and on and specify lots and lots of subcategories. Now, these fancy terms sound complicated, but one can take any of these approaches with quite simple texts and in quite simple ways with quite young readers. What was happening in science and in politics in the time of Mary Shelley or H. G. Wells, and how did these affect Frankenstein and The Time Machine? So, this is the second largest category of things that people do when “reading literature,” and ALMOST NONE these things is ever done on these exams.
So, if that’s what people do when they read literature, and if the tests don’t have students do those things, then they are not tests of reading literature, are they?
QED
Tell me, when was the last time, except for a test prep exercise, that you read a short story to find out which of four techniques of character development was used in paragraph 8?
When we approach literary texts with students in classrooms, we first set them up, providing necessary context, and then we encourage our students, first and foremost, to take the journey—to fall down that rabbit hole, step through that wardrobe, fly through that rip in the space-time continuum, and HAVE THE IMAGINARY EXPERIENCE that the text occasions. Then, we encourage them to reflect on and react to their experiences, the text, the author, the period—any of a great many matters, depending on the text, the approach we are taking, and what we want to accomplish. All this takes time and is COMPLETELY UNLIKE anything that is done on standardized tests.
If you want to find out whether students can read literature, you ask them to read literature—in hundreds of different ways, over the course of years, and you evaluate what they have done in hundreds and hundreds of different ways, depending upon what you wish to accomplish.
The test makers have taken a tiny, tiny, tiny fraction of the reading of literature and decided to test that. Why? Because that’s what tests as they conceive them are capable of. THEY HAVE GRABBED A HAMMER AND SET ABOUT TREATING EVERYTHING AS IF IT WERE A NAIL. Because they are having students engage in such a WEIRDLY NARROW SUBSET of activities from the realm of reading literature, what they are asking kids to do is NOT VALID as a test of the whole. It’s rather as though someone had decide to test ability to play baseball by asking questions about how to calculate batting average. Is this person a great pitcher? A great fielder? A great batter? Sure, she can calculate batting averages with an 80 percent accuracy.
I have written at length, elsewhere on this blog, about the invalidity of the writing done for these tests. So, I won’t repeat that here. Suffice it to say that if we are taking writing well, we are teaching the entire writing process, often using a collaborative workshop model. For the tests kids do something that cannot by any stretch of the imagination be called writing as it is done in the real world or as it is done in our classes, if we are still teaching writing and have not, instead, yielded to the pressure and begun to teach Formulaic InstaWriting for the Test. Writing and Formulaic InstaWriting are completely different abilities. For the summative tests, students do a variety of the latter. And, of course, accomplished writers have mastered a vast body of specific, concrete techniques and ropes and structures, ALMOST NONE OF WHICH are tested on the standardized tests.
So, whatever these tests test, it is not reading literature and writing.
But because of “accountability,” people end up throwing out real teaching of reading literature and real teaching of writing and, instead, teaching courses that might be called “InstaReading and InstaWriting for Tests.”
The tests dramatically distort and narrow curricula and pedagogy, and incalculable damage is done as a result.
The Invalidity of VAM
The evidence against VAM is overwhelming, so I am not going to enumerate that again here. I just wanted to mention one critique that, I think, is especially damning but that isn’t often heard. Teachers plant seeds. These often take quite while to grow. Suppose that I have a kid in my class whom I turn on to science fiction. Maybe this kid wasn’t a reader before, but I got him or her interested in Ender’s Game or Slaughter-house Five or the Hitchhiker’s Guide to the Galaxy or Neil Gaiman’s American Gods. And maybe, then, over the course of a young adulthood, this person takes to reading science fiction books and perhaps even joins one of those fan fiction sites and starts contributing. Perhaps he or she becomes, as a result, a reader. I will have done my job. But that seed will have taken a long time to grow.
The test will punish the acorn because it is not yet an oak.
And the kid will say, “F that,” and all my work will be for naught.
cx: tropes, not ropes, of course. A number of typos in this, which was written hastily. My apologies for those.
cx: So, since students do not engage in anything like real reading and writing when taking these tests, their actual reading and writing abilities are not, ipso facto, being tested.
Oh, to be able to edit these posts! 🙂
VAM is based on the dubious proposition that these crude tests do a better job, on average, of evaluating students and teachers than do the millions of teachers and administrators who are actually on the scene. The Deformers are say, “We cannot trust you people. We do not respect your judgments. We are going to do the judging here, in our way. And we are going to apply the same criteria to everyone.
And so they set about testing the fish, the eagle, the mole, the grizzly, the ocelot, and the dolphin on their burrowing ability.
Idiots.
VAM. Value-Added Measurement, or Vacuity-of-curriculum-and-pedagogy Acceleration Mechanism; means for enforcing the reduction of the complex, unquantifiable, humane enterprise of teaching and learning to a number intended to measure the extent to which a teacher has
a) effectively narrowed his or her curricula to the bullet list of “standards;
b) based his or her pedagogy on extrinsic punishment and reward;
c) robotically parroted his or her canned scripts;
d) modeled for his or her students proper obsequiousness to superiors; and
e) identically milled his or her differing students to specification, via test preparation, thereby inuring them to gritful performance of meaningless tasks and preparing them for the low-wage service jobs of the future.
See data-driven decision making (Rheeformish numerology) and technocratic Philistinism (Rheeformish creed).
more:
BILL GATES IS BAD AT MATH!
Here’s more or less how I will break down the ASA report on VAM to students, faculty, and the community when I introduce the film Standardized at Wayne State this evening, and in my doctoral seminar tomorrow evening. If I’m reading this incorrectly, peeps, please correct me:
The ASA is not simply saying that there is a strong correlation between poverty and test scores. While that’s certainly true, VAM supporters (including the Gates-funded Ed Trust) would counter by saying, “Yes, but a growth model, such as VAM, takes that into account, and looks only at growth starting from where students are when a teacher first gets the students.”
The ASA is saying, no— growth attributable to a teacher is not measurable by VAM or any statistical modeling, because the growth itself is mostly (86 to 99%) attributable to factors the teacher does not control. Only 1 to 14 percent of GROWTH is a result of the teacher. GROWTH itself is mostly determined by things like changes in the home- and community-based realities (health, jobs, nutrition, family relationships) of the student. Therefore, using VAM would unfairly reward teachers whose students saw their home- and community-based conditions improving, while it would unfairly punish teachers whose students’ home- and community-based conditions were worsening. Thus, VAM would create incentives for teachers to choose teaching environments where conditions were improving, and to turn away from teaching environments where conditions were worsening– because they would be more likely to be fired there in a VAM model. If we lived in a society that was intent on redistributing wealth from the top 50 percent of society to the bottom 50 percent of society, then teachers would be wise to leave affluent districts and teach in poorer districts. In that scenario, the VAM scores of the poorer districts would unfairly reward teachers who taught there. Unfortunately, as we well know, that is not the typical trend in American society. Instead, if the past twenty years are any evidence, we live in a society where those on the bottom are falling further and those who are on the top are gaining even more.
If Michigan as a state accepts VAM, then we are rewarding some teachers for gains they did not produce, and we are punishing others teachers for results that they did not produce. But in the end, what’s even worse is the effect on students. VAM creates one more incentive to not teach in environments where a community’s fortunes are fragile or declining. VAM hurts the neediest kids. Supporters of VAM, therefore, if they choose to ignore the ASA, are knowingly and culpably hurting Michigan’s neediest kids.
This is an “Obomaination”! So if you have a student who is very slow mentally, you, as a teacher, are tied to his responses on a test ? This really has no basis in equating the student’s score with a teacher’s ability to convey information. My job, as a thirty-year veteran of the teaching profession, is to instill a love of learning, despite the barriers a student may present. It is necessary to have scale and product evaluations, but to hog-tie these measures with the teacher’s ability to teach has much less connection than obvious environmental and historical factors influencing the test results.
VAM is almost as accurate as using this method to evaluate firefighters. http://protectportelos.org/new-nyc-firefighter-evaluations/
Let’s nit further that the Supreme Court of NY has stated that teacher observations contain “NO FACTS.” Elentuck v. Green 1994