Last week, Kevin Huffman and John Ayers resigned. Huffman was state commissioner of education in Tennessee, and he employed every possible strategy to make testing a centerpiece of education policy. Ayers was director of the Cowen Institute at Tulane University in New Orleans, which was greatly embarrassed when it released–and then rescinded–a “research” report claiming amazing gains in the charter schools of New Orleans. Both were big boosters of using student test scores to judge the quality and effectiveness of teachers, a methodology referred to as VAM, or value-added-modeling.
Audrey Amrein-Beardsley, one of the nation’s expert researchers on teacher evaluation, looks at the two resignations as evidence that the VAM-mania is failing and claiming victims. There is as yet no evidence that VAM improves teaching, improves student achievement, or correctly identifies the strengths and weaknesses of teachers. As its critics have said consistently, VAM results depend on many factors outside the control of the teacher and may vary for many different reasons. A teacher may get a high VAM rating one year, and a low VAM rating the next year. VAM ratings may change if a different test is used. Yet those who stubbornly believe that everything that matters can be measured with precision can’t let go of their data-driven mindset.
The lesson: proceed with caution with a methodology that has no record of success and that inevitably places far too much importance on standardized tests.
Diane Ravitch's blog 
I’m not the optimist Audrey is. I think Huffman and Ayers (and Arne would make 3) will reemerge heading up some or another reformy organization. Like in the horror movies where you think the heroine has escaped from the slasher, only to have him reappear in her daughter’s bedroom.
All the reforms are full steam ahead in my district Christine.
Christine, Love it! Good belly laugh!
I’m glad…best medicine and all..
Sent from my iPhone
>
These VAM cheerleaders resigned because they failed making the grade by whatever VAM measured their performance. They will be replaced and when the replacement fails, then that replacement will resign and be replaced by the next guy who promises to make it work.
VAM cheerleaders dreaming of $$$$$$$$$$ will wait in line for their chance and Bill Gates and the other VAM driven corporate minded oligarchs will keep hiring as the door revolves until they get what they want—anything that makes VAM look like it works even if based on false and/or cherry picked data that will silence the resistance.
VAM – another measure created by the reformers to end the teaching profession, right? VAM won’t work on TFA or TNTP temps because they don’t stick around long enough for the results. These reformers, they literally have no consciences, no soul, no heart. Perhaps they are empty meat bags with dollar bill stuffing?
When VAM disappears, something similar will take its place. They are already creating their spin. I don’t know how they come up with this stuff, all to further their agenda of privatization/profitization, while vilifying teachers, and the spin they put on everything is downright masterful. Lucifer himself couldn’t do better.
I hope reform measures continue to implode, and I wish “the enforcers” would go down with the ship. Sadly, I agree that those who step down will go onto greener pa$ture$ and work for a think tank or a charter or lobby company, or….whatever. I have to say tho, and this will be the only compliment they’ll ever get from me…..a) they have strong constitutions to endure all the criticisms (tho they have no consciences) and b) they all seem to be made of teflon. That alone tells you the character of Gates, Broad, Bloomberg, Koch, Walton – that they would spend all their billions on these soulless turds who daily are criticized for their evildoings, even tho the 1%ers adore them.
VAM is a scam. The opinion of the American Statistical Association has not changed the policies in state departments of education or USDE.
And it has not stopped educational researchers and economists from pumping up studies of VAM as if these are credible.
A case in point is the Aug/Sept issue of Educational Researcher, an article reworking data from the Measures of Effective Teaching Project (MET) funded by more than $64 million by the Gates Foundation. Four researchers and a Ph. D. candidate are still massaging data, trying to figure out why VAM calculations based on math scores for grades 4 to 8 (six districts, different states) are not highly correlated with scores on a teacher observation protocol for math called PLATO.
Of course the districts and teachers got perks for participating (no high stakes), and the “observation “ ratings were not really in classrooms. The observation ratings were based on 15 minute videos submitted by the teachers, scored by math teachers trained as “certified raters” who earned that status by looking at some “master-scored videos” and getting their ratings into a 70% exact agreement with the master scoring. These ratings were “good enough” for the study because it was “just a study”—teachers would not lose their jobs. The proxy of video snippets in one subject is “good enough” to produce another peer-reviewed study aiding and abetting the propaganda mill for VAM.
The preoccupation with pseudo-scientific measures and quests for precision in rating teachers is out of hand. These researchers want high correlations between VAM and observations of teachers. They want mutually reinforcing measures that point to one-and-only-one view of “quality teaching.”
Chasing one-size fits all measures of teacher quality should be ridiculed. Gates-funded researchers are engaged in perpetuating this nonsense. They need to be called out as navel-gazers. The tables, graphs, and footnotes in this study show how absurd these efforts are–how totally disconnected. The work of real teachers cannot be understood by constructing scree plots and doing Horn’s parallel analyses, and thinking about eigenvalues.
Teaching is not a mechanical engineering problem. It is not OK to use VAM to rate teachers. It is not OK to convert VAM into units called “months-of-learning units” initially defined as “months-of -schooling gained.”
This is nonsense parading as if a meaningful discussion of teaching and learning.
Navel-gazing researchers are indulging themselves in nit-picking—giving excessive attention to data gathered from a botched Gates-funded “Measures of Effective Teachers” project at the expense of a wider and grounded view of the work of teachers. This is tragic and also inexcusable. The equally absurd are SLOs and variants, recommended for rating about 70% of teachers. SLOs have been marketed since 1999 by some Boston-based travelling salesman and more recently by a salesforce hired by USDE.
Very well put.
You are exactly right.
Rather than accept what ASA and others have told them (that VAMs are very “noisy” and hence unreliable and don’t even account for much of the variation in student test scores, which, in turn may have little to do with later “success” in life), the VAMbots search for something — anything — that will “validate” their invalid VAM approach.
They just “know” (have a preconceived notion) that VAMs are a valid, reliable way of evaluating teachers and weeding the “bad” from the “good” and are willfully ignoring/disregarding contra-indications.
Their mind was made up at the very start and no amount of evidence showing the emptiness of their position is going to change that.
What these people are doing is not science. It’s quackery.
The university “researchers” who support the use of VAMs for firing teachers are particularly deserving of harsh criticism (even ridicule), especially by their university colleagues.
Personally, I would welcome a groundswell of parental outrage and protest nation-wide as they begin to understand their children’s impending future. May they soon realize the damage ahead facing their families.
Let me guess. This could be an interlude to “Nightmare on the VAMpire Street 3: Revenge of Duncansteiner”
The Mismeasure of Students and Educators
The latest in the propaganda war against teachers and students is the notion of Value Added Assessment or Analysis. This purportedly accurate measurement uses a statistical technique based on a minimum of three data points in the past to predict the likelihood of a student achieving a certain level of proficiency on a subject area standardized test. This seemingly impressive quantitative data conveys an aura of scientific legitimacy for what really is a fallacious reductionist view of knowledge as a discrete number or percentage. It calls to mind the scathing and definitive rebuke of IQ testing and the reification of intelligence in Stephen Jay Gould’s brilliant book The Mismeasure of Man. Lest one think that the late Dr. Gould didn’t have the background to appreciate quantitative data, Stephen Jay Gould was the evolutionary biologist who developed the Punctuated Equilibria Theory which has significantly contributed to our understanding of the evolution of life on earth. Unlike those reductionists who quantify learning into a percentage, Dr. Gould was able to get the big picture and come up with profound ideas.
In defending the Value Added approach, it is argued that it doesn’t matter what the student’s economic background, because each student is in effect his/her own control since the data being compared are his/her own data from year to year. Anyone working with children knows that test grades can vary from week to week and certainly from year to year. Test grades for the same student may also vary greatly from subject to subject. If one looks at test data over a number of school years, there are often major differences. Children change. Child development and adolescent behavior enter into the equation, and the variables are many and are not controlled.
Parents will be quite surprised and likely dismayed to learn that during the fall of the 2014 – 2015 school year the State of Pennsylvania’s Department of Education will use data to predict the supposed likelihood to a tenth of a percent of a student scoring Below Basic, Basic, Proficient, or Advanced for each student who will be taking a Keystone Exam 9 months later in May. That is before the child has ever taken a biology class, the state will say that little Susie or Johnny has a 14.7% chance of scoring proficient on the Biology Keystone. They will derive this percentage from a minimum of three data points from previous years’ standardized test results. Never mind that the data points are from unrelated subjects. Who cares if we mix apples and oranges as long as we have a number! If Johnny’s or Susie’s teacher decreases the 14.7% chance to pass to 14.2%, bad teacher. The student data for that teacher will be included in that instructor’s evaluation and will have a negative impact if the collective “value subtracted” is greater than the “value added.” If the teacher has increased Johnny’s or Susie’s chance to pass to 16.1%, then that teacher is a Value Added educator. Never mind the fact that the state is giving little Johnny or Susie a very slim chance of receiving a passing grade before he/she has ever taken the course. Sounds like the casino approach to education. It is strongly suggested that these quantitative experts go back and take a high school science class. Perhaps they need to learn the basics of scientific method. You can’t have reliable data if you are measuring different types of things for the same experiment.
It also appears that the number crunchers are actually divorced from the testing instruments. In Pennsylvania two different companies are responsible and what is actually asked seems to be of no consequence to those doing the computations.
The tests are written by a relatively small player in the testing business, Minnesota based Data Recognition Corporation with $118 million in revenue. Many states are now part of PARCC (Partnership for Assessment of Readiness for College and Careers,) a coalition of states that want to use the same exams written by Pearson which is also a major player in the American high school textbook market. 2013 revenues for Pearson were $7.925 billion. It is likely that state standards, textbook content and test questions will be similarly aligned in part due to the dominance of this company, for better or worse.
The Pennsylvania Department of Education refuses to release old exam questions or even old Classroom Diagnostic test (a practice test) questions. When asked about the paucity of materials for study and curriculum purposes, the response has been that it costs thousands of dollars to write a question and that launching it out there is like launching your baby into the world. You can’t just give it away. It is probably safe to say that many of these questions are likely to be reused on future tests. The only questions that have been released are a limited number of sample questions, some of which are poorly written and ambiguous. When asked about the sample questions, it was stated that those were the rejects that were not used on the tests. Yet those samples are the only questions published by the Pennsylvania Department of Education for teachers and students to prepare for these high stakes assessments. This stands in stark contrast to the longstanding New York Regents Exams which are not written by a multimillion or billion dollar corporation but rather by teachers working with the New York State Department of Education under the authority of the Board of Regents of the University of the State of New York. These subject area exams are given three times a year. No sooner than they are administered, the exam questions are made available online, along with an answer key and scoring guide for constructed response essay questions. No one is kept in the dark about the types of questions or content one can expect because of the extensive material made readily accessible to all concerned.
The North Carolina based corporation responsible for the number crunching for Pennsylvania’s Value Added Assessments is SAS, known for business analytics software and services. Its 2013 revenue in the US was $3.02 billion. The last five years has seen a growth in revenue of about a billion annually. The revenue growth has been buoyed by the increase in school districts jumping on board with Value Added Assessment. Enriching the private sector through lucrative contracts while cutting state funding to public school districts seems to be a trend over the past decade. One area of Pennsylvania educational funding that has been on the rise is for standardized testing. The 2013 – 2014 budget had approved $53,691,000 for assessments. The 2014 – 2015 budget has $58,291,000 allotted for testing, an additional $4,600,000 or an 8.57% increase at the same time that there has been no increase in state funding for basic education.
As indicated previously, while SAS determines if teachers and students are “value added” or not, it has nothing to do with assessing the validity of the test instrument itself which is written by employees of the Data Recognition Corporation.
Since it is not difficult to poke holes in the methodology employed by the number crunchers not to mention the lack of transparency in the construction of the exam, its reliability, alignment with curricula, etc., one must then ask what is the real motivation for all this testing and supposed quantification of the growth of student learning and value of teachers?
There is an all-out war against public education being waged in the United States and it is happening quickly. It has been a multipronged assault which includes the funding of charter schools, the defunding of public schools, attacks on teachers, their unions, seniority, salary scales, and pensions. The overuse and abuse of standardized testing has been a powerful weapon to help accomplish this. Poor scores on these tests are deliberated and unfairly correlated to mean poor teachers and bad schools. Hence get rid of unionized teachers and public schools. Replace them with schools run by corporations and send teacher salary scales back to a prior century when teachers were known to have the worst salary scales for professionals with that level of education.
It is already a fait accompli in poor New Orleans’ Recovery School District, where the remaining traditional public schools closed permanently in June. 510 of 600 employees lost their jobs. 33,000 students had to apply for a seat by lottery to get into one of the 58 charter schools.
When a major American city gets rid of all its public schools alarms should sound. It threatens to undermine the essence of our democracy since without public education, as in centuries past, only the children of the privileged and powerful will have access to a quality education.
With regard to real educational growth rather than a fixation on standardized test scores, it is instructive to heed the advice of a historic mover and shaker in American education, John Dewey, “Were all instructors” (to that one should add educational policy makers) “to realize that the quality of mental process, not the production of correct answers, is the measure of educative growth something hardly less than a revolution in teaching” (education) “would be worked.” (Democracy and Education)
Robin Williams
Biology teacher
The oligarchs don’t need to educate the rest of us—-only their children, the ruling class, the royalty of the future, who will inherit foundations from their parents and grandparents to rule over the 99% for millennia.
Why don’t they need to educate the other 99%. Because they have computers that are getting smarter all the time that won’t question authority—-computers and robots that will replace the 99% in the workplace.
99% of humanity is quickly becoming irrelevant. Even future armies are going to be drones and robots.
Very powerful. I took a lot away, but the question I have is…did every kid “win” the lottery? “33,000 students had to apply for a seat by lottery to get into one of the 58 charter schools” – and the politicians don’t even care.
We can hope that parents, and even students, across this nation continue to wake up and say no. If we starve the testing companies by saying no, their stocks go down and fickle investors will jump ship.
When we are successful in voting out the posers, we starve their backers as well.
They believe in God in a twisted fashion. They believe they will be God and decide what to do with the rest of us. Jefferson warned of corporations, they were not people then and they were limited. They also kept legacy oligarchies from forming through inheritance taxes. We need to do the same, but alas, it may be too late.
The way the 1% currently thinks reveals that too many of them consider themselves gods with the power of a god, but they are not God—-because their power comes from $$$$$$$$$$$$$
Take away the $$$$$$$$$$$$$$$, and they are no different than everyone else.
The real God, if there is One, has no money. He doesn’t even have a bank account, a credit card or a credit rating.
If God applied for a credit card, He’d probably be denied.
He only created the universe with the big bang, then He shared some rules to live by that appear in the Old Testament—something He shared with Moses several thousand years ago—and He gave us free will and later, much later, judgement by only Him, and He really hates false gods who speak in His name.
Let Bill Gates, the Koch brothers, the Walton family and the oligarchs, who think they are gods, match that.