The Sham of VAM: Evidence-Free Policy

By dianeravitch

July 3, 2015 //

This article about the lack of evidence for VAM is open access for one week only.

It was written by VAM critic Audrey Amrein-Beardsley and Jessica Holloway-Libell.

Categories VAM (value-added modeling)

24 Comments Post your own or leave a trackback: Trackback URL

tultican says:

July 3, 2015 at 6:26 pm

Thank you. I have it.

LikeLike

Reply
liberalteacher says:

July 3, 2015 at 6:31 pm

I printed it out for my records. An excellent article that describes all of its flaws. It amazes me how so many can be bamboozled by our modern snake oil salesmen.

LikeLike

Reply
- Debbie Rakestraw says:
  
  July 3, 2015 at 10:37 pm
  
  Amen. Have you heard the PSAs on the radio with a woman talking to her son about how Common Core means everyone will learn to carefully no matter what book you’re reading? Talk about trying to brainwash the public…..my question is if CC is so great, why on earth is it necessary to run ads to convince people of it’s value? Can you remember another time educational policy required advertising to justify its worth?
  
  LikeLike
  
  Reply
Laura H.Chapman says:

July 3, 2015 at 6:43 pm

This is a brilliant article.

I hope the authors can rework their analyses of the philosophical and statistical assumptions in VAM for a much larger audience and expose the really meager peer-reviewed research that even deals with the subject.

This study has a damning pie chart for those who engage in praise for “evidence-based” policy and practice. It illustrates the extent to which VAM has been promoted by spin, into what I would call a doctrine of false objectivity. That doctrine has been sold to policy makers by economists.

This article has nailed the philosophical assumptions that are made and promoted about the whole enterprise of “accountability in education” and the chosen gurus of our time–economists and statisticians.

Congrats to the authors. This article is a major achievement. It harks back to the olden days when analytical philosophy had a place in education.

If you don’t have access to this article, Get yourself into this. Don’t be stymied by the obligatory forest of references. It is a fitting addition the educator’s quest for an Independence Day from VAM.

LikeLike

Reply
- Ed Detective says:
  
  July 3, 2015 at 6:52 pm
  
  In truth, this article is a piece of evidence that should inform policy and practice. In real evidence-based policy and practice, VAM would be regarded as a sham.
  
  LikeLike
  
  Reply
- Debbie Rakestraw says:
  
  July 3, 2015 at 10:38 pm
  
  Excellent comments- agree whole-heartedly!
  
  LikeLike
  
  Reply
retired teacher says:

July 3, 2015 at 7:31 pm

A copy of the VAM analysis should be sent to Arne Duncan and every governor. Even the “cliff notes” summary in the boxes would be enough to potentially enlighten those in charge. The analysis shows what happens when false assumptions and conclusions are applied to education for political and monetary gain. In other words, the acceptance of VAM as fact highlights the corruption of our representatives in the government and the woeful inadequacy of those leading teachers’ unions.

LikeLike

Reply
- Debbie Rakestraw says:
  
  July 3, 2015 at 10:32 pm
  
  True!
  
  LikeLike
  
  Reply
KrazyTA says:

July 3, 2015 at 9:08 pm

Corporate education reform uses snappy slogans that suggest that the eduproduct being pushed is very different from what it really is.

For example, any objection to the use of the scores generated by standardized tests for faux accountability purposes is typically met with something along the lines of the following sneering and crushing reply: “So you don’t want to know if the student has properly learned, if the teacher has properly taught, if the school measures up?!?!?! Another defender of the failed educational status quo that defends cheating students of an education and covers up the incompetence and bigotry of teachers and administrators.” There is more but it comes down to blustering blather in defense of selling points and its self-aggrandizing salespersons.

Very simply, contrary to what one can find out by reading experts in the field—including the bean counters that have produced value-added systems—VAM is peddled to the general public as if it is measuring something independent, separate, complete in itself, an entity that has definite shapes and contours and dimensions. Hence, failure to produce the numbers that define how big or small that entity is, i.e., “learning” and “teaching,” is simply evasion and obfuscation and cheating and fraud.

Wrong. Wrong. Wrong. It’s all relative. Literally. So by its very nature VAM, in practice, gives wildly unpredictable—and predictably wild at the same time!—ratings. The toxic inconsistencies of VAM are, in the strictest sense of the word, completely to be expected because in the real world all the factors that affect learning and teaching are constantly changing.

Yet the appearance of certainty, bolstered by mathematical intimidation and obfuscation, knows no bounds when $tudent $ucce$$ is part of the calculation. That is the metric that matters. The bottom line. Black is good. Red is bad. Leave the rest to the “experts.”

Here’s my bottom line: if VAM is so certain and useful and helpful and accurate, let the zealots of this bludgeon of measure-and-punish like Chetty and Hanushek and Sanders et al. come to this blog, with a sternly worded challenge to the owner of this blog—

“Next year at the annual NPE meeting we want an entire panel devoted to the mysteries, wonders and delights of our creation and what impact and effect it has on the real world. Bring on Audrey Amrein-Beardsley and Bruce Baker and Jersey Jazzman and deutsch29 and GF Brandenburg and Gary Rubinstein and the rest of your nay-saying midgets of math and data analysis. We say, in the words of Mr. Bill, there is a 98% chance of certainty that we will crush your puny objections with the mere flick of a Gates Foundation grant. To paraphrase that mental giant of our merry band of VAMsters, the NJ Comm. of Education, we double down on whatevers in daring you to take up our challenge!”

😏

They don’t and won’t issue such challenges because they understand, as only the VAM creators do, that they don’t have a mathematical leg to stand on. Their shameful and very public cowardice in the face of reasoned criticism is the surest sign that they know they can’t defend the sad metrics that measure the contents of their wallets and the number of fawning lines about them in the MSM and the number of their seats at the tables of the BBBC [BoredBillionaireBoysClub].

To riff off of Gertrude Stein’s famous line:

“There is no there there.”

And there never was.

And we didn’t have to wait ten years [thank you, Mr. Bill Gates!] to figure that out.

😎

LikeLike

Reply
Debbie Rakestraw says:

July 3, 2015 at 10:29 pm

Interesting reading….but the people who NEED to understand the issues with VAMs will probably never see it. Just can’t wait to be evaluated on my students’ growth next year using PARCC and MAP data, and the Danielson framework for teaching. It’s gonna be a great year…..

LikeLike

Reply
- retired teacher says:
  
  July 4, 2015 at 8:43 am
  
  VAM is the mental construct of an economist whose false assumptions have been applied to teaching for the purpose of creating chaos. It serves no purpose as it cannot improve instruction because it is a “house of cards” of invalid metrics. Those harmed by VAM should file class action suits against this voodoo science, or perhaps the teachers’ unions could wake up and take action to support their members. Why should this junk science be accepted as gospel allowing school districts to fire teachers based on a bogus score? Teachers should not remain silent when the ax man comes allowing judges to skirt the issue of invalidity simply because it is the “law.” Remember slavery was once the law too.
  
  LikeLike
  
  Reply
K. Friedrich says:

July 4, 2015 at 7:34 am

Every school should evaluate their own tests http://mathisconceptual.blogspot.com/2015/07/reflective-test-validation.html
http://mathisconceptual.blogspot.com/2014/11/guide-for-evaluating-data-tools.html

LikeLike

Reply
Sad Teacher says:

July 4, 2015 at 9:08 am

Even sadder is that only a certain group of teachers (Grades 3-8) of selected subjects are targeted for VAM. If you are a teacher in this group like I am, you are mighty unlucky. My VAM for next year will be based on the unfair PARCC, so my life as a teacher is not worth two nickels. I am already fully expecting to be rated “ineffective” even with a good evaluation from my principal. Their evaluation can not overcome that crazy VAM. There is starting to be a lot of turnover in third grade teachers, because they cannot stand the stress of the “Third Grade Guarantee.” So, what happens when the VAM teachers are fired, when all teachers are not under the vise grips of VAM? I am so relieved to be close to retirement. My dedication, my love for my students, and my long hours are not enough anymore.

LikeLike

Reply
SomeDAM Poet says:

July 4, 2015 at 9:34 am

Einstein once said that one should make things as simple as possible, but no simpler.

For economists, violating his second rule is like breathing.

It’s no real surprise that they believe that they can simply “tweak” a construct designed to model cattle growth so that it applies to students and teachers.

Because, you know, students are just like cattle and teachers just like cattle fodder and better fodder means more student growth.

Isn’t it obvious? (I don’t know about anyone else, but I know my own fodder helped me grow — my mudder too)

And controlling for all the other factors (in addition to teachers) that impact student learning is just as easy as controlling for all the factors (in addition to fodder) that influence cattle growth .

…because, you know, students are grown in stalls just like cattle with feed troughs where they get measured amounts of feed (antibiotics, etc) with no outside influence. You did know that, didn’t you?

“If I have seen fodder, it is by standing on the horns of cattle” — William Sanders, who applied VAM for cattle to teachers and just won a prize for it

LikeLike

Reply
- NY Teacher says:
  
  July 4, 2015 at 10:00 am
  
  “I was like a boy playing on the sea-shore and diverting myself now and then, finding a a rotting fish or toxic oil spill, whilst the whole ocean of mismeasurement lay all unused before me.” – William Sanders
  
  LikeLike
  
  Reply
  - SomeDAM Poet says:
    
    July 4, 2015 at 10:24 am
    
    Ha ha ha ha ha!
    
    TAGO!
    
    The possibilities are endless.
    
    “Do not worry about your difficulties with Reality. I can assure you mine are still greater.” — William Sanders
    
    LikeLike
  - Ed Detective says:
    
    July 4, 2015 at 2:54 pm
    
    “Everything that counts can be counted. Everything that is counted counts.”
    
    -William ‘VAM’ Sanders and The Economists
    
    LikeLike
- Duane Swacker says:
  
  July 4, 2015 at 12:52 pm
  
  Speaking of cattle: VAM is bullshit!
  
  LikeLike
  
  Reply
  - SomeDAM Poet says:
    
    July 4, 2015 at 8:21 pm
    
    “The Cowbull Prize”
    
    William Sanders did surmise
    (for which he got the “Cowbull Prize”)
    That students are like cows and sheep
    A finding that’s both strong and deep
    
    LikeLiked by 1 person
  - Duane Swacker says:
    
    July 6, 2015 at 10:34 am
    
    The worse type, by far, of that “strong and deep” is of porcine origin. It gets sprayed on fields around here defiling the air giving new definition to “country freshness”!
    
    LikeLike
- Duane Swacker says:
  
  July 4, 2015 at 12:57 pm
  
  “(I don’t know about anyone else, but I know my own fodder helped me grow — my mudder too)”
  
  LikeLike
  
  Reply
drext727 says:

July 4, 2015 at 10:20 am

Reblogged this on David R. Taylor-Thoughts on Texas Education.

LikeLike

Reply
Ken Watanabe says:

July 4, 2015 at 12:38 pm

The only evidence innovators and promoters of VAMpire want is pool of data sample for BLOOD.

LikeLike

Reply
Duane Swacker says:

July 4, 2015 at 12:51 pm

From the article: “Value-a-dded models (VAMs) are statistical instruments intended to objectively measure the amount of “value” that a teacher “adds” to (or detracts from) student learning and achievement from one school year to the next.”

VAM proponents may state that “VAMS are statistical instruments [in other words, bullshit] intended to objectively measure . . . ” Intentions do not reality make. THERE IS NO “OBJECTIVE MEASURE” of the teaching and learning process as there is no agreed upon standard of measurement and no measuring device. No amount of statistical machinations and fudging by psychometricians and/or economists can ever make the teaching and learning process “measurable” and/or epistemologically and ontologically sound/true. IT CAN’T BE DONE RATIONO-LOGICALLY. Falsehoods and errors abound in the two educational malpractices that are educational standards and standardized testing making the whole regime COMPLETELY INVALID.

To understand why, read and understand Noel Wilson’s utter destruction of those two malpractices in his never refuted nor rebutted treatise “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700

Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine.

1. A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.

2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).

3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.

4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”

In other word all the logical errors involved in the process render any conclusions invalid.

5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.

6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.

7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”

In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?

My answer is NO!!!!!

One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self-evident consequences. And so the circle is complete.”

In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.

LikeLike

Reply