Audrey Amrein-Beardsley, one of the nation’s leading researchers of VAM, says Néw York is going from bad to idiotic in doubling the importance of test scores in teacher and principal evaluations.
The chair of the Board of Regents, Merryl Tisch, and Governor Cuomo are “pushing for a system in which these scores would “trump all,” and in which a teacher rated as ineffective in the growth score portion would be rated ineffective overall. A teacher with two ineffective ratings would “not return to the classroom.”
This is not only “going from bad to worse,” says Amrein-Beardsley, but going from bad to idiotic.
All of this is happending despite the research studies that, by this point, should have literally buried such policy initiatives. Many of these research studies are highlighted on this blog here, here, here, and here, and are also summarized in two position statements on value-added models (VAMs) released by the American Statistical Association last April (to read about their top 10 concerns/warnings, click here) and the National Association of Secondary School Principals (NASSP) last month (to read about their top 6 concerns/warnings, click here).
Diane Ravitch's blog 
The one positive thing, if they go this route, is they get it out there in the open, something very real and concrete to fight.
Unfortunately, too many teachers and parents stay on the sidelines, hoping things will pass (as they usually have) until they actually feel the pain.
This is governor Cuomo, he is vindictive and hateful of teachers. Especially because we did not vote for him. I think there is going to be a lot of problems in the near future, perhaps even a state wide strike. I’m ready.
Funny how this has developed. Years ago the reform line was that principals should have more power to run their schools, including real power to hire and fire. Now the proposal is to strip principals of all that authority.
FLERP!: I know you realize that “actions speak louder than words” so I will just reinforce [from my POV] what you wrote with that old standby:
If someone wants to know what the leaders and enablers and enforcers of the self-styled “education reform” have in mind then best to—
“Follow the money.”
And I would add what is at times almost as important:
What fattens their egos [as well as their bank accounts].
So principals are demi-gods or useless ornaments on the EduXmas Tree, whatever is the self-serving selling point du jour.
Just my dos centavitos worth…
😎
So, with the ASA and others coming out against the use of VAM for these evaluations, I have to wonder — have they ever been challenged legally? Have they ever been challenged and lost?
It would seem that somewhere out there something must have been brought to court. Or are we all still waiting?
There is a case against VAM in NY that hopefully will be heard
In Florida, teachers sued when they were evaluated based on scores of students they never taught. The judge said it was unfair but not illegal. Incredible.
People like Cuomo and crew who know nothing about education and more importantly how teachers affect education should keep their noses out of the education tent. Like a good ostrich they should bury their heads in the sand and let those who know lead. Why is it that teaching as a profession has so little say about their profession? How would people feel if we let people like Cuomo tell doctors how to practice medicine? He probably knows less about education than he does about medicine. As a former teacher I fear for the fate of our education in NY and the detrimental effect it will have on the future of NY.
Everyone has been to school, everyone has had experiences with teachers who inspired them and teachers who did not, therefore everyone thinks they know what makes a good teacher. They don’t understand the hard work, dedication, and professional support that fosters good teaching.
On top of that, in the higher-income suburbs of Westchester County, NY, many parents think too many public school teachers are lazy, disinterested, and out of touch. They believe the teachers’ union is protecting them. I don’t agree with these people; as a newly retired teacher I have been trying to educate them…but as soon as I start talking their eyes glaze over and they change the subject. They’ve made up their minds.
I have heard that parents can starve the data beast and invalidate the test scores. Does anyone know the actual tipping point? The supposed requirement has been 95%. At what point does the lack of participation nullify scores and VAM and APPR?
NYT
“At what point does the lack of participation nullify scores and VAM and APPR?”
At the starting point.
The tests are COMPLETELY INVALID to begin with as proven by Noel Wilson. See: “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
By Duane E. Swacker
I understand your “starve the beast” idea but I think the percent of participation is tied to aid…that is the caveat…
No school district has lost any federal or state aid as a result of parent opt outs. Some districts in NY had over a 40% opt out rate. So far not a penny has been denied. The idea of losing money is just a fear based tactic with no substance behind it.
Here in Massachusetts the rate of participation affects the ranking of the level of the school. Schools must have a 95% participation rate to qualify as level 1 (the highest) and of 90% to be a level 2. (Level 4 qualifies a school for churnaround and 5 for state takeover.) It can be seen as a coercive measure to get test participation.
What are the implications of school ranking? Is it just prestige?
School rankings have been sold as helping parents make choices. I wonder if any data show that parents actually use these data and reports, whether it’s school report card K-12 rankings or US News college rankings.
Of course, the pending USDE regulations for accountability in teacher education just rollover all of the requirements for current accounting based on student test scores that are present in NCLB, soon to include the PARCC and SBAC tests for math and ELA. USDE has deliberately snubbed all scholarship showing the invalidity and unreliability of VAM, adding insult to injury to USDE’s vaudevillian posturing about “evidence-based” practices…must have those, always, all the time (except when formulating federal policies.)
And please remember that an estimated 70% of teachers who do not have VAM have an equally fraudulent measure–some variant of SLOs/ SLT’s/SGOs/ also called SMART goals. These schemes basically force each teacher to comply with a districtwide or a statewide formula for meeting “targets for achievement” and differentiated targets for student “growth measures” typically meaning gains in scores pretest to posttest and/or year-to -ear. These schemes are too rarely mentioned because VAM dominate discussions.
I reviewed four recent USDE publications on the SLO process and variants that are comparable in absurdity to VAM and now have a choke-hold on a majority of teachers. These USDE publications included a review of the literature (significant omissions), as well as the use of SLOs in various pay-for-performance schemes, beginning in Denver in 1999. There is no evidence of improved learning across the full spectrum of subjects for which this “alternative to VAM” is promoted. There is no evidence of the validity of the SLO process with recourse to VAM–a great case of reasoning by jumping from one invalid measure to argue another is valid.
The SLO process has been pushed forward relentlessly by the Boston-based consulting firm, CTAC—the Community Training and Assistance Center and by USDE publications farmed out to promote SLOs.
One facet of this marketing campaign began in late 2010 with a $43 million grant to IFC International, a for-profit consulting firm with sub-grants to skilled marketers of RTT and related policies, some of these PR firms headed by former employees of Gates who promoted the Common Core.
So, unlike VAM, the metrics for SLOs are less a product of a statistical construct than they are a version of 1954 (Drucker) management-by-objectives on steroids. In at least 21 states all teachers are faced with some version of SLOs as well as VAM, if there are statewide tests. SLOs were recently sold to key officials in the State of Maryland by the Community Training and Assistance Center.
The process of writing and scoring an SLO assumes that the content for a course or year should be fully mapped, as in programmed instruction for training. The process also assumes every course is/should be part of a vertically aligned system of prerequisites.
Thus, students who have not mastered pre-requisites, identified by baseline information and pretests, need to learn at a faster rate than other students in order to keep up with others. So, a teacher of any new course, or any exploratory course with no prerequisites, has no “baseline data” and is faulted for that.
My analysis of the marketing of SLOs as a proxy for VAM and “close reading” of the four USDE studies is not yet unpublished. It is available on request in exchange for comments on it. chapmanLH@aol.com
The way I deal with SLO and VAM-related stress is to sit down with my guitar and play. And write an occasional song parody…
Here is the link to my song parody page (mostly earth science related), but the first song is “S.L.O. Man”.
http://www.nscsd.org/junior_high.cfm?subpage=49429
This webpage is not accessible from my regular website. I use this page for educators and other interested persons who would like to hear and download my songs, but don’t want to wait until I post them in class. All songs are in mp3 format, and I’ve also provided pdf files of the lyrics. Please do not link to this page — I want to keep it “hidden”.
The song lyrics are copyrighted, and all I ask is that you do not distribute my songs or lyrics without my permission.
GregR
North Syracuse, NY
SLOs can be a smart teacher’s best friend, though. You cannot design an SLO system that cannot be “gamed”, and if you are a bit more clever than the people evaluating you, you can basically hand yourself a “4” grade.
The trick is to think like a corporate education reformer. Take care to always tell the (technical) truth, but manipulate the frame of reference and anything else you can in order to make yourself look fantastic.
It isn’t even hard. And most administrators are far too busy to pay the sort of attention to nuance you’d need to pay to even notice how you’re rigging the game (and if they DO notice, there’s not a lot they can do since you’re technically not doing anything wrong).
All of this assumed teachers are allowed to write their own SLOs, as they are here in RI.
ELA scores reflect a lifetime of language acquisition. The sum total of the conversations, books, writing, texting, travel experiences, TV and video game viewing, radio listening, and more. By the time an 8th grader sits down in early April to take the NY Pearson ELA assessment, they will have spent approximately 80,000 waking hours, engaging in language acquisition activities. If you are an 8th grade ELA teacher you will have had a grand total of 60 hours (that is the equivalent of about 7.5 work days) to influence and improve the reading and writing skills tested. T put this into perspective, if the waking lifetime (80,000 hours) of a 14 year old were compressed into just a 2 hour time frame, the ELA teacher has all of 5 seconds to exert their influence on 25+ students at once.
Just some food for thought for all of the edu-fakers that expect miracles from a profession that they have no understanding of. No clue as to the constraints and limitations of how little influence we can have, from a simple time perspective. And when the edu-fakers include the the underlying (but often false) assumption that our students attend school regularly, pay close attention to all we say, and give their best effort on all we ask, they have no idea to what degree our hand are tied.
Great SLO analysis. A perfect example of compliance (for non-tested subjects) and certainly not best practice.
We’re finally getting some dissenters in Ohio:
“Democrats on the state school board have banded together to push for better state funding of schools, better controls over charter schools and a reduction in the amount of standardized tests kids have to take.”
Ohio had a head start on NY because we’ve had all of “Cuomo’s” innovations for the last 17 years. NY will end up undoing them too. In 15 years or so 🙂
Cuomo could just travel to Ohio and see all his proposed policies in action, but I guess each state has to make the exact same mistakes. I think NY is still in the “reckless experimentation” stage of ed reform. Next comes denial, then regret, then the political battle to undo some of the damage.
http://www.cleveland.com/metro/index.ssf/2015/01/state_school_board_democrats_v.html
Unfortunately for NYS students, Cuomo thinks he’s smarter and that he’ll get it right and head to the White House. I would love to see him brought down.
Nevada is just getting the reform ball rolling with our tea party legislature arguing over which KKK nut job is presentable to the public to speak for them. They plan to finish reforming education this session. These things will end in Nevada in a generation or two I suspect.
I always wondered how maternity leaves would factor into this, when a teacher normally doesn’t get to choose who replaces her when she takes leave. If a teacher who was out on maternity for say, 12 weeks ( standard FMLA) was rated ineffective based on test scores alone, would she be able to sue on grounds of discrimination?
A great question. And what about student teachers?
Long term illness?
At this point, having to continually deal with drugged up kids, I will be glad to get out of the classroom. I wanted to teach, but I didn’t want to have to count noncompliant behaviors all day, chart them on a graph, and grovel on my knees before getting any help. I am considering myself as burned out.
The first paragraph from today’s NYT editorial.
They just don’t get it.
“New York’s schools are the most segregated in the nation, and the state needs remedies right away. That was the message delivered to the governor and the Legislature last week by the chancellor of the State Board of Regents. Minority children are disproportionately trapped in schools that lack the teaching talent, course offerings and resources needed to prepare them for college and success in the new economy.”
http://www.nytimes.com/2015/01/10/opinion/racial-isolation-in-public-schools.html?_r=0
What we need now is another Albert Shanker and a statewide if not nationwide STRIKE. if we don’t stand up for ourselves, who will? Educators are the authorities when it comes to school and we need to take charge by whatever means or surrender to tyranny.