A new report prepared by Andy Porter, dean of the graduate school of education at University of Pennsylvania, and Morgan Polikoff of the University of Southern California caution about value-added-measurement, basing teacher evaluation on test scores, because this method has “a weak to nonexistent link with teacher performance.”
Why are at least 30 states using this flawed measure? Because Arne Duncan made it a requirement of eligibility for Race to the Top and for state waivers. Despite the lack of evidence or negative evidence, states have passed laws tying as much as 50% of a teachers’ evaluation on scores.
“Morgan Polikoff and Andrew Porter, two education experts, analyzed the relationships between “value-added model” (VAM) measures of teacher performance and the content or quality of teachers’ instruction by evaluating data from 327 fourth and eighth grade math and English teachers in six school districts. The weak relationships made them question whether the data would be useful in evaluating teachers or improving classroom instruction, the report says.”
The article quoted the recent American Statistical Association report. How many more artifices and reports will it take before D.C. attention:
“In April, the American Statistical Association issued a statement criticizing the use of value-added model, saying teachers account for between 1 and 14 percent of the variability in student test scores.
“Ranking teachers by their VAM scores can have unintended consequences that reduce quality,” the statement said. “This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.”
Diane Ravitch's blog 
Using VAMs to evaluate teachers is like reading chicken entrails to predict the weather, like playing Russian Roulette with teachers’ lives and careers and our kids education. — Edd Doerr (arlinc.org)
Links?
From the New Yorker piece on Newark ed reform:
“President Obama and Secretary of Education Arne Duncan created Race to the Top, a $4.3-billion initiative to induce states to approve more charter schools and to rate teachers based on student performance.”
Wow. Was that the entire goal?
“We know what works,” Booker and other reformers often said.
Well, maybe not, as it turns out.
http://www.newyorker.com/reporting/2014/05/19/140519fa_fact_russakoff?currentPage=all
This was also interesting, at least to someone who hasn’t followed this story closely:
Now- Senator Booker:
“I define public education not as a publicly guaranteed space and a publicly run, publicly funded building where our children are sent based on their Zip Code,” he said. “Public education is the use of public dollars to educate our children at the schools that are best equipped to do so—public schools, magnet schools, charter schools, Baptist schools, Jewish schools.”
I think it’s great he admits it. Now if we could just get the rest of them to admit it, we could have an actual debate on privatization instead of claiming this is about “improving public schools”
It’s about replacing public schools with an elaborate voucher system.
I figure they keep 10% “public” to act as safety net schools, like Medicaid and the private insurance exchanges in the health care law.
“Baptist schools, Jewish schools…”
You know, I truly wonder about this.
How do people (normal everyday tax payers) really feel about tax money going to religious schools?
Especially when it is pointed out that there are religions with which you might not agree. Those schools will still be eligible for your tax money.(I suppose most people are fine with the idea as long as they perceive religious school to equal my religion for my kid.)
I cannot wrap my head around it.
Here is a letter I wrote about the FL VAM.
——————————————-
The Golden Rule of Accountabilty:
VAM unto others as you would have others VAM unto you
To: School, District and State Educational Administrators
From: Your Passionate, Professional and Committed Teachers
Re: VAM (Value Added Models, or Very Arbitrary Measures)
Dear Administrators and Pedagogical Theoreticians:
Since you believe teachers should be held accountable for results and have a proportion of their paycheck tied to test results, then you must also believe that you too should have part of your paycheck determined by test results. If you disagree with this then tell us why; give us a rationale for your hypocrisy.
Teachers complain that VAM statistics ignore some basic variables (ex. socioeconomic levels) and that they will be blamed for poor performance (aka: low test scores) for factors that are beyond the control and scope of their classrooms and influence. They will even point out all the variables that lie within the affective domain (desire, perseverance, work-ethic, love to learn, will to self-actualize [self-efficacy]) are primarily developed within the domain of the family (after all, family support-input is the primary predictor of success in school and post-secondary pursuits). Yet, under VAM teachers will get the blame for low performance, even though the equations are supposed to consider primary factors (but ignore socioeconomics and family). So even when we do our best some kids will not learn in spite of our efforts, though our paychecks will suffer.
So, then you too dear administrators must get blamed for events beyond your control, because if “it takes a village to raise a child”, then all in the village should suffer the negative reinforcement (lower paycheck) when the child fails.
We, teachers, may decide to blame you for your possibly deficient leadership or ineffective mentoring, or buying junky curriculum, or…? If, in the metaphor of the learning-village, part of our effectiveness as teachers is tied to your effectiveness as leaders and guides, then you too should be held accountable!
Are VAM (Value-Added Model, or Very Arbitrary Measure) truly unbiased and take into account as many covariates, or confounding variables (and factor these out), so that the “signal to noise ratio” is significant. One analysis by the American Statistical Association showed that only 1-14% of VAM data variability was affected by teacher input, the other 86% to non-teacher factors. So, the signal-to-noise ratio of most districts’ VAMs are 86% noise, and at best 14% signal (effects “created” by teacher input).
Yet, if this is true, then should not next year’s merit pay schedule only show, at most, a 14% salary gain for high VAM values? Why should salaries change more than 14% if teacher influence on test scores is less than or equal to that value?
By the way, I have asked UTD and MDCPS (my principal) when and where the 2 salary schedules (PSC, CC or Merit Pay?) for next year, when all of SB 736 is to be implemented, yet nobody knows where to find them. Is MDCPS working hard and fast to get these published, so that teachers can see their choices for next year? I though by August 2014 the full Merit Pay choice is supposed to be implemented, but I hear nothing from the district about this.
If VAMs don’t consider socio-economic levels, or past or current family life, then they are biased and will ignore the work of those who teach lower-level students, or may be handicapped against those who teach upper-level students. For example, is the value of helping out a low income, inner-city, child go from a 1 to a 2 in the FCAT, the same as that teaching in a rich suburb and having a score go from a 3 to a 4? Intuition would say no; that it might take more work (and therefore more value?) to help out the student who grew up in a less-fortunate environments.
Will VAM statistics factor this into account? If so, how, and if not then they are seriously flawed. Should VAMs be handicapped, so that gains made in the lower 1/3 of the student population are weighted more than the upper 1/3, because it takes more teacher-effort to raise the lower 1/3? If little or no gains are made by the upper 1/3 is it because they have poor teachers, or that it is more difficult to create gains with groups already in the 80-90 percentiles? I don’t believe VAM equations look at these variables, but then the real mystery is that most of the equations districts use are hidden and cryptic, only found in research journals that analyze and comment upon them, but not out in the open for all teachers to see.
Several times I’ve asked the UTD and my own principal to show me where at the MDCPS website is our VAM posted, but have never received an accurate answer. One would expect that at the Teacher Portal there would be a big bright banner stating “MDCPS VAM Equation” because I thought stakeholders are valued to the degree they are informed about decisions that affect them. Yet, when it comes to VAM I guess the district’s policy is the inverse of this logic; that teachers should be ignorant and left in the dark about the VAM, and this will make them feel “added value”? Are all Board members knowledgeable of the VAM, so that they can have meaningful discourses about it with their constituents?
If we “VAM unto others as we would have others VAM unto us”, then what kind of rubrics do we use with administration, district and state education leaders, and the legislators that make the mandates?
District administrators may feel this is unfair. Some might even admit to “uneasy feeling” of seeing their teachers get punished by the VAM (next year when all the finalities of SB 736 are implemented, teachers could see smaller pay for lower VAM), yet with no consequence to themselves. Though, if students do poorly because of poor teachers (though research says that that is only 14% true), then poor teachers reflect poor administration? Though administrators are never “vammed”?
Administrators, local and district, may complain that variables beyond your control should not be the reason for you lower paycheck under a Administrator-VAM statistic, but sorry no double standard. District leaders may grumble that FL DOE leaders, or FL legislators, made choices that negatively affect their performance at the district level. Well, so, should you not be held accountable anyway? Should there not be a VAM to unfairly punish you, as the one being used for teachers? FL DOE leaders may blame the federal government for poor funding, bad curriculum mandates, unsound/invalid pedagogic assessment models, and whine that getting smaller paychecks is unfair.
After all, should we not all just blame the President? No, of course not, individuals should and must be held accountable for producing excellent products; the “buck must stop somewhere”, agreed. Yet, why does it stop at the teachers? Why are we the only ones who will have a part of our paycheck tied to a VAM statistic? Why not leaders too? Why not parents too? Hey, why not penalize the future salaries of students who deliberately choose not to learn with a student-VAM?
But no, we, the teachers, will take all the blame for failing students (not even the parents get blamed) without any accountability (tied to salary) for our leaders? We are flattered that you leaders believe we have that kind of power in the classroom; that we can lead the horse to water and make it drink; that we can plan and cook the meal and make someone eat it too.
How we wish, as teachers, the assimilation of knowledge was so easy; that all our input equated to student output, but we all know this is false.
Does not Maslow’s hierarchy of needs teach us that no student will self-actualize and desire to learn for the delayed future reward of a good job, or the “love of learning”, whose underlying primary needs of love, support and home-life are not robustly provided. Students today, in general, come into the classroom with so much “baggage” (ex. uncertainty of family support after 2 divorces, abusive authorities, excessive premarital sex, an Internet that exposes them to less-than-desirable behaviors, having some of the adults they used to look up to disappoint them, being bombarded with inane, vain and useless internet entertainment, etc.).
So, teachers are expected to produce a superior product in spite of the defects in the raw materials that enter the classroom? As a chemistry teacher I believe I can help the student (ore) refine itself, and will do so with all my passion, but I cannot do alchemy; I cannot make efficacious teaching and learning happen when the ore is unwilling to be refined.
Do we even teach and test on the affective domain of hard work, perseverance, diligence, honesty, self-sacrifice, altruism, grace, mercy and self-control? Yet, are not there variables as important, if not more, to being successful in the real world, are what most employers are looking for (not just cognitive potential). Yet, our VAMs never consider these variables, and therefore are limited in their predictive power!
Did our test-metric-engineers forget about the maxim: not all that counts matters, not all that matters can be counted, not everything counted has value, and not everything of value may be counted?
After all, what do test scores prove, if not nothing more than having good test-taking skills (ex. cramming to fill short-term memory and organized mental-schema that help one store and access data). Research shows time and again that there is very little transfer of knowledge, or skills, across content areas in high school students because their underlying knowledge base and mental-schema are still in the developmental stages and have not had enough time and experience to make the deep and profound connections, that occur with more maturation and study.
We believe students should analyze and solve geometry proofs and logic-tests, hoping that it will transfer to their language arts classes, whereby they will be more able to deconstruct texts and analyze authors’ intent and purpose. Yet, research paints a picture that is more correlational than causational; students that can think, analyze and produce do so in most classes, regardless of the teaching strategy or assessments used. Which again, proves the goal of a good liberal-arts pedagogy of “teaching students how to think, not what to think”. “How to think curriculum” is more messy and sticky; it requires long-term studies and research, interdisciplinary explorations and teacher collaborations; it is not so neat and easy as teaching a “what to think” class with an EOC. Yet, how-to-think skills are used more in the real world, than the short-term (shallow measure) memory skills of EOC tests and the curriculums determined by them. After all, if a test goes into VAM, and VAM affects paycheck, then teacher will more likely “teach to the test”, and the overall quality of education will suffer…..duh!
In the end of it all, the Big-Picture, I always point out to my students the “Graph of School Predictions”, where the Predictors go on the X-axis (ex. GPA, AP classes taken, AP exams passed, FCAT scores, etc.) and Criterions go on the Y-axis (ex. future socio-economic class, contribution to the GDP, “being a prepared and responsible citizen”, “self-value/actualization/fulfillment”, etc.). Then, I point out that there is little, to none, long-term research and data to show that there is a positive correlation between predictors and criterion. We assume students with higher GPA will make a greater contribution to the nation’s GDP, but where is the convincing data. What about the outliers (if the real pattern is even known), like the Una-Bomber, who probably scored high in school, but was very-low in life?
In the Big-Picture, 90% of my chemistry students will forget 90% of the content on an EOC within 9 months, unless they restudy it because they take college chemistry. That can be said of most classes; unless the content is repeated and restudied (and then “permanently” assimilated), the learning was just temporary (and any short-term measure of it, ex. EOC: end-of-course exams) and has little to no effect and/or value in one’s adult life and career choice. So, why do we make such a big deal of EOC in high school, when they don’t even do this in college? If our tests have little power/significance in preparing, or predicting, success in the adult-life, then why are they such an important term in the VAM metric? Previous generations never suffered through all this testing, and teachers never scrutinized by VAMs, and yet the high school graduates of the 50s-80s were well prepared for college. So, what happened to the mantra: “if it ain’t broke, don’t fix it”?
So…..
Dear Superintendent, if we are going to get smaller paychecks because of an unjust VAM statistic, then why does it not work the other way? Why not divide up the money won for the Broad Award (or all other awards for education) and share the winnings with your teachers? Though, we might feel guilty because if we do get higher wages because of VAM incentives, then should we not share some of that with our students. For they, after all, were the ones performing on the tests. Just where does “the buck stop” or the reward end? Should some merit-pay go back to the families (their love, support, concern and accountability) that are the number one predictor of student success?
Mr. Governor, should not a proportion of your salary also be tied to student test scores? Is not the principle “a servant is not greater than their master” be applied to you? If we fail in the classroom, then your leadership must have something to do with it? Of course, the families never get any blame?
Mr. Scott if the FL GDP does not rise during your tenure, should not your salary be impacted? You may complain that federal policy has tied your hands. Well, do you now empathize with how teachers’ input is limited? Teachers strive for excellence, in spite of many of the socio-cultural variables that inhibit learning (ex. multiple divorces, inane and excessive entertainment medias, etc.). So, please treat us with knowledge and respect we deserve, and therefore drop VAM policies. If you disagree, then please have some statistician write a FL Governor VAM metric (and load it with variables you have no control over, ex. Senate bills passed) and then you might be more able to empathize with teachers.
Dear FL State Legislators, you approved VAM, so then apply it to yourselves. 50% of your pay should be based on the number of bills your author, and that PASS the legislature. We don’t care how hard you work, how many joules of energy you expend, or the product/profit you produce. No, we only care if your bills get passed (metaphor for test scores). Oh, but you will complain that the rate of bills passed is due to variables beyond your control. So, deal with it; suck it up and be treated in the same way you want to treat your teachers.
Florida public servants, ex. Firefighters and Police, should there be a VAM for you too. Should your evaluations, and part of your salary, be based on actual crimes caught, or those that were prevented? Is it the fires you put out, or your daily service to the community, that matters? If you do a good job and crime rates drop, and there are less crimes to catch, then should your VAM score get lower and paycheck too?
I think we all see the fallacy of “product-only” VAM measures, that take no account of the daily service and commitment of the worker (ignore all “process related” activities). If only test scores matter in my VAM, then why should I even care about collaborating with my peers, helping set up labs for the science department, leading local trainings and peer mentoring, writing curriculum and seeking best practices (if they have no influence on an EOC exam)? Why go the “extra mile” in my daily teaching, when all FL cares about is the “one-inch” of test data.
If one really considers the possible implications of a “dog eat dog” competitive environment of teaching to VAM tests, where teachers only care about their own students’ performances, and nothing else (since a significant part of their salary is at stake), education will become a bleak and barren wasteland of test-obsessed pedagogy, diminished in real and diverse learning and low in any true cognitive stimulation.
So, to all leaders who are unwilling to have a VAM equations applied to their own salary, STOP the hypocrisy and join us in a more justice and equitable society. Be willing to walk in our shoes, before you arrogantly and pretentiously tell us how they should be worn.
Mr. Rick Lapworth
Science Teacher, Felix Varela High School
15255 SW 96 St, Miami, FL 33196
Ed.S., NBCT 1999-2019
305-752-7900, x 3259, rm 259
Fax: 305-386-8987
This is the witness list for an Ohio House hearing on teacher evaluations.
There are 20 witnesses.
Two witnesses oppose the test-based teacher rankings.
Eighteen witnesses support the test-based teacher rankings.
Of the 18 in support of administering tests to students in order to grade teachers, SEVENTEEN come from StudentsFirst and the other one once started a charter school.
This is all very fair and balanced and scientific, as you can tell! 🙂
Good Lord. It’s truly a bubble. Ohio’s House members listened to StudentsFirst lobbyists recite talking points for nearly the entire hearing. What an absolute joke.
http://www.ohiohouse.gov/committee/education
There seem to be a few website articles about this. Here’s the actual research itself.
http://epa.sagepub.com/content/early/2014/04/11/0162373714531851.full.pdf+html?ijkey=Uwvo4Eg6.hQHI&keytype=ref&siteid=spepa
Try again — link to article Diane is referring to: http://www.usnews.com/news/articles/2014/05/13/report-finds-weak-link-between-value-added-measures-and-teacher-instruction
Tried to post this before but it didn’t show up.
Yes, that’s a good article about it. My link is what the two authors published in Educational Evaluation and Policy Analysis, to which the US News article refers. I notice their funding came from Gates (bottom of research article before references); I wonder if they will continue to get funding from Gates when they produce these results… 🙂
Teachers may want to consider a labor market reason for the reduction of labor protections with all these various schemes. Charter schools are non-union. In order to compete for teachers with public schools, they have get rid of teachers unions or it puts them at a competitive disadvantage. That’s the way it works here with private sector union shops versus non union shops. When there’s an opening in a union shop, the best workers flee and the union shop gets hundreds of applicants. My very own son just recently went this route, where a non-union shop trained him (manufacturing) and he fled the moment there was an opening in a union shop.
He wasn’t alone. They were interviewing in groups of three 🙂
It’s not complicated. They pay better and he gets some protections.
Great news. We told them so!
Here is another link to the study, complete with a video explanation by Morgan Polikoff, one of the study’s co-authors. I commend the video to your viewing. Interestingly, the study was funded by The Gates Foundation.
http://educationbythenumbers.org/content/researchers-say-pennsylvanias-measurement-teacher-effectiveness-doesnt-measure-good-teaching_1238/
Interesting today in the Ohio Dispatch paper an editorial pushing for more collection of student attendance and personal data at the state level. Timing or coincidence?
The Dispatch has been heavily pushing a school “data rigging scandal” using a lot of ink. They work hard to keep the issue in the news. Their idea is that since some student records were changed, no school can be trusted so the state needs to keep more private data.
Of course, the govenor never quite seems to merit the same scrutiny.
Reblogged this on peakmemory and commented:
“The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.”
Reblogged this on rightfulwriter.
All, read Kim McGuire’s article in the Minneapolis StarTribune (May 10, 2014) “In Cincinnati they’re closing the achievement gap.” Rothenberg Academy (charter? not really sure) is 99% black students living in poverty. This school feeds them, clothes them, lauders their clothes, gives them medical care and immunizations…they do whatever is necessary to care for the child who happens to be an elementary school student. The results have been phenomenal. It isn’t rocket science or testing. It’s just plain meeting kids’ needs. They don’t talk about closing the achievement gap — they talk about helping all of their students. Read and rejoice!
It’s a shame that all these resources being poured into assessment and evaluation are burning up money that could be spent on Head Start and other programs that would address the real issue of the effect that poverty has on early childhood.