Parents Across America has called on states to take advantage of the Every Student Succeeds Act and abolish VAM.
PAA has been critical of high-stakes testing.
PAA also produced a one-page fact sheet to demonstrate the failure of value-added-measurement.
David P. Cleary, chief of staff to Senator Lamar Alexander, responded to my questions about the Every Student Succeeds Act.
This is part 2:
The stakes attached to testing: will teachers be evaluated by test scores, as Duncan demanded and as the American Statistical Association rejected? Will teachers be fired because of ratings based on test scores?
Short Answer:
The federal mandate on teacher evaluation linked to test scores, as created in the waivers, is eliminated in ESSA.
States are allowed to use federal funds to continue these programs, if they choose, or completely change their strategy, but they will no longer be required to include these policies as a condition of receiving federal funds. In fact, the Secretary is explicitly prohibited from mandating any aspect of a teacher evaluation system, or mandating a state conduct the evaluation altogether, in section 1111(e)(1)(B)(iii)(IX) and (X), section 2101(e), and section 8401(d)(3) of the new law.
Long Answer:
Chairman Alexander has been a long advocate of the concept, as he calls it, of “paying teachers more for teaching well.” As governor of Tennessee he created the first teacher evaluation system in the nation, and believes to this day that the “Holy Grail” of education reform is finding fair ways to pay teachers more for teaching well.
But he opposed the idea of creating or continuing a federal mandate and requiring states to follow a Washington-based model of how to establish these types of systems.
Teacher evaluation is complicated work and the last thing local school districts and states need is to send their evaluation system to Washington, D.C., to see if a bureaucrat in Washington thinks they got it right.
ESSA ends the waiver requirements on August 2016 so states or districts that choose to end their teacher evaluation system may. Otherwise, states can make changes to their teacher evaluation systems, or start over and start a new system. The decision is left to states and school districts to work out.
The law does continue a separate, competitive funding program, the Teacher and School Leader Incentive Fund, to allow states, school districts, or non-profits or for-profits in partnership with a state or school district to apply for competitive grants to implement teacher evaluation systems to see if the country can learn more about effective and fair ways of linking student performance to teacher performance.
This is an essay that was written by a South Carolina school superintendent named John Taylor in 2002. It was originally called “Absolutely the Best Dentists.” It was supposed to be a satire, but reality overtook the jokes, and it was retitled “No Dentist Left Behind.” It was reprinted again and again. It was supposed to be funny.
Imagine a government system to rate dentists by the number of cavities their patients have. When the dentist in the story says that he chooses to serve patients in a poor neighborhood, and he can’t control how often they brush their teeth or what they eat, he is too to stop making excuses.
A dozen years ago this story was a satire. It was laughable and absurd on its face. Now it is federal policy, an integral part of Race to the Top. No one is laughing.
John Thompson, historian and teacher, takes a closer look at the New Mexuco court decision that imposed a preliminary injunction on the use of value-added measurement to evaluate teachers and to punish or reward them. In his judgment, VAM is on the way out.
New Mexico District Judge David K. Thomson granted a preliminary injunction preventing consequences from being attached to the state’s teacher evaluation data. As Audrey Amrein-Beardsley explains, “can proceed with ‘developing’ and ‘improving’ its teacher evaluation system, but the state is not to make any consequential decisions about New Mexico’s teachers using the data the state collects until the state (and/or others external to the state) can evidence to the court during another trial (set for now, for April) that the system is reliable, valid, fair, uniform, and the like.”
This is wonderful news. As the American Federation of Teachers observes, “Superintendents, principals, parents, students and the teachers have spoken out against a system that is so rife with errors that in some districts as many as 60 percent of evaluations were incorrect. It is telling that the judge characterizes New Mexico’s system as a ‘policy experiment’ and says that it seems to be a ‘Beta test where teachers bear the burden for its uneven and inconsistent application.’”
A close reading of the ruling makes it clear that this case is an even greater victory over the misuse of test-driven accountability than even the jubilant headlines suggest. It shows that Judge Thompson made the right ruling on the key issues for the right reasons, and he seems to be predicting that other judges will be following his legal logic. Litigation over value-added teacher evaluations is being conducted in 14 states, and the legal battleground is shifting to the place where corporate reformers are weakest. No longer are teachers being forced to prove that there is no rational basis for defending the constitutionality of value-added evaluations. Now, the battleground is shifting to the actual implementation of those evaluations and how they violate state laws.
Judge Thomson concludes that the state’s evaluation systems don’t “resemble at all the theory” they were based on. He agreed with the district superintendent who compared it to the Wizard of Oz, where “the guy is behind the curtain and pulling levers and it is loud.” Some may say that the Wizard’s behavior is “understandable,” but that is not the judge’s concern. The Court must determine whether the consequences are assessed by a system that is “objective and uniform.” Clearly, it has been impossible in New Mexico and elsewhere for reformers to meet the requirements they mandated, and that is the legal terrain where VAM proponents must now fight.
The judge thus concludes, “New Mexico’s evaluation system is less like a [sound] model than a cafeteria-style evaluation system where the combination of factors, data, and elements are not easily determined and the variance from school district to school district creates conflicts with the [state] statutory mandate.”
The state of New Mexico counters by citing cases in Florida and Tennessee as precedents. But, Judge Thomson writes that those cases ruled against early challenges based on equal protection or constitutional issues, as they have also cited practical concerns in implementation. He writes of the Florida (Cook) case, “The language in the Cook case could be lifted from the Court findings in this case.” That state’s judge decided “‘The unfairness of this system is not lost on this Court.’” Judge Thomson also argues, “The (Florida) Court in fact seemed to predict the type of legal challenge that could result …‘The individual plaintiffs have a separate remedy to challenge an evaluation on procedural due process grounds if an evaluation is actually used to deprive the teacher of an evaluation right.’”
The question in Florida and Tennessee had been whether there was “a conceivable rational basis” for proceeding with the teacher evaluation policy experiment. Below are some of the more irrational results of those evaluations. The facts in the New Mexico case may be somewhat more absurd than those in other places that have implemented VAMs but, given the inherent flaws in those evaluations, I doubt they are qualitatively worse. In fact, Audrey Amrein-Beardsley testified about a similar outcome in Houston which was as awful as the New Mexico travesties and led to about 1/4th of their teachers subject to those evaluations being subject to “growth plans.”
As has become common across the nation, New Mexico teachers have been evaluated on students who aren’t in the teachers’ classrooms. They have been held accountable for test results from subjects that the teacher didn’t teach. Science teachers might be evaluated on a student taught in 2011, based on how that student scored in 2013.
The judge cited testimony regarding a case where 50% of the teachers rated Minimally Effective had missing data due to reassignment to a wrong group. One year, a district questioned the state’s data, and immediately it saw an unexplained 11% increase in effective teachers. The next year, also without explanation, the state’s original numbers on effectiveness were reduced by 6%.
One teacher taught 160 students but was evaluated on scores of 73 of them and was then placed on a plan for improvement. Because of the need to quantify the effectiveness of teachers in Group B and Group C, who aren’t subject to state End of Instruction tests, there are 63 different tests being used in one district to generate high-stakes data. And, when changing tests to the Common Core PARCC test, the state has to violate scientific protocol, and mix and match test score results in an indefensible manner. Perhaps just as bad, in 2014-15, 76% of teachers were still being evaluated on less than three years of data.
The Albuquerque situation seems exceptionally important because it serves 25% of the state’s students, and it is the type of high-poverty system where value-added evaluations are likely to be most unreliable and invalid. It had 1728 queries about data and 28% of its teachers ranked below the Effective level. The judge noted that if you teach a core subject, you are twice as likely as a French teacher to be judged Ineffective. But, that was not the most shocking statistic. In Albuquerque, Group A elementary teachers (where VAMs play a larger role) are five times more likely to be rated below Effective than their colleagues in Group B. In Roswell, Group B teachers are three times more likely to be rated below Effective than Group C teachers.
Curiously, VAM advocate Tom Kane testified, but he did so in a way the made it unclear whether he saw himself as a witness for the defense or the plaintiffs. When asked about Amrein-Beardsley’s criticism of using tests that weren’t designed for evaluating teachers, Kane countered that the Gates Foundation MET study used random samples and concluded that differing tests could be used in a way that was “useful in evaluating teachers” and valid predictors of student achievement. Kane also replied that he could estimate the state’s error rate “on average,” but he couldn’t estimate error rates for individual teachers. He did not address the judge’s real concern about whether New Mexico’s use of VAMs was uniform and objective.
I am not a lawyer but I have years of experience as a legal historian. Although I have long been disappointed that the legal profession did not condemn value-added evaluations as a violation of our democracy’s fundamental principles, I also knew that the first wave of lawsuits challenging VAMs would face an uphill battle. Using teachers as guinea pigs in a risky experiment, where non-educators imposed their untested opinions on public schools, was always bad policy. Along with their other sins, value-added evaluations would mean collective punishment of some teachers merely for teaching in schools and classes where it is harder to meet dubious test score growth targets. But, many officers of the court might decide that they did not have the grounds to overrule new teacher evaluation laws. They might have to hold their noses while ruling in favor of laws that make a mockery of our tenets of fairness in a constitutional democracy.
During the last few years, rather than force those who would destroy the hard-earned legal rights of teachers to meet the legal standard of “strict scrutiny,” those who would fire teachers without proving that their data was reliable and valid have mostly had to show that their policies were not irrational. Now that their policies are being implemented, reformers must defend the ways that their VAMs are actually being used. Corporate reformers and the Duncan administration were able to coerce almost all of the states into writing laws requiring quantitative components in teacher evaluations. Not surprisingly, it has often proven impossible to implement their schemes in a rational manner.
In theory, corporate reformers could have won if they required the high-stakes use of flawed metrics while maintaining the message discipline that they are famous for. School administrators could have been trained to say that they were merely enforcing the law when they assessed consequences based on metrics. Their job would have been to recite the standard soundbite when firing teachers – saying that their metrics may or may not reflect the actual performance of the teacher in question – but the law required that practice. Life’s not fair, they could have said, and whether or not the individual teacher was being unfairly sacrificed, the administrators who enforced the law were just following orders. It was the will of the lawmakers that the firing of the teachers with the lowest VAMS – regardless of whether the metric reflected actually effectiveness – would make schools more like corporations, so practitioners would have to accept it. But, this is one more case where reformers ignored the real world, did not play out the education policy and legal chess game, and anticipate that rulings such as Judge Thomson’s would soon be coming.
Real world, VAM advocates had to claim that its results represented the actual effectiveness of teachers and that, somehow, their scheme would someday improve schools. This liberated teachers and administrators to fight back in the courts. Moreover, top-down reformers set out to impose the same basic system on every teacher, in every type of class and school, in our diverse nation. When this top-down micromanaging met reality, proponents of test-driven evaluations had to play so many statistical games, create so many made-up metrics, and improvise in so many bizarre ways, that the resulting mess would be legally indefensible.
And, that is why the cases in Florida and Tennessee might soon be seen as the end of the beginning of the nation’s repudiation of value-added evaluations. The New Mexico case, along with the renewal of the federal ESEA and the departure of Arne Duncan, is clearly the beginning of the end. Had VAM proponents objectively briefed attorneys on the strengths and weaknesses of their theories, they could have thought through the inevitable legal process. On the other hand, I doubt that Kane and his fellow economists knew enough about education to be able to anticipate the inevitable, unintended results of their theories on schools. In numerous conversations with VAM true believers, rarely have I met one who seemed to know enough about the nuts and bolts about schools to be able to brief legal advisors, much less anticipate the inevitable results that would eventually have to be defended in court.
This just in from the group that led New York’s historic opt out movement. One of every five students did not take the state tests last spring, over 220,000 students.
FOR IMMEDIATE RELEASE: December 15, 2015
More information contact:
Lisa Rudley (917) 414-9190; nys.allies@gmail.com
NYS Allies for Public Education http://www.nysape.org
Parents Will Continue to Opt Out Until Ed Law Repealed &
Real Change Seen in the Classrooms
The Governor’s Common Core Task Force released a list of recommendations last Thursday, Dec. 10th. The recommendations while a reflection of the parent and educator voices around the state do not alone restore trust in Albany. How the recommendations and other issues get addressed is the key and parents are watching this very closely.
Until there is a halt of the Common Core standards, repeal of the Education Transformation Act, major changes to the state tests, a reduction of unnecessary testing, protection of data privacy, and local control restored, parents will continue to Opt Out in large numbers.
The recommendations deliberately state that Governor Cuomo’s ‘signature’ legislation that enforces many of these harmful policies doesn’t need to be touched. On the contrary, this law is the prescriptive blueprint to these harmful policies that was passed by the legislature as part of the budget last spring.
One of the recommendations to put a 4-year moratorium on evaluating teachers based on the flawed Common Core state tests was officially voted into emergency regulations by the Board of Regents at today’s board meeting. Until the law is repealed, this moratorium does not reduce testing it actually does the opposite, increases testing and further puts a strain on school districts’ budgets to comply.
NYSAPE is calling on parents to Opt Out of state tests and any local tests that are linked to this corrupt and invalid evaluation system that clearly doesn’t provide value for the students, educators, or schools.
“The task force recommendations have opened the door to change. Much of these harmful policies came in through our legislature when they passed the Education Transformation Act against the will of the people they serve. Our State Assembly and Senate must now reverse this harmful legislation so that changes will be meaningful and substantial. Parents will be vigilant in following these changes every step of the way. We will continue to refuse to allow our children to participate in this system until ALL harmful reforms are removed from our classrooms,” said Jeanette Deutermann, Long Island public school parent and founder of Long Island Opt Out.
“Until specific laws and policies regarding standards, student assessment, teacher evaluation and school ranking are changed, parents will continue to boycott any system that ties high-stakes to standardized assessments.” Chris Cerrone, Erie County public school parent, educator, and school board member.
Jamaal Bowman, Bronx public school parent and middle school principal said, “Although I consider the task force recommendations to be a step in the right direction, it is merely a single step. At this point, there is too much uncertainty to get excited about where we are headed in our public schools. Until we know how the recommendations will be implemented, and by whom, and until the law tying teacher evaluations to test scores is revised or repealed, we will not be able to move forward and properly meet the holistic needs of our children.”
“How the Common Core, testing, and other education policies are revamped to be in the best interest of the children will be watched very closely by parents. I will not be opting my children into any unnecessary tests including local assessments that do not provide important feedback for my children,” said Lisa Rudley, Westchester County public school parent and founding member of NYSAPE.
“While there is much talk of high standards, there is little discussion of the non-curricular resources required to ensure that all students can succeed in the face of poverty and lack of adequate funding. It is disappointing that the task force failed to raise the question, if disadvantaged students were struggling prior to the implementation of the Common Core, how will simply raising the bar increase student achievement,” said Bianca Tanis, Ulster County public school parent, Rethinking Testing member and educator.
“After so much time and money has been wasted in forced implementation of flawed policy, students and educators of New York have been hurt and trust has been broken. We must repeal the APPR imposed by politicians who did not understand the domain. Scholars in schools of education and professional educators should design the best systems to achieve goals for public education,” said Katie Zahedi, Dutchess County, principal.
Marla Kilfoyle, Long Island public school parent, educator and BATS’ executive director said, “Teachers and parents do not trust NYSED, the ‘Tisch’ Regents’ majority, the legislature, or the Governor to be in charge of education. What they have done to our public education system and to our children is unconscionable. I have been an educator in NY for over 25 years and the mass destruction their policies have caused will take years to repair.”
NYSAPE, a grassroots organization with over 50 parent and educator groups across the state, is calling on parents to continue to opt out by refusing high-stakes testing for the 2015-16 school year. Go to http://www.nysape.org for more details on how to affect changes in education policies.
###
Carol Burris, executive director of the Network for Public Education and former principal at South Side High School in Rockville Center, Long Island, New York, has subjected the report of Governor Andrew Cuomo’s task force to a close reading.
But not the kind of close reading where you forget about context and prior knowledge. She notes that Governor Cuomo has no intention of amending or repealing the law he pushed through last June, which requires that teachers are evaluated by test scores that count for 50% of the evaluation.
There is the elephant in the room–the evaluation of teachers by test scores. When it comes to the damage done by APPR, the report is strangely silent. It is as though the committee never heard a complaint on how evaluating teachers by test scores increased both anxiety and test prep. The only place where it is addressed is in Recommendation 21 that states that until a new set of standards are phased in, the results of Common Core 3-8 assessments should be advisory only. Cuomo immediately seized on the ambiguity of that statement and issued the following:
[Cuomo statement] “The Education Transformation Act of 2015 will remain in place, and no new legislation is required to implement the recommendations of the report, including recommendations regarding the transition period for consequences for students and teachers. During the transition, the 18 percent of teachers whose performance is measured, in part, by Common Core tests will use different local measures approved by the state, similar to the measures already being used by the majority of teachers.”
The Education Transformation Act was the bill Cuomo pushed through the legislature to raise the percentage of test scores in teacher evaluations to 50 percent. Like a teenage boy who doesn’t get that the relationship is over, Cuomo cannot let go of his APPR, even though more researchers agree that evaluating teachers by test student scores makes no sense.
And more ominously, she describes the new testing corporation that New York has contracted with for the next five years.
Truth be told, no matter what recommendations the report made, at least half of the horse is already out of the testing barn. The new direction in assessment was set with the July approval of a $44 million contract with Questar that locks the state in for five years. If parents are looking for relief from test-driven instruction, they will not find it with Questar. You can read about the company’s philosophy of continuous assessment-driven instruction here. Below is an excerpt:
…after every five minutes of individualized tablet-based instruction, students would be presented with a brief series of questions that adapt to their skill level, much as computer-adaptive tests operate today. After that assessment, the next set of instructional material would be customized according to these results. If a student needs to relearn some material, the software automatically adjusts and creates a custom learning plan on the fly. The student would then be reassessed and the cycle would continue…
The practice of adaptive, computer-based learning, known as Competency Based Education (CBE), is a reincarnation of two other failed reforms from the last century — Outcomes Based Instruction and Mastery Learning. As the tests roll out, Questar will be marketing their CBE modules for test prep, and schools desperate to increase scores will buy them.
Thus far, Governor Cuomo has gotten the press he wanted: banner headlines in the New York Daily News and Long Island’s Newsday, proclaiming prematurely that Common Core is dead. No, it is not. What happens next is up to the Governor.
The good news is that he has an outstanding educator advising him, Jere Hochman, former superintendent in Bedford, New York. Hopefully, Hochman will help the Governor understand how to get out of the hole he dug for himself and how to take concrete steps to remove the disruption and constant churn that the State Education Department and the Governor’s interventions have imposed on schools. It is time for some stability and sanity at the helm. At the moment, teachers and students see a battle for control of the wheel, and the ship is lurching from side to side. I won’t torture the analogy any more. But I do hope that Governor Cuomo listens to Jere Hochman’s advice and takes the task force report seriously.
The Journal News of the Lower Hudson Valley in New York, referred to as Lohud, has been critical of the mess that Andrew Cuomo has made with his constant meddling in education policy.
Today, Lohud praised Cuomo’s task force for listening to the parents who opted their children out of the Common Core testing. The number of children who opted out were about 225,000. That is a huge number of people expressing no-confidence in the state’s testing regime.
Lohud thinks the task force listened to parents and educators and hit all the right notes:
The task force released a report Thursday that accurately and even passionately captures the confusion and disarray unleashed on schools by Albany over the last several years. Consider this slap at New York’s educational leadership, which sounds like it came from a group of outside critics:
“The implementation of the Common Core in New York was rushed and flawed. Teachers stepped into their classrooms in the 2012-2013 school year unfamiliar and uncomfortable with the new standards, without curriculum resources to teach students, and forced to administer new high-stakes standardized tests that were designed by a corporation instead of educators.”
Hey, that’s what happened.
We messed up
Without naming names, the report is a pretty stunning rebuke of Board of Regents Chancellor Merryl Tisch and former state Education Commissioner John King (soon to become acting U.S. education secretary), who refused to heed the legitimate and plentiful concerns of educators and parents. As a result, New York will wind up spending more than a decade rewriting education policies over and over, without any guarantee that students will be better off in the end….
Interestingly, the report does not explore the merits and failings of New York’s teacher-evaluation system, which is perhaps most controversial for grading teachers, in part, on student test scores. Instead, the task force recommends that test scores not be used to evaluate teachers or students until 2019-2020. (State law already bans including the test scores on student transcripts or using them to make student placement decisions through 2018.)
This rather vague recommendation leaves the teacher-evaluation system in place, and would likely require school districts to replace test scores with another measure for the next several years.
The task force did not take the next, necessary step of declaring the evaluation system a failure and calling for the development of a new system that would not only hold teachers accountable but give them the information they need to improve their performance and student achievement. But the panel covered a lot of ground in a few short weeks, and it should not be up its 16 people to solve all of New York’s problems.
Should Cuomo and the state Legislature move ahead with the development of new standards and testing, a new evaluation system would have to be next. Otherwise, the education wars will continue.
There’s no telling, at this point, whether Cuomo will endorse the task force’s work in whole or part or whether the recommendations would be carried out in such a way as to win back the loyalty of disenchanted parents and educators. We’ll likely find out where the governor stands when he delivers his State of the State address next month.
Unless the Legislature repeals or amends the law that was passed last June and tucked into the state budget, teachers will still be evaluated by test scores, counting for up to 50%, then local measures will not replace what the law requires. Their evaluations won’t lead to punishments, but presumably they will go onto their permanent records. Thus, for the task force’s recommendations to have any teeth, the Legislature must act to change the objectionable law. The task force’s recommendations do not trump state law.
Lohud credits the parents for forcing the task force to listen. Now, let’s see what Governor Cuomo does. It would be nice if he walked back his statement that he hopes to bust the “public education monopoly,” which he said right before he was re-elected. That would be a good start, especially for the parents of more than 90% of the children in the state who attend public schools.
Peter Greene is not impressed with the Cuomo Task Force report on the Common Core, the tests, and teacher evaluation. He calls it a “nothing Sundae.”
He goes through the recommendations one by one. But his big beef is that the report does not question the value of the CCSS, does not question the testing, and does not get to the problem of test-based accountability for teaching. The report assumes that the problem all along has been poor implementation, not that any of the fundamental ideas need to be changed or dropped or replaced.
Another blogger points out that the Task Force report includes this curious statement:
The Education Transformation Act of 2015 will remain in place, and no new legislation is required to implement the recommendations of the report, including recommendations regarding the transition period for consequences for students and teachers. During the transition, the 18 percent of teachers whose performance is measured, in part, by Common Core tests will use different local measures approved by the state, similar to the measures already being used by the majority of teachers.
The blogger writes:
Yes, tests will still count for 50% of a teacher’s evaluation.
The task force appointed by Governor Andrew Cuomo to review the Common Core standards, testing, and teacher evaluation will recommend a moratorium on tying teacher evaluation to test scores--as much as four years–and a reboot of the standards and tests.
Why Cuomo is making these decisions is unclear because the New York State Constitution gives the governor no role in education. The New York State Board of Regents is the legal authority, not the governor, but this governor decided to take control of education.
Meanwhile, we wait to hear from Governor Cuomo to see what in the task force report he agrees with, since he has made himself the Decider.
John Thompson, historian and teacher in Oklahoma, has reviewed the work of economists Raj Chetty. You may recall that Chetty, a Harvard professor, was co-author of a study that purported to show that teachers could be evaluated by the test scores of their students. An effective teacher, one who raised test scores, would raise lifetime income, increase high school graduation rates, prevent teen pregnancies, and have lifelong effects on students. Raj and his colleagues John Friedman and Jonah Rockoff were cited on the first page of the New York Times (before the study was peer-reviewed), appeared on the PBS NewsHour, and were hailed by President Obama in his State of the Union speech in 2012. Their study became the #1 talking point for those who thought that using test scores–their rise and fall–would be the best way to identify effective and ineffective teachers. As Professor Friedman told the New York Times, “The message is to fire people sooner rather than later.”
Critics thought the findings were fairly modest. Even the Times said:
The average effect of one teacher on a single student is modest. All else equal, a student with one excellent teacher for one year between fourth and eighth grade would gain $4,600 in lifetime income, compared to a student of similar demographics who has an average teacher. The student with the excellent teacher would also be 0.5 percent more likely to attend college.
That works out to about $105 a year for a 40-year career, or $2 a week. But the Times then looked at the results in the aggregate and calculated that the aggregate of gains for an entire class would be $266,000 over the lifetimes of the entire class, or millions of dollars in added income when multiplied by millions of classrooms. Pretty great stuff, even though it means only $2 a week for one student.
The Obama administration bought into the Chetty-Friedman-Rockoff thesis whole-heartedly. Fire teachers sooner rather than later. Use test scores to find out who is a great teacher, who is a rotten teacher. It all made sense, except that it didn’t work anywhere. The scores bounced around. A teacher who was great one year was ineffective the next year; and vice versa. Teachers were rated based on the scores of students they never taught. Tests became the goal of education rather than the measure. It was a plague of madness that overcame public education across the land, embedded in Race to the Top (2009) and certified by Ivy League professors.
Thompson writes:
As it becomes more clear that value-added teacher evaluations are headed for the scrap heap of history, true believers in corporate reform continue to respond with the same old soundbites on the ways that their statistical models (VAMs) can be valid and reliable under research conditions. But, they continue to ignore the real issue and offer no evidence that VAMs can be made reliable and valid for evaluating real individuals in real schools.
Gates Foundation scholar Dan Goldhaber recently replied to the American Educational Research Association (AERA) statement which “cautions against VAM being used to have a high-stakes, dispositive weight in evaluations.” His protest recalls the special pleading of VAM advocates Raj Chetty, John Friedman, and Jonah Rockoff in reply to the American Statistical Association’s (ASA) 2014 statement warning about the problems with using VAMs for teacher evaluations.
Goldhaber criticizes the AERA by citing a couple of studies that use random samples to defend the claim that they can be causally linked to a teacher’s performance. Using random samples makes research easier but it also makes those studies irrelevant to real world policy questions. Goldhaber then cites Chetty et. al and their claim that low-stakes 1990s test scores resulted in the increased income of individuals during the subsequent economic boom in New York City during the 2000s.
Interestingly, Chetty’s rebuttal of the ASA cited the same two random sample studies, as well as his own research that was cited by Goldhaber. Like Goldhaber and other value-added proponents, he acknowledged the myriad of problems with value-added evaluations, but added, “School administrators, teachers, and other relevant parties can be trained to understand how to interpret a VAM estimate properly, including measures of precision as well as the assumptions and limitations of VAM.”
That raises two other concerns. First, if educators should be trained in the arcane methodologies, assumptions, and limitations of regression studies in order to use VAMs, should economists not be trained in the logistics of schools so they can conduct research that is relevant to education policy? Secondly, even if they ignore the nuts and bolts of schools, isn’t it strange that Chetty and his colleagues ignore economic factors when explaining economic effects? Why are they so sure that education – not economic forces – explains economic outcomes?
These questions become particularly interesting when reading Chetty’s web site. If he was really committed to the use of his Big Data methodology to help improve schools and students’ subsequent economic outcomes, would he not engage in a conversation with practitioners, and ground his methods in reality, so information from his models could be used to improve schools? After all, architects run plenty of quantitative structural analyses of their construction projects but they also interview their clients and listen to how they will use their buildings.
Chetty could have gone back and learned what he didn’t know about schools before he joined in the social engineering experiment known as school reform. Instead, he is rushing off to promote policies for problems which seem to be equally beyond his realm of knowledge. And, he seems equally uncurious about the new people he wants to “nudge” into better behavior. His method for studying anti-poverty policy is to ignore what actually happens in schools and communities and to “treat behavioral factors like any other modeling decision, such as assuming time-separable or quasi-linear utility.” The goal of his new project is to create incentives so that policy-makers can rid poor people, especially, of their “loss aversion, present bias, mental accounting, [and] inattention” so they will move to better places.
I’m not an expert on Chetty’s new The Equality of Economic Opportunity Project but my reading of the evidence is that Robert Putnam, who combines qualitative and quantitative research to document the decline of social mobility, makes a much stronger case than Chetty, who believes social and economic mobility hasn’t declined. It seems to me that Putnam is right and that we must take a generational view in order to show that economic opportunity for the poor has been reduced. I also believe that Derek Thompson nails the case that each generation since the first half of the Baby Boom is seeing an economic deterioration.
I can’t help but wondering why Chetty doesn’t stop scurrying around complex social issues, pontificating on simplistic quick fixes, and study issues in depth. He seems more intent on promoting his Big Data methods, and defeating traditional social science, than actually solving real-world problems. Chetty (and other VAM true believers?) appear preoccupied with academic combat against traditional social scientists who still respect falsifiable hypotheses and peer review. Education and child poverty appear to be just the battlegrounds for academic combat with researchers.
Traditional school improvement was based on the imperfect process of drawing upon the scientific method to diagnose problems, policy debates, and the imperfect democratic process known as compromise. To do that, educators and researchers studied the history and the nature of the causes and effects of underperformance. Corporate reform sought the opposite. Rather than study and debate the nature of our schools’ shortcomings, problems and solutions, the contemporary reform movement attempted a series of bank shots. Ignoring their actual targets, they sought incentives and disincentives that would prompt others to devise solutions. The job of economists’ regression studies was to suggest rewards and punishments that would make educators improve.
An illustration of Chetty’s disdain for evidence-based, collaborative conversations about school improvement is the first graph on his web site. It shows the surge in student test score growth which occurs when a “High VA Teacher Enters,” and replaces a low performer. If Chetty sought to articulate a hypothesis or discuss how his hypothesis, if proven, could improve teacher quality, he would have addressed some issues. But, the graphic resembles a political attack ad more than a presentation of evidence for school improvement.
Chetty’s graphic is strangely opaque about what he means by “high VA” teachers or how many of them there are. In fact, those gains he showcased are the educational equivalent of a White Rhinoceros.
Chetty emphasizes the incredible size of his database. His data spans the school years 1988-1989 through 2008-2009 and covers roughly 2.5 million children in grades 3-8. Because there are 974,686 unique students in the dataset, his Power Points seem impressive. But, it is extremely difficult to find the key number which a traditional social scientist would have volunteered at the beginning of a study. Chetty’s graphs that illustrate such dramatic gains are based on samples as small as 1135. In other words, about 12 to 17 of these top-performing New York City teachers transferred, per year, into low value-added classrooms.
Chetty doesn’t ask why such transfers are so rare. Moreover, he makes it extremely difficult for a reader to learn the most important facts that would prompt that essential question and a constructive discussion of solution. Instead, he indicates that the answer is using VAMs to fire low-performing teachers and, without evidence, he implies that there are enough top 5% teachers who would respond to modest incentives and transfer to those low value-added classrooms. Otherwise, Chetty’s work on transfers might earn him academic awards but it is just theory, irrelevant for real world policy.
Sadly, it looks like Chetty’s new studies are equally simplistic. The problem, he implies, is not that the economic ladder out of poverty is broken. The problem is getting poor families to move from places without opportunity to places where there is opportunity. So, we in Oklahoma City should forget that Supply Side economics incentivized the mass transfer of good-paying jobs to the exurbs. In Oklahoma County, where poor children’s economic opportunity is in the bottom 17% of the nation, we should incentivize the movement of poor families to Cleveland County where social mobility hasn’t declined.
Presumably, the additional good-paying jobs for the influx of poor families would magically appear. In other words, Chetty’s logic on moving to opportunity is the first cousin of his faith that top teachers will flock to the inner city because they want to be evaluated with an algorithm which is biased against inner city teachers.
I wish I didn’t feel compelled to sound so sarcastic. I really do. But, for every complicated question, there is an answer that is quick, simple, and wrong. Why are Chetty et. al so quick to conclude that it is schools – not the totality of market and historical forces – that drive economic outcomes? Even though the market has undermined the futures of poor families, why does he remain convinced that it can fix schools?
And, the inconsistencies of Chetty and other corporate reformers drive me up the wall. He now proclaims, “We find that every year of exposure to a better environment improves a child’s chances of success.” Were he consistent, Chetty might understand that exposure to education environments might improve his chance of studying education in a way that improves his chances of successfully helping students.
Why does Chetty not take the time to understand the environments of poor children, and build better school environments? Why not help create learning environments that would attract high value-added teachers, not drive them out of the profession? Rather than demand that teachers and poor families learn to look at their worlds the way Chetty does, why not listen to the people who he says he wants to help?
Diane Ravitch's blog 