Audrey Amrein-Beardsley posted a guest blog by a rising star in the Academy, Jimmy Scherrer of North Carolina State University, who previously taught in LAUSD.

Scherrer wrote:

“As someone who works with students in poverty [see also a recent article Scherrer wrote in the highly esteemed, peer-reviewed Educational Researcher], I am deeply troubled by the use of status measures—the raw scores of standardized assessments—for accountability purposes. The relationship between SES and standardized assessment scores is well known. Thus, using status measures for accountability purposes incentivizes teachers to work in the most advantaged schools.

“So, I am pleased with the increasing number of accountability systems that are moving away from status measures. In their place, systems seem to be favoring value-added estimates. In theory, this is a significant improvement. However, the manner in which the models are currently being used and how the estimates are currently being interpreted is intellectually criminal. The models’ limitations are obvious. But, as a learning scientist, what’s most alarming is the increasing use of the estimates generated by value-added models as a proxy for “effective” teaching…..”

“Typically, research studies on teaching and learning are framed using one of three perspectives: the behaviorist, the cognitivist, and the situative. Each perspective is associated with a different grain size. The behaviorist perspective focuses on basic skills, such as arithmetic. The cognitivist perspective focuses on conceptual understanding, such as making connections between addition and multiplication. The situative perspective focuses on practices, such as the ability to make and test conjectures. Effective teaching includes providing opportunities for students to strengthen each focus. However, traditional standardized assessments mainly contain questions that are crafted from a behaviorist perspective. The conceptual understanding that is highlighted in the cognitivist perspective and the participation in practices that is highlighted in the situative perspective are not captured on traditional standardized assessments. Thus, the only valid inference that can be made from a value-added estimate is about a teacher’s ability to teach the basic skills and knowledge associated with the behaviorist perspective.”

This, he writes, is “intellectually criminal” and “intellectually lazy.”

Tell it! VAM is Junk Science.

Laura Chapman, a regular contributor to the blog, has worked in arts education for many years.

She writes:

This desire to churn the teaching workforce is not just a push from Bill Gates and lawsuits to dismantle unions.
Six economists/statisticians brought together at the Brookings Institution offered a similar plan. These number crunchers said that district-wide VAM (value-added) scores should be used to determine the most effective teachers, irrespective of the subjects and grade-levels they teach.

This proposal is efficient and absurd. It is based on the assumption that a district’s value added scores are so highly correlated with “non-value added” measures that employment decisions for all teachers can be based on the performance of teachers with value added scores.

Under this system, all teachers would also have a composite evaluation based on multiple measures such as end of course test scores, observations, and student surveys. Even so, the teachers with VAM scores would determine the employment fate of all teachers. How is this conclusion reached?

Here is the magical thinking: “For example, we would assume that the correlation between observationally-based ratings of teachers and value-added (scores) in math would be the same in history, where value-added measures are not available.”

In other words, the statisticians freely invent (impute) a missing metric for the history teacher by assuming a math teacher’s rating on a classroom observation protocol can be used as a substitute for the history teacher’s missing value added score.

Those inferential leaps are just the beginning of a larger plan that would make all teacher evaluations “comparable” without any distinctions in grade level, or subject, or conditions under which teachers work.

The Brookings policy articulates principles for dismissing up to 25% of teachers in a district, on the assumption that this action plan would increase test scores and be “fair” to every teacher. The only exception to this formula might be for teachers of exceptional children. This case of econometric thinking ignores the educational, ecological, and substantive importance of different job assignments. See Corerelation, Para 5 in http://www.brookings.edu/research/reports/2011/04/26-evaluating-teachers

The Brookings paper is not radically different, (except for the 25 % churn) from a USDE plan for all teachers by a collective VAM for a school, but limited to one of the “priority” subtests such as reading or mathematics. In Florida, for example, the school wide VAM in reading or math is assigned to art and other teachers of nontested subjects. In other words, the curriculum and instruction that really matters is narrowed to the three R’s.

The use of a collective VAM focused on reading or math is a rapid and cost-effective way to meet federal or state requirements for teacher evaluation. Moreover, in 2014, a U.S. district judge ruled that evaluators in Florida are allowed to disregard a teacher’s job assignment in rating performance. The judge ruled that this practice is legal, even if it is unfair.

Teacher ratings based on a collective value-added score are likely to increase in states where Common Core State Standards (CCSS) are adopted and tested. The CCSS call for all teachers to improve student proficiency in English Language Arts and mathematics.

Although the American Statistical Association has denounced the practice of using VAM for rating individuals, that measure is unlikely to disappear as a tool for churning the workforce.

In the Obama/Duncan/McKinsey & Co. “RESPECT” project, for example, a teacher can only be judged “highly qualified” by producing more than a year’s worth of growth (gain in test scores) in three out of every five years. Teachers without that designation have shorter up-or-out criteria to meet.

This stack-ranking system, like the Brookings plan, banishes job security and churns the teaching workforce by insisting on one-size-fit-all criteria for “effective” teachers. http://www.ed.gov/blog/2012/02/launching-project-respect/

Peter Greene here evaluates a report by two analysts at Bellwether Education, a DC think tank, about how teachers should be evaluated. His post is a model of how to tear apart and utterly demolish the musings of people far removed from the classroom about how things ought to work.

He begins by situating its sponsor:

“I am fascinated by the concept of think tank papers, because they are so fancy in presentation, but so fanceless in content. I mean, heck– all I need to do is give myself a slick name and put any one of these blog posts into a fancy pdf format with some professional looking graphic swoops, and I would be releasing a paper every day.

“Bellwether Education, a thinky tank with connections to the standards-loving side of the conservative reformster world, has just released a paper on the state of teacher evaluation in the US. “Teacher Evaluation in an Era of Rapid Change: From ‘Unsatisfactory’ to ‘Needs Improvement.'” (Ha! I see what you did there.) Will you be surprised to discover that the research was funded by the Bill and Melinda Gates Foundation?”

He reviews what they describe as current trends and pulls each one apart.

Here is an example of a current trend and Greene’s response:

“3) Districts still don’t factor student growth into teacher evals

“Here we find the technocrat blind faith in data rearing its eyeless head again”

The authors say: “While raw student achievement metrics are biased—in favor of students from privileged backgrounds with more educational resources—student growth measures adjust for these incoming characteristics by focusing only on knowledge acquired over the course of a school year.”

“This is a nice, and inaccurate, way to describe VAM, a statistical tool that has now been discredited more times than Donald Trump’s political acumen. But some folks still insist that if we take very narrow standardized test results and run them through an incoherent number-crunching, the numbers we end up with represent useful objective data. They don’t. We start with standardized tests, which are not objective, and run them through various inaccurate variable-adjusting programs (which are not objective), and come up with a number that is crap. The authors note that there are three types of pushback to using said crap.

“Refuse. California has been requiring some version of this for decades. and many districts, including some of the biggest, simply refuse to do it.

“Delay. A time-honored technique in education, known as Wait This New Foolishness Out Until It Is Replaced By The Next Silly Thing. It persists because it works so often.

“Obscure. Many districts are using loopholes and slack to find ways to substitute administrative judgment for the Rule of Data. They present Delaware as an example of how futzing around has polluted the process and buttress that with a chart that shows statewide math score growth dropping while teacher eval scores remain the same.

“Uniformly high ratings on classroom observations, regardless of how much students learn, suggest a continued disconnect between how much students grow and the effectiveness of their teachers.

“Maybe. Or maybe it shows that the data about student growth is not valid.

“They also present Florida as an example of similar futzing. This time they note that neighboring districts have different distributions of ratings. This somehow leads them to conclude that administrators aren’t properly incorporating student data into evaluations.

“In neither state’s case do they address the correct way to use math scores to evaluate history and music teachers.”

After carefully pulling apart the report, here are the conclusions, theirs and his:

Greene reviews their recommendations:

“It’s not a fancy-pants thinky tank paper until you tell people what you think they should do. So Adelman and Chuong have some ideas for policymakers.

“Track data on various parts of new systems. Because the only thing better than bad data is really large collections of bad data. And nothing says Big Brother like a large centralized data bank.

“Investigate with local districts the source of evaluation disparities. Find out if there are real functional differences, or the data just reflect philosophical differences. Then wipe those differences out. “Introducing smart timelines for action, multiple evaluation measures including student growth, requirements for data quality, and a policy to use confidence intervals in the case of student growth measures could all protect districts and educators that set ambitious goals.

“Don’t quit before the medicine has a chance to work. Adelman and Chuong are, for instance, cheesed that the USED postponed the use of evaluation data on teachers until 2018, because those evaluations were going to totally work, eventually, somehow.

“Don’t be afraid to do lots of reformy things at once. It’ll be swell.

“Their conclusion

“Stay the course. Hang tough. Use data to make teacher decisions. Reform fatigue is setting in, but don’t be wimps.

“My conclusion

“I have never doubted for a moment that the teacher evaluation system can be improved. But this nifty paper sidesteps two huge issues.

“First, no evaluation system will ever be administrator-proof. Attempting to provide more oversight will actually reduce effectiveness, because more oversight = more paperwork, and more paperwork means that the task shifts from “do the job well” to “fill out the paperwork the right way” which is easy to fake.

“Second, the evaluation system only works if the evaluation system actually measures what it purports to measure. The current “new” systems in place across the country do not do that. Linkage to student data is spectacularly weak. We start with tests that claim to measure the full breadth and quality of students’ education; they do not. Then we attempt to create a link between those test results and teacher effectiveness, and that simply hasn’t happened yet. VAM attempted to hide that problem behind a heavy fog bank, but the smoke is clearing and it is clear that VAM is hugely invalid.

“So, having an argument about how to best make use of teacher evaluation data based on student achievement is like trying to decide which Chicago restaurant to eat supper at when you are still stranded in Tallahassee in a car with no wheels. This is not the cart before the horse. This is the cart before the horse has even been born.”