Last spring, the Néw York legislature passed a budget that included a harsh and punitive teacher evaluation plan. This was done at Governor Cuomo’s insistence because he was angry that the state teachers’ union did not support his re-election in 2014. There were no hearings, discussions, or debates on the governor’s plan. It was passed because he wanted it.
The following comment was sent by Lisa Eggert, a specialist in education law who lives in Néw York. She wrote in response to this post.
“Thank you Diane for posting this! And here’s more to say about whether “this is the law”–
“1. The law (Education Transformation Act) required that the Regents pass rules on the evaluation plan by June 30, 2015, which they did, so they will not be in violation by voting no. The legislature surely realized that the tight time constraint meant that only temporary 90-day emergency rules could be passed, and it did not require a subsequent rule to be passed when the emergency one expires.
“2. It’s the job of the Regents and Ed Dept to set the plan’s cut scores that determine who is effective and who’s not. The plan of now sets an effective rating at a whopping 75% of students meeting targets. The School Administrators Assn. suggests 55%. What science or research supports 75%?
“3. The law actually requires that the public be told all the specifics regarding research and studies on which the plan is based, when the Ed Dept publishes a Notice of Rulemaking (the Notice is also required by law). But when the Ed Dept published the Notice, it gave a non responsive answer, identifying no study or research and just acknowledging that it had to work with experts. This is a legal violation of NY’s State Administrative Procedure Act, which protects the public’s right to have input into rules that have the force of law.
“4. The law is also being violated because the Admin Pro Act requires that any member of the public who asks be allowed access to any underlying studies. The Notice says to contact Kirti Goswami at the Ed Dept. I’ve emailed and spoken with her several times to find out how to access any underlying studies supporting the plan or, alternatively, to confirm that in fact no studies or research were relied on in creating the plan. She has been unable to provide anything or confirm anything, all in violation of the Admin Pro Act. (It feels like an awful run-around.)
“5. So, in talking to the Regents, feel free to point out that yes, the law is being broken –the law that protects the public’s right to understand and assess proposed rules and give input. I don’t mean to sound hoaky but this is the law that protects the democratic process, giving the public a voice when unelected officials, like the Commissioner Elia and the Regents, make rules. The Regents need to stand up for these laws that protect our basic rights.
“6. And also, from the state’s inability to point to any underlying science, it strongly appears that these rules, including the harsh cut scores, are entirely unfounded. They should be voted down so that a researched-based plancan be created by experts.”
Diane Ravitch's blog 
Reblogged this on David R. Taylor-Thoughts on Education.
I wonder if other states (Florida) have similar laws?
Utah actually has a law that specifically allows opting out. But it’s not advertised, so few parents know about it. That way the legislature can have its cake, (“We support parents!”) and eat it, too, (“Look at the test scores! Public schools are awful.”).
Hopefully, her message gets out and people pay attention.
“Destroy the School to save it”
Destroy the school to save it
”The school reformer’s saw
“Destroy the road the pave it”
“Destroy it, it’s the law!”
The question becomes, Why have the unions not sued the regents as a result?
I am waiting for a frequent commenter on this blog to complain that this posting impugns the integrity of rheephorm educrats (and their political enablers) everywhere because it casts doubts on their willingness and ability to gather up & make available all those precious hard data points.
I mean, just look how well data collection & dissemination of the education establishment has worked and is working in NOLA!
😱
Yep, practically as bad as claiming that the LATIMES and NYTIMES have generally been partisan players in the ed debates.
Ok, I ‘fess up: Guilty as charged.
And proud of it.
😎
Rules and regulations regarding education should have some evidence based requirement. Otherwise, special interest groups, as in the case of “reform,” can come in a dictate policy based on their own agenda, bias or madness. Students in the state should be protected from unfounded, political or religious notions about what is appropriate for students in public education.
I think that the cut score issue is NOT a matter that can be settled by empirical research. Setting a cut score is a decision. It is a judgment about performances. Just using the word “cut” tells you that the people using the system have little regard for nuanced judgments. What is valued? Anything that can be measured by scorekeeping and related metrics.
There are, or should be, criteria for making decisions about metrics. Metrics do not speak for themselves and the process of setting criteria should be public–published, open to review and modification. The legal issues are not just about procedures, they are about the value system that the rules and procedures are intended to support and defend. The cut score problem will not be solved by empirical “research.”
Consider the absurd expectation for 100% proficiency on NCLB tests. That was called an “aspirational goal.” Not an ounce of research pointed to that goal as a feasible, wise, or prudent policy. The aspirational goal of perfection was an over-the-top expression of faith that impossibly high expectations would guarantee “breakthrough results” for everyone and also close the achievement gap, and improve our ratings on international tests, and possibly save the economy as well. That 100% proficiency measure, as acted upon by federal and state officials, ensured that everyone would: a) fall short of the expectation and b) try to comply and meet the target by any means possible even if there was no moral or educational justification for those contortions.
Setting cut scores, and using other performance measures to produce a stack rating of individuals and institutions has everything to do with valuing schemes that too often function as a zero-sum game, declares winners and losers, values survival of the fittest.
The myth that cut scores can be made “objective” by empirical research needs to be purged from the law. If any research is produced, I would like to see it, and see it dissected–laid out–to show the values that are honored.
When the social scientists co-opted measurement from the natural scientists and misapplied it to human cognition, it was like Adam and Eve taking a bite of the forbidden apple.
It was the original sin from which all other sins have followed.
And when economists took a bite of the apple, all Hell broke loose.
Those who think cut scores are an objective exercise supported by research should read Gene Glass’s 1978 paper on them. They have been abused since the 1970’s and are at best pseudoscience. http://www.gvglass.info/papers/standards/
Lisa Eggert, can you explain further on the “cut scores” in #2. What are the 75% targets? What are the 55%?
Q1: Are those achievement scores or growth scores?
Q2: Are the “targets” actually 75% passing or is there a target set for each school/district based on past performance and/or current SES?
Q3: Are the “cut scores” what kids need to answer correctly to pass the test? Or are these set for the targets in Q2 above.
Q4: What are the penalties for not meeting targets?
By law, the targets cannot be based on achievement. If 100 percent of seniors taking physics pass the physics Regents exam, but their teacher has projected targets a student may miss by one point (sometimes because the numerical score does not exist on the state conversion chart) that is counted against the teacher. It doesn’t matter if 100 percent pass, ONLY meeting a speciously-set target counts. Absolutely invalid. The penalty for not meeting the target could be a rating of ineffective. get two of those, and a teacher has a hearing and faces being fired.
Nimbus: if what you wrote is true, then—
It’s like saying that one of the greatest heroes of self-styled “education reform,” Michelle Rhee, didn’t take “her” students from the 13th to the 90th percentile.
And that would mean that the entire rheephorm experiment is based on rigged measure-to-punish schemes.
Say it isn’t so…
😎
The proposed rule has the following about target scores. The rule doesn’t explain how the state determines how it calculates the state provided growth score. An “effective” requires a score of 15. To get a 15, 75-79% of a teacher’s students must meet their target scores.
(http://www.regents.nysed.gov/common/regents/files/meetings/Revised%20Subpart%2030-2%2030-3_0.pdf — pages 33-34)
********************
From the rule:
(3) Each measure used in the student performance category (State provided growth score, SLOs, State-designed supplemental assessments) must result in a score between 0 and 20. The State will generate scores of 0-20 for measures using a State- provided growth score. Districts shall calculate scores for SLOs in accordance with the table below; provided however that for teachers with courses with small “n” sizes as defined by the Commissioner in guidance, districts shall calculate scores for SLOs using a methodology prescribed by the Commissioner in guidance. For all other measures that are not State-provided growth measures, scores of 0-20 shall be computed locally in accordance with the State provided or approved growth model used.



SLOs
Percent of Students Meeting Target
Scoring Range

0-4% 0
5-8% 1
9-12% 2
13-16% 3
17-20% 4
21-24% 5
25-28% 6
29-33% 7
34-38% 8
39-43% 9
44-48% 10
49-54% 11
55-59% 12
60-66% 13
67-74% 14
75-79% 15
80-84% 16
85-89% 17
90-92% 18
93-96% 19
97-100%, 20
(d) Overall Rating on Student Performance Category.
(1) Multiple student performance measures shall be combined using a weighted average pursuant to subdivision (c) of this section to produce an overall student performance category score of 0 to 20. Based on such score, an overall student performance category rating shall be derived from the table below:

Overall Student Performance Category Score and Rating
Highly effective, 18-20
Effective, 15-17
Developing, 13-14
Ineffective, 0-12
Unbelievable – 75% of targets – and all of those targets are subject to huge standard deviations, AND are evaluated relative to a bell curve.
Nevermind that those targets are far from objective and teachers are NOT informed of what the expectations of them are until the school year is over and then they are evaluated relevant to everyone else.
Doesn’t it say something that teachers don’t know the expectations for each child at the beginning of the year – and that NYSED CAN’T produce that information? And then teachers are held for 75% of students achieving a standard they don’t know, for 100-150 children, with evaluations that are so defective at the very least of producing numbers that are consistent within the realm of what’s being tested for students (note I’m not saying student learning – just that by its own standards it’s inconsistent whatever they measure).
Frankly, this is bulls–t. What if they set the goal at this 75% and mass numbers of teachers fail?
The bad press alone will dry up the pool of teachers faster than it is now. That they have the ability to go back and change expectations SAYS this isn’t scientific. This is manmade crap made by people who are not experts in psychometrics and just need a number that sounds good for the public and that we’re not allowing crap teachers in the classroom.
The problem is that those political numbers effect real people who will be hired, fired, and not enter the profession based on them. In the name of protecting the children they are very haphazardly hurting a lot of those adults charged with caring for those children with saddling them with unknown expectations that are likely out of their realm of influence.
I really think we need to call it psycho metrics, not psychometrics.
Yes, and the practitioners of all the pseudo-science involved in scaling should be called psycho magicians not psychometricians.
You’re right.
It isn’t science. Has nothing to do with it, actually.
It is complete and utter bullshit and the ones making up the growth models and forcing them on teachers are bullshit artists extraordinaire.
And make no mistake. What these people are doing amounts to scientific fraud, which makes them fraudsters as well.
Hi Diane,
Need to clarify something you said in the first paragraph. In June or July of 2014, Governor Cuomo sat on a Forbes panel and basically said, in front of your friend and our national union President, Randi Weingarten, that he wanted to break the public education monopoly. From what I can gather, Randi kept this information to herself and most members didn’t find out about Cuomo’s comments until the videotape was released in December, after the election.
Even though we weren’t aware of Cuomo’s comments made at the Forbes panel, a week or so before the election he repeated his desire to “break the public education monopoly” publically at least twice. Members were outraged yet Randi Weingarten actually defended (or as Leo Casey maintains “dismissed”) these comments as “campaign rhetoric”. Randi maintains that she sent a personal letter/email to Governor Cuomo. Why a personal letter? Aren’t members entitled to see what our national union President said to stand up to Cuomo’s attacks?
Thirdly, and to me this is unforgiveable, knowing that Cuomo had vowed to destroy the public education monopoly at least three times, and after NYSUT had voted to refuse a Cuomo endorsement, Randi Weingarten made robocalls for Cuomo’s running mate Kathy Hochul. Randi’s actions were unanimously deemed “wildly inappropriate and undermining” by NY local union presidents.
So, as horrible as Cuomo is, Randi Weingarten, who is PAID a very handsome salary to represent teachers, is just as much to blame for the current situation we find ourselves in.
Randi’s allegiance to the Democratic Party at all costs, helped re-elect a Democratic Governor who is following through on his campaign promises to break the public education monopoly.
I believe Randi is detrimental to our union and should be removed from office but I know it will never happen because the largest local, the UFT, is so undemocratically run and requires its 750+ delegates to the AFT to sign “loyalty oaths” where they agree to vote whichever way union leadership tells them to vote.
I don’t know how, but we must find a way to take back our union from the destructive forces of Randi Weingarten. If anyone has any suggestions, please let us know.
When we stop believing that social science is pure science, we will be done with the notion that it can and will tell us anything about a human being’s college and career readiness, success, growth, effectiveness, intelligence, etc…
An interesting article:
http://www.theguardian.com/science/2015/aug/27/study-delivers-bleak-verdict-on-validity-of-psychology-experiment-results
Mamie Krupczak Allegretti, nice article and I think it’s useful to see how reliable the results are. I have an article notched on my bcdavison twitter feed a couple years back showing that most drug trials fail their second or third tests. That happens because you have to take 95% confidence with a grain of salt. If I predict drug A will improve outcome 1, conduct a trial and it works, then I have confidence drug A will pass rounds 2 and 3. But, if I take drug B and see if it will improve any one of outcomes 2-100, I’m pretty much assured I will get a false positive on one of those conditions. Then, I can get early approval for sales while I am conducting an rounds 2 and 3 specifically to see if it improves the condition from round 1. Most of the time it does not because round 1 was a false positive.
It’s similar in social science where it is hard to develop similar conditions. That’s why some got positive results from these PBL tests with small sample sizes. There’s no legitimate evidence PBL works over time and on large scales. But I have an ideologue who pushed it down my district’s throat in a single year even though 60%+ of the teachers oppose it. We now have no more finals (I’m not talking about SOLs, I’m talking teachers aren’t given a 3-hr block to give final exams and midterms/finals must count less than 20% of that semester’s grade. I guess only homework is “reliable” anymore.
In any case, this fivethirtyeight.com article describes how CFR had its studies replicated exactly on other districts within about a year, even by critic Rothstein. So while I agree in general on your social science critique, it’s completely irrelevant in this case. Next question?
Lovely. Time to stop complaining and litigate the nys boe