Herb Bassett is a teacher in Louisiana. He teaches music, but like Jersey Jazzman, has the ability to understand statistics and how they work in real classrooms.
This is a letter that he wrote about Louisiana’s new teacher evaluation system, which is as incoherent as teacher evaluation systems in other states:
State Superintendent John White showed his true colors when he recently praised four FirstLine charter schools that “fell in the top 10 percent of Louisiana schools in terms of improving test scores, yet ranked fewer than 10 percent of their teachers highly effective.
‘Amazing results,’ he wrote.”
He did not mention that one of the four schools, while ranked in the 99th percentile of improvement, declared 68 percent of its teachers Ineffective. Most of its teachers are now on a fast track to dismissal.
In each of the other three schools, at least 69 percent of their Value-added Assessment Model (VAM) teachers ranked Highly Effective, but none received an observation rating of Highly Effective. Not one.
If the VAM computer model ranked so many teachers Highly Effective, why could the principals not find at least one example of Highly Effective teaching in an observation?
These results clearly do not reflect student achievement or teacher quality. They deserve condemnation, not praise.
What does this bode for teachers and students on the coming Common Core assessments? White has predicted that due to the “rigor” of the new standards, achievement scores will go down.
As strange as it seems, teachers will not see lower ratings under VAM – even with the dramatic drop predicted for student scores. The VAM computer model simply ranks the teachers from highest to lowest. No matter whether the scores rise or drop dramatically, there will always be a bottom ten percent ranked Ineffective and a top twenty percent ranked Highly Effective. These quotas were set by the Louisiana Department of Education. Yes, the Department arbitrarily decided that ten percent of teachers are Ineffective and twenty percent should be Highly Effective.
Then why does the Compass Report show that only four percent of all teachers are Ineffective?
The computer model does not rank all teachers. The majority of teachers are not subject to the quotas. The purpose of the Compass Report was to show the discrepancy, and to coerce evaluators of the non-VAM teachers into matching the VAM system quotas.
White, however, seems to relish the thought of evaluations that cut short the quota for Highly Effective teachers.
Superintendent White now controls the cut-off scores for the achievement levels on the new assessments. Having seen him praise unjustifiably low teacher evaluations, should parents trust him to decide whether their children pass or fail the new assessments?
I’m beginning to realize that the people who rig these evaluations do not know they are stupid, much like dead people do not know they are dead. It is the frightening, blinding stupidity of the bureaucrats who ultimately control us all.
Reblogged this on Crazy Crawfish's Blog and commented:
More evidence that our teacher evaluation systems are ridiculous and our leaders profess success regardless of the actual underlying data.
So scientific, this VAM. Very impressive.
This blog and others serve an invaluable function in pointing out the ‘man behind the curtain’ [WIZARD OF OZ].
The leading charterites/privatizers and their accountabully underlings love to cower people using “mathematical intimidation” when they present their “objective” ratings systems but strangely [?] fail to mention the many subjective judgments behind VAM scams.
At the risk of oversimplifying: what goes into the VAM calculations and what gets left out? Those decisions are called “judgment calls.” In this particular case, I pose a very simple question: why was 10% chosen for “Ineffective” and 20% for “Highly Effective”?
What was the ‘objective’ and ‘scientific’ and ‘obvious’ reason for those percentages?
I am not a betting person but I would love to make some easy money. I am willing to wager that one of the most oft-repeated responses would be [if we could get the proper officials to comment] “that’s not my department, you have to ask the experts” — i.e., a non-answer.
Just consider the comment by “Vernon Coffey, Director of Grainger County Schools and former Tennessee Commissioner of Education” during TN House Education committee hearings in 2004 on the pioneering VAM system called Tennessee Value Added Assessment System. He stated “that he believed TVAAS to be as reliable and valid as SAT and ACT, while admitting that ‘I don’t understand all the numbers, but I’m not supposed to.’” (quoted in THE MISMEASURE OF EDUCATION, Jim Horn and Denise Wilburn, 2013, p. 105)
Does the phrase “rights and responsibilities of citizenship” mean anything anymore or is that not part of “twenty first century cagebusting skills”?
Or perhaps the 10% and 20% are a repeat of what Andrew Lang pointed out so long ago: “He uses statistics as a drunken man uses lamp posts — for support rather than for illumination.”
🙂
@krazyTA – don’t forget too that SAS employs TVAAS and EVAAS software and refuse to share the process by claiming it’s proprietary.
The closest you’ll come to seeing the math, and computer programming, is in Millman’s “Grading Teachers, Grading Schools” (1997). Even then you’ll probably need advanced degrees in statistics and computer programming to figure out what in the world is going on.
This means we are being evaluated with a system that only a select few know and understand…multiple matrices, BLUE, BLUP, and other B.S.. It will take court orders to beat this system – we are that deep into it, and I’m not sure if a group of teachers will have the wherewithal to take this system to a judge.
What’s most sickening is the concentration of attention to test scores. A teacher can beat all the B.S. statistics by teaching to the test or doing even worse. There’s no variable in their regression process that can account for teachers taking matters into their own hands – none. That is called a lurking variable, and there are probably so many, many more that simply cannot be taken into account by these econometricans and psychometricians.
Those of us on the ground, that actually understand what’s going on, see this movement for what it is – an attack by people on public schools that BELIEVE (with no evidence) that our public schools are failures along with the teachers in those public school. There is in fact plenty of evidence to the contrary.
Let me leave you with this gem of a quote from Millman’s “Grading Teachers, Grading Schools” – its from one of the moron authors that reviewed TVAAS:
“There are at least a few heroes out there in the public schools. We should support those such as Sanders, Saxton, and Horn who are doing their best to identify them and reward them.” (Millman, 1997, p. 168)
-quote from Richard B. Darlington
Kind of gives you a clue as to their angle on things.
It is stupefying to see how far into Fantasy World they think they can take this. Arrogance is our friend. This is like trophy fishing. You have to have patience to properly set the hook so it does not get away and then be ready for a big fight. In the end with the information getting out and those with knowledge laying it down with documented facts their Fantasy Game is over. Example: Yesterday at an LAUSD Committee Meeting concerning Title 1 and Parent Involvement our friends and collegues from the California Title 1 Parent Union led by Yolande Beckes, from England, and Walter Richardson laid down the illegality of the district concerning Title 1 for more than two years which led to a $2.5 billion Federal and State lawsuit presently in litigation. A friend and myself personally served Maria Cassillas, head of the Parent Division, with that lawsuit. They must be confronted to make change. You must line up your ducks. Yolande intimately knows education internationally. Walter has been in education advocating for 50 years for parents. They have full training programs including Title 1 and the first “Parent Trigger” training we know of.
Focus like a laser beam.
Reblogged this on Parents of PVMS.
“The VAM computer model simply ranks the teachers from highest to lowest. No matter whether the scores rise or drop dramatically, there will always be a bottom ten percent ranked Ineffective and a top twenty percent ranked Highly Effective. These quotas were set by the Louisiana Department of Education. Yes, the Department arbitrarily decided that ten percent of teachers are Ineffective and twenty percent should be Highly Effective.”
That is scary. I didn’t realize that some of the state’s VAM was based on a bell curve. Glad that ours in OK is being held up… hopefully, it will never make it at all.
But seriously, there is no good or meaningful purpose to assume that only 20% of teachers are highly effective and 10% need to be dismissed. Would a legislator send their child to a school in which only 20% of students will ever get an A?
Asinine!
Welcome to Jindal and Whites Louisiana, where eduction policy continues to be identical to the Twilight Zone taking place inside the Bermuda Triangle.
What happens when you declare 10% of your teachers automatically ineffective?
How do we “know” that every year at LEAST 10% of teachers should be ineffective – that means every teacher is actually in competition with every other teacher for a coveted “above ineffective” spot which is partially decided by which community they decided to work with and how needy they are..
If I was working in Louisiana, I would now have every incentive to try to not help any of my colleagues improve since what if their students “grow” more than mine and I could end up close to that 10% cut off?
Absolute insanity.
Join the club – what good are PLC’s now? I’m not sharing anything. I’ve got to feed my kids.
Here’s a whole bunch of reasons why all school evals are invalid. What are they afraid of? http://savingstudents-caplee.blogspot.com/
Apparently, higher education faculty are still spewing the poison…What is it that blinds them so?
http://www.huffingtonpost.com/marc-f-bernstein/raising-the-bar-for-teachers_b_3915418.html?utm_hp_ref=%40education123
I looked at Compass Final Report for 2012-2013 for the schools mentioned by Herb and downloaded the data.
http://www.louisianabelieves.com/teaching/compass/compass-final-report-2012-2013
Herb’s critique makes too many assumptions. First, this is apparently the first year for Compass so one should expect a host of teething problems used to what appears to be a complex process. Second, Herb’s focus on the FirstLine Schools and the absence of any “Highly Effective” teacher ratings raises another reasonable explanation: FirstLine expects more from it teachers and rates harder. Certainly the fact that their schools are ranked 2nd, 4th, 5th and 7th out of 26 K-8 Schools in RSDNO makes it worth closer and objective scrutiny. Third, the Compass assessment process appears to set guidelines for rating teachers rather than fixed percentages. (See:
http://www.louisianabelieves.com/docs/teaching/compass-briefing-presentation.pdf?sfvrsn=2 )
This means that no definitive conclusions can be made about the distribution of ratings without more details.
Compass may or may not be an effective tool for improving schools in Louisiana. FirstLine may or may not be leading their schools effectively. I think Herb and others need to wait for more data or cite additional data before drawing such dire conclusions.
Bernie 1815,
Thanks for looking into this. This letter was prepared to fit the length required for publication as a letter to the editor (in Louisiana) and therefore assumes that the reader is aware of some local context. Let me share more details here:
For background info, my letter is a response to this press release by LDOE where Supt. White praised 18 high-performing schools with low teacher evaluations. My quote is from a Baton Rouge Advocate article where Supt. White singled out the four schools. That is why I focused on them.
Louisiana’s Compass System gives teachers a final rating based on two components: 1) Professional Practices – i.e. principal observations using a modified Charlotte Danielson Rubric – and 2) Student Growth Measures – VAM for teachers of 3-8 ELA, Math, Science, Social Studies, and Algebra I and Geometry, or teacher-written, principal approved “Student Learning Targets” for non-VAM teachers.
Each counts as 50% unless the score in either component is “Ineffective”. Even a Highly Effective (top rating) on one half is overridden if the other half is “Ineffective”.
Four rating categories are given in the Compass system: Ineffective, Emerging, Proficient, Highly Effective.
So, to address your third point, the ten percent failure (“Ineffective”) quota is indeed fixed by LDOE, since an “Ineffective” VAM ranking overrides the Professional Practices score. LDOE has given principals the flexibility to adjust a teacher’s VAM ranking between the categories of Emerging and Proficient, but not to override an “Ineffective”. LDOE has never provided a rationale for the 10 percent failure quota.
Now, the Compass Report also notes that about 8.5 percent of VAM teachers rank as Ineffective. Why is it not ten percent?
Some VAM teachers teach in two or more different content areas. The failure quotas go by each content area. (All the Math teachers are ranked; all the ELA teachers are ranked separately, etc.) These teachers essentially get a second chance within the VAM system. If they are “Ineffective” in one content area, they may score high enough in another content area to pull it up. (LDOE said it used a weighted average of the percentile ranks, but removed that info from their website just before release of the Compass Report).
So teachers of multiple VAM content areas appear to get a break; by my best estimation they have about a five percent chance of ranking “Ineffective”, while teachers of a single VAM content area are subject to the ten percent quota. LDOE has inadvertently created a tiered system: about one percent of non-VAM teachers are rated Ineffective, multiple VAM content-area teachers probably rank “Ineffective” at about five percent, and the single content-area VAM teachers are subject to the ten percent quota.
And to top it off, Supt. White in June 2013, on his own authority, “held for further review” the VAM rankings of “roughly 50 teachers statewide” because he did not like their results. That is about three percent of the teachers ranked “Ineffective” by VAM. However, in January, White prevented our State Board of Education (p.19 in the linked document) from making a rule to do exactly what he did in June. He promised that LDOE could adjust the VAM system to avoid a particular problem. Obviously, LDOE failed to do so because the problem still existed in June.
Now to your second point. I believe (but cannot prove) that the FirstLine Charter School principals intentionally gave low ratings for the purpose of showing just how “rigorous” they were being.
LDOE made its first annual report to our legislature on the Compass System on March 1. That report called for work to be done so that the principal’s evaluations would line up with the VAM results to provide better feedback (p. 13 in that Compass Report, not to be confused with the current September Compass Report).
Here, in the FirstLine schools, the principals clearly have not aligned their observation ratings with the VAM results. 69 to 77 percent of the VAM teachers were ranked Highly Effective, but none were recognized as Highly Effective in their observations.
Since White has called for an alignment of ratings, he should have condemned the mis-alignment.
Technically speaking, his praise was targeted at the teachers’ low scores on the Student Learning Targets, but these, too, are subject to influence from the principals. As a matter of responsibility, he should have looked closer at the data and noticed the extreme non-alignment of principal observations with VAM ranks. The teacher evaluation system was promoted as a way to reward great teachers. Think what you want, but I take from the data that rewards for great teaching have been withheld. In three of the four schools, no teacher was given the final rating of Highly Effective.
While I disagree with what I believe these principals have done, my objection is to Supt. White’s choice to hold them as examples for teacher evaluations. Supt. White had no obligation to praise these schools for their Compass system results.
Your first point was that these are teething problems. This spring, the LA House of Representatives voted unanimously to put off the Compass results for this year. Four Senators on the Senate Education Committee blocked it from consideration in the Senate. My opinion is that the system should have been put on hold until the problems were worked out. But that may never happen with this system.
The purpose of the current Compass Report was not only to show how teacher evaluations are not uniform throughout the state, but more importantly to show that the ratings given by the principals were not as harsh as the arbitrary quotas set by LDOE for the VAM system. Supt. White chose to praise these schools’ teacher ratings not because of their rigor, but because they were more harsh than the quotas.
Bernie 1815,
Sorry for the missing links. Here they are:
http://www.louisianabelieves.com/newsroom/news-releases/2013/09/05/18-high-achieving-louisiana-schools-set-highest-bar-for-student-learning-using-compass-tool
http://theadvocate.com/news/6985949-123/teacher-testing-shows-performance-gaps
http://archive.constantcontact.com/fs129/1104839106897/archive/1113911596984.html
http://www.bese.louisiana.gov/docs/bese-official-minutes/2013-01-16-bese-meeting-minutes.pdf?sfvrsn=4
http://www.louisianabelieves.com/docs/teaching/2011-2012-value-added-report.pdf?sfvrsn=2
Herb B.
Thanks.
Herb:
Thanks for the thorough and thoughtful response. I will read White’s comments and respond more fully.
I will briefly note a couple of general things about this type of performance management system. ( I was involved in advising clients on performance management and stack ranking assessment procedures for nearly 30 years, though not in schools.)
First, this type of large scale and novel process really needs to be implemented incrementally – even if it has had a successful track record in another State or major School District. There should also be some pretty grounded, case by case discussions of the outcomes of the process, i.e., were those rated as ineffective actually ineffective and to what degree.
Second, this type of system is predicated on having effective and respected principals and administrators. Going back to train folks after a system wide implementation is not good for anyone. Putting this type of system in the hands of weak leaders is asking for big trouble. Whatever the system, it needs to be applied to this level of management first. Bluntly, they have to eat their own dog food.
Third, I am not opposed to fixed percentages of individuals rated as ineffective and highly effective. Frequently it is the only way turn around poor performing organizations and to drive more consistency of management and supervision through large organizations. Those percentages, however, have to comport to some commonsense assessment of what is happening in the organization. At the bottom end, if measurable outcome results are good and there is a real learning learning curve, then 10% in this bottom category is way too high. 20% is way too large for the top category under most circumstances. Such a large % will quickly lead to legitimate questions about the process, essentially because the difference between the person in the 20th percentile and the 21st percentile is likely very small. We would always argue for 10% or lower at the top end. Most folks will accept that they may not be the very best and have no issues with recognizing the really outstanding performers. If you have a very poorly performing system then the % rated as ineffective should be based on the shortfall in performance and the organization’s ability to select and train replacements. Given other sources of turnover, my guess is that 10% represents the upper bound for a troubled school. If the same % is kept for 3 years an effective principal will have a much stronger staff with better results and can drop that percentage dramatically. In other words there should be no mandatory District wide designated percentages.
I will get back to you after I have read White’s comments.
Bernie 1815,
I know this issue is of limited relevance, but here is my e-mail to White about his June decision to hold the VAM rankings and his response. I had this exchange with him before I discovered that he had pulled from BESE consideration the rule which would have covered these teachers. Note that he did not answer my very specific questions with data, and I still hold that if there was no bias, there was no reason to hold the VAM rankings for further study. Either the VAM computer model gives reliable results or it doesn’t, but there is no middle ground for looking at the results after they are in and throwing out just the ones you don’t like:
Superintendent White,
This email regards the Compass Process FAQ July 8 update addressing a concern that the VAM system did not accurately reflect the effectiveness of certain teachers:
A very small number of teachers’ scores (roughly 50 statewide) are undergoing further study before they are finalized. In these cases, the majority of students were performing at ‘advanced,’ ‘mastery, or ‘excellent,’ but the ratings placed the teacher as Ineffective. These teachers’ value-added data will not appear in the Compass Information System. Evaluators should use student learning targets to assign the student growth rating for these teachers.
1) You did not give the necessary context to support the contention that the VAM rankings inaccurately reflected those teachers’ effectiveness. LDOE materials have specifically stated that it is a myth that “teachers of high-achieving students are at a disadvantage because they may not show growth with students on the value-added model.”
What was the total count of teachers AT ALL LEVELS (Ineffective, Effective Emerging, Effective Proficient, and Highly Effective) who had a majority of students score Excellent, Advanced, or Mastery? This information is needed to show that VAM disproportionately ranked this class of teachers as Ineffective.
2) The decision is unfair to similar teachers in the 11th to 20th percentiles of VAM. Their highest possible score would be lower than that of the teachers in the bottom decile.
Some of the teachers whose VAM rank was Ineffective (1st-10th percentile) will receive the final Compass rating of Highly Effective since they will be evaluated on their SLTs instead of their VAM rank.
Similar teachers in the 11th to 20th percentiles could at best could receive the final Compass rating of only Effective: Proficient. By current Compass rules, they must receive a score of 2.00 on the Student Growth Rating. Given a perfect Professional Practices score of 4.00, such a teacher has a maximium possible overall Compass rating of 3.00 – Effective: Proficient. These teachers would be affected by the same inaccuracy (if it exists) and arguably be better teachers.
Teachers in the 21st to 80th percentiles at least in theory could achieve a final rating of 3.50 Highly Effective. Even this would arguably be harder to achieve this through the use of VAM ranking (they would get a 2.00 or 3.00) than by SLTs (they could get up to a 4.00 or 3.50) for the Student Growth Measure.
Possible solution – at a minimum, extend to the 20th percentile such teachers who will have their Student Growth Measures based on their SLTs.
Otherwise, there would be the potential for appeals and/or lawsuits as there would be a documentable monetary effect on teachers in the 11th to 20th percentiles who would have received a rating of Highly Effective if they had fallen in the lower decile. Remember that pay scales now reward teachers differently based on their evaluation results.
The best solution would be to make VAM results be for informational purposes only to all teachers and administrators – not just the teachers of high-performing students.
Herb Bassett
Herb, thanks for the note. Out of 50,000 teachers, and out of 16,500 who received value-added data, there have been a small number of instances where I have asked for the data to be reviewed further. In the aggregate, yes, there are not pronounced biases of the kinds you discuss. But, whether as a matter of policy or a mater of procedure, there are bound to be some instances in which we really need to understand better what has happened and whether the data should be used to determine someone’s rating.
I think that’s a fair position to take.
John White
Louisiana Department of Education
Twitter @LouisianaSupe
Plain scary!
I hope people aren’t thinking this is just Louisiana’s problem.
The VAM scores are only one measure, right? So, you’re all assuming that the rest of the “multiple measures” is something cogent, like observations? No, it isn’t.
Here in Massachusetts, I have to set my own quantitative “SMART” goals, in addition to the VAM component, and they must be “data driven”, based on some kind of scores on the nearest commercially administered standardized test. Then, I have to specify goals in my teaching to improve those scores. That’s the non-VAM part of my evaluation. Confused? Here is a clarification, form a longer email I got Wednesday at work:
“The data I was referring to is from part 1 of the Self-Assessment (we skipped this part last year). Regulation 603 CMR 35.06 (2)(a)1 requires that teachers complete: “Part 1: Analysis of Student Learning, Growth, and Achievement: Briefly summarize areas of strength and high-priority concerns for students under your responsibility for the upcoming school year. Cite evidence such as results from available assessments. This form should be individually submitted by educator, but Part 1 can also be used by individuals and/or teams who jointly review and analyze student data.” This data does not need to be connected to last year’s SMART Goals but may be depending on the goal the teacher selected. The data should be aggregated – not individual.
Some teachers will have data on their current students available immediately through EDWIN (we will ALL be learning about this on the September 27th PD day).
(anticipating next question): Even though the regulations cites “high-priority concerns for the students under your responsibility for the upcoming school year”, educators can use data linked to other students to inform practice for THESE students.
For example, if my MCAS data from my students last year shows my kids struggle with Open Response questions, this is a high priority concern for my current kids. I would cite this data (perhaps my statement would be “only 4 of my 28 students earned 3 or 4 points on open response questions last year”) and select my new professional practice SMART goal to be “increase my knowledge of and the effective implementation of tasks of high cognitive demand.”
Well, hell, who is opposed to increasing knowledge of and the effective implementation of tasks of high cognitive demand? Wait, though. How exactly will I be evaluated, though?
VAM rhymes with SCAM
This is the exact thing that happened at Microsoft with the same attitude. It stifled creativity and people torpedoed each other to prevent changes in who got canned or ahead. So we have the problems with their stuff we do now. I hate Windows 7 when it goes somewhere and I did not click it. Gates, I know more of what I want than you or your software does. This is your problem: You think you know it all. No one does. You think you know all about process. You would not last a minute at the old Skunk Works with Kelly Johnson with that attitude. Think about what he did ahead of schedule and under budget. Do you know that with the first deliveries of the U-2 he gave the govt. money back and more planes than the contract called for. Now, that is a real hero to me. I worked there when top secret on one of the hottest parts. No where were all treated with respect like there. Be professionals, we know people make mistakes, just tell us if that happens and nothing will happen. If you hide it you are out the door. Fair enough for me on a top secret project. And talk about security.
A wonderful characterization of good management and the antithesis of usual “government work.” Has ANY public school system EVER delivered on time and under budget Has any government agency other than the Skunk Works ever done the same?
I know people who have turned totally failed schools around rapidly and they all do the same simple thing. You bring in real education and show them they are the most important. Nothing is 100% like saying all will graduate. That is an outright lie. You will get the rate down as low as you can is truthful. Why don’t they put these people in a room to come up with a plan? I mean only those who have taken the worse schools and done a dramatic turnaround like happen a few years ago at Sun Valley Middle School. One of my employees at the time had their daughter there before the takeover and it was a full on mess. Only three people were changed: The principal and the two vice principals. The two vice principals gave up principals jobs to do this and prove what they had dreamed of proving. That is power and $30,000/year at least. In two weeks 200 truancies to 20. Also, no more graffiti or trash. They handed out tickets for the weekly drawing in which you could even win a computer for yourself if you have a ticket. You got a ticket if they saw you picking up a piece of trash and putting it in the can and if you were reading during lunch and break. I took a two hour tour of the school with one of the vice principals during the lottery one Friday. Amazing. Motivation is a key.
Later, they moved these people to the district offices and the school once again fell into the pit. The Fish Rots from the Head.
There never has been and never again as far as we can see will another Skunk Works like Kelly Johnson’s exist. My dad worked for him from 1937. He was a brilliant anomaly. He was an expert in every field and the control freaks could not stand him. After all, he did it right and made others look like fools. If they tried to get him to do something that just was crazy he would give them back their money and tell them they are not going to waste their time on that. Imagine that in education today.
Well, Harlan, help us get back control of the Superintendent of Instruction at LAUSD. If the public has input on this position at that district it will spread nationwide through demand to not be hoodwinked again by having someone incompetent shoved down their throat. What do all of you think? Should we stop them again at LAUSD? You helped us put board member Ratliff there and now there is upset with the billionaires, should we give them another headache?