A reader who teaches in the District of Columbia wrote to ask my advice. He couldn’t understand his evaluation based on the District’s complex IMPACT evaluation system. I sent it to a teacher evaluation expert, Audrey Amrein-Beardsley. She wrote a post trying to make sense of the evaluation report.

Categories District of Columbia, Teacher Evaluations

33 Comments Post your own or leave a trackback: Trackback URL

Peter says:

December 21, 2013 at 7:16 am

Thanks for the link. Somehow I had missed Vamboozled on my required blog reading list, but it’s great

Reply
Dan McConnell (@DMaxMJ) says:

December 21, 2013 at 7:38 am

I have heard suspicions expressed that administrators are being pressured to identify ineffectiveness whenever possible, and supporting success isn’t really the purpose.

Reply
- bookworm 23 says:
  
  December 21, 2013 at 8:50 am
  
  I believe that’s true, and happening in other places across the nation. It’s not just this evaluation system. It’s all a game.
  
  Reply
  - Jane says:
    
    December 21, 2013 at 9:02 am
    
    Last year, during my first APPR evaluation, in answering a question, a seventh grade student of mine said she didn’t know what a canoe was. (The student later came up to me and said she actually did know, and apologized, because she gets nervous speaking in front of the class.) But the observing administrator spent 10 minutes discussing this point in our post-evaluation meeting. He suggested that I always carry pictures of canoes with me when I do this lesson to avoid such confusion. I had to breathe deeply to keep from laughing. Surreal!
  - bookworm 23 says:
    
    December 21, 2013 at 10:27 am
    
    Lol!!! Good grief, that is SO ridiculous and yet not at all hard to believe!
- jean sanders says:
  
  December 21, 2013 at 11:08 am
  
  Dan, you are correct and it has been going on for years. A friend of mine was hired as middle school principal with the express interest that she knew how to fire teachers.
  
  Reply
artseagal says:

December 21, 2013 at 8:08 am

It is clear from reading both of the “evaluator’s” comments that the evaluation is a highly controlled and extremely “directed” survey in typical “ed reform” style in which the entity creating this “survey” wants very targeted information in very targeted language. The main problem is that the survey is only as good as the entity who creates it. I recently heard an individual administer a “survey” which will measure the student’s “opinions” if their teacher for a group of special education students in an upper elementary grade. The students didn’t know why they were doing this evaluation on the named teacher (they were not in her classroom at this time but had switched into another teacher’s class). They did know that they were answering questions about their teacher. Just to give you one example of the ridiculousness of this.. they were asked a question “Do you feel you understand the material introduced after the teacher has given the lesson?” They had 4 subtly different choices that went something like: I always understand the material introduced, I frequently understand the material introduced, I sometimes understand the material introduced.. One student asked.. what is “material”.. another student wanted to know which was the correct answer since they had been told it was not a test… As each question was read to the students (most read at a very low level), I could not believe my ears! And then I think about how the teachers who teach special ed are forced to teach curriculum at grade level that is WAY ABOVE the abilities of most of their students and the same students are forced to take high stakes tests WAY ABOVE their abilities. So if a student answers that no, they do not understand the material after it is taught, whose fault is it really? I would answer… the entity that imposes this forced one-size-fits-all curriculum on students. All the teachers in my county (myself included) are going to be forced to have a portion of their “evaluation” based on leading and obtusely worded questions and answers. How many students hate school nowadays thanks to all the testing and prep for testing? Who is responsible for this tragic disengagement? I answer that is surely is not the teachers whose hands are tied and are forced to teach in ways THEY NEVER WOULD NORMALLY. So as students answer these questions they either have to lie if they like their teacher and can figure out why the questionnaire is being given or they tell the truth and it reveals a scathing reality of THE MESS OF THE ED REFORM CURRICULUM!

Reply
- bookworm 23 says:
  
  December 21, 2013 at 9:01 am
  
  You have really put this very well. I am a special ed teacher and it means so much to know that there are others who understand the particular plight of my students. Everything you’ve written here is spot-on. Thank you.
  
  Reply
schoolgal says:

December 21, 2013 at 8:37 am

There was a big discrepancy between the principal and master teacher. This is why evaluations should be following the PAR method. This lesson would be discussed with the PAR committee. Then the teacher would be brought in to defend the lesson as per due process. The principal could in fact be overruled on the U-rated or in this case 1 decision. If the committee did agree that the lesson was ineffective, they would provide the teacher with assistance if they thought the teacher had potential.

Reply
- jean sanders says:
  
  December 21, 2013 at 11:12 am
  
  The instrument being used “stinks”; adding a committee to that does not make the instrument valid. Committee cannot take “trash” and turn it into factual information. I know I am using vernacular here; I just want to specifically say that a committee cannot clean up a poorly chosen or designed instrument. The committees I have attended are whoever happens to show up that day; or, do you want a “supreme court”. I don’t think the way you describe the committee is how “due process” works in the school districts that I am familiar with but then I realize my experiences are unique to me.
  
  Reply
- Ellen T Klock says:
  
  December 22, 2013 at 6:04 am
  
  You make it sound like the teacher is on trial, or, even worse, they are awaiting sentencing. Is being grilled a part if a teacher’s job description.
  
  “You stated the objective but you did not point to it on the board as you spoke.” Was a comment for a teacher who was observed by 4 administrators from downtown plus the principal. An example of our surprise “walk throughs” to make sure we were teaching what was posted on our schedules. She was literally shaking afterwards. One of my favorite teachers. Sigh!
  
  Reply
GE2L2R says:

December 21, 2013 at 8:54 am

Could it be an instance of Educational Dissociative identity disorder, EDID. Or perhaps EDID not . . .

Reply
- Duane Swacker says:
  
  December 21, 2013 at 10:26 am
  
  TAGO!
  
  Reply
carolcorbettburris says:

December 21, 2013 at 9:25 am

A few thoughts. First, this may well be the result of the observers observing two different classes. That would explain the differences in student behavior. Because Mr .T does not get a growth score, he is probably a secondary teacher with different classes. Second, they were two different lessons. Even master teachers have an off day. Third, one may have been announced, and the other unannounced. (That can swing both ways. I have seen teachers blow announced observations because of nerves or do sloppy lessons when they don’t expect to be observed).
Impressions–the second observer makes statements but provides no evidence. The first observer at least makes an attempt (heads down). I am in my second year using this kind of observational tool. It is unhelpful and tedious. I now go through the motions and I have returned to what I have always done–have serious conversations with the teacher about what I saw and what the teacher experienced and saw in the lesson. This is then reduced to a few concrete recommendations if needed. Sometimes the conversation is about all the things the teacher did so well. What you expect to see is that recommendations are put into place over time. When they are, I have seen great growth–especially in novices.
I have observed teachers for 18 years. Here is what it is all about–good time management, clarity and student engagement that is continuous and challenging.
The first two define a good teacher. All three define a master teacher.

Reply
- readingexchange says:
  
  December 21, 2013 at 9:54 am
  
  Carol, if I read the article correctly, both observers saw the exact same lesson at the same time.
  
  “… NOT on the same evaluation page, again, when observing the same thing during the same time at the same instructional occasion.”
  
  Reply
  - Duane Swacker says:
    
    December 21, 2013 at 10:34 am
    
    Why would you expect an administrator to be able to “close read”?
    
    (and Carol that is not a stab at you as you’ve proven yourself to be what an administrator should be, just my general disdain for about 99% of the administrators I have known.)
    
    Hell, one of my administrators put down that I didn’t implement district curriculum in my lesson. I asked him if he had read the curriculum. Yep, you know what the answer to my query was: “No, I didn’t even know where to find it, I had to call the assistant superintendent to find it.” He could have asked me as I had just written it earlier last summer. Reality is definitely stranger than fiction.
  - Duane Swacker says:
    
    December 21, 2013 at 10:35 am
    
    And folks, don’t get me wrong as I’ve misread things many a times before-I think I’m human although some would dispute that.
- LG says:
  
  December 21, 2013 at 10:41 am
  
  We have been told that that knowledge of the curricular continuum and a clearly stated objective/purpose are required of all effective teachers.
  
  The number one question children have on their minds when they enter my classroom is, ” What are we doing today?” I strive to carefully set the expectation for my students that they will know right up front what we are going to be learning–whether or not we are going to play any games, sing any songs, or play instruments. I work to provide this to students on their developmental level.
  
  In my district, we have a very specific expectation that students must not only know the objective of the lesson, but they must be able to clearly state it when questioned by the observer, a practice that I find inconsistent and often times pedagogically unsound. Young children may not be able to fully articulate nor understand the true purpose of every lesson, nor should they always need to. With my youngest students, I don’t include the fact that we are going to be moving to a specific pattern to prepare them for reading notation that will be used for an instrumental ensemble. I do not tell them that our vocalise exercises are building the kinesthetic and aural kills necessary to match pitch. Yet an observer with zero knowledge of music pedagogy might not get what we are doing, and my score will reflect that I’m just not all that effective in that category. I need to spend the time educating the observer on the musical practices of my classroom. This would happen no matter what kind of evaluative instrument was being used, but this time, a number is given to me by a judge who is not an expert, and this number is a reflection on my worth as an educator which becomes a tangible piece if evidence as to whether or not I should be in a classroom in the first place.
  
  An administrator should be able to account for the variables in the job. One the biggest challenges of music certification stems from the fact that we are supposed to be experts on child development due to a K-12 certification. The powers-that-be decided to cut a position and moved me from middle school to elementary because my certification allowed it. It was a political move, I found out later, but one that could not be questioned because there was no contract violation. I thrived as a teacher in middle school with students continually raising the level of their skills year after year, yet felt simply average when I was moved to elementary school.
  
  So how does an administrator account for the fact that a teacher’s strengths are not being utilized? We are expected to teach to the varying learning styles of our students, but the reality is we cannot always take teaching styles into account when assigning teachers to positions. I would never take our master Kindergarten teacher and put her in a fifth grade classroom. She does not belong there. Yet, as a professional, she might be asked to go there. Should her observations as a newly placed fifth grade teacher reflect that she has a lot to learn about teaching? No. She is an expert learning a new developmental level and curriculum. She will struggle in some areas but not in others. Yet, her evaluation may show that she is just average in some areas. Is this fair! Well, yes. As stated, she is a professional and expected to do the job. However, she has two years to be effective. A wise administrator once told me that it takes a good five years to really get to know a job. The state thinks that it should happen in four plus one day (at least for tenure), yet a teacher cannot have a learning period of more than two or else he could lose his job (despite tenure). Does all of this make any sense?
  
  Teaching is a personal art, and teaching skills are honed through practice, reflection, and collaboration…not numbers. We need to sever the high stakes connection and bring the evaluation process back to a constructive purpose.
  
  This evaluation system will serve to expose how little time teachers in this country have to devote to planning, preparation, and articulation with colleagues. Until we find a way to provide more time for reflection–as happens in countries like Finland, our teachers will continually be over-worked and ripe to burn out. This evaluation system seeks to destroy us, one by one. Even seasoned professionals with 30-40 years of experience in my district are being evaluated at a level 2 out of 4 by administrators who are still learning the system. Our teachers have taken time-consuming effort to defend their performances and have offered artifacts and rebuttals explaining the reality if what went on in their classrooms during the observation. We have frustrated teachers who are swamped by CCSS, and now they must put in even more time to defend their evaluations. Many are being told they need to “study the videos a lot” as if the time to do this will magically appear in the day.
  
  The best practice of our evaluators is to support us, defend us, and to fight a system that sucks the life-energy out of both the evaluator and the evaluated so that the process can be more stream-lined and productive instead of convoluted and destructive. Morale is down, yet we need to put on a happy face for our students. Enough is enough.
  
  Reply
  - Ellen T Klock says:
    
    December 22, 2013 at 6:25 am
    
    LG – it’s one thing to move a teacher to a different grade level because of a real need or at their request, it’s another to willfully change teachers around on a whim or as a punishment or to placate their favorites. Sometimes the principals like to play God, and often at the student’s expense.
    
    You were definitely out of your comfort zone, but I’m sure what you consider average was more than adequate. On a positive note (and here is my assumption you are male, if not my apologies), there are not enough male teachers in the elementary grades. I know schools who refuse to have a man in a classroom under fourth or fifth grade. What a waste – the best Kindergarten teacher I ever worked with was male. Some children lack male role models in their lives, and they need exposure to positive experiences. So out of all adversity (and as educators we live it) comes something positive.
  - LG says:
    
    December 22, 2013 at 10:21 am
    
    Ah, Ellen, I am actually a woman, but I grew up with older brothers and have been in many musical ensembles as pretty much the only female so I wonder if perhaps I sometimes come off as a “male” in my commentary? My husband might disagree. 😉
    
    In all seriousness, I absolutely agree that we need more men in elementary school. This year, we have two more men in my school, with one teaching Kindergarten. The other has a 4th grade homeroom where 50% of students have an IEP. The returning male staff member is a Phys.ed/health teacher who is the loving daddy of twin 6 year-olds. Our principal is only 1 out of 2 male elementary principals. We have 4 female elementary principals and 1 out of 2 male middle school principals. Every supervisor we have is female. We need more male elementary educators in general, especially in the classroom.
  - Ellen T Klock says:
    
    December 22, 2013 at 10:32 pm
    
    Sorry for the gender confusion. I should have known, if you were a male they wouldn’t have switched grade levels on you – unless you ticked somebody off.
    
    We had an EVIL principal that used to love messing with teacher’s heads. He’d use them for his personal agendas then get rid of them. Switching grade levels was the least of their worries. He knew how to use the system – there was always donated pizza and snacks at the faculty meetings. Still, bribes couldn’t make us like him.
    
    He retired this Fall. They posted the retirement flyer more than once. I always wondered if anyone showed up to his party.
  - LG says:
    
    December 22, 2013 at 10:57 pm
    
    No apologies necessary. I applaud gender confusion in this case as I do not believe genders are intellectually typical.
    
    I briefly worked under a principal that most people really did not like, yet I had no issues with her. She admitted that she had no people skills. She was all business, but that’s part of what I liked about her–there was no pretense, and you knew never to take anything personally. I didn’t miss the warm fuzzies since she kept it real, albeit a bit colder than I like.
  - Ellen T Klock says:
    
    December 23, 2013 at 6:03 am
    
    The best principals/administrators are the ones who inspire you to go beyond your best. I can count them on one hand. I would have done anything they asked and their responding encouragement and praise thrilled me to no end.
    
    I was on maternity leave. I was using up all my sick days, finishing out the year, and going to a new district (Buffalo). The Assistant Superintendent called me and asked me to help with the culminating activity for the Parp reading program I had helped set up for the district (about 8 elementary schools). I created a list of books for each grade, located a local vendor who had most of the books available (at a discounted price), went to the warehouse and personally selected the books, had them delivered to the cental office, sorted them out and personally went to the schools to make sure they were distributed properly (with help from the parents) so every elementary student received an appropriate book.
    
    Now – I was leaving. I was angry with the way the library program was run. The libraries were in locker rooms, on a stage, in a storage area with no seating, in the back part of a cafeteria, etc. I had three schools, traveled daily, was overworked, under appreciated, etc. I was more than ready to move on.
    
    However, I would do anything for this one particular man. His appreciation appeased my anger and I decided to go ahead with his request. It was a nice gesture for the kids, but if anyone else had asked, the answer would have been “NO!” But I did it – on my time, without pay, and with a newborn daughter.
    
    May we all be inspired to go above and beyond – for the sake of ourselves, our children, our country, and our world.
- jean sanders says:
  
  December 21, 2013 at 11:18 am
  
  Instruments of an kind need to be proven statistically reliable and valid. We had a wonderful questionnaire designed at Boston Children’s Hospital and it took two years to get it properly administered in a school district so that it would have high inter-rater reliability (agreement ). When essays are scored in an English class ETS showed the school districts how to check for inter-rater reliability. Poor design of instruments and the lack of agreement among/between individuals using the instruments (rubric, test, etc) needs to be carefully described… I haven’t done well here but it is important any type of measurement whether we are judging our students or people are judging teachers. I don’t want a neophyte standing over my doctor when he takes my appendix out; or my eye surgeon when he closed the retinal hole in my eye. I have always believed we are a profession…. these insane measurements destroy the profession. Now, that does not mean I am against accountability; I just don’t throw the baby out with the bath water. The public goes along with the drumbeat that “teachers need to be fired” and then the pundits and mandarins create these nonsensical designs that they implement in policy without appropriate safeguards.
  
  Reply
LG says:

December 21, 2013 at 9:32 am

[This] “supposedly ‘objective’ measure seems to be anything but.”

Evaluations have always been very dependent on the evaluator, except now the practice has been broken down in little categories with numbers, as if numbers are valid indicators of anything. Feedback needs to be useful, non-confrontational and frequent. Now reformers are expecting teachers to be miracle workers every minute of every day in every class. This is next to impossible.

Teaching is a diverse, human practice that cannot be encapsulated in a few observed lessons due to the varying nature of students, circumstances, content, and lesson material. Supervisors know this, yet many of them refuse to stand up for the education profession and actually defend this–instead, they have dutifully embraced the burden of these new evaluation trends.

Any educator worth his or her salt knows that these evaluations are meant to be harmful, not helpful. We are all becoming trained monkeys doing “specific” tricks for the best scored bananas or else we lose our job at the circus and have to go out in the wild training to be some other animal lest we get eaten.

Reply
- Duane Swacker says:
  
  December 21, 2013 at 10:51 am
  
  About time for a daily lesson from Wilson! (Jeez, it’s already almost 10:00 a.m. CDST).
  
  “. . . as if numbers are valid indicators of anything. . . ”
  
  “It requires an enormous suspension of rational thinking to believe that the best way to describe the complexity of any human achievement [teaching and learning], any person’s skill in a complex field of human endeavour [again, the teaching and learning process], is with a number that is determined by the number of test items they got correct [the number of points they got on an evaluation rubric]. Yet so conditioned are we that it takes a few moments of strict logical reflection to appreciate the absurdity of this.” [my emphasis]
  
  From Noel Wilson in “Educational Standards and the Problem of Error”, Chapter 15, page 4, found at:
  
  http://epaa.asu.edu/ojs/article/view/577/700
  
  For that gem and much more all should read and understand what Wilson has proven in this never rebutted nor refuted study. If one understands what he has written one can’t help but come away with the realization of the sheer absurdity, the sheer insanity, the complete invalidities involved in these types of regimes-standards, standardized testing and assessment and the grading of people whether students or teachers.
  
  Reply
Duane Swacker says:

December 21, 2013 at 10:23 am

That reminds me I need to buy a bunch of holiday Hershey kisses (hopefully on sale) so that when the newly instituted student survey of their teacher is taken by the students I can offer them a “pick me up” before doing it. They told us the date and which hour that they are going to take the students from our class and have the librarian proctor the survey, I’m assuming they will be doing it on a computer. And of course they just so happened to pick my worst behaving class out of seven.

And of course the survey will be part of our evaluation. You know, I may just have to go over the questions ahead of time with the students so that they know what the questions and responses mean. (We haven’t been told that we couldn’t do that, ha ha ha ha, gotta find that chink in the armor-eh!)

Reply
- jean sanders says:
  
  December 21, 2013 at 11:24 am
  
  Thanks, Duane
  Duane, at the college level I usually take a student’s evaluation of my work in light of their GPA….. now that could mean that some students who are low scoring hate me anyway? Or, it could mean that I grade on the curve? or it could mean that I am just reinforcing my own belief system about my self- indulgence… but that is the point …. What do all these gathering points of so called “data” really mean? and, if any of it is valid , how do I adjust my teaching accordingly? When I felt at the college level I had to adjust my teaching to the “lowest” common denominator I knew it was time to stay home and spend more time with the dog. These are real issues about the profession and I guess I am trying to be fatuous or facetious or just provoke some kind of discussion. Thanks for all you do and for assisting your students. I keep Wilson in mind (and in my lap top as well).
  
  Reply
  - Duane Swacker says:
    
    December 21, 2013 at 12:11 pm
    
    Jean,
    
    “What do all these gathering points of so called “data” really mean? and, if any of it is valid, how do I adjust my teaching accordingly.”
    
    To answer your question: Epistemologically and ontologically** not much as all data relies on context, and that data is used for a description of some sort of “event”/”interaction” in a specified context. As to validity, well Wilson, as you know, has shown the complete invalidity of these processes so that attempting to “adjust” the teaching and learning process because of this very limited and invalid data would be in vain and a waste of time. The ol crap in crap out thingy to put it a little less philosophically.
    
    “These are real issues about the profession.”
    
    Exactly, see below for what are probably the most overlooked and “under taught” concepts in teacher education and administrator preparation. Most programs will do a light touch on the subjects but generally shy away from them:
    
    **Oh no, those big philosophical words/questions, but two of what should be the most important considerations in the teaching and learning process.
    
    Epistemology being “the branch of philosophy concerned with the nature and scope of knowledge and is also referred to as “theory of knowledge”. It questions what knowledge is and how it can be acquired, and the extent to which knowledge pertinent to any given subject or entity can be acquired”, and I contend how to assess whether that knowledge has been acquired. And what descriptive devices do we use to declare so.
    
    And ontology being “the philosophical study of the nature of being, becoming, existence, or reality, as well as the basic categories of being and their relations”, in other words what constitutes our perceptions of what something “is”. (quotes from Wiki)
  - jean sanders says:
    
    December 23, 2013 at 8:02 am
    
    Duane, are we able to actually talk about the means and ends? I always appreciated Mario Cuomo senior because of his philosophy derived from Martin Buber. But, the current generation of governors seems to be of a different ilk…. I gather a lot from Eliot Eisner’s purposes he describes for education — why have we allowed the political forces to narrow down this purposefulness? to the pursuit of increasing Thatcher’s “free markets” when I thought the agreement for to create environments for free people?
- Ellen T Klock says:
  
  December 22, 2013 at 6:33 am
  
  Please note, the librarian is proctoring and the library is closed, not just for your class, but for all the classes she will have to proctor. Such a good use of her time because, who knows, if left to her own devises she might spend the day reading books and eating “Hershey Kisses”.
  
  Reply
  - Duane Swacker says:
    
    December 22, 2013 at 9:23 am
    
    TAGO!
Da Coach says:

December 21, 2013 at 1:37 pm

Formal evaluations are nothing more than a dog and pony show, and are ineffective without ongoing “drop-ins” to the classroom throughout the year. Mr T certainly knew he was being evaluated, so the low marks from the administrator are surprising to me. The high marks from the master teacher are unsurprising, though. When I observed other teachers and noticed something problematic, I didn’t write it down for public consumption. I spoke to the teacher privately and did a number of followup “drop-ins” to see what progress was being made.

Reply