Education has always been buried in jargon, and today’s reforms have made the pile of jargon steeper than before. The Common Core brings us “close reading” (reading with background or context). The demand for test-based evaluation of teachers brought us VAM (value-added-modeling or value-added-measurement), and also SLO (student learning objectives). In this comment on the blog, Laura Chapman says there is even less evidence for the validity of SLOs than for VAM.
Chapman writes:
I appreciate Audrey Amrein Beardsley’ great work on the VAM problem, now at the threshold of becoming a major component of evaluation for teacher education programs.
Even so, equal attention needs to be given to the use of SLOs for evaluating teacher education in so-called untested and non-tested subjects.
It has been estimated that about 65-69% of teachers have job assignments for which there are not state-wide tests. SLOs (and variants) are the proxy of choice for VAM. An SLO is a writing exercise for teachers required in at least 27 states SLOs must include pretest-posttest and/or baseline to post-test reports on student growth.
Four reports from USDE (2014) show that there is no empirical research to support the use of the SLO process (and associated district-devised tests and cut-off scores) for teacher evaluation.
The template for SLOs originated in Denver in 1999. It has been widely copied and promoted via publications from USDE’s “Reform Support Network,” which operates free of any need for evidence and few constraints other than marketing a deeply flawed product.
SLO templates in wide use have no peer reviewed evidence to support their use for teacher evaluation…not one reliability study, not one study addressing their validity for teacher evaluation.
SLO templates in Ohio and other states are designed to fit the teacher-student data link project (funded by Gates and USDE since 2005). This means that USDE’s proposed evaluations of specific teacher education programs ( e.g., art education at Ohio State University) will be aided by the use of extensive “teacher of record” data routinely gathered by schools and districts, including personnel files that typically require a teacher’s college transcripts, degree earned, certifications, scores on tests for any teacher license and so on.
There are technical questions galore, but a big chunk of the data of interest to the promotors of this latest extension of the Gates/USDE’s rating game are in place.
I have written about the use of SLOs as a proxy for VAM in an unpublished paper titled The Marketing of Student Learning Objectives (SLOs): 1999-2014. A pdf with references can be obtained by request at chapmanLH@aol.com
Diane Ravitch's blog 
My small K-8 school is abandoning SLOs. That might seem like a good thing, but the effect is scary. Most teachers who had to give SLOs will most likely now be basing their evaluations on the middle school results of the NYS ELA exam (6-8). It makes a certain sense in an insane situation. However, our school is so small we have only one middle school English teacher, me. That means I will become personally responsible for the evaluations of about half the staff in my school.
Just me. That’s my reward for having this state test. No pressure. No pressure at all.
It’s like something out of “The Onion”, except it’s true and happening to me.
LikeLike
Me too: I am the only 10th grade ELA teacher, and one of 2 9th grade. We are growing our high school so we only have those two grades now. ELA only counts this year, as math doesn’t. It’s SE s have been changed so they are phasing them in. All on us. No pressure there.
LikeLike
My child’s 5th grade teacher “forgot” to tell the students that they were not suppose to know the material on the pretests.
LikeLike
Yes, SLO’s are nonsense on stilts, and in Rhode Island at least just a huge administrative time-suck. It has taken a few years for everyone to realize that these aren’t just crazy ideas coming from their state/district administrators, but essentially a de facto national policy.
LikeLike
Not quite a national policy but marketed by USDE contractors to every state that applied for an RTTT grant, whether or not they received a grant. The marketing of SLOs was organized by a grant from USDE to professionals in public relations, under the umbrella of the “Reform Support Network.”
LikeLike
I beg to differ. My understanding of David Coleman’s interpretation of “close reading” is that it is to be done WITHOUT the teacher tapping into students’ background knowledge or providing information about context. This is one reason why “close reading” is so horrendous to genuine educators, who know that learners attempt to make sense of the world and will fill in the blanks if they aren’t given guidance, often with misconceptions or nonsense.
To demonstrate this, I’ve played the telephone party game in my classes with college students, where a sentence like “Ontogeny recapitulates phylogeny” is the message that is whispered in the ear of each person sitting around a table. Invariably, most people have no clue what this means and the message morphs into gibberish. By comparison, a message like “50 stars for 50 states” is much more likely to resist being twisted by personal interpretations and typically survives as originally told, instead of turning into something like “15 scars for 15 skates,” because most Americans have the background knowledge necessary to understand it and comprehend the unspoken context.
With this major faux pas, Coleman has demonstrated that he has no understanding of learners or Cognitive Science.
LikeLike
Yes, I was going to make the same comment. I think “without background or context” was intended.
But, I like your “telephone” example, Teacher Ed!
LikeLike
Having read Diane’s commentary on close reading in the past, I think the definition of close reading was a proofreading error and should have said “reading without background or context.”
LikeLike
We are just getting our SLOs together here in IL and as a specials teacher I get to make mine all Type 3 (portfolio and the like). Well this is going to be a cake walk for me. What grade shows the most improvement in my subject, KINDERGARTEN! I’m doing both my type 3 assessments in that grade. I’m only doing them the years I’m evaluated. And, yes, I will totally ace them. But nothing…and I mean nothing will be gained by either myself or the students. So, yes SLOs are worse but only in that they do absolutely NOTHING for anyone!
LikeLike
SLOs are a teacher’s best friend, assuming they are able to design their own. All you have to do is think like a corporate reform face man. Remember to always (technically) tell the truth and you’ll be golden — if you’re smart about what you choose to measure.
My favorite trick is to set my baseline on day one, right when the kids come into their first class. Give them something like a compare and contrast essay starter handout with no preteaching and tell them that you’re setting a baseline so it doesn’t count for their grade. You can imagine how serious your high school English students will take this, which is exactly how seriously you WANT them to take it.
Then, a few weeks later, give them an entire unit on compare and contrast essays (or whatever you gave them on day one). Use as much scaffolding as you can manage and make sure the kids all know that this is for a MAJOR grade.
That right there is almost guaranteed to give you all the “growth” you need, without needing to lie about anything.
Note that many administrators are either too overworked, too apathetic, or just too dull to pay too much attention to exact wording when they sign off on your SLO. If two out thirty kids meet standard on the baseline, and four out of thirty kids meet standard later on, you’ve just shown 100% growth!
SLOs have been pulling my evaluations up to 4 out of 4 since we started using them in my state.
LikeLike
You have learned to play the SLO game. That is also a major finding from USDE funded research on SLOs. Nevermind. Policy-makers who have mandated the SLO process market it as a “best practice” for teachers.
LikeLike
Laura, it is indeed a game, and a very wasteful one at that, as I’m quick to point out to anyone who will listen. But here in Rhode Island, no one with authority over the matter seems to want to listen, so — game on!
Don’t hate the player…
LikeLike
I did not mean to imply that you were doing something nefarious. Every teacher who must do this writing assignment and participate in the guessing game is caught in a Catch 22. The little research available shows that teachers do, in fact, learn to comply with the process and also are highly critical of it, for very good reasons. The marketers of SLOs want to have the tests and targets for learning standardized at the district level, in order to meet a federal requirement that tests “be comparable across classrooms.” Rhode Island is part of the USDE funded “Reform Support Network,” meaning that it is among the states that is highly dependent on “borrowing” ideas from other states. State departments of education are drowning in requirements that are conjured by federal officials who do not have to live with the consequences of policies.
LikeLike
My state has Student Learning Goals–someone decided that “slugs” were better than “slows”. Every teacher I have talked to does what werebat suggests: make the pretest as easy to get a 0 on as possible. We are also supposed to make tiered goals, based on the students’ scores on the pretest. E.g., If a student scored 40%-50% on the pretest, they should get 90% to 100% on the post test. All I’ve managed to do is show that pretests–even when made as a subset of questions from the actual post test–have almost no predictive value. This year, the correlation coefficient for pre-test to post-test was just above 0.2. The scatter plot looks like a shotgun blast.
LikeLike
You are being asked to participate in process that is so convoluted that it should be banned as snake-oil. From my paper:
“The categories and criteria for a typical SLO forward the illusion that every step in the process is scientific. Thus, students in classes are dubbed a “population.” Records from prior grades (or a pretest) become “baseline data” for profiling and grouping students. “Expected growth” is a prediction of posttest scores, but stripped free of any theory that might leverage reasoning about the expected outcomes. “Growth” is a euphemism for an expected increase in scores, pretest to posttest, even if scores are based on rubrics.
In effect, SLOs are framed and rated as if the teacher is documenting a one-group pretest-posttest experiment for the population named in the SLO, but with no control group, and with an arbitrary demand for multiple standards, measures, and research-based teaching strategies.
Given all of these variables and criteria, no reliable and valid inferences can be made about the effect of the teacher on the posttest scores. None. In this respect, the use of SLOs to justify judgments about a teacher’s effectiveness is not only blatantly unscientific but also unethical (Berliner, 2002; Popham, 2014).”
LikeLike
There is a word for the combination “blatantly unscientific and unethical”: fraud.
If the people supporting the use of SLOs and VAMs for high stakes decisions like firing and denying tenure were working in pretty much any other field (with the exception of economics, of course), they would not be working there long because they would be run out for being the quacks that they are.
Imagine if someone claimed that an instrument could reliably screen for cancer when in fact it did no such thing. The purveyors of such nonsense would not only be driven out of the medical profession but undoubtedly brought up on criminal charges for defrauding the public.
.
LikeLike
I worked on a committee to create examples of SLO’s for my state. It took about 16 hours to create one portion of the document. Where will teachers find the extra time to produce this? You can bet that it will be done on their personal time in addition to lesson plans and grading. It is only Wednesday and I have already given my 40 hours. I’ll finish teaching, planning and grading for the next two days without earning time and a half other salaried workers get. In fact I’ll do it without any extra compensation. Professionals do,this sort of thing. With professionalism comes an expected measure of respect. I find all of this disrespectful, demoralizing and demeaning.
LikeLike
SLOs are a writing assignment. A “complete” SLO–one marketed by the Reform Support Network (RSN) in multiple states–requires a teacher to meet about 25 criteria.
RSN has a library of SLOs organized by state, grade level, and subject. All examples have annotations made by a “trained evaluator.” As of November 2014, five states had contributed examples of SLOs in the arts. The comments on each example are framed to make the teaching of art fit SLO criteria. It is clear from these examples that the arts evaluators have little understanding of the aims and methods of instruction in the arts or the conditions under which teachers in the arts work. The evaluators are concerned with mustering strict compliance with each part of the SLO. The SLOs must fit the evaluators’ concepts of “proper education” in the arts. No teacher in this sample had a fully compliant SLO. https://rtt.grads360.org/#communities/slo/slo-library
LikeLike
Salaried workers typically do NOT get “time and a half” compensation for work hours beyond 40, at least not in NYS. Salaried workers get paid a salary that pays them the same whether they work 37.5 hours in a week or 67.5 hours.
One of the reasons those outside our profession get annoyed with us when we speak about how many hours we work, how much work we take home, etc is because they do it too. Before I began teaching 11 years ago I worked twenty years as a consulting professional geologist. I averaged 50-55 hours per week on the job, sometimes as much as 60 – and I took work home nearly every night.
Our complaining in this specific way just irritates those who already do the same, especially in the professional salaried ranks. We do ourselves a grave disservice when we include statements like that in the defense of our profession. Better to state that ” We, like most other professional work many more hours beyond those in the classroom because that’s what professionals do – they work until the job is done”.
I’ve jousted too many times on too many internet forums with non-educators about this. Believe me, if you frame it as I suggest, it shuts them down immediately.
Peace.
LikeLike
I would say something like “Do you want your children’s teachers spending their time on regulatory red tape?” is probably more in the right direction regarding the time issue.
LikeLike
Statements like this (i.e. “salaried workers typically do get time and a half compensation for work hours beyond 40”) also convey that the speaker has about as much familiarity with private sector employment as George Herbert Walker Bush has with supermarket checkout procedures.
LikeLike
SLOs…what a joke.
You can easily and completely game them, so they are of zero value.
LikeLike
I’ve been working with our district for a while now on SLO’s, and while they are not the best, they are better then VAM.
By the way the best case scenario would be trust. We all know that’s not happening.
I teach a non-tested area (CTE) and I’m horrified by the idea that part of my evaluation in on stuff I do not teach. I’ve heard many attempts to force a justification of how core academic areas should be part of EVERY class, but shoving a fatman into a little coat is nothing but comedy. At least with SLO’s I know that the objectives are relevant and valid, because I get to right objectives that are relevant and valid.
I’m just wondering how many hour (days?) we are going to sink into this before the next panacea comes along?
LikeLike
At least 27 states require all teachers to write SLOs, even if the teachers are receiving evaluations via the VAM scam.
LikeLike
I served on our SLO evaluation committee this year and last. I chose to do this to make sure that someone was there questioning and challenging. My experience was full of personal conflict. I understand how to play the game and write SLOs that are not only approved, but also are somewhat useful and do not take too much instructional time. I saw incredibly long assessments like a semester exam. I read pre-assessments that included material not included in their standards in the name of rigor. One teacher included the pre-asessment in first quarter grades so no child received an A or B. Much of the work of the committee became exactly what you said, a writing assignment for the teacher. We went round and round with some teachers who simply couldn’t figure out how to follow directions and write it “correctly”. Then there were the inconsistencies between graders. We were trying to figure out as we worked what fit the criteria for approval. We were inconsistent. If the grader was less familiar with the subject, it was hard to know if the assessments were appropriate. Some teachers left the SLO group this year because they were giving assessments from the approved vendor list (Ohio). This made the remaining teachers angry since they were still left with their writing assignment. Furthermore, teachers would ask colleague in other districts what their SLO process looked liked and shared examples that were seemed relatively easy compared to ours.
Personally, in my teaching position, I prefer being in the SLO group as there will never be standardized assessments in my area, and I want no part of shared attribution. I have basically figured out how to make my SLO useful to me, but otherwise harmless.
The state superintendent included in his recent recommendations regarding testing
“Eliminate the use of student learning objectives as part of the teacher evaluation system for teachers in grades preK-3 and for teachers teaching in non-core subject areas in grades 4-12.”
However, the whole report is not worth much.
LikeLike
Those of us who have worked in the private sector where workers typically receive annual reviews can readily see what a sham teacher evaluations are today, and they are affecting teachers in programs from PreK through Grad school. Because they are ongoing, there is nothing annual about them. I teach college courses and I have received at least four “annual” reviews this year! And the required PDs are such a heavy workload that they are comparable to graduate courses, but without the grad school credits. For those of us who have been doing this for years, it all feels more like the probationary period in the private sector and that the primary purpose is to identify people who can be fired. Feeling like I am on probation in perpetuity makes me really wish that I could change professions, but after 40 years of teaching and no pension, it’s way too late in the game for me.
LikeLike