I agree with the following comment by RRatto. If checklists for teacher evaluation were so great, how come they are never used in the nation’s elite private schools? They are a remnant of factory thinking, and unworthy of any profession. Checklists are great for auto mechanics and home builders. They are distinctly non-professional.
RRatto writes:
“anyone who claims they know how to measure a teacher’s effectiveness is full of #%^.
Because of our immensely diverse population and the need to differentiate I can tell you most teachers must change the way they teach almost daily. There is no magic rubric that can measure that.
Once we start aiming to check off the must do’s of a rubric we are no longer are teachers. We become circus monkeys performing for our masters.”
Diane Ravitch's blog 
When admins come to observe with their I Pads tucked under their arms ready to write something in each Danielson Rubric box it is insulting
here’s the rub.. most admins find in revolting as well. Yet they too perform for their master as well.
Once again, an accurate assessment from Ralph. He tells it like it is.
Some dictators, I mean administrators, love it and use it to intimidate. Heaven forbid lesson plans aren’t perfectly manufactured, I mean written. Makes for an extremely toxic environment.
Do the nations elite private schools require teachers be certified? Do they require due process for firing teachers? Do they increase pay automatically with each year employed and higher degrees independent of field?
It seems to me there are many differences between elite private schools and my neighborhood school. Which do you think he public schools should adopt?
Most private schools pay significantly less, but many public school teachers are deciding that the extra money is just not worth the STRESS and humiliation of public school teaching. Can’t blame them. Private tutoring can often supplement their income and make up the difference. If you notice, the best long-established private schools have very little teacher turn-over.
Given the low teacher turnover, would you recommend public schools adopt an “at will” employment standard like the private schools?
Sure, and nobody would work there, at least not any quality people.
You MUST provide people incentive to work in a public school. We are forced to deal with the effects of poverty involving drugs, gangs, violence, sex, etc…
None of those things occur in most of our private schools, and if they do, the kids are rich enough to cover it up or not get caught.
If we keep going down this road, there will be a teacher shortage that this country has never seen. TFA types will dominate – in for a few years, and gone.
If we don’t pay our public school teachers appropriately, and give them the respect they deserve, nobody will do it.
TE,
Not sure how you come up with the concept that since a private concern has at will employees that that is the cause of low employee turnover. Maybe those who teach at these posh tony schools are independently wealthy and have no need to earn a livable wage. That’s just as likely a scenario as what you suggest.
Not in my experience, Duane. We did have picked students, small classes, and we were expected to “deliver” (whatever that means), but stability of the teaching force in my school arose from all of us being damn glad to have the job and damn good at doing it. As in many businesses, you were expected to do your job, and if you did it, you were ok. When enrollment dropped, the headmaster was fired. When the new one came in, enrollment rebounded. For many of us, it is true, the spouse also worked, and that permitted the families to subsist. So there may be that kind of subsidy in private schools. But two working parents is sort of standard everywhere now, isn’t it?
In NYC, they don’t always require certification, they don’t offer defined-benefit pensions, they offer less generous healthcare, and they can and do fire teachers without due process. Private schools generally have much lower labor costs. Everybody who sends their children to these schools knows this. And yet somehow they feel that these schools do a better job educating their children.
Well, until all, or at least the vast majority of public education teachers and administrators realize that the root and/or heart of this problem are supposed educational standards and standardized testing due to the myriad logical errors in the process of developing, implementing, giving of the tests and the dissemination of the results which renders the whole process invalid* nothing will significantly change.
These evaluation schemes are nothing more than an extension of the supposed educational standards upon which the evaluation rubrics are based.
*See Noel Wilson’s “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700 . Here is a very brief summary:
1. A quality cannot be quantified. Quantity is a sub-category of quality. It is illogical to judge/assess a whole category by only a part (sub-category) of the whole. The assessment is, by definition, lacking. The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing we are lacking much information about said interactions.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. This is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher)
.
3. Wilson identifies four “frames of reference” each with distinct assumptions
(epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the
Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.” In other word all the errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms shit-in shit out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it measures NOTHING as the whole process is error ridden and therefore invalid. And the whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
Copied/Pasted to Word Document. Now let me concentrate and read…
Thanks for the share.
Your welcome.
Hopefully reading my “Cliff Notes” version will whet your appetite enough to read Wilson’s work. For a shorter version of the invalidity problem I also suggest reading his takedown of the testing bible put out by AERA, APA, and NCME “Standard for Educational and Psychological Testing”. Wilson’s work “A Little Less than Valid: An Essay Review” can be found @ http://www.edrev.info/essays/v10n5.pdf
By the way I am interested in any and all feedback concerning Wilson’s work. I have tried to find a professional rebuttal to his dissertation and have not succeeded. In email discussions with Noel he noted that in the time since it’s been published (1997) their has been no professional rebuttal and only about a half dozen people have contacted him about his dissertation. A very kind gentleman he is!
Duane, for the love of God will you please stop citing this repeatedly as if it were some sort of brilliant analysis. It’s not. It’s needlessly verbose dressing-up of a simplistic and ridiculous ideas. Granted I haven’t read the full article, but if your summary here is accurate there’s little point in reading it because it’s predicated on absurdities. No one is arguing that tests are perfect. “All the errors involved in the process render any conclusions invalid”? NOTHING IN LIFE is “perfect” or free of “error” except certain elements of mathematics and deductive logic. Error doesn’t make something meaningless. And this insistence that “sorting and separating” is some sort of awful thing is similarly ridiculous. People go on to do different things in life and need different types of education and training. There’s no escaping certain types of sorting and separating.
Standardized tests are being badly misused these days, that’s a given, and something we need to fight against. The idea that “all tests are meaningless” does not help things. We need to combat the current abuse of testing with common sense, not going completely off the rails.
Jim,
This pretty much sums up your argument: “Granted I haven’t read the full article. . . ”
“. . . but if your summary here is accurate there’s little point in reading it because it’s predicated on absurdities.”
What are those “absurdities” Well, you’ve not pointed out any of those absurdities yet. The last post you used a couple of ad hominem attacks and a couple of straw man arguments to attack my summary of Wilson’s work (which I have stated is just a bit of a Cliff Notes version and that one has to read the whole work to understand it) such as above where you stated “The idea that ‘all tests are meaningless'” which is something I have not nor has Wilson stated (but how would you know what Wilson has stated since you admit to not having read his work).
““All the errors involved in the process render any conclusions invalid”? NOTHING IN LIFE is “perfect” or free of “error”.
I agree that “NOTHING IN LIFE is perfect”, that is not what I have stated. So now you use a “red herring” rhetorical device-Red herrings work well if the distracter is something many people will agree with or seems to be closely related to the original issue-from http://www.education.com/study-help/article/distracting-techniques/#heading2 . Neither I nor Wilson has stated that perfection is the goal. What we say is that when there are errors in a process it, by definition, precludes an accurate conclusion, i.e., results in invalid conclusions. This is a simple logical and scientific principle.
“It’s needlessly verbose dressing-up of a simplistic and ridiculous ideas.”
If you are an educator then it’s kind of sad that you think that complex ideas/thoughts/analyses are just a “verbose dressing up of a simplistic and ridiculous ideas” (sic).
I have written seven summaries of what I consider Wilson’s most important points. Nothing of what you have written has addressed those points. If you have problems with the concepts then let’s hear them and enough of the ad hominem, red herring and straw man rhetorical devices.
Duane
Duane, I believe I have addressed them. They aren’t complicated ideas, as you are stating them. You’re attacking testing (ALL testing, not merely bad standardized testing) by arguing that tests aren’t perfect, therefore they are “invalid.” When I point this out you accuse me of logical fallacies and suggest that it isn’t what you or Wilson are saying, but it IS what at least you are saying, in such statements as “all the errors involved in the process render any conclusions invalid”, and other similar assertions. My argument is that this is absurd; you’re holding tests to such a high standard that no endeavors in life would ever meet such a standard, save mathematical proofs and maybe certain types of precise engineering.
Jim,
First off, I appreciate your comments as they help me to better state what needs to be stated. Thanks for responding.
The logical fallacy is you saying that Wilson and I, by extension, are saying that something (standards and tests) have to be perfect. That is not what is being proposed at all. (In my mind perfection is an absurd concept to begin with. I would never propose that “standard”.) That-perfection as a goal/standard is a totally different concept than “in the process of making educational standards and standardized testing the many errors (which are either denied or minimized by the test makers) in that process produces invalidities in the “output”-scores, outcomes etc. . . . ” Two different concepts.
For a shorter read on invalidity and unreliability I ask that you read Wilson’s takedown of the testing bible’s guidelines for validity and reliability put out by AERA, APA, and NCME “Standard for Educational and Psychological Testing”. Wilson’s work “A Little Less than Valid: An Essay Review” can be found @ http://www.edrev.info/essays/v10n5.pdf .
Wilson was in the belly of the beast having been in charge of producing standardized tests for the state of New South Wales in Australia. He thoroughly understands what the issues of reliability and validity are when it comes to standards and standardized testing. Except that he turns the discussion around to one of invalidity and reliability and how those aspects of the standardized testing industry are minimized and fudged over to make the standards and tests appear to be valid and reliable when they aren’t. Please take the time to read this short (15 page) essay review to begin to understand where he is coming from. He’s the expert, I’m just trying to keep his ideas alive. (and obviously not doing that great of a job at that)
Thanks again, I wish more people would respond.
Duane
Duane,
Perhaps you could explain why these are large errors and how you measure the size of the erroros.
TE,
The majority of these “errors” are not amenable to being measured. These errors are ones of epistemological and ontological logical types. In other words they are errors in describing the nature of reality and being. Now the psychometricians will have you believe that they can measure these types of concerns through things like “standard estimate of error” and such. What Wilson does is to show how, through their own language/statements, that their descriptions of error or lack thereof contradict their own statements of accuracy (epistemological logical error). And that is just one source of error of thirteen identified by Wilson in the whole process.
Now I know that most folks, even probably most college professors unless they are philosophy majors, have not been exposed to concepts involving epistemology and ontology. ‘Back in the Goodle days” (cue Johnny Hartford about now: http://www.youtube.com/watch?v=Hfmr-Zn_nRI ) most university grads were probably more versed in those two concepts. But questions of what is the nature of knowledge very much inform our perceptions of and what constitutes a proper teaching and learning environment.
Duane,
I hang out with philosophers, so I am somewhat familiar with the major fields. I have never found continental epistemology to be very enlightening. I am much more comfortable with the Anglo-American analytic approach.
I see good and not so good points from both schools of thought, not that I necessarily consider myself to be a philosopher, just looking for better ways to ascertain this world, both physical and mental in which we live. Just an old fart Spanish teacher with this crazy desire to see the world become just a tad more just and equitable for all. And we can’t have that without an enlightened, in the classical sense, and critical thinking citizenry. How we get that, might be considered one of the main reasons for this blog.
Unfortunately just the words epistemology and ontology sends shivers up most folks spines, mainly because they are so unfamiliar with the concepts even though they experience them on a daily basis without realizing it. Maybe public schools are failing on that account, eh!
Duane,
I’m sorry, I know I’m coming across as intellectually lazy for not reading the original source. When the school year is over and I have some time to myself I’ll read it, so when it’s alluded to again I’ll be more directly informed.
Duane,
Here are a couple of classical works on anglo-american empiricism that you might be in. Empiricism and the philosophy of mind by Wilfred Sellars and Two Dogmas of Empiricism by William Van Orman Quine.
The depth to which we were requiredto write our lesson plans for every single lesson was absurd. In an elementary classroom, we teach reading, writing, spelling, language, math, science, social studies. The reading and spelling were divided into groups according to levels. Using Word Work, Marzano’s vocabulary building, guided reading groups, based on DRA tests takes a huge amount of time. I finally decided that I was NOT going to write what each group would be doing every second. It would have taken me 10 hours to type it up every Sunday. And, we had to do it for the upcoming week, when everyone knows that every day requires modification of what you do THAT day, let alone what you do the NEXT 4 days. That is why it is called TEACHING … tending to the needs as they arise, not prescribing some formula that will guarantee that everyone will learn. They will not, not at the same time. I finally figured out a format that I plugged in the generalities to each week and kept it “open” because I might spend 35 minutes with the low group one day and 5 minutes with the high group. I always tried to do everything I was told to do. It drove me to panic attacks and depression. That drove me to bad health, with heart issues, neuropathy and pre-diabetes. All of this insanity … and I was already teaching well enough before this began. In fact, I believe my students actually did more and did better before we began this crazy testing mania.
Deb, as I read your entry, I can’t help but think that some sadistic SOB came up with the insane lesson plans, which serve no one, and this evil sociopath sits back enjoying the collective witch hunt from afar. We can fill in names for such characters. We should ALL refuse to do any of this nonsense and Boycot! Only option to stop this madness.
Please take good care of yourself.Find a kind place to continue your teaching career. They still exist, but may not pay well.
We had Topic, Student Objective, Standard, Materials (page and source), Activities, Evaluation (every lesson), COS#(s) and connections to previous and future lessons in every lesson plan for every subject. Insane. It was a whole day’s work to plug in all this info. You would think you could use the same thing every year, but there were perpetual changes in the language to meet the new standards (and definitions) and whatever in the heck they wanted to add.
I’ve said before that if a cook had to list all of his/her ingredients, including source, date found, calories, type of stove/oven used, word for word recipes, instead of just the NAME of the recipe, and, of course, a blow by blow description of every step in the creation of the dish, complete with an evaluation of how it tasted and was received by the people who were served … who would be a chef for long? It wears you out.
the ‘checklist’ approach, besides squashing differentiated instruction, is a throw-back to ‘Taylorism’, devised to increase the efficiency/ production of factory workers. Strange, Taylorism lost favor in the factory world and it has been appropriated for teacher evaluation. As many educators have written, the one size checklist/rubric approach is unsuited for the diverse and complex world of the classroom, but makes a one sizefits-all cookie cutter.
And, now we have the Six Sigma mindset that has permeated industry and has reached its .tentacles into the education circles. Efficiency. Students aren’t merely raw materials to be manufactured into products. They are indivuals, gloriously individual. The beauty of teaching is to manage the classroom almost as a conductor conducts an orchestra. Each group of instruments work together to produce a concert, each group having its distinct purpose as a part of the whole. Each group having several distinct parts in the harmonies that make the concert as pleasant as possible. And, each listener enjoys that concert from a distinct perspective. One checklist is meaningless to use for comparisons because each administrator using it is influenced by individual factors and relationships with the person being observed. The “efficiency” to be observed in teaching should be the gentle but firm manner in which each student is led to find his greater self.
How do you have a standard for a variable as wide as this one?
If the “this one” is the teaching and learning process then one can’t have a standard, whatever the hell an educational standard is to begin with.
There are several reasons public schools do not have well conceived evaluation systems but the easiest to explain is this: the span of control in public education is broader than anything one encounters in any educational organization. In districts where I served as Superintendent we typically had one Principal for an elementary school of 500 students or fewer and added an assistant principal for larger schools. The Principal was expected to evaluate the teachers, instructional assistants, secretarial staff, related service providers, and—in some cases—bus drivers, custodians, and cafeteria workers. Oh… and the Principal is the final arbiter in playground disputes among children, supports teachers in the management of misbehaving students, serves on IEP teams, meets with social services when referrals are made to parents, works with the PTO chair to organize fund raisers, signs off on spending and purchasing, makes certain computers operate and the cafeteria food is decent, the list goes on…. PLUS, the Principal has several daunting annual activities to tend to: budget development, planning and implementing standardized tests, and hiring staff. Finally,in many New England districts the Principal prepares board meeting packets as well. So with that range of staff to over see and that long list of responsibilities, is it any surprise that when it comes time to conduct evaluations that checklists are used?
If the public wants to provide good evaluations they need to provide TIME for evaluations which means either hiring more administrators or providing release time for teacher/coaches…. and in education, like any business, TIME means MONEY… and in public education (or any public enterprise) money means more taxes…. and in the end, it is easier to offer easy and inexpensive “market based” solutions to schooling than it is to suggest tackling the complicated systemic solutions… and so checklists!
“. . . the span of control in public education is broader than anything one encounters in any educational organization.” Excellent point wgerson! Except that the span of control is far broader in public education than most everywhere in the business sector.
Many complain about administrative costs of public education but the span of control issue is never brought up. When I bring it up with my conservative critic of public education friends it knocks their argument out of the water. Not only span of control but for the same level of responsibilities in the private sector a public school administrator would be making anywhere from 3-10 times as much as they actually do.
There was a study done in the 90s (and I don’t have it at hand) about public education vs private sector span of control, the results being that in the private sector, on average, there is one supervisor per five employees whereas in the public education sector that figure is 1:23. Tell me which is “more efficient” especially taking in the salary differences. Seems to me the public gets quite a bargain on administrative costs.
I agree as that is precisely what “standards-based” now means–that a teacher runs through a number of “must do/ be present” and presto chango, standards are met.what about trying to get a hamdle on learning rather than “achievement” or compliance?
How ironic…every time one of these scripted curricula is brought up at faculty meetings, I use the circus monkey analogy as well…
Our district is in SW Ohio, but our superintendent was quoted as saying in the Cleveland Plain Dealer (in NW Ohio) that our district union was against joining RttT because we “didn’t want to be evaluated every year”. Our district was to get a paltry $25K for putting ourselves through this insanity. We have been on the Ohio Excellent with Distinction list for 5 years. Our teachers are great teachers. We think evaluations are bogus, particularly in light of who does them.
Our union voted to pay $52 per month in additional dues to support a campaign to fight this. Our building rep, who was against paying dues or taxes at all … but was rep so he could have a “voice”… not for the teachers … went to the state house as a delegate and voted “no” because they changed the amount to $54. Of course, it passed and we had to pay, but this is the kind of brown-nosing that we have to deal with. No one else wanted to be the rep and he volunteered.
People have such huge house payments that they will tolerate anything to be able to keep their jobs, it seems.