Fairtest: The Fight Against Test Madness Continues

41 Comments Post your own or leave a trackback: Trackback URL

1. Duane Swacker says:

   June 17, 2014 at 1:09 pm

   They forgot the most important one:

   “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700

   Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)

   1. A quality cannot be quantified. Quantity is a sub-category of quality. It is illogical to judge/assess a whole category by only a part (sub-category) of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as one dimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing we are lacking much information about said interactions.

   2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).

   3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.

   4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”

   In other word all the logical errors involved in the process render any conclusions invalid.

   5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.

   6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.

   7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”

   In other words it measures “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?

   My answer is NO!!!!!

   One final note with Wilson channeling Foucault and his concept of subjectivization:

   “So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”

   In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.

   LikeLike

   Reply
   • Jim says:

     June 17, 2014 at 2:20 pm

     Duane – If no quality can be quantified and quantities are a subcategory of qualities then it follow that no quantity can be quantified.

     LikeLike

     Reply
     • Duane Swacker says:

       June 17, 2014 at 3:59 pm

       That statement certainly makes absolutely no sense to me.

       Please expound!

       LikeLike

     • Jim says:

       June 17, 2014 at 4:28 pm

       It’s called logic, Duane. Possibly you’ve heard of that.

       LikeLike

     • teachingeconomist says:

       June 17, 2014 at 6:09 pm

       Duane,

       Let me give it a shot.

       First, your opening sentence: No qualities can be quantified.

       Next, your second sentence: Quantities are a subset of qualities.

       If no qualities can be quantified and quantities include as a subset quantities, it must follow that no quantities and be quantified.

       LikeLike

     • teachingeconomist says:

       June 17, 2014 at 6:10 pm

       The way out would be to say that some qualities can be quantified, among them, quantities.

       LikeLike

     • teachingeconomist says:

       June 17, 2014 at 6:13 pm

       I want to be clear, so I have to fix my last paragraph in the longer post:

       If no qualities can be quantified and qualities include as a subset quantities, it must follow that no quantities and be quantified.

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 10:46 am

       TE & Jim,

       What you are pointing out at first appears to be a conundrum but is actually the mistake that Wilson points out. That the sub-category, subset, is not the whole and to ascribe characteristics of the whole to the subset/sub-category doesn’t hold.

       LikeLike

     • teachingeconomist says:

       June 18, 2014 at 11:01 am

       But statements about the whole must also apply to the subset, right? Catfish have whiskers. The flathead is a subset of catfish, therefore flatheads have whiskers.

       Qualities cannot be quantified. Quantities are a subset of qualities, therefore quantities can not be quantified.

       See the parallels between the two sets of statements?

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 11:12 am

       Cars: Cars have engines to propel them: The Chevy Volt is a car therefore it has an engine to propel it.

       No, it doesn’t.

       LikeLike

     • teachingeconomist says:

       June 18, 2014 at 11:15 am

       Duane,

       The Volt does have an engine, in fact I think it has four.

       If the volt did not have an engine, your premise that all cars have engines would be false.

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 11:17 am

       Close but no cigar, TE. Yes the Volt has an engine but it is not used to propel the car. The engine is for generating electricity to supply the motor that propels the car.

       LikeLike

     • teachingeconomist says:

       June 18, 2014 at 11:22 am

       Duane,

       So your statement that the Volt does not have an engine is false?

       I would go with a pure plug in as an example, but if you are going to say that a mother is not an engine, well I would have to say that your premise that all cars have engines is false because there are cars without engines.

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 11:28 am

       Reread my statement. What I said about the general-cars, does not hold true for the subset Volt.

       And by the way “Despite their name, not all catfish have prominent barbels [whiskers]; members of the Siluriformes order are defined by features of the skull and swimbladder.”

       LikeLike

     • teachingeconomist says:

       June 18, 2014 at 11:51 am

       Duane,

       What is true for the super set must be true for all members of the set. If all X have property Y, it must be the case that x, an element of the set X, has property Y. If X does not have property Y, either x is not an element of the set X or not all X have property Y.

       We can use any characteristic of catfish that you like. If all catfish have property Y and the bullhead is a catfish, bullheads have property Y.

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 11:53 am

       No doubt a bullhead is a catfish. And provides a good eatin.

       Around here most claim the blue catfish (the largest, I believe, in North America) is the best eatin.

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 11:55 am

       By the way, I am not arguing about set theory but about using a sub characteristic to describe the whole set. One can do that but it leaves out a lot of information about the whole set.

       LikeLike

     • teachingeconomist says:

       June 18, 2014 at 12:09 pm

       I think this is where your confusion lies. Your point one says that not being quantifiable is a property of the whole set “quality”, not a property of some subset. I am applying the property of the whole set “quality” to what you say is a subset of “quality” called “quantity”. I am not using a property of a subset to define the superset, i am applying your property of the super set to what you define as a subset to come up with the statement quantities are not quantifiable.

       There are two ways out of the logical problem. You could say that some qualities are not quantifiable (I think that is correct) or you could say that quantity is not a subset of quality (which I think is probably wrong).

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 12:14 pm

       I see your point but again I am not wanting to argue set theory. I am arguing descriptive language games that may or may not coalesce with set theory (which is one way of many to look at these problems).

       I’ll go with your first way out except to say that almost all are not quantifiable.

       LikeLike

     • teachingeconomist says:

       June 18, 2014 at 12:57 pm

       Duane,

       I think that would be a great start.

       LikeLike

     • Duane Swacker says:

       June 18, 2014 at 1:03 pm

       I think the line will go something like “It is impossible to adequately quantify a description of quality.”

       LikeLike

   • Jim says:

     June 17, 2014 at 2:26 pm

     Duane – Wilson’s elucidation of “error” in #4 begins to sound a lot like the gibberish quoted by Les Perelmann in a recent thread.

     LikeLike

     Reply
     • Duane Swacker says:

       June 17, 2014 at 4:03 pm

       Jim,

       Pleas link to what Perelmann wrote. TIA!

       What about what Wilson explains about “error” is it that you don’t understand/disagree with. Please expound and explain your position better than “sound(s) like the gibberish”.

       LikeLike

     • Jim says:

       June 17, 2014 at 4:30 pm

       There’s nothing to agree or disagree with. It’s empty pompous verbiage lacking any discernable meaning.

       LikeLike

     • Duane Swacker says:

       June 17, 2014 at 5:11 pm

       “It’s empty pompous verbiage lacking any discernable meaning.”

       Pot knowingly calling the kettle black, EH!!!

       Basta ya contigo.

       LikeLike

   • Jim says:

     June 17, 2014 at 2:33 pm

     What utter tripe. The Three Stooges could probablly come up with something more intelligent.

     LikeLike

     Reply
     • Dienne says:

       June 17, 2014 at 3:04 pm

       Speaking of intelligence, this one’s for you, Jim: http://blogs.plos.org/speakingofmedicine/2014/06/10/im-scientist-learning-disabilities-thats-okay/ (It’s linked above, but I wanted to be sure you didn’t miss it.)

       LikeLike

     • Duane Swacker says:

       June 17, 2014 at 4:07 pm

       And why is Wilson’s study “utter tripe”? What exactly about his work do you disagree and/or find objectionable?

       I’ve been looking for over a decade now for a rebuttal or refutation of his work and haven’t found any yet other than a few opinions like “utter tripe” or “there’s nothing there” (from TE). So please, feel free to rebut/refute what you find objectionable.

       LikeLike

   • KrazyTA says:

     June 17, 2014 at 11:08 pm

     Señor Swacker: a little support from Banesh Hoffman, THE TYRANNY OF TESTING (2003 republication of the 1964 edition of the 1962 original), Chapter 9, “Challenge to the Testers” that begins with the following:

     “UNDERMINING BLIND FAITH in statistics is crucially important. But it does not exorcise statistics—nor is it meant to. For statistics, when viewed with healthy skepticism, can be a valuable and often indispensable tool.

     Unfortunately, even when the layman is amply forewarned he is apt to be helpless against experts who wish to influence him by statistical arguments. Undermining blind faith in statistics, therefore has only limited effect. It may weaken the impact of statistical arguments, but that alone is not enough to bring about reforms in testing.” (p. 149)

     He then offers general criticisms of standardized tests before going on to more specific points. I think the former would be apt here (p. 150):

     [start quote]

     The tests deny the creative person a significant opportunity to demonstrate his creativity, and favor the shrewd and facile candidate over the one who has something of his own to say. Unlike essay examinations, they are mainly concerned with predetermined intellectual snippets, and not with the crucial ability to conceive, design, and actually carry out a complex undertaking in an individual way.

     They penalize the candidate who perceives subtle points unnoticed by less able people, including the test-makers. They are apt to be superficial and intellectually dishonest, with questions made artificially difficult by means of ambiguity because genuinely searching questions do not readily fit into the multiple-choice format.

     They take account only of the choice of answer and not of the quality of thought that led to the choice.

     They too often degenerate into subjective guessing games in which the candidate does not pick what he considers the best answer out of a bad lot but rather the one he believes the unknown examiner would consider the best.

     They neglect skill in disciplined expression.

     They have a pernicious effect on education and the recognition of merit.

     [end quote]

     Of course, El Gordo y El Flaco will disagree with the above and Wilson, but always keep in mind “no pedir peras al olmo.”

     Just my dos centavitos worth…

     😎

     LikeLike

     Reply
     • Duane Swacker says:

       June 18, 2014 at 10:37 am

       Gracias por el dicho.

       Many English proverbs/sayings have their origin in Spanish.

       LikeLike

2. Peter says:

   June 17, 2014 at 1:26 pm

   Thanks to Bob and Fairtest for the work they do.

   LikeLike

   Reply
3. Dienne says:

   June 17, 2014 at 2:17 pm

   From the Florida Mother link: “She correctly discerns that the problem in schools is not the concept of accountability, but the implementation.”

   But it *is* the *concept* of “accountability”. There is no good way to implement it because the very concept assumes that teachers otherwise have no accountability. Most of us in the real world have one or a small handful of bosses with whom we work regularly, who know us and who know our work. Those are the people who decide whether we get raises and promotions or whether we get the boot. That’s the way it used to work – and how it should work – with teachers. Principals and other administrators know who the good teachers are and who the bad ones are and there always have been ways for them to recognize the good, remediate the mediocre and eliminate the few truly bad. Any administrator who says they can’t fire or discipline teachers under their supervision is either lying or admitting incompetence and they’re the ones who should be fired.

   I can’t think of any other profession that is held “accountable” by the performance of other people over whom they have little if any control, at least not without taking into consideration all the other variables that would affect such performance. (For example, yes, a doctor whose patients all died would probably be considered suspect. But whether that doctor was a general practitioner in an affluent area or a geriatric specialist in a poor area would probably be taken into account.)

   LikeLike

   Reply
   • NYS Teacher says:

     June 17, 2014 at 2:47 pm

     There are many teachers being held accountable by the performance of students they do not teach and subjects that they do not teach. This should fact should end any counter argument. There is no justification for this that is acceptable to any sane person.

     LikeLike

     Reply
     • TC says:

       June 17, 2014 at 3:35 pm

       Yes, that is the end of the debate. To go beyond that is a futile journey beyond reason.

       LikeLike

4. Dr. Bob Blomeyer says:

   June 17, 2014 at 3:35 pm

   Cudos and thanks to Bob Schaeffer & Fair Test!

   LikeLike

   Reply
5. Barry Elliott says:

   June 17, 2014 at 6:01 pm

   This is a story copied from a site just visited. I didn’t think to document the source, but I don’t think anyone will question its veracity.
   A Pennsylvania woman was found dead in her cell June 7, halfway through a two-day sentence in Berks County Prison for unpaid debt. Eileen DiNino, 55, owed $2,000 in fines and court costs related to dozens of truancy violations her children racked up since 1999, the Associated Press reports.

   DiNino was a mother of seven, four of whom still lived with her, and she did not work. Over the years, she skipped or was unprepared for hearings, but the 48 hours in jail would have eliminated her debt. The cause of DiNino’s death is unknown and, while sudden, isn’t considered suspicious.

   Parental imprisonment for truancy isn’t unheard of in Berks County (northwest of Philadelphia) — since 2000, more than 1,600 people have been jailed there over truancy fines, two-thirds of them women, according to the Reading Eagle, a local newspaper. Under federal law passed in 1833, people cannot be jailed on outstanding debt, but every state has its own set of rules.

   For example, if you fail to appear in court over a debt dispute in some states, you can be jailed for contempt of court. Some states also imprison parents who fall behind on child support.

   These modern-day iterations of debtors’ prisons are just some of the many ways debt can be destructive. Collection accounts, missed loan payments and other factors often related to an inability to pay will wreck a consumer’s credit standing, which in turn makes it difficult to access some financial services, housing or employment.

   SO, WHAT IS THE POINT OF THE STORY HERE?
   The point is that, if they can put parents in jail for the “truancy” of their kid(s), they can test all they want. The noise y’all make may slow them down a little and put a slight crimp in their style, but in the end, you are tilting at windmills and trying to spray up a rope, if you are familiar with that crude image. You relinquish your rights and your autonomy by quietly aquiescing to laws that create arbitrary authority and that subject yourselves and your children or students to the will and decisions of powerful people and interests, regardless of their lack of knowledge or competence, and then you bitch and complain for the rest of your life about what is being done to you or those you care about. Save your breath or your ink if you aren’t willing to fight to take that power away from those ostensible experts and authorities. You are deluding yourselves about making a meaningful long-term difference for children. You lost the battle and the war when the attendance laws were passed making schooling and education political issues. But, hey – what do I know?

   LikeLike

   Reply
   • Chiara says:

     June 17, 2014 at 6:24 pm

     It’s much, much bigger than truancy. They’re putting people in jail because they can’t pay fines and fees.

     “Several courts in Ohio are illegally jailing people because they are too poor to pay their debts and often deny defendants a hearing to determine if they’re financially capable of paying what they owe, according to an investigation released Thursday by the Ohio chapter of the American Civil Liberties Union.”

     The best part of the ACLU lawsuit was they listed the Ohio courts who are doing this.

     Nothing happens until you name an individual. Things moved very fast once the names were published 🙂

     http://www.csmonitor.com/Business/Latest-News-Wires/2013/0404/Debtors-prisons-Thrive-or-serve-jail-time

     LikeLike

     Reply
   • NY teacher says:

     June 17, 2014 at 8:51 pm

     “You are deluding yourselves about making a meaningful long-term difference for children.”

     You obviously don’t understand the nature of the profession. Most veteran teachers are making a difference in the lives of our students because we ignore most of the BS that we rant about here and do what we’ve always done. We close our classroom doors and we do what’s best for children.

     “Save your breath or your ink if you aren’t willing to fight to take that power away from those ostensible experts and authorities.”

     Are you recommending non-violent or violent civil disobedience. Do you think we should be Rioting? Picketing? Pillaging?
     Punching? Threatening? Litigating? Other?

     And if are “willing to fight” what makes you think that will work.

     Finally if our ranting here is bothersome to you I suggest you ignore us. For many of us this blog helps get us through the day. I know for a fact that the conversations here have made me a much better teacher. That alone makes it worthwhile.

     LikeLike

     Reply
     • Barry Elliott says:

       June 18, 2014 at 11:05 am

       The reason the ranting and the demonization of individuals is bothersome is because it reveals that the participants have chosen to ignore the fact that virtually everything discussed here is a recycling of issues, problems and complaints that have been around for generations. In every generation there are extraordinary teachers who circumvent the rules in various ways in order to benefit students and who find ways to compensate in part for the damage done by the authoritarian structure. Meanwhile, there are people screaming bloody murder because those are the exceptions and profound damage is being done by the teachers who are not extraordinary and by a system that chews up and spits out even some of the best teachers. Many others either choose not to enter the profession or drop out before their career gets off the ground. It is irresponsible and cowardly in my opinion to accept these conditions and to participate in a system that is so destructive and anachronistic merely because your ego is placated by your good feelings about defeating the negative aspects part of the time for some of the more fortunate kids. Coercion and learning are antithetical, as I’ve said here a dozen times. You can pretend that isn’t true, or you can work (and fight) to eliminate the coercion, which is a direct consequence of the legalization of schooling. It is a fight because nearly everyone clings to the delusion that trying to educate by force is OK or possible. I don’t advocate violence, but much violence is borne of this horrible state of affairs. The place to start – the only place to start – is in removing attendance laws. That is where the powers that be get their (artificial, arbitrary, and illegitimate) power. If you think of yourself as a good teacher and a good person, then you should be a conscientious objector. By going along to get along and not dealing with the fundamental issue and the mythology, you are supporting the whole corrupt enterprise and contributing to the harm done to millions of children.

       LikeLike

6. Donna says:

   June 17, 2014 at 7:28 pm

   and yet, none of it matters….the wealthy WILL own the schools and your children’s futures; perhaps the wealthy will even DETERMINE your children’s futures.

   LikeLike

   Reply
7. Barry Elliott says:

   June 19, 2014 at 2:46 pm

   A follow-up:Is there a credible scientist in any field on the face of the Earth who would try to make a case that many of the most prevalent and serious problems of youth have not always been corellated with school attendance of the mandatory variety? Would any sentient adult claim that truancy is not the precursor to all manner of juvenile delinquency, defiance, deviance, and rebellion? The list is long. From teen-age pregnancies, to drug and alcohol abuse; from bullying and dropping out to stealing cars for joy rides and vandalism (very frequently targeting school property), and from suicide to homicide (again, school shootings and violence and even rapes on school property and directed against students and teachers have become common) the problems of youth are inextricably and intimately linked to the discontent of children and young adults resulting from sensations of oppression, disaffection, powerlessness, anonymity, and alienation as students.
   We have had a virtual epidemic of self-loathing, low self-esteem, and paranoia among our youth and particularly among graduates for many decades. Drop-outs, on the other hand have actually historically comprised a healthy proportion of the great entreprenuers and successful business owners in the US, as well as many of our model citizens and parents. Drugs such as Ritalin and anti-depressants have been prescribed to children on a massive scale for many years and certainly long before the Common Core, the big charter school push, or Bill Gates and the other interploers stuck their noses in. To fail to make the connection between schooling that is involuntary and that requires an authoritarian hierarchy in order to comply with attendance laws and their corrolary demands and institutional needs is to wear blinders in a windowless room with black out drapes and zero lighting.
   Some ignorance is excusable. Willful ignorance and self-delusion on the part of prominent figures who have high-paying positions in education or blogs about education are not excusable in my humble opinion. I’m sorry. This charade has been going on for far, far too long. We are supposed to be responsible adults. The people posting on this blog claim to care about children and education. Pretending that those millions of kids who are paying a dear price for having not been able to fit into the mold or for having the good sense to resist the programming and the abuse is criminal negligence. If the shoe fits, it’s time for a pedicure or a lobotomy or or something.

   LikeLike

   Reply