Our policy makers are in love with standardized tests. They
can’t talk about education without talking test scores. If I could
wave a magic wand, I would have every politician, every pundit, and
every state commissioner take the 8th grade math test and publish
their scores. Or take the PARCC test for 8th grade. If they did,
the results would be interesting and there might be less
complaining about our kids, our teachers, and our schools. Peter
Greene explains
here why standardized tests are meaningless. I think they
may be useful for diagnosing problems and helping kids. I think
they are useful for trends. But their limitations and gym flaws are
too great to use them to rank and rate children or determine their
life chances.
A site to discuss education and democracy
33 Comments
Leave a comment
-
Diane Ravitch's Blog by Diane Ravitch is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
Based on a work at dianeravitch.net.
Diane Ravitch's blog 
Diane, In LA, they are giving the end of year test next week. We still have 6 weeks of school left. They give them now to get the results by the end of the year. The student must pass the test to be promoted to the next grade. Wonder what would happen to the student if the parent opted the student out of the test?
Of course, the students are being taught under the Common Core, but they are being tested on the old state standards. Of course, none of this is aligned.
Gretchen Lampe
Sent from my iPad
>
Peter Greene is quite correct in his astute criticism of standardized testing and its relationship with the teaching and learning process. He has hit on a number of basic areas where standardized testing fails the “logic test” one of the main ones being:
“People in the testing industry have spent so much time convincing themselves that aspects of human intelligence can be measured (and then using their own measurements of measurement to create self-justifying prophecies) that they’ve lost fact of that simple fact:
You cannot know what’s in another person’s head.”
But there has been a person from the testing industry who has completely destroyed the validities of standardized testing and the accompanying educational standards, Noel Wilson. Read and understand his never rebutted nor refuted “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
1. A quality cannot be quantified. Quantity is a sub-category of quality. It is illogical to judge/assess a whole category by only a part (sub-category) of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as one dimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing we are lacking much information about said interactions.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. As a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it measures “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
Maybe our fallback position needs to be to reply to any reformer pushing tests, “Have you taken one? What was your score?” Pushing something they do not use (and likely their kids have not undergone) is a little like someone pushing a miracle cure that they themselves have not tried. And that reminds me of snake-oil salesmen.
Kindly explain why the North Carolina passing rate for ED kids is 17% while the passing rate for NED kids is 49% (for Asians it is 56% – the highest). Does that have anything to do with fewer ED kids stay in school and graduate and can read their diploma when they graduate? Does that have anything to do with the fact that over 50% of the kids entering college (community or four year) require remedial help to do the work?
If one does not give tests how can one be sure that the kid is learning the material? How can one be sure that the kids are ready to move on to the next building block of knowledge? How can one be sure that the kid who is graduating from high school can read their diploma?
Perhaps part of the problem is that the starvation diet of funding in North Carolina and elsewhere cuts the services that Emotionally Disturbed kids need (I am assuming that is what your ED acronym means). Class sizes balloon, special education and other services are cut, materials are lost, classes are cut, etc. The problem with all of this standardized testing is not only does it tell us absolutely nothing, but it cost hundreds of millions of dollars per state to give and grade them. These tests tell us nothing because no one can see the test questions and so cannot guide instruction by what students have missed. The money that these tests consume could be MUCH better used giving Emotionally Disturbed and all other students smaller class sizes, enriched course offerings, clean, well-kept buildings, and services for those who need extra assistance.
And saying that huge numbers of students “can’t read their diplomas” is disingenuous, at best. I teach middle school, and while I have had my share of students with low reading levels, I have NEVER had a student with such a low reading level. I have taught for 13 years, including three in an alternative school, and eight others in Title I schools, so if these supposed students existed, I think I would have seen them by now.
Hello,
Just for information — ED stands for Economically Disadvantaged or poor while NED stands for Not Economically Disadvantaged or not poor.
Those are the headings used in the NCLB or No Child Left Behind reporting individually for every school (public and charter) in North Carolina. The schools are aggregated to the school district and then the state.
“Does that have anything to do with the fact that over 50% of the kids entering college (community or four year) require remedial help to do the work?”
The latest Big Liar Contest winner on this topic was none other than Arne the Duncan when he proclaimed on his secret trip to Massachusetts to visit a charter school that forty percent of Massachusetts high-school graduates are taking remedial classes when they go to four-year universities.
Carol Burris refuted this handily here:
http://www.washingtonpost.com/blogs/answer-sheet/wp/2014/03/17/are-american-students-grossly-unprepared-for-college/
About a third of the incoming first year students at my institution take a remedial mathematics class. There are no remedial English classes, but a large fraction of the incoming first year students place out of the lowest level English class.
Are American students grossly unprepared for college?
http://www.washingtonpost.com/blogs/answer-sheet/wp/2014/03/17/are-american-students-grossly-unprepared-for-college/
“Meanwhile, Americans need to know that the sky is not falling. The National Center for Educational Statistics provides a far more accurate picture of remedial rates than those the Common Core reformers would have the public believe. It is also important to know that the United States is No. 4 in the world — just behind Canada, Israel and Japan—on the percentage of adults with college degrees.
There are two important takeaways from this story. First, is that misinformation is part of a continuing strategy to paint a picture of American public schools as failures in order to sell the public the Common Core, charter schools and the corporate reform agenda.
The second takeaway is that fact-checking is too often ignored by the press. The distortions are repeated even when, as in this case, common sense should call them into question. It is the responsibility of the press, not the public, to get the story straight and make sure the truth is told.”
Greene’s explanation on his blog is relevant to neuroscience. They say that the fetus has all the neurons in the first 3 months which only those that are actively connecting to those will survive (synaptogenesis) and the excess connections will go through synaptic pruning especially in the first year after birth. The implications of this as we all know is that environment plays a huge role very early in life and well into young adults years as the brain continues to develop.
IMHO, this process and exposure in early years is what makes us all different in how we learn and who we are. While there is much debate between neuroscience and education research, we can agree that we all the same in that we are different intelllectually. So how can anyone propose that a high-stakes standardized test is a valid and reliable measuring tool that determines if a student deserves a high school diploma and ready for college?
Answer: Reformers know that this isn’t about “doing what is right for kids.” If it was, they would selfishly keep it to themselves and in their childrens’ schools. Because they don’t allow their innovations to be applied to their children, it would seem suspicious and harmful. I perceive that reformers are spreading a virus, but immunizing their children from being infected. Isn’t that an act of a terrorist? Or am I being to lenient?
Consider this: scores on math tests are related to many things, stressors included, but it seems perseverance is key to doing well on testing. I believe it is true. A student has to care enough to do his/her best and to keep on trying. Confidence in doing things correctly and stopping when you think you have done enough is also necessary. Point being: so many factors other than “knowledge” and “learning” go into test taking. Many of these things are intrinsic in test takers. To use a test to prove that a teacher is unfit is absurd.
http://m.theatlantic.com/education/archive/2014/04/the-countries-where-kids-are-terrified-of-math/360102/
OK, Peter Greene is my Brother from Another Mother.
Peter even plays jazz trombone! I am a jazz guitarist (when I am not, like Peter, writing, teaching, and railing about education deform generally and the idiocy of the new “standards” and of standardized testing in particular).
Another great piece, Peter. Thank you!
All who frequent Diane’s blog. Please take time to read Peter’s beautiful, profound essay, and the many others on his blog, which are so informed, reflective, compassionate, witty, erudite, and humane.
Keep testifyin’, Brother Peter.
And signifyin’
Start of a good thing?
The Edudeformers’ Nightmare Club Band featuring Peter Greene on trombone and Bob Sheperd on guitar. Get your seat tickets now!!
Any others joining in???
Joanna??
It would be fun to do to the Deformers what Mingus did to Faubus.
http://www.jazzwax.com/2009/05/charles-mingus-fables-of-faubus.html
Duane Swacker & Bob Shepherd: I don’t know why, but reading your comments I suddenly heard “Rasheed” [INTO SOMETHING] by Yusef Lateef playing in the background.
Followed by “Someone to Watch over Me” [IMPROVISATIONS] by Stéphane Grappelli.
Ending with “Jazzman” [HER GREATEST HITS] by Carole King.
“Where words leave off, music begins.” [Heinrich Heine]
😎
Will have to check them out later. Just came back from DU meeting/fish fry on how much we raised on our dinner a month ago. Not into the thinking mode at the moment, time for bed. Will check it out tomorrow.
Ah, Krazy! So fine!
Recently, I came across two guys on the street, a guitarist and a violinist, channeling Django and Grapelli.
Playin’ real good for free.
Don’t know that one could get bet than Django and Grapelli!
You got lucky!
Bob Shepherd, Peter Greene makes me laugh out loud.
Me, too! I LOVE this guy!
If we changed the title of Peter Greene’s post to “High school diplomas tell nothing” how many other things would need to be changed in the post?
There are a few words that might have to be changed, like instead of “standardized tests are a joke” there might be “non-standardized teacher written tests are a joke”, but for the most part everything he says about standardized testing would apply to high school graduation (or college graduation for that matter).
TE, the high-school kids’ transcript reflects how he or she performed on an enormously varied set of tasks in the judgment of an enormously varied set of people, his or her teachers in many classes, over several years.
Yes, some of what Peter says applies to any measurement that we do of another’s abilities or knowledge, but some of it applies, in particular, to these narrowly conceived standardized test measures.
Go read the responses on TestingTalk.org., TE. Find out what teachers think of the standardized tests that they are seeing from these testing consortia, about how valid they think those instruments are.
cx: kid’s, of course
Which part of Peter Greene’s post does not apply to earning a high school diploma?
I have a hard time finding any part that does not apply to measurement of another ability or knowledge.
I tell you what, TE, why don’t you and I and Peter get together over dinner. This is going to be a long discussion. I would have said over beers, but I don’t drink.
And where did you learn this technique of answering every post with a question? Is this a teaching technique? Are you a reader of Plato, perhaps? Are you the wise, Socratic gadfly on Diane’s site, here to pose penetrating questions to straighten out less-than-careful thinkers? Do you sometimes ask questions to which you have no preconceived answer? If so, what sorts of questions? Are they the sorts of questions that can appear on standardized tests? Do you consider yourself a Platonist generally? Do you hold a correspondence theory of truth, and if so, how do you avoid falling into a naive realism, and to what sorts of statements does it apply?
Sorry, TE. I’m just ribbing you. Busy with work here. Best, Bob
TE, here is the key line from Peter’s piece:
“Recognizing that performance tasks are complicated and bubble tests aren’t, standardized test seemed designed to rule out as many factors as possible.”
So, as you can see, he agrees that his ideas are broadly applicable but that they are PARTICULARLY and ESPECIALLY applicable to standardized tests.
And he is right about that.
I would interpret that line to referring to all multiple choice tests, not just standardized tests.
Unfortunately VAM has given standardized tests an unintended meaning
Slam VAM thank you maam????
Diane: Have you read and written any review comments of James Heckman’s 2014 book “The Myth of Achievement Tests”? I’m about 20% of the way into the book at it’s intriguing, to say the least. The book discusses the recent fixation on measuring performance and really focuses, specifically, on the GED. Would love to hear your thoughts about the book. Thanks. Loren