In response to my post earlier today about the growing movement against testing–and its misuse for rewards and punishments–Robert D. Shepherd sent the following comment:
“I think that it’s empirically demonstrable that these tests aren’t even valid and reliable as tests of reading, writing, and math abilities, much less of teacher and school performance. What’s next–shall we use the tests to measure the performance of the the neighborhoods the schools are in? the cities and towns? Crazy.”
If, as Shepherd says, the tests are neither valid nor reliable, then what is happening to American children and teachers must be considered the Crime of the Century.
On the other hand, consider the next logical step, which he proposes: based on test scores, we begin closing down towns and cities and renaming them, or giving them to charter operators or emergency managers….wait, that’s already happening in Michigan.
Diane Ravitch's blog 
It depends on your view of “worthless”. If you are a school-aged child, the parent of a school-aged child, a teacher, a principal or anyone else impacted by the negative uses and abuses of these tests, the tests are beyond worthless – they’re downright destructive.
But if you’re an edupreneur who wants to sell all the solutions to America’s “failing school crises”, the tests are priceless.
Well said, Dienne, well said.
Ditto.
1. The tests around the country are a mixed bag in terms of reliability and validity. Some are terrible. Some are fairly accurate for limited purposes. Bear in mind that reading, writing, and mathematics ability are all complex, multifaceted phenomena, and there are aspects of all of these that the tests simply do not cover.
2. In order to show validity, one has to correlate outcomes on the test to outcomes on other measures. The question then becomes, to what other measures does one correlate the state test outcomes? to grades? to graduation rates? to NAEP? to standardized tests like the Stanford 10? The answer should be “all of the above.” Certainly, comparisons of NAEP outcomes to state test outcomes has shown, in some states, large discrepancies between the measures. (One of the reasons for NAEP in the first place was to keep the states honest.)
Again, the state tests are a mixed bag. Some are well vetted. Some are not. NONE should be used for purposes other than the fairly specific, limited purposes for which they are appropriate.
The coming national tests will doubtless be subjected to a LOT of official and independent scrutiny, including official and independent tests of validity and reliability. That’s a good thing. It’s going to be interesting to watch that process unfold. I suspect that it’s not going to be pretty.
In business, many companies have recognized the danger of relying on single measures of overall health, and a great deal of time, thought, and money has gone into developing “balanced scorecards” that look at a broad range of measures, quantitative and qualitative, hard and soft. This has been done because businesspeople have learned how dangerous it is a) to rely on single measures and b) to overgeneralize from those measures. Last year’s financials alone won’t tell you whether you have the capacity to innovate to meet coming changes in the market, what the competitive picture looks like in the future, whether your customers are loyal or looking, etc.
Tests can be valuable FOR LIMITED APPLICATIONS. But lean on them too hard, make them do work they are not designed for, and you can do a lot of damage.
lohvfl,
Any and all standardized tests, and the accompanying “standards” are invalid as shown by Noel Wilson’s 1997 dissertation “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700 or for an easier read: “A Little Less than Valid: An Essay Review” found at: http://www.edrev.info/essays/v10n5index.html or http://www.edrev.info/essays/v10n5.pdf .
Read and understand these works and your will understand why.
Duane
As they are currently being abused, the tests are harmful to students, schools and teachers. I think standardized tests can serve useful purposes as well. Sometimes they point to the potential and skills of students that teachers haven’t uncovered.
Validity and reliability are not all or nothing– but many points in between. And the critical component of validity is “purpose”.
I believe the original purpose of standardized tests was to guide instruction. Am I wrong about that? We call this school year ‘the year of the test’ at my school… state testing, online benchmarks, writing assessments in ELA and in math (CRAs), system benchmarks, etc., etc., etc….. and the resulting data is, as much as anything, a punitive weapon….
One of the questions one asks with regard to reliability of tests is how likely it is that the same student, taking the test another time, will get a similar score. But what does that sort of reliability even mean when one can change outcomes this dramatically by making an intervention that has NOTHING WHATSOEVER to do with the student’s ability (in read, writing, math, or whatever)?
http://www.slate.com/articles/health_and_science/science/2012/09/how_children_succeed_book_excerpt_what_the_most_boring_test_in_the_world_tells_us_about_motivation_and_iq_.html
Such results should give people pause. Tests aren’t and can’t be everything, however well constructed they are.
Here’s a regression analysis of the ISAT (Illinois’ state achievement test) results in Chicago Public Schools with respect to elementary school demographic information: http://bit.ly/YQTxp1 In short, you can predict 60% of the variance in school performance (as measured by the % of students meeting or exceeding state standards) from the percentage of a school’s student body that is low income, black, or special education. If you are measuring performance by the % of students exceeding state standards, those three variables will predict 70% of the variance.
Basically, ISAT performance is a great measure…of who’s attending which schools. But since we know that information already, maybe we should stop wasting two weeks of the school year on testing and umpteen weeks on test prep. And maybe we could also stop basing school closing decisions on ISAT performance as well. But that would mean admitting the ugly truth that school closure decisions are based on race and class.
A parent shared her heartfelt story of her autistic son and the harm of high-stakes testing with me. I would love to see the politicians and bureaucrats who make education policy visit a special education classroom on the days that the ELA assessment is given.
http://atthechalkface.com/2013/03/03/guest-post-high-stakes-testing-hurts-our-most-vulnerable-students/
Detroit is out of money. That’s why Snyder is proposing an emergency manager, in order to SAVE the basic services to the people of the city and to keep city workers employed. It is embarrassing and galling for a city to be subjected to outside management when it can’t manage itself by the usual democratic processes. But it has proven it can’t, or at least that it won’t. The city has in the last 10 years claimed that the state owed it money, all the while being governed by a mayor currently on trial for taking kickbacks himself and also arranging city contracts for his father and buddies. It’s just basic theft and corruption and dishonesty and incompetence. City government was seen as a way of personal enrichment by it’s elected leaders rather than good government for the benefit of all the citizens.
Testing likewise exposes the reality of an individual’s ability to manage himself and his learning. Back in the day (as the kids say) there was only one test, the SAT, and that was a college admissions test. Its letters stood for “Scholastic Aptitude Test,” and it was supposed to let colleges know whether you were good enough at learning to be admitted whether you went to school at a ritzy prep school or a weak inner city or country high school. Even back then curricula were to some extent warped in the direction of test prep. In math I was prepared for the test but not for college math.
There probably has to be some kind of test, at some point, to sort out the students, but Diane is totally correct in her view that good teaching depends on the individual teacher’s knowledge of the individual student. When the emphasis is on real learning, any outside test, in my opinion, will take care of itself. WHY individual teachers are no longer trusted to teach but are subjected to the current perpetual testing regimens is an historical question to which I do not have the answer, but the public perception of “failed schools” in “failed cities” seems to be part of it.
Neither democratically elected city administrations nor democratically elected school boards are getting the job done well in the inner cities. Thus the external oversight.
Anyone who believes that any government entity is simply “out of money” may be interested in buying a bridge I have for sale.
Chicago is supposedly “out of money” too, which is why we need to close “underutilized” schools. Oddly enough, though, we have the money to open dozens of new charter schools. Funny, that.
I agree with lohvfl – I’m not sure who Robert D Shepherd is, but I’d be curious about his conclusion that “I think that it’s empirically demonstrable that these tests aren’t even valid and reliable as tests of reading, writing, and math abilities, much less of teacher and school performance.” That statement is too broad, as there is variability between tests, and also multiple elements of “reading, writing, and math abilities” to be measured.
Diane, since you’re posting this I’m guessing Shepherd is a trusted source and that you’ve seen this empirical support. Would you mind passing along references you’ve seen to support this assertion?
Ededed these conclusions are well documented in Dr. Ravitch’s book, The life and death of the great American school system. Fascinating, eye-opening read for all.
What does it mean to graduate from high school? What should a high school graduate be able to do?
That is a great question, teachingeconomist. Another great question is “how long should a student be given in order to reach the graduation goals, and what do we do with a student who exceeds the time limit?”
I do not have any useful idea about how long the time limit should be, but if it is exceeded the student should not be given a high school degree. Graduating from high school should mean something.
Graduating from any educational or training experience means something… but it never means the same thing for every person– and when we try to make it so, we end up creating more damage or more illusions– or both.
What does having a high school diploma mean to the outsider?
There was a time not too long ago when my university accepted any in state student with a high school diploma. At that time a high school diploma meant that the student was capable of doing college level work. Graduating from high school no longer has that meaning.
I have similar questions regarding the tests: are they serving the intended purposes? If not, why? If so, what metrics are being used?
But why do these conversations seem to involve all the peripheral stakeholders but not the parents or children, the target population? I believe lack of these stakeholder’s involvement from the beginning reduce the chances of success and sustainability. This may explain our past failures. Parents and children have opinions and resources necessary for success.
I have come to the regrettable conclusion that tests are a demonstration of the power of the few who are so elite, no one would think of giving them a test. If you have to be winnowed you take a test. If your status in our culture is assured, you don’t. Tests are a way of proving yourself to the already proven, so it doesn’t matter that they are not valid. If the guys (mostly) who rule say they are valid, well then, they are. The fact that the sale of these measures lines their pockets and contributes to their already considerable power is the icing on the cake. Sorry this sounds so bitter, but I come from a generation that was vaulted into a career by public schooling and the more I read about this the more I realize we have a plutocracy, and a nasty one at that.
BTW, when students learn music, their standardized tests improve. This is a huge irony since this over testing regimen has consistently wiped out music programs. My husband’s full band program (that he created over the last 10 years using private funds) in his PS/MS in the South Bronx was shut down completely this year because the school needed the time for testing. Students weren’t doing so well on their standardized tests and therefore, the school is in danger of being shut down or renamed or cleaned out to make way for a corporate charter.
Here’s one of many studies:
“TEN-YEAR STUDY SHOWS MUSIC IMPROVES TEST SCORES
Regardless of socioeconomic background,
music-making students get higher marks in standardized tests.
UCLA professor, Dr. James Catterall, led an analysis of a U.S. Department of Education database. Called NELLs88, the database was used to track more than 25,000 students over a period of ten years.
The study showed that students involved in music generally tested higher than those who had no music involvement. The test scores studied were not only standardized tests, such as the SAT, but also in reading proficiency exams.
The study also noted that the musicians scored higher, no matter what socioeconomic group was being studied.
Reference: Dr. James Catterall, UCLA, 1997.”
And….
Guess what? Learning a musical instrument teaches kids how to learn!! They also get to feel the joy of accomplishment from practice. Imagine that! Music is a natural way to instill the joy of learning into our students from everywhere, including our most underserved neighborhoods. They don’t know their underserved. They just know that they can play “Mercy Mercy” on their trumpet with their band mates and swing. Then they get the triumph of performing for their school, receiving applause and honors.