California: Testing Incoherence Continues

16 Comments Post your own or leave a trackback: Trackback URL

1. Yvonne Siu-Runyan says:

   August 27, 2017 at 2:06 pm

   This is why all this online stuff is plain stupid.

   LikeLike

   Reply
2. Marian Cruz says:

   August 27, 2017 at 2:17 pm

   The word, incoherence, fits this testing frenzy well. It’s utter nonsense, it’s harming many students and money spent on testing could be used for music, art, drama and after school programs!

   LikeLike

   Reply
3. Bob Shepherd says:

   August 27, 2017 at 3:53 pm

   Is it because, as in Florida last year, the scores are so low that the state has to take more time to figure out how to massage them because they can’t set cut scores below the level one would achieve, on average, by random guessing?

   LikeLiked by 1 person

   Reply
4. Duane E Swacker says:

   August 27, 2017 at 4:15 pm

   Start with crap, end with crap.

   That’s all one needs to know about the CA test score situation.

   LikeLike

   Reply
5. LeftCoastTeacher says:

   August 27, 2017 at 4:37 pm

   I can think of a few possibilities for the “data issue”. The testing website had another one of its many glitches or crashes? The data security was breached and everyone’s personal information stolen? The data were accidentally deleted? The California Department of Education is thumbing its nose at the U.S. Department of Education and the required attacking of 5% of schools with greatest needs according almost solely to test scores instead of California’s more comprehensive school info Dashboard? (I’d bet that one.) The California Charter Scams Association lobbied to ensure charter scam scores were better than those of public schools and they weren’t? Another coverup of cheating by reformies? The scores were altogether either ridiculously low or high? Eli Broad’s dogs ate them? Kevin Johnson and Michelle Rhee ran out of billionaire support and are holding the data for ransom to pay mansion mortgage? Someone in Sacramento suddenly realized that Common Core is a joke? A very unfunny joke? A painfully horrendous, remarkably cruel prank of an unfunny joke?

   LikeLike

   Reply
6. JS says:

   August 27, 2017 at 4:47 pm

   The odd thing is that some schools have received scores, and so have some families. Others have not. Who knows with this garbage.

   LikeLike

   Reply
   • LeftCoastTeacher says:

     August 27, 2017 at 5:36 pm

     I was wondering why some of my colleagues were talking about “their” scores last week.

     LikeLike

     Reply
7. Threatened Out West says:

   August 27, 2017 at 7:33 pm

   Utah had to regrade all of the essays this year in several grades because the rubric was way too hard, and the scores were really low as a result.

   LikeLike

   Reply
   • Duane E Swacker says:

     August 27, 2017 at 7:57 pm

     Solo puedo decir ay ay ay.

     LikeLike

     Reply
   • Bob Shepherd says:

     August 28, 2017 at 2:31 am

     One of the many nasty little secrets of all this testing madness is that the tests can have any results that the test makers and state decide that they want. How to manipulate the “data” before and after the test date. . . let me count the ways. . . . No, I’ll skip that here. Too long and boring.

     Test-making is a very lucrative racket for otherwise failing educational publishers. They can churn this stuff out for a pittance and not spend a dime on any real INDEPENDENT validity and reliability testing or norming (the state tests are not normed–they are so-called “criterion-referenced” tests). And they don’t have to spend a dime on any real critique by INDEPENDENT experts (imagine English professors and linguists looking at the reading tests instead of a few educrat bobblehead consultants paid to rubber stamp this crap–the tests would never survive THAT). So, the publishers can charge a fortune for a product that, unlike textbooks, has negligible manufacturing cost. Some of these publishers would have tanked completely by now, for their textbook divisions are doing really badly, if it weren’t for their lucrative testing scams.

     I think of these state standardized [sic] tests as the game booths of the education carnival midway. Many points of comparison there. But at least the carnival games are fun, if you like that sort of thing.

     Because few people know much mathematics, the test publishers can hide what they are doing behind pseudo-mathematical mumbo-jumbo, knowing that no one in any state education department and no one at the Dept of Education will call them out, and they can hide the sloppiness of the questions they write by not releasing the items (which also makes it possible for them to recycle these items again and again, further reducing their meager development costs). When they are forced to release items, they naturally try to release the very best, but the very best that they can find in their item banks are TYPICALLY just awful. The released items don’t survive any scrutiny, which is why, in recent years, publishers have tried to release as few as possible and have made discussing problems with items on past tests a crime via nondisclosure agreements. Every test-taker and every proctor has to sign one. Nonetheless, after every test, the kids come screaming away from them talking about nothing else but the utter ridiculousness of the questions. And guess what? The kids are right, but you won’t know that, because you can’t look at the questions.

     I have a deal for you. I have a car for you. You pay me a LOT of money, but you can’t look the product before you buy it. Trust me on this. I also have some nice waterfront property for you down here in Florida, but you can’t look at that before buying it either.

     But, amusingly, the test publishers sometimes get tripped up by their own greedy haste. This is fairly common with grifters. They OFTEN do their jobs so poorly that they are left with the embarrassing situation of having a test they’ve already given with scores so low that they have to choose between passing almost no one or setting the cut scores at a level barely above what a student would get, on average, by simply guessing.

     Of course, the worst thing about the state standardized tests is that, in ELA, none of them have any reliability or validity. They simply don’t measure what they purport to measure.

     In ELA, the “standards” are so vague and abstract that one can write a test question for any one of them at any level of difficulty whatsoever and there’s no particular thing that a given “standard” describes but, rather, a vague, amorphous, ill-defined or actually undefined range of phenomena. In other words, the standards are often, in ELA, so vague that they cannot be rationally operationalized sufficiently to measure them reliably or validly.

     And, in ELA, the test makers try to use what people used to call “objective test formats”–multiple-choice questions–to measure sophisticated, complex thinking, which is entirely inappropriate, so they end up with test questions that are extraordinarily confused and confusing–for which, arguably, as written, none or more than one of the given answers is correct or the question is not actually answerable (again, as written) at all.

     And then, of course, the data [sic] from these tests comes in many months after the students have moved on to another grade level and other classes, so even if it were valid and reliable and actionable (which it’s not), it is not, as a practical matter, actionable by the teacher who had those students when they took the test.

     Let’s listen in on a Principal/Teacher “Data Chat”: “Well, our data [sic] show that your kids last year were having problems with their analyzing multiple interpretations of a literary work skill.” ANYONE WHO TAKES A STATEMENT LIKE THAT AT ALL SERIOUSLY–WHO DOESN”T SEE THE MANY PROBLEMS THERE–SHOULD BE IMMEDIATELY REMOVED FROM ANY POSITION OF RESPONSIBILITY FOR SCHOOLS, for he or she simply CAN’T THINK AT ALL CLEARLY.

     And, of course, traditional English curricula are now completely out the window. English class is now, in most schools, across America, test prep class.

     I’ll just give you one more small taste of the ludicrousness of these supposed measures. THE CCSS for ELA, Grades 11 and 12, have a standard calling for the students to be familiar with foundational works of 18th-, 19th-, and early 20th-century American literature. In a typical lit text from ten years ago, that period would cover some 300 or so authors and a dozen or so multi-decade periods of literature. This is, BTW, one of the few references to content in the otherwise almost content-free Common Core State Standards for ELA. So, a typical state standardized ELA test will have one or two questions on one or two paragraphs from one or two texts by one or two authors from this 250-year period and purport to have measured whether the students have met that standard–are familiar with the major or foundational works from 250 years of American literature. And those one or two questions won’t get at familiarity with the author(s) or the work(s) or the literary period(s). They will be just a couple more extraordinarily badly written skills questions. So, teachers can just skip this standard because it’s only one of many, and it would take an entire curriculum actually to meet it. Easier to do what they are now doing, practice those skills standards and questions, covered by all the rest of the test, day in, day out.

     Our politicians and many of our educrats have been suckered in by people peddling numerology as data. . . .

     Numerologists are running and ruining our schools.

     How about a little accountability on the part of the accountability folks, for a change?

     How about INDEPENDENT special investigators to look carefully at the multi-billion-dollar, nationwide fraud that is the standardized state testing industry?

     For that’s what it is. Fraud.

     LikeLike

     Reply
     • dianeravitch says:

       August 28, 2017 at 8:11 am

       Bob, so true.

       The tests are a social construct. The only thing standardized about them is the machine scoring. The scores arrive when the student has a different teacher, and the teacher is not allowed to learn what the students got right or wrong. They are junk science. Make that junk pseudo-science.

       LikeLiked by 1 person

     • ciedie aech says:

       August 28, 2017 at 1:31 pm

       YES.

       LikeLike

8. Laura H. Chapman says:

   August 28, 2017 at 8:58 am

   Bob has given readers a primer on the 10000000 ways that scores on tests are contrived to be educationally meaningless while being persuasive to anyone fixated on ranking people and their accomplishments.

   The marketing of tests is replete with claims of precision, calibration, objectivity, reliability, validity, high quality, and the rest. Snake oil.

   My experience is not as deep as Bob’s but he did not mention another deception: field testing items on the cheap by adding questions to a high stakes test, hoarding that data, but eliminating responses to those questions from the final score of the student.

   The extra items are then placed in a pool of ” field tested items” for possible use in another test. These items, administered to many students, generate enough information for test makers to classify them as relatively easy or difficult. When a new test is constructed, it is designed to include easy-to-difficult items in proportions that will yield scores approximating a bell curve.

   The testing apparatus enables much posturing about grade-level proficiency, and other grand myths associated with meeting, exceeding, or failing to exceed grade-level “performance.”

   The demands for confidentiality protect the test maker’s bottom line. I speak from formal studies of test making and experience on item development and interpretation of results on two NAEA tests…among other informative misadventures in education.

   LikeLike

   Reply
   • Bob Shepherd says:

     August 28, 2017 at 11:04 am

     “informative misadventures” . . . LOL . . . well said

     LikeLike

     Reply
9. Bob Shepherd says:

   August 28, 2017 at 11:59 am

   The interesting thing is that almost all English teachers know that the state ELA testing is pseudoscience. Astrology, phrenology, numerology, Dianetics, Tarot reading, wearing magnetic bracelets to cure cancer, state standardized ELA testing–all the same sort of thing. They know this, but they keep their mouths shut because they can lose their jobs if they speak up. They dutifully put up their data walls and nod affirmatively at their data chats and then look at one another when their doors are closed and shake their heads about people at the highest levels in our education system being duped by this stuff.

   When you ask a test question, that question operationalizes what you are measuring. The question stands in for the larger thing being measured. So, you have to ask yourself, does that question actually measure what people claim that it is measuring–in other words, is it valid? And you have to ask whether it reliably does this–in other words, are the results repeatable across trials or populations. In ELA the state tests simply are not valid or reliable, and almost every English teacher knows this. And almost every one of them now lives in terror of his or her administration finding out that that’s what he or she actually thinks. People don’t want to be fired for not being with the program, and at the same time, they know that the program is a sham. It’s a classic double bind.

   One problem is what’s being measured–the “standards” for ELA. It’s a fairly easy matter for me to write test questions to determine whether you know your multiplication tables up to 12 times 12. So, if the standard says, “The student will have memorized the multiplication table through 12 times 12,” then that is measurable. But if the standard says something incredibly broad and abstract and vague like the student will be able to “Analyze multiple interpretations of a story, drama, or poem . . . evaluating how each version interprets the source text,” then it’s simply impossible to write one or two multiple-choice test questions that will validly–accurately–measure attainment of this “skill.” Consider any part of that standard. What is meant by analyze? Is the student to look at two interpretations, break those into their parts, show how those parts are related to one another (that’s what “analyze” means) and then compare them? And this is supposed to be done in one multiple-choice question? Crazy. What multiple interpretations are we talking about–there are literally hundreds of quite distinct approaches to interpreting literary texts, and mastering any one of them is a long-term, complex undertaking, and the various approaches to interpretation, while overlapping, are not commensurate. There is no such thing as measurement of interpretation skill IN GENERAL. And what texts are we talking about? Any story, drama, or poem? Tony the Tank Truck or Waiting for Godot? There are hundreds of different types of stories, dramas, and poems, and learning how these hundreds of different types work–what their parts are, how they are structured, what techniques used in them, what and how they meant in the historical contexts in which they originated–all this is prerequisite to being capable of analyzing, authoritatively, interpretations of them.

   But it ought to be clear enough to anyone that the one case (measuring whether people know their times tables) and the other (measuring whether they are able to analyze multiple interpretations of a literary work) are different kinds of thing altogether. The former can be validly done with a few multiple-choice questions, and the latter cannot. And if it can’t be validly measured, then the test is a sham.

   Another problem, of course, is that the vague standards leave out much of what traditionally constituted the curriculum in literature class–genres, periods, specific rhetorical techniques and types of figurative language, specific authors and texts but have been taken by curriculum developers as a map to the curriculum. So, English class has become Common Core test prep skills class. And as a result we’ve lost one of the important traditional functions of English class, which was to turn kids on to the great authors and literary works of our past–to give them the lay of the land there–and to provide the scaffolding to make those accessible.

   LikeLike

   Reply
10. bradinsocal says:

    August 28, 2017 at 3:39 pm

    Well said, Bob. Even funnier, last year in California, a huge proportion of the ELA test seemed to be about science, history, and “report writing.” (I say “seemed” because we were honor-bound not to look, but sometimes students had computer issues, so I couldn’t avoid looking at the screen.) When teachers ourselves did the practice exams on computer beforehand, I was reminded how soul-killing the whole exercise was. How any sane person could hope to get any meaningful data out of all this is remarkable. I guess the payoff is “worth it” for the data overlords. Another reason I was glad to retire in June. I can’t see all the craziness ending soon. Good luck to all still at it.

    LikeLike

    Reply