Jason Stanford writes a great blog about Texas politics. He has developed a special interest in education. After all, Texas was once home to the “Texas miracle,” and gave the nation No Child Left Behind.
In this column, he reports on what happened when a friendly legislator tried some questions on the fifth grade math test and found the wording to be confusing. He gave the same question to some of his colleagues, and the language puzzled them too. People with law degrees couldn’t understand the question. What should have been a straightforward math question turned into a verbal tangle because of the way the question was posed.
Pearson has a nearly $500 million contract with the state of Texas. Parents are furious. Legislators are thinking of canceling the contract.
This question does Pearson no good.
Diane Ravitch's blog 
Can we see the question?
LikeLike
If you read the article, the question is included.
LikeLike
Questions: http://www.genefortexas.com/docs/STAAR_combo.pdf
Answers: http://www.genefortexas.com/docs/STAAR_answers.pdf
LikeLike
Questions: http://www.genefortexas.com/docs/STAAR_combo.pdf
Answers: http://www.genefortexas.com/docs/STAAR_answers.pdf
LikeLike
Tip of he iceberg, of course.
On the NY State Regents, my students got the question on Darwin wrong
because the ‘correct answer’ was that Darwin believed in ‘survival of the fittest.’
As a bit of scholar on Darwin and Social Darwinism, I had told them that
‘survival of the fittest’ is not Darwin’s term, but came from Herbert Spenser,
who tried to apply the theory to social phenomena.
It is an important distinction, one that Stephen Jay Gould wrote on, because to
say something is fit because it survives is to define ‘fit’ only as regards current
circumstances and does not take into account possible changes in the environment.
Darwin did not fall into that trap, but spoke merely of ‘natural selection.’
I wrote the Regents board but never received a reply, at least not until two years
later when another ‘survival of the fittest’ question showed up. The phrase gives the impression that those that survive were preordained to survive and prosper. Hmmm . . . maybe it is accidental, but why would our current ed reformers want us to accept such a phrase as a scientific fact?
LikeLike
@Brian Ford, my guess would be that the”ed reformers” aren’t really that concerned about facts or the truth for that matter.
LikeLike
Well in CANADA they believe that Darwin coined the term survival of the fittest…
LikeLike
To all readers,
I am a Supt. in Texas and I just received this email. When I saw the words “school funding portability” I knew something wasn’t right. Does anyone know anything about this organization? ben.carson@leisd.net
Dear BEN,
I’m writing to share with you Reason Foundation’s new handbook on student-based budgeting.
http://reason.org/studies/show/handbook-student-based-budgeting
Click to access student_based_budgeting_handbook.pdf
We are in a transition period in the United States, moving from funding institutions to funding students. The growth of student-based budgeting in school districts and states mirrors a national trend toward more decentralized school funding where the money follows the child. More than 30 “school funding portability” systems (in cities like New York, Baltimore, Denver, Hartford and Cincinnati, and states including Rhode Island, Hawaii and Indiana) are funding students through student-based budgeting mechanisms. Los Angeles Unified has more than 100 pilot schools funded on a per-pupil basis. In California, Colorado, Connecticut, Georgia, Nevada, Ohio and Utah there are ongoing legislative debates about fixing the state school finance system through a weighted student formula.
Student-based budgeting proposes a system of school funding based on five key principles:
1. Funding should follow the child, on a per-student basis, to the public school that he or she attends.
2. Per-student funding should vary according to the child’s needs and other relevant circumstances.
3. Funding should arrive at the school as real dollars—not as teaching positions, ratios or staffing norms—that can be spent flexibly, with accountability systems focused more on results and less on inputs, programs or activities.
4. Principles for allocating money to schools should apply to all levels of funding, including federal, state and local dollars.
5. Funding systems should be as simple as possible and made transparent to administrators, teachers, parents and citizens.
While student-based budgeting is not a silver bullet, it is a school funding practice that makes resources more transparent, increases school level equity for students with similar characteristics, and allows the funding to follow the child. When parents and students have portable funding and can choose between schools within a district it provides a financial incentive for those schools to improve education practices to attract and retain families.
Thanks for your support of Reason’s education work.
All the best,
Lisa Snell
Lisa Snell
Director of Education
Reason Foundation
310-391-2245
http://www.reason.org
http://www.reason.com
http://www.reason.tv
LikeLike
Reason Foundation is a libertarian organization that supports vouchers.
When they talk about “portability,” they mean vouchers.
LikeLike
A “portability” bill just appeared in the Utah State Legislature. In this case, it’s not exactly vouchers, but more charter schools. I know, same old, same old. But Utah is a bit shy about doing vouchers ever since the citizens overturned vouchers by initiative several years ago.
LikeLike
I do a little bit of tutoring in an urban elementary school, and the first question that comes to mind when you see the kids try to prep for the state math tests is how much of their trouble is with reading rather than the actual math.
Add in some poorly written questions, and they’re really up the creek.
LikeLike
@corey, click on the link.
Ya know, for 27 years I’ve been teaching 5th grade and 5th grade math to Gen. Ed. students. When I started it was The Stanford Achievement Tests, then the Iowas. 4 days of testing, for maybe an hour a day, then done. Mail them out and when the results came back it gave information on each kid that might validate or nullify any concerns one would have academically for said student. Were they a great tool? No, but semi-useful. Now it’s test for 3 days each for math and ELA , spend god knows how many hours prepping (well I used too), and then we get pulled from our class to correct the written portions in May and June, another 2 days away from kids. The results don’t tell me a damn thing about the student, and since I’m in NY part of my APPR comes from these results. It’s so far beyond inequitable it defies comprehension. Why our local and NYSUT ever agreed to this is baffling. I can only hope that the lawsuits will be forthcoming and soon.
LikeLike
I was subbing for a middle school resource room teacher and a para was helping a student with his math homework, which was a page from the official workbook (just to make it clear it wasn’t teacher error). For several questions, the student/para came up with an answer that didn’t match the answer sheet, at which point the following process ensued: para discusses question with the other para-they come up with original answer, para brings question to me-we come up with original answer, para bring question to teacher next door-they come up with original answer, para brings question to math teacher-math teacher explains and it turns out that the either there was missing information or the question that FOUR educated adults thought it was asking was not the question it was asking. Para returns to class and we all agree that whoever wrote the curriculum should be bought up on charges.
It’s as if the questions were originally composed by a math person (hopefully), but edited for space by someone who knows nothing about math!
LikeLike
That’s what everyone needs to do. Get hold of questions and see how they’d do. Every legisator who supports testing of this sort should be obliged as part of his or her vote to take these tests and have their scores posted–i rank order. Maybe they shoud be allowed to chose the grade level they think they’re up to. Deb
LikeLike
The question is, indeed, poorly worded. It shows four pictures, each depicting bills and coins. The question reads, “Look at the amounts of money shown below. In which amount of money does the digit 4 represent four cents?”
The question requires the reader to figure out that the pictures represent money in varying amounts and that “amount” should be construed as a number made up of digits.
The question should read as follows: “Each picture below shows bills and coins. If you wrote the amount of money shown in each picture as a number, in which number would the digit 4 represent four cents?”
The question as I have reworded it fixes the semantic inaccuracies that the original question had. However, the question is now longer and so has a greater “reading load,” as test designers call it. A test of math skills should try to isolate those skills. To the extent possible, it should not ALSO be a test of reading rate. A slight increase in the length of questions can be a problem over a long test. Some kids who have decent math skills might not finish the test simply because they read too slowly, and those kids will have completed not a test of their math abilities but a test of their reading and math abilities.
All this raises an important issue. Politicians, pundits, and state administrators tend to think that testing is straightforward. They commonly think that if you want to find out whether someone knows something, you can just give him or her a test. This presupposes that everything to be tested is as simple as “What is the sum of 2 and 2?” or “What is the capital of Nigeria?”–that students will either know the answer or not. But it’s not that simple, not at all. Writing good test questions is extraordinarily difficult, so difficult, in fact, that one could easily create a graduate degree program dealing solely with the complex of skills involved in doing the job well.
I see problems like the one in this test question ALL THE TIME, on every state and national test that I review. It’s QUITE COMMON for questions to be so poorly worded that the answer the test designers are looking for is actually incorrect and one of the “distractors,” or incorrect answers, is actually correct. And, as a commentator above noted, the tests are full of factual errors like attributing the phrase “survival of the fittest” to Charles Darwin. (There’s a complicated history there that I won’t go into.)
Here’s the rub: If a test is poorly designed AND is being used for high-stakes determinations like whether a student will pass to the next grade, whether a teacher will receive a bonus or keep his or her job, whether a school will stay open or be closed, then no one is well served. However, designing tests well is extraordinarily difficult, and there are few people with the requisite skills, even among professional writers and editors for educational publishers. Inevitably, flawed instruments will be used to make these determinations, and that’s a shame for all affected.
BTW, both John F. Kennedy and Barack Obama misused the word “affect” for “effect” in Inaugural Addresses. They would have got wrong a standardized test question on that usage. It would have been pretty crazy to make high-stakes determinations about their overall abilities based on whether they knew the difference between these two words. My point is that reading, writing, and mathematics abilities are complex. I’ve known great scholars who didn’t punctuate well. Harriet Monroe, who published T.S. Eliot, Ezra Pound, and Wallace Stevens said that none of these literary giants could spell worth a damn.
LikeLike
You call too different situations “high stakes”, and I have often wondered about the difference. When it is an issue of promotion to the next grade, it is high stakes for the student taking the exam and presumably low stakes for the teacher. When it is an issue of teacher evaluation, it is a low stakes exam for the student taking the exam, but high stakes for the teacher. Is it important to differentiate between these two situations?
LikeLike
TE: They are high-stakes in both scenarios. Teachers who go through traditional teacher preparation programs are typically trained to avoid using the results of a single test taken on one day in a students’ life for making high-stakes decisions (the child’s dog could have died that day, etc.) Teachers are advised to use multiple measures and to include authentic assessment, such as student portfolios with work samples collected over time to demonstrate progress.
LikeLike
What about when the test is both?
There are several tests in GA that are required for graduation and are also used to evaluate the teacher.
LikeLike
The test might be high stakes for both, but that does not seem to be the norm. More interesting is when it is not. The differences between the two create far different incentives. If it is high stakes for the student, the student has a reason to study for the exam and do his/her best on the exam, and the teacher may well be trusted to grade the exam fairly. If it is only high stakes for the teacher, the student may just as well draw a Christmas tree with his answers on the scantron sheet (as did Bart Simpson), and the teachers and administrators can not be trusted to handle the exams, much less grade the exams.
One of the principal principles in economics is that incentives matter.
LikeLike
High-stakes testing is not considered best practice in education. The examples given are policies designed by non-educator politicians and corporate know-it-alls. For children with disabilities, the results of a single test may never be used to make any major decisions, such as regarding retention and promotion. It’s mandated that multiple measures must be used and it was educators and parents that made sure that was written into the law, Do you think that’s really only best practice for a population subset? I don’t believe so, and I was trained and have experience as both a regular ed teacher and a special ed teacher.
LikeLike
I am simply making a point that “high stakes” for the student is not necessarily “high stakes” for anyone else and “high stakes” for the teacher and administrator is not necessarily “high stakes” for the student. The two should be thought about differently.
Even if there are multiple measures, wouldn’t it be true that whatever is used to confirm previous findings regarding retention and promotion is “high stakes”?
LikeLike
I know of no test (in GA) that is high stakes for the student (required to graduate and or counts toward a large % of a class grade) that is not also used to evaluate the teacher and or the school.
You are correct, however, that some of the new tests that are currently being rolled out count toward teacher evaluation but are not part of student grade, etc.
I agree the second type of test is especially problematic. Very little incentive for the student to put much effort on it. Especially those kids that have a lot on their plates, so to speak. Why study for something that “does not count” when there are AP exams and SAT’s to prepare for? And as you mentioned, the struggling student who has been tested to death is very likely to “christmas tree” the answer sheet to “get it over with”.
The first type of test (“high stakes all around”) is NEVER graded by the teacher.
In fact, we are not supposed to even look at them.
The second type does have a small portion (free response) that is teacher scored.
LikeLike
I was under the impression that much of the gains on test scores in DC and Atlanta were the result of cheating on the exam on the part of teachers and administrators. To do that, the teachers and or administrators must have had access to the exams at some time. Was that an old practice that has now been eliminated?
LikeLike
“Even if there are multiple measures, wouldn’t it be true that whatever is used to confirm previous findings regarding retention and promotion is “high stakes”?”
Sorry, I can’t teach you how to be an effective teacher in one easy lesson, but I can tell you that this is not it. Look up how teachers use ongoing formative, summative and authentic assessment to promote mastery learning and you’ll get an idea about how teachers use multiple measures to inform instruction and facilitate progress, not “to confirm previous findings.” .
LikeLike
Other Spaces,
So there is never any single price of information about a student that would cause a teacher to change their evaluation of that student? Nothing that would change the grade given at the end of the term? My children have all had final exams in high school. I thought the performance on those exams would have some impact on their grades.
LikeLike
Though i agreevwith this comment, the funny thing about this test item is that all you have to do to get it right is find the picture with the 4 pennies. No need to add up the money, as there would have been ifbone had 4 pennies but added to $4.39.
LikeLike
Don’t you realize that misleading and ambiguous questions are only discovered by pointy-headed journalists and liberals… and pro-public school bloggers from Brooklyn!
LikeLike
Actually, Monroe said that of the poets she published, only Eliot could spell worth a damn. You see? It’s very easy for professional editors of educational materials to make errors, and creating error-free tests is very, very difficult. Given their inherent limitations, it’s a mistake to have standardized tests be used for a dozen different high-stakes determinations about this that and the other. They are the wrong tool for many jobs. Using them is like using a butter knife to turn a Phillips head screw or a screwdriver to butter bread.
LikeLike
Robert Shepherd, I need some posts from you about standardized testing!
LikeLike
I suppose the same problems would apply to tests written by individual teachers.
LikeLike
Sure, but a teacher can realize that there’s an error and drop the question (I have to do that sometimes). By the time a problem on a big standardized test is discovered, the scoring is done, and often makes it difficult or impossible to fix a problem.
LikeLike
But, unlike with standardized tests, individual teachers can give credit to students when questions on teacher-made tests are confusing, too arbitrary, easily misinterpreted, etc., and we can edit the questions for future test-takers.
I do this all the time at the college level. One reason I believe I have to be on top of that is because we are now teaching the NCLB generation of students and some of them are very alert to trick questions that they believe are intended to set them up for failure. For example, I have found that if there is even a minor spelling error in the correct answer to a true/false test question or to a word in the correct answer of a multiple choice question, some students will mark it false or not identify it as correct just because of that misspelled word.
LikeLike
I’ve always said the adults who choose the test should also take the test…if you can’t understand it or pass it why should your children be able to?
LikeLike
The profit motive drives an incentive for lower scores. Confusing questions on secret tests are the best case scenario for a corporation who wants broad power and revenue.
LikeLike
There were a lot of questions on history and civics, which were probably included in Wu’s sample because he was sending them to legislators. However, the curriculum in many districts across the country has been narrowed due to high-stakes tests that don’t often include Social Studies (nor does the Common Core), so you have to wonder how much about that students have been actually taught in the past decade.
I’ve often wondered if the real purpose of focusing on informational texts in the ELA Common Core standards was to make English classes/teachers do double-duty, in order to ultimately justify eliminating such disciplines (and those who teach them) by declaring them redundant, as in, “We can cut out Social Studies altogether because students learn history, civics, etc. in English classes now.”
LikeLike
Finally! Why has it taken so long to present the facts about these tests? Poorly written test questions are prevalent.
LikeLike