Computer adaptive assessments are all the rage. They are supposed to be not only cost effective but they allegedly are objective and standardize grading. Also, and not incidentally, they are big business in an age of mass testing.
The idea behind them is that the student answers a question (picks a bubble), and if it is the right answer, gets a question that is slightly harder. If the answer is wrong, the next question is slightly easier. In this way, the computer soon figures out what the student’s level of competence is. Watch for the next round of computer assessments that score student essays. Expect an end to imaginative writing as computers are not programmed to understand what they have never before encountered.
This parent explains why her daughter doesn’t like computer adaptive assessments and how she copes with them. It appears that her daughter has never taken a test that asked her to show what she knows, just to pick the right bubble.
My kid doesn’t like online adaptive assessments. She likes knowing there are 50 questions in 40 minutes. She hates tests that give you many more difficult questions when you answer correctly. The test seems to go on forever.
So, one time she decided to hit buttons randomly and get a bunch wrong. Then the computer spit out fewer, easier questions, and she was able to finish the test at last.
Diane Ravitch's blog 
To be fair, isn’t this a problem with all assessments–that students can easily guess randomly, give up, or not answer?
Not with ALL assessments, only with standardized tests.
Excellent point. I should have been more specific.
The other HUGE problem with this test is that you don’t know why a student answered the way they did.
But, again, you don’t know that with other standardized tests either. I remember taking written tests in elementary school when a question like “what would be the best title for this article” would be asked. I’d always choose the most interesting sounding title, even if it didn’t capture the main idea, which was really what the question was asking. My teachers didn’t know why I selected that answer, perhaps assuming that I didn’t understand main idea. I just don’t think the “problems” laid out here are specific only to computer adaptive tests. I was hoping for more.
it’s easy to ask them. With a computer test that’s depending on the software but the students could be asked to have a small paper and write down the item numbers they did guess or had a hard time.
Some computer systems also track how much time the student needs for each question.
One year I gave the same Regents practice test to 4 classes of sophomores and one class of seniors who had failed a number of times before, but who were very motivated to graduate. The sophomores with nothing to lose scored passing grades, while the seniors did poorly. They were test-fatigued and stressed, and the more they were put under pressure the worse they did.
When I gave these computerized tests to my students with special needs, it was heartwrenching to watch. From the many (more appropriate) assessments we had already given to create and improve on the children’s IEPs, we already knew many of our 3rd, 4th, and 5th grade students were reading at the K-2nd grade levels. To have a computer screen full of 5th grade level text pop up, and then seeing the kids expressions was horrible. I encouraged them to guess, and “try their best”, but many just clicked randomly through the answers. At times, I had to physically lift some of the kids’ hands from the mouse. The tests did begin to get easier, but the lowest level tests only went to the 2nd-3rd grade level, much higher than some of my kids were reading. And the tests just kept going! The computer was probably confused due to the random answers. I had the kids take breaks (according to their IEPs) and gave out candy. But it was really hard to watch. I had to do that to those kids three different times that year.
And then to add insult to injury, my very reformy, data-driven principal, had meetings with each of the staff regarding the results of the data. Even after explaining how useless the tests were to gauge anything at all about my students, my prinicipal still forced me to have a conversation about that meaningless data! I felt like I was in an episode of the twilight zone. Principal: “But how will you change your teaching based on these results?” Me: “Um, these results are not valid…” Principal: “But how will you teach differently?” Me: “But these tests don’t really tell me much. I have other assessments and a portfolio of work…” Principal: “HOW WILL YOU CHANGE YOUR TEACHING?” Me: “(Pause) I will focus on phonics and vocabulary instruction.” Principal: (Scribbles note on official form) “Ok, you can go.” Surreal…
The comedian Lewis Black has a great line for the educational reality in which we find ourselves: “I took LSD when I was younger to prepare myself for these times”.
KatieO: That line of reasoning must be something they teach at principal school. I’ve been in that situation too. Not much fun.
The administrators know how to game the system too …… It is called moving the shells around. It is called cheating.
No one is talking about how computerized tests require a totally different way of teaching and testing. You can’t do math all year long using a paper and pencil, and then suddenly have to do all of your calculations (including graphing and algebra) on a computer. And did I mention the fact that multiple choice is going away? So suddenly, students must do math on the computer. For an special education student, this takes a math word problem from a 4-6 step situation to a 10-15 step situation. Not only must they solve it – they must input it correctly. Is this fair?
Let me give you an example of something I saw on a computerized supposedly-it-doesn’t-count “data-gathering” assessment done in my state last year for children on modified testing. (PS -I am paraphrasing, but you will get the idea).
If you add 1/4 cup plus 1/4 of a cup together, what do you get? The answer should be 1/2, correct?
NO!!! For these students, some with IQs of 70 and below, the test demanded up in the header (not individual question – just in the header) that in order to write their answer in the answer box given, they must convert it to a decimal.
So the correct answer is .5
Any student who typed in 1/2 was incorrect.
Is this fair?
DUH. I had never thought of how disconnected the computerized testing and standard teaching methods (paper/pencil; chalk/board, smartboard, etc.) could be. Of course. The answer??? Digital learning. Let’s watch out for that response.
One of my concerns comes from having watched some of my kids from poverty struggle with the mere negotiation of the computer, the keyboard, the mouse, the navigation tools on the screen. They put out as much energy of trying to manipulate the computer as they do taking the test. Meanwhile, the kid who has had access to a home computer since age three can whiz through that aspect of testing. The digital divide counts.
Fair? It’s criminal!
Sounds like online surveys for retail stores. I know if I click there was a problem or someone helped me I will get more questions so I always avoid them. So I do what the kids do I avoid the extra questions and game the system. Test makers like retailers really aren’t getting accuracy from their subjects.
The stories above demonstrate why computer adaptive tests have to be high-stakes in order to be valid. I have proctored enough non-adaptive standardized tests to know that many students just want the test to be over! If they can “click through” items on these new-fangled tests to finish them without any tangible ill effects, many will. Can’t say that I blame them because for them these assessments are generally not related to what they are doing in class when they are administered, and are therefore often perceived as a waste of time.
Students do not suffer any ill effects now from standardized tests, only their schools and teachers do. Parents and students are not “accountable.” It is all the teacher’s fault. Life isn’t multiple choice, we learn from failure. Thomas Edison wouldn’t have a chance today, except he wouldn’t care, he’d just go on inventing things.
Reblogged this on Transparent Christina.
I was searching for something different and found this critique on adaptive assessments and incidentally I did an implementation of such a system for a web application. Not an easy task, mind you, but the mathematics behind adaptive systems are highly interesting and worth studying those publications (going back to the 70s, this is nothing brand-new).
The parent’s and student’s statement that such tests can go on forever is simply not true, even if you are answering the assessment in an extreme manner (everything correct or wrong) the test will end because there will always be a point when another question won’t significantly change the score. There’s nothing new to be learned from the 16th question if the 15 before were answered correctly. For the test to go on forever the questions’ difficulty would have to rise forever, something which simply is impossible.
50 questions in 40 minutes: personally I’d finish such a test in 30 minutes and be bored for another 10 or worse, start distracting fellow students. That’s why my school years were like and with adaptive assessments my boredom would have been much shorter.
I was searching for something different and found this critique on adaptive assessments and incidentally I did an implementation of such a system for a web application. Not an easy task, mind you, but the mathematics behind adaptive systems are highly interesting and worth studying those publications (going back to the 70s, this is nothing brand-new).
The parent’s and student’s statement that such tests can go on forever is simply not true, even if you are answering the assessment in an extreme manner (everything correct or wrong) the test will end because there will always be a point when another question won’t significantly change the score. There’s nothing new to be learned from the 16th question if the 15 before were answered correctly. For the test to go on forever the questions’ difficulty would have to rise forever, something which simply is impossible.
50 questions in 40 minutes: personally I’d finish such a test in 30 minutes and be bored for another 10 or worse, start distracting fellow students. That’s why my school years were like and with adaptive assessments my boredom would have been much shorter.