This is Matt Di Carlo’s best post ever.
Matt is a brilliant and careful social scientist who has more faith in quantification than I do.
But I read what he writes because I almost always learn something.
In this post, he explains that tests are not a cause of success in life, they are a signal.
Our policymakers think that if they can just get scores higher and higher, everyone will succeed, but this has led them to overdose on testing. They assume that more tests cause higher scores, and that higher scores will produce many other good results. As a result, they are investing scarce public resources in testing, instead of using the tests as a measure and a signal.
Diane Ravitch's blog 
Here’s the last paragraph of my research/opinion paper from the fall – DiCarlo hits the mark here – see citation.
*Reforms, such as the use of VAM, will continue forward until proven successful or unsuccessful in raising student achievement. In foresight, these policies look as though the ends will justify the means, but at a cost. “If we mold policy such that livelihoods depend on increasing scores, and we select and deselect people and institutions based on their ability to do so, then, over time, scores will most likely go up” (Dicarlo, 2012). Using testing data to evaluate teachers and create policies will become a norm, thus creating a chain reaction that will include the labeling of schools and teachers – keeping only those that increase test scores. In the end, the students at the heart of our system will score better on tests, but not be better off for it in the long run.*
LikeLike
There is a long running debate in labor economics about why people with high school diplomas and college diplomas earn more than those without diplomas. One explanation is that graduating from high school and college gives a student the skills that make him or her more productive. The other explanation is that only the relatively more productive students can graduate, so being a graduate is a signal of productivity.
If the signal group is correct (or mostly correct as graduation could be both), making sure that everyone gradutes from high school distorts the signal that high school graduates are relatively more productive. This might explain why some jobs that used to only require a high school degree now require some college or even a college degree.
LikeLike
“If the signal group is correct (or mostly correct as graduation could be both), making sure that everyone gradutes from high school distorts the signal that high school graduates are relatively more productive”
Are you channeling Ahmad with that one??
LikeLike
I am not sure who you are referring to.
LikeLike
Ahmad Hashan (sp?) shows up everynow and then although with some interesting mish mash of languages that generally don’t make sense.
He had commented on a recent article and I went back to find it and it’s gone. I don’t know if Diane is deleting them as she comes across them or if it has something to do with the internet here as school vs mine at home.
LikeLike
I delete nonsensical comments as they are usually computer generated
LikeLike
And you don’t delete the economists comments???
(Just joking TE!)
LikeLike
TE,
See my response @ 8:38 below for some of “Ahmads” writing (which Diane deleted).
I thought your statement had a little bit of “not making sense” until I further contemplated it.
LikeLike
Good point teachingeconomist, and I was also thinking that – even skills and signaling aside – there is an advantage simply because one can qualify. But, Di Carlo does acknowledge that eventually – if graduate requirements lower (and they have) – that HS graduation will mean less (and it does). Still, it does mean something, and I would venture that at least part of it is simply due to the paper presented on graduation day.
It would be interesting to see a study (and there may already be one out there) that compares HS grads from non-HS grads but controls for all of the variables graduation is supposed to signal. In other words, holding things like social skills and reading fluency steady, what benefit does HS graduation give someone today. Granted, that may change over time if graduation is cheapened as Di Carlo said, but I still do think that HS graduation is probably not JUST a signal, but a cause as well.
LikeLike
There is a growing liturature that compares students accepted to elete colleges and universities who end up attend non-elite universities like mine (we have a 93% acceptance rate) to students who graduate from the elite institutions. The two groups seem to have the same outcomes, at least if measured by earnings.
LikeLike
teachingeconomist it’s always interested me how some schools seem to get better or worse reputations. While selectivity certainly may indicate quality, it may also simply indicate popularity, which could be due to anything from a good football team to a beautiful campus. This isn’t to say elite schools aren’t good, but my experience through college (when talking with friends and family) as that many folks seem to have both good and bad experiences regardless of where they went.
Of course, the finding you mention could also not have any bearing on quality of education at those various tiers of school. It could be that quality education doesn’t necessarily translate into marketable skills/knowledge.
LikeLike
I suspect that lifetime earnings was used because it was available for every former student and easily compared across the former students.
There is also some research that suggests that students who apply to schools like Harvard have the same average earnings later in life as those that are admitted to Harvard.
LikeLike
That’s pretty interesting teachingeconomist – thanks for sharing. Something to think about.
LikeLike
“Experience as we learn together with new solenoid, all sheep to be possible with good signature and go on living!”
Have to agree with that one, eh!
LikeLike
Can’t fatten a pig by weighing it.
LikeLike
No, but the more scales you use, the wealthier the scale manufacturers get.
LikeLike
Weighing the pig will let you know if the pig is thriving, however.
LikeLike
Right, TE, because you could never tell by looking how fat a pig is.
LikeLike
Should the buyer of the pig simply take the farmers word that the pigs are healthy?
LikeLike
TE, we are–I am–talking about children. Real children. Each one is different. Unique. A difficult concept for an economist to grasp, granted, but you are a father, so maybe you can see that children’s value to you as a parent cannot be weighed or measured by standardized tests or any other scale.
LikeLike
Dr. Ravitch,
You are correct, I am a father and, somewhat unusually for posters on this blog, the father of a current public school student.
I understand the diversity of students, and that is why I advocate for a diversity of schools. I don’t understand how people can argue that students all have individual needs, on the one hand, and argue that they should all be sent to a one size fits most traditional public school. If the zoned school doesn’t fit, students should be free to find one that does even if their parents can not afford a private school.
When you say a child’s value can not be weighed or measured by standardized tests or any other scale, do you mean to advocate for the ending of teacher determined grades? The exams in those classes are the ones that are high stakes for the students.
LikeLike
The buyer of the pig usually intends to slaughter it and so is looking for the amount of meat. If that’s what you intend for children, by all means, weigh away.
LikeLike
Nope. Not unusual that you are the parent of a public school student. Many us are.
LikeLike
Especially if the focus is on the scale and not the pig (or the farmer). Test scores are for people looking for simple, low complexity measures assessing the health of the pig. Reinhart and Rogoff should be causing an intellectual riot in the blind use of statistics in setting policy. Honest and learned academics would tell you that studies and measures are only as good as the integrity of the creators and the assumptions of the models. No sane business person would roll the entire future of a company on the mass release of an unproven and untested product. Yet we are doing that very approach in education.
LikeLike
or for an alternative measure to teacher assigned grades.
LikeLike
Sandy: I agree that you can’t “fatten a pig by weighing it.”
But beware the following end-around run of your impeccable logic: you can tell if the pig is fit and healthy—right?—because if it weighs more now than last month or the month before that, it must be doing well. I mean, unassailable logic, wouldn’t you say?
¿?
Unfortunately, here on Planet Reality a pig that has been fattened up to the point where it has reached a certain weight is considered “fit”—for slaughter!
😦
The pig is doing just as well as a lot of turkeys on Thanksgiving Day.
Or kids in hospitals for life-altering surgery with regard to high-stakes standardized testing.
Perhaps in the future we can refrain from reminding pigs and turkeys of why we weigh them. Why ruin their day and ours?
And maybe, just maybe, we could refrain from inflicting similar damage on children.
Just keepin’ it real.
Not Rheeal.
🙂
LikeLike
Pig farming is not teaching
LikeLike
Maybe not, but it’s a smelly proposition.
LikeLike
I agree – great post, and I usually enjoy reading his comments. Definitely highlights some of the potential benefit of accountability and testing, but some of the unfortunate byproducts as well, particularly when we go to far with it or don’t implement policy/practice in the right way.
LikeLike
“Definitely highlights some of the potential benefit of accountability and testing. . .” and says absolutely nothing about the inherent errors of the process that render it completely invalid.
By the way, have you had a chance to read my email to you or any more of the Wilson studies?
LikeLike
Hi Duane – I actually sent you a pretty lengthy response on 4/25. Just checked my outbox to make sure it sent and it looks like it did. Maybe check your spam?
In any case, I’ve been enjoying the conversation and looking forward to it continuing. At this point I haven’t been convinced that all assessment using standardized tests is inherently invalid.
LikeLike
This mornin, we were discussing our school improvement plan in light of this video:
The general thought is that we need to change a lot of what we do to meet the demands mentioned in the video. Students need opportunity for authentic, integrated learning. Testing demands do not allow for this. When will it change so we can truly prepare out children for the future, including those jobs that currently don’t exist.
LikeLike
No we don’t. First of all predictions about the future are nearly always wrong. Therefore trying to prepare for the unknown and the wrong is impossible. Think of all the educational trends over your career and how ______ is going to change education. It didn’t. From film strips to TV to video to web based educational technology-it didn’t. The essential human element across time is to strive for meaning. Encouraging kids to explore their culture, their talents and use the unique human qualities of empathy and reason is education at it best. You can do this with a stick in the dirt, around a camp fire or across the world on a wiki, the means do not matter. This is the effrontery of the testing movement, another fad that is being dealt a blow in Texas-it does not make sense to real educators because there is no deep meaning.
LikeLike
Time to review Herman Melville’s Bartleby the Scrivener. Reminds me of the post about the girl with Rett Syndrome who preferred not to take the test.
http://www.bartleby.com/129/
LikeLike
“Weighing the pig will let you know if the pig is thriving, however.”
In practice, it will let you know if it’s ready for slaughter.
LikeLike
Weighing the pig will tell you how much mass the pig has when a given gravitational force acts on it. Also, he who dies with the most toys is dead.
LikeLike
Perhaps in the case of education we should think of it as graduation.
LikeLike
A professional hog farmer, using his experience and expertise, can probably tell if a pig is thriving without having to weigh it much
LikeLike
Why are we comparing student learning to the raising of pigs in the first place? Is it because some animals are more equal than others?
LikeLike
Or are we just removing the pork from education? 🙂
LikeLike
Th, th, th, th, th, . . . That’s all folks!
http://www.bing.com/videos/search?q=Porky+the+pigs+that%27s+all+folks&view=detail&mid=782BE02D23D7BF954126782BE02D23D7BF954126&first=0&FORM=NVPFVR&qpvt=Porky+the+pigs+that%27s+all+folks&adlt=strict
LikeLike
Accountability in teaching is often misunderstood by business and academics. In business, if an employee is a “problem” or a customer base is unprofitable, you can fire the employee or shift to a different market segment. Are the “accountability” advocates suggesting teachers “fire” difficult students or schools should exclude certain segments of the population? In higher education, universities purposefully weed out struggling students rather than have a responsibility to educate them. Academics have a difficult time understanding the fact public school teachers do not and cannot simply walk away from a student in the classroom. We are in fact penalizing teachers who want to teach all students and, instead, forcing teachers to game the system by picking the students that have the highest return on standardized tests.
LikeLike
Well said, sir!!
LikeLike
An excellent article by Di Carlo. I would add only these three points:
First, the reason why tests have to be proved to be valid measures is that, as everyone knows, that it’s tautological that what a test really measures is the set of abilities required to take the test successfully. To the extent that a test is invalid–to the extent that it measures a narrower set of abilities than those that it is purported to measure, people will, in a high-stakes environment, concentrate on teaching those abilities. That is, instead of teaching writing or reading, they will teach taking the writing or reading test. This inevitability distorts both curricula and pedagogy. Most educators understand that a high-stakes environment demands replacing the teaching of writing with the teaching of taking writing tests, and most policy wonks don’t.
Second, in a high-stakes environment where the pressure is on to teach not reading nor writing but the taking reading and writing tests, time will be allocated to the latter AT THE EXPENSE OF the former. We are seeing this all around the country, where schools are now spending a third of the school year doing test prep, practice tests, and the tests themselves and where the pedagogical decisions throughout the rest of the year are being made based on the criterion so well-captured by kids’ time-honored question, “Will this be on the test?” There is, of course, enormous opportunity cost to replacing reading and writing instruction with reading and writing test-taking instruction. As a teacher recently said to me, well, three months before the end of the school year, well, it’s FCAT season, so basically done for the rest of the year.
Third, the most valid sort of testing is the work itself. Imagine that you have been hired by the Guild of American Luthiers to design a test of guitar-making ability. Whatever pen-and-paper test you design, it won’t be as valid as it would be to have the apprentice luthier build a bunch of guitars and to assess the parts of the work all along the way. How accurate was the measurement of the spacing of the frets on this guitar? What is its tonal range? How resonant is it? How precisely was the rosette inlaid? The actual abilities that go into being a capable luthier are simply too complicated, too numerous, to be captured by any particular test, and while it is possible to use statistical means to ensure that a given test is a valid sample of the larger ability space, making the test high stakes inevitably leads to concentration on that narrower set of abilities–to gaming of the testing regimen. So, ideally, in order to be valid, the testing would DISAPPEAR into, would BECOME, the tasks performed throughout the instructional process. To the extent that this is not done, the teaching and learning will become distorted.
LikeLike
I have no problem with offering a more rigorous school environment or with helping students use critical thinking skills to an optimum. There are difficulties with this, however. As long as we are forced to evaluate students using grades and percentages as opposed to rubrics, we will not be able to fully implement the changes. I have always found a “grade average” to be confining and misleading as to students’ real abilities. Learning, to me, is a progression of understanding, not an average of errors along the way. We don’t view students as a finished product but as an average of all their attempts to succeed. Unfortunately, not everyone is qualified to assess another person’s critical thinking skills. Suppose a child has a great observation but the evaluator doesn’t comprehend the direction in which the child is going with his/her thoughts. Would that child be evaluated accurately? I have usually been fairly perceptive as to the abilities of my students. I have come across several students, who, in 4th grade, were obviously better critical thinkers than I am. I have always tried to keep my options/judgments open as to the “plausibility” factor and the process that the students use. They don’t have to feed my own ideas back to me in order to be correct. I was the sort of person who, in college, would answer questions on my tests, and then write in the margin, “This is the answer you want, but I don’t believe it because …” We have a huge task ahead. But, if we expect the kinds of abilities necessary for equipping educators with the know-how.
LikeLike
yes yes yes!
LikeLike
I recently got an e-mail inviting me to work as a scorer for our latest round of state tests. The e-mail came not from CTB/McGraw-Hill, but from Kelly Temps. I would be paid a whopping $11.05 an hour:
http://ctb.appone.com/MainInfoReq.asp?R_ID=608911&B_ID=46&fid=1&Adid=
To add a little perspective, when I worked as a freelance proofreader for McGraw-Hill in the late eighties, I was paid $12.00 an hour.
If the only result of the Common Core transition were to get the profiteers out of the student testing business, and keep them out, the switchover would be well worth it. I nearly applauded when our principal announced that testing will be taken over by one of the two new NONPROFIT agencies, created to design and implement the tests and ONLY that.
McGraw-Hill, Pearson, etc. know very well that our students are THEIR captive audience. If the test questions are poorly written, or the answers are ambiguous, few of our students are going to complain. For the most part, they’ll just sit there frustrated, thinking, “I’m too dumb to figure this one out.” And that’s just a damned crime, but these companies think nothing of cheaping out on quality control in their testing divisions, because they know they can.
Don’t get me wrong…I’d love to see these goofy, time-consuming, education-gobbling tests go away. To my mind, the whole steaming pile of assessment could be whittled down to three tests: One that the children take in fifth grade, and must pass before they can proceed to middle school. One that they must take in eighth grade in order to proceed to high school. And of course the high school exit exam, which kids in California already take, and which could easily be reworked to whatever necessary degree in order to meet state and federal requirements for assessing school performance.
But as long as we are currently stuck in the thrall of testing mania, at least the corporate goons will no longer be running the show. Undoubtedly, however, their well paid lobbyists are pushing, at this very moment, to regain their luxy seats on the gravy train.
LikeLike
I think there’s a basic conceptual flaw with the Di Carlo piece. He’s saying that some amount of testing, used skilfully by well meaning-people can be used for school and teacher accountability, but that we “may” have gone too far and are undermining the “good” kind of test-based accountability. From what I’ve read, though, the actual evidence is that most of the accountability systems that exist in the wild are tainted to some degree by conflicts of interest, problematic models, age inappropriate application, insufficient basis in research, being used to justify administrative decisions unrelated to what is actually being measured, etc. So in saying that he thinks test-based accountability is basically a good thing, but suffers from poor application, he’s really begging the question of whether it actually can be a good thing.
it seems to me that the only legitimate approach is to recognize that in effect we are right now in the trials phase of test-based accountability research. There’s a hypothesis that standardized reading and math tests can be used to measure and incentivize students in ways that will improve their performance/outcomes, that such improvements can be correlated to something other than the tests themselves, and that the costs of this mode of performance management can be compared to other modes. Variants on that hypothesis are right now being tested in schools systems accross the country. If afther some reasonable amount of time (which may not yet have come) all we see is the bad kind of test-based accountability, we have to be open to the idea that there is no good kind. I am actually open to the idea that there’s a good kind, but I don’t see it yet at this stage of the trials.
LikeLike
Meant to say “There’s a hypothesis that standardized reading and math tests can be used to measure and incentivize students and educators …”
LikeLike