As we saw in Atlanta, people will do all sorts of things, good and bad, to reach targets. Data can be very pliable.
Gerald Grob, Professor of the History of Medicine Emeritus at Ritgers University, published a book in 2014 titled: AGING BONES: A SHORT HISTORY OF OSTEOPOROSIS. It includes the following example of the creative use of statistics.
Grob analyzed clinical trials of such drugs as Fosamax. He wrote to tell me, “Merck reported a 50% reduction in hip fractures, and the drug made billions for the company. The 50% figure, of course, was the relative reduction, which has no meaning. The absolute reduction was from 2.2% in the placebo group to 1.1% in the treated group. The absolute reduction was this 1.1%, a hardly impressive statistic. Moreover, it did not take into account the adverse effects of the drug. Above all, it ignored the fact that about three-quarters of all hip fractures occur among people with normal bone mineral density for their age and result from falls.”
Fun with numbers!
Diane Ravitch's blog 
Yes, reporting results only by percents should always raise a red flag. Recently, a study was reported that the prolonged use of drugs like Benadryl (diphenhydramine) are associated with a 50% increase in your chance of getting Alzheimer’s. One can’t know how bad that is if you don’t know the initial rate.
LikeLike
Mike Barrett: quite so.
I strongly recommend to readers of this blog Gerald Bracey, READING EDUCATIONAL RESEARCH: HOW TO AVOID GETTING STATISTICALLY SNOOKERED (2006).
While I give a few of his “Principles of Data Interpretation” just below, he gives many excellent and pointed examples and explanations in his book, in plain English and with documentation to back them up. From p. xix:
#5: “Be sure the rhetoric and the numbers match.”
#9: “Be aware of whether you are dealing with rates or numbers. Similarly, be aware of whether you are dealing with rates or scores.”
#11: “Be aware of whether your are dealing with ranks or scores.”
And there’s always Darrell Huff’s slim classic first published in 1954, HOW TO LIE WITH STATISTICS (1993, paperback).
Thank you for your comments.
😎
LikeLike
Thanks.
Sent from my iPhone
LikeLike
Thank you so much for sharing. I was given Fosamex about 15 years ago for beginning osteopenia. I refused to take it. Something said no to me. Every drug I’ve been given has side effects and often are said to cause cancer ten years later. Fosamex causes problems with jaws and other side effects. I decided to just exercise and lift weights. Much healthier. Love this used as an example of how deceptive statistics can be.
LikeLike
I likewise refused hormone replacement and statins. I never regretted my choice, and I also found alternatives. Sometimes you have to follow your instincts. Both hormone replacement and statins are now linked to cancer risk with statins also linked to diabetes.
LikeLike
The classic for my generation was How to Lie with Statistics. Then Gerald Bracey kept exposing the shell game with numbers in education.
I have written about the econometric turn in education meaning that the inferences from research conducted by economists, often based on inferential leaps through thin air, has been given more credibility and publicity than it often deserves.
A case in point is the continued use of VAM for teacher evaluation…with one of the first studies in the 1970s from economist Eric Hanushek. Since then he has over 500 publications in education, is the expert of choice in the defense of test scores for evaluating teachers.
Or consider the Gates funded Measures of Effective Teaching (MET) project taken as if authoritative, from economists at Harvard, who thought you could just pay teachers and administrators to randomly assign students to classrooms, and classrooms to teachers in order to fit the statistical assumptions in VAM calculations. Those assumptions reflect an appalling ignorance of how schools are organized, and how principals actually mediate classroom assignments. The failure to achieve random assignment did not prevent the researchers from churning out some ratings of teachers as ” effective,” or not based on student test scores. Student test scores are known to be invalid for such ratings. The scores are not ” instructionally sensitive” … That is one reason why the same teacher working with similar students and the same basic curriculum can be highly effective one year, and rated ineffective the next.
The MET study included Charlotte Danielson’s observation rubrics. The researchers wanted to figure out how many observations of videos it would take in order to get decent reliabilities. So videos submitted by teachers, who received some perks for their trouble, were used for this research, as if the Danielson protocol would not benefit from some serious analysis for its credibility. And as if videos made and selected by teachers have the same value as seeing teachers in action and following up its some face-to-face conversation. And the spillover can be seen in the use of video snippets for Pearson’s test of teacher candidates.
The MET study also included a student survey, designed by an economist, and with seven constructs that produce scores favoring teachers who assign homework, run a very tight ship, and so on.
So these economists, and this Gates funded project, got publicity–including some Congressional testimony from the lead researcher– as if some gold standard with three major components for teacher evaluation had been established.
Not.
LikeLike
Mark twain likes the statement, “There are lies, damned lies and statistics!”
LikeLike
That was before VAMs were invented or he would have put that at the end.
LikeLike
What of major concern to me is this also:
Science in our major universities used to be about finding “BASIC” science, looking for scholarly truth, for looking for new information about BASIC things..
Now it is becoming the fact more and more that even our university science is funded by corporations. If the “facts” do not come out the way the corporations wish – no more funding.
Science turned upside down.
LikeLike
“On fleek.” We need to work that into the education vernacular.
LikeLike
my previous supervisor was a former President of the Massachusetts School Business Association; he insisted that every technology that was brought into a public school setting have proven evidence that it was cost effective . The only good research we had available was from Henry Levin and H.L.’s student did a study for Massachusetts showing that the Pat Suppes math program on computers was cost-effective (that means it brought results at a reasonable cost).
A lot of “stuff” has been foisted onto the schools since that time but Henry Levin has been in the field far longer — if you cannot meet the R&D standards of Henry Levin for an evaluation then you are just providing marketing studies; that has become common as Pearson has it’s marketing arm reaching out into every state and the Commissioner of Education in Massachusetts has become a full time marketing agent for Pearson pushing experimental tests on every state where parents and students don’t want them and the states would like to get out of the Memorandum of Understanding. So M. Chester is carrying the gun holster for Arne Duncan when he holds the gun to the head with threats about waivers and denying federal money. I am enraged by what is going on for the profit$ of corporations that does not one thing for students (other than to hype technology).
LikeLike
quote: “The MET study also included a student survey, designed by an economist, and with seven constructs that produce scores favoring teachers who assign homework, run a very tight ship, and so on.
So these economists, and this Gates funded project, got publicity–including some Congressional testimony from the lead researcher– as if some gold standard with three major components for teacher evaluation had been established.”
Laura this is a huge problem,; we are also seeing it in climate change where a Harvard/Smithsonian researcher (I think his name is Wong?) is sending out items that go direct to Senator Inhofe; Smithsonian is putting out a press release on all the wonderful things they do about climate change but the harm has been done and it reverberates through the media, senators aides office etc.
We have standards and fire teachers and close schools but the standards for what is an evaluation report and what is a marketing piece need to be brought to the attention — I’m doing what I can emailing the newspapers and blogs…
LikeLike