Gary Rubinstein has a somewhat startling habit of insisting on accuracy. He gets very annoyed when educators or pseudo-educators make claims that are false or only half-true or embellishments. I have worked with him on several occasions to track down the facts about “miracle schools” that turned out to be schools with high attrition rates or some other explanation of a dramatic spike in test scores or graduation rates.
In this post, he examines a claim made in an article by Louisiana Superintendent John White and Massachusetts Commissioner Mitchell Chester. Both of them are members of Jeb Bush’s Chiefs for Change, which is a strong indication that they are wedded to test scores and school choice.
Chester comes from a state that has historically been the highest-performing in the nation.
What bothers Rubinstein is that White uses the article to claim some sort of Louisiana “miracle” on his watch, and he cites NAEP scores. That sets off alarm bells for Rubinstein.
This is White’s claim:
In Louisiana, radical change means that 128,000 fewer students attend schools rated D or F than did in 2011. That’s had a powerful impact on the historically disadvantaged children too often consigned to failing schools, vaulting the performance of African-American fourth graders into the middle of the pack on the National Assessment of Educational Progress in 2015. In 2009, for example, black fourth graders ranked 43rd and 41st in the nation for proficiency in reading and math, respectively. Those rankings jumped to 20th and 23rd in 2015.
Rubinstein writes:
As far as the 128,000 fewer students attending schools rated D or F, since they are the ones who assign those ratings and since the criteria for getting a D or F has changed over the years, I don’t take that one too seriously.
But I was interested in ‘fact checking’ that NAEP statistic since that was one I hadn’t heard of before. I knew that Louisiana as a whole had very low NAEP scores and they were not improving very much over the years the way, for example Tennessee and Washington D.C. have, otherwise we’d be hearing about Louisiana NAEP much more.
White says that black fourth graders ranked 43rd in reading and 41st in math in 2009 and now rank 20th and 23rd. So I went to the National Center for Education Statistics website and dug into the data.
Since NAEP isn’t just for 4th graders, the first thing I checked was what their current ranking was for black 8th graders and saw that for 8th grade math they actually dropped from 39th to 44th between 2009 and 2015. For 8th grade reading they dropped from to 43rd to 45th between 2009 and 2015. So it is obvious why they don’t mention their 8th grade change in rankings.
I also checked how they have done in math for all 4th graders regardless of race. I found that in 2009 they were 48th while in 2015 they were not much better, at 44th. In reading they went from second to last in 2009 to 8th to last in 2015. A jump, but not the sort of thing that John White would ever use to prove his point about his knowledge of improving schools.
But he went on to inquire about the statistical significance of the fourth grade gains.
What he learned will surprise you.
Diane Ravitch's blog 
We need people like Rubinstein and other statisticians that can investigate privateers’ many, often erroneous, claims. They tend to over state their assertions, and sometimes they use “alt. facts.” In this case they are cherry picking data, a favorite means of making false claims. Instead of “chiefs for change,” Bush should call his group “cherry pickers for choice,” masters of the not “significant.”
LikeLike
Gary is a high school math teacher who doesn’t like phonies.
LikeLike
Most of the real statisticians are quiet as crickets on a frigid winter’s night when it comes to this sort of stuff.
They say little to nothing and thereby give the “statusticians” (Chetty, Hanushek and others whose primary concern is status and prizes) free reign to make their bogus claims.
I guess they figure debunking economists is beneath them.
They are probably right about that.
LikeLike
Quite apart from the significance issue, cherry picking is not an honest “mistake”.
It is meant to receive.
In other words, it is fraud.
LikeLike
Meant to deceive.
LikeLike
FRAUD is the NEW NORMAL and so is ALTERNATIVE FACTS. SAD.
LikeLike
This morning’s Wall Street Journal editorial worked hard to prove that attacks on charters were unwarranted, just teacher union desperation. They tried to abstract several meta-studies ( studies of studies) to prove charters are superior to public schools. I am sure Gary could demolish the claims, but why bother talking to editors at the WSJ.
Gary’s clarity and homey illustrations should be in a book, and marketed as well as Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.
For a brief moment I was reminded of the ridicule-worthy “Oak Tree Analogy,” intended to help teachers understand VAM. It crashed and burned but it is still available on line. To believe it, you have to believe that trees have brains and that students are no different from trees.
I urged Gary to construct and send to Diane a plain language explanation for statistician/economist Eric Hanuseck’s insistent claims that charter schools produce X “days of learning” (or months of learning, or years of learning) when compared with public schools. I view these claims as statistical fictions, untethered from any understanding of life in real schools where multiple subjects are taught, students and teachers are not always focussed on test scores and so on.
LikeLike
I am so tired of the portrayal of the “evil empire,” the teachers union, when the reality is the union is a sleeping giant on life support. Thanks for all your research and debunking of fake assertions; keep “blinding them with science.”
LikeLike
The data for Tennessee showing tremendous gains is cherry picked as well. I cannot recall which test was responsible for the recent claim of the “nation’s fastest improving”, but it was something like science test for fourth graders.
LikeLike