Bruce Baker has written an important post about the inability of pundits (and journalists) to read NAEP data.

Part of the misinterpretation is the fault of the National Assessment Governing Board, which supervises NAEP. It has a tight embargo on the scores, which are widely released to reporters. It holds a press conference, where board members and one or two carefully chosen outsiders characterize the scores.

He writes:

“Nothin’ brings out good ol’ American statistical ineptitude like the release of NAEP or PISA data. Even more disturbing is the fact that the short time window between the release of state level NAEP results and city level results for large urban districts permits the same mathematically and statistically inept pundits to reveal their complete lack of short term memory – memory regarding the relevant caveats and critiques of the meaning of NAEP data and NAEP gains in particular, that were addressed extensively only a few weeks back – a few weeks back when pundit after pundit offered wacky interpretations of how recently implemented policy changes affected previously occurring achievement gains on NAEP, and interpretations of how these policies implemented in DC and Tennessee were particularly effective (as evidenced by 2 year gains on NAEP) ignoring that states implementing similar policies did not experience such gains and that states not implementing similar policies in some cases experienced even greater gains after adjusting for starting point.

“Now that we have our NAEP TUDA results, and now that pundits can opine about how DC made greater gains than NYC because it allowed charter schools to grow faster, or teachers to be fired more readily by test scores… let’s take a look at where our big cities fit into the pictures I presented previously regarding NAEP gains and NAEP starting points.
The first huge caveat here is that any/all of these “gains” aren’t gains at all. They are cohort average score differences which reflect differences in the composition of the cohort as much as anything else. Two year gains are suspect for other reasons, perhaps relating to quirks in sampling, etc. Certainly anyone making a big deal about which districts did or did not show statistically significant differences in mean scale scores from 2011 to 2013, without considering longer term shifts is exhibiting the extremes of Mis-NAEP-ery!”

But if NAGB wanted intelligent reporting of the results, it would release them not just to reporters but to qualified experts in psychometric s and statistics. Because it refuses to do this, NAEP results are reported like a horse race. Scores are up, scores are down. But most journalists never get past the trend lines and cannot find experts who have had time to review the scores and put them into context.

I have a personal beef here because I was given access to the embargoed data when I blogged at Education Week and had 4,000 readers weekly. Now, as an independent blogger with 120,000-150,000 readers weekly, I am not qualified to gain access to the data until after they are released (because i do not work for a journal like Edweek.) I don’t claim to be a statistical expert like Bruce Baker, but surely the governing board of NAEP could release the data in advance to a diverse group of a dozen qualified experts to help journalists do a better job when the scores come out.