Some guy who works for StudentsFirst–the organization that promotes vouchers and charters and wants to strip teacher of all due process–wrote a criticism of me on Huffington Post because he doesn’t like the way I interpret NAEP data. This is silly because I served on the NAEP board for seven years and know its strengths and limitations. NAEP was designed to serve an audit function, never to be used for high stakes. Like every other standardized test, NAEP reflects socioeconomic status. The kids with the most advantages score at the top, and those with the fewest advantages cluster at the bottom. NAEP is generally known as “the gold standard” because no one knows who will take it, no student takes the whole test, and no one knows how to prepare for it. NAEP scores may reflect demographic changes or other factors.

Here Mercedes Schneider takes him to task for his misinterpretation of what I wrote.

The release of the 2013 NAEP results set off cheering among advocates of corporate reform because DC and Tennessee showed big gains. But, I pointed out, states following exactltly the same formula showed small gains, no gains, or losses.

He missed the point.

Editor’s note:  While Diane is on a somewhat reduced blogging schedule, she has invited members of the Education Bloggers Network, a consortium of people who blog about education issues on the national, state or local level to contribute to her blog.  If you are a blogger who supports public education and would like to join the Education Bloggers Network, contact Jonathan Pelto at jonpelto@gmail.com.

This guest blog is written Paul Thomas

During her tenure as Secretary of Education (2005-2009), Margaret Spellings announced that a jump of 7 points in NAEP reading scores from 1999-2005 was proof No Child Left Behind was working. The problem, however, was in the details:

During President George W. Bush’s tenure, NCLB was a corner stone of his agenda, and when then-Secretary Spellings announced that test scores were proving NCLB a success, Gerald Bracey and Stephen Krashen exposed one of two possible problems with the data. Spellings either did not understand basic statistics or was misleading for political gain. Krashen detailed the deception or ineptitude by showing that the gain Spellings noted did occur from 1999 to 2005, a change of seven points. But he also revealed that the scores rose as follows: 1999 = 212; 2000 = 213; 2002 = 219; 2003 = 218 ; 2005 = 219. The jump Spellings used to promote NCLB and Reading First occurred from 2000 to 2002, before the implementation of Reading First. Krashen notes even more problems with claiming success for NCLB and Reading First, including: 

“Bracey (2006) also notes that it is very unlikely that many Reading First children were included in the NAEP assessments in 2004 (and even 2005). NAEP is given to nine year olds, but RF is directed at grade three and lower. Many RF programs did not begin until late in 2003; in fact, Bracey notes that the application package for RF was not available until April, 2002.”

With the 2013 release of NAEP data, then, shouldn’t we be skeptical of Duncan’s rush to claim victory for education reform under Obama?:

This year, Tennessee and the District of Columbia, which have both launched high-profile efforts to strengthen education by improving teacher evaluations and by other measures, showed across-the-board growth on the test compared to 2011, likely stoking more debate. Only the Defense Department schools also saw gains in both grade levels and subjects.

In Hawaii, which has also seen a concentrated effort to improve teaching quality, scores also increased with the exception of fourth grade reading. In Iowa and Washington state, scores increased except in 8th-grade math.

Specifically pointing to Tennessee, Hawaii and D.C., Education Secretary Arne Duncan said on a conference call with reporters that many of the changes seen in these states were “very, very difficult and courageous” and appear to have had an impact.

Duncan’s claims, in fact, have prompted The Wall Street Journal to announce “School Reform Delivers”:

Education Secretary Arne Duncan hailed this year’s National Assessment of Educational Progress (i.e., the nation’s report card) results on Thursday as “encouraging.” That’s true only if you look at Washington, D.C., Tennessee and states that have led on teacher accountability and other reforms….

However, a handful of states did post significant gains, and the District of Columbia and Tennessee stand out. Until very recently, Washington, D.C. was an example of public school failure. Then in 2009 former schools chancellor Michelle Rhee implemented more rigorous teacher evaluations that place a heavy emphasis on student learning. The district also tied pay to performance evaluations and eliminated tenure so that ineffective teachers could be fired.

Between 2010 and 2012, about 4% of D.C. teachers—and nearly all of those rated “ineffective”—were dismissed. About 30% of teachers rated “minimally effective” left on their own, likely because they didn’t receive a pay bump and were warned that they could be removed within a year if they failed to shape up.

Clearing out the deadwood appears to have lifted scores.

As I warned on the release date of NAEP, we should anticipate this careless and unsupported eagerness to use NAEP data as evidence of corporate reform success.

Jim Horn has highlighted that NAEP shows a powerful picture of the growing problem with re-segregation and the entrenched reality of racial and socioeconomic achievement gaps—messages ignored by Duncan. At the very least, then, Duncan is cherry-picking.

Gary Rubinstein has also dismantled the DC “miracle,” and G.F. Brandenburg provides a clear chart showing that DC gains are a continuation of a trend pre-Rhee. As Rubinstein concludes:

I’m still pretty confident that in the long run education reform based primarily on putting pressure on teachers and shutting down schools for failing to live up to the PR of charter schools will not be good for kids or for the country, in general.  I hope politicians won’t accept the first ‘gains’ chart without putting it into context with the rest of the data.

With the USDOE at Duncan’s disposal, it seems careless and inexcusable to make unproven claims that policy has caused test score changes when no one has had time to analyze the data in order to make such claims

Like Spellings, Duncan is showing that he is either unqualified to be Secretary of Education due to a lack of understanding of statistics or that he is willing to place partisan politics above what is best for children and public education. Either way, this is yet another example of failure from the top in the world of education reform and politics.

The indefatigable Bruce Baker is at his best in this post, where he puts the NAEP scores into perspective. As he notes, it is not useful to look at a two-year test score change as a reliable indicator. It is far wiser to look at scores in a longitudinal fashion and, when possible, look at other factors that may affect test scores. Then, too, he notes that the NAEP results do not align well with Michelle Rhee’s scorecard for the states. Some of the states she considered to be tops don’t do well on NAEP, either short term or long term.

Even with Baker’s fine analysis, it makes me uneasy to see this maniacal national and international race to get the highest scores. As long as our policymakers and federal policy continue to ignore the undying factors of child health and well-being, and the well-being of families and communities, the NAEP scores are like shadows on the wall, interesting but a distraction from the more important factors that create the conditions for a good life, including a respect for and love of learning.