Bruce Baker has a great post about Matt Di Carlo’s April 1 post on the Maryland NAEP scores. Baker knows that Di Carlo was using his post to mock the misuse of NAEP data, but Baker shows how people like Chris Cerf frequently use misleading graphs to make a point.
When Di Carlo put up his April Fool’s Day post about a “miracle in Maryland,” he forgot to label it as a satire. He updated it.
I posted it in the original version, and noted that neither Matt nor I believe in miracles. When Matt updated his post, I updated my post.
But the interesting point is that the graphs in Matt’s post about the gains posted by Maryland were not made-up. They were real. Maryland has made impressive gains in reading and math.
Matt’s point was that NAEP data cannot be used to explain causation. He extrapolated possible causes as a way of satirizing the way certain big-name reformers like Jeb Bush use NAEP data and claim credit for their policy preferences. Doing this is wrong. NAEP shows trend lines, but it doesn’t say why the scores went up or down. In most states and cities, there are many things happening at the same time: demographic changes, state law changes, federal mandate changes, and unexplained changes. The trend lines can’t tell you which of those changes, if any, caused the scores to change.
Diane Ravitch's blog 
Great post by Dr. Baker,
Some of the data and graphs he exposes are almost comically bad. Seems like being data driven means nothing if you do not understand some basic statistics.
Can the BROADIEs really be this ignorant of “data”? Or are they assuming “we” do not care ?
LikeLike
To your first question, NO, the broadie’s aren’t necessarily that ignorant of the data, their just good a manipulating data to suit their ideology.
And to your second, NO, they don’t necessarily assume we don’t care, they just don’t care themselves if their conclusions match what the data says. The only thing is to get and stay on the “message” using pretty graphs and graphics. They know most people have no clue how to analyze the graphs and data (if they even give a source for the data, in other words they might have just made it up as pointed out by B. Baker.)
LikeLike
I would argue that these are particularly good manipulations of data. In fact, they are embarrassingly bad… ridiculous. Indeed, providing sources and sufficient detail is always a good start. For example, if whoever made the graph for Cerf had even included… perhaps lighter shaded/faded to background the rest of the data points… and/or had explained what percent of schools were included/excluded… that would be a start toward a less ridiculous graph. I urge you to check out the other graphs from his presentation (linked in the post). There are consistent absurdities… like comparing total numbers instead of shares/percents where appropriate – kind of like the Cato graph which takes the money doesn’t matter graph to a new level of stupid. But these are the kind of things I would really hope that a reasonably informed reader would catch. This is middle school level math content (having been a middle school science teacher in a past life!).
My concern is that we’ve reached a new level of completely shameless outright fabrication, misrepresentation and disinformation in the school reform (and public policy) debates.
LikeLike
Sorry… goofed… these AREN’T good manipulations… they’re just dumb.
LikeLike
Thank you,
I remember something Nate Silver has proposed. Not an exact quote here, but…
In the world of BIG DATA, it seems less likely that people will use it wisely; more likely they will scour for something to CONFIRM what they choose, want to believe, or already believe.
Maybe the PARCC and SMART folks can use Cerf’s graphs as part of their exams; find five problems with presenting data like this!
Thank you for a great post and informed commentary.
LikeLike
I am dizzy from all the reformy spin! Stop the ride! Stop the ride!
LikeLike
I’ve got some dramamine for you if you need it!
LikeLike
Education “reform” gone bad.
(Highlighting is mine) -Rich
Bill Ayers:
The road to the massive cheating scandal in Atlanta runs right through the White House.
The former superintendent, Dr. Beverly L. Hall, and her 34 obedient subordinates now face criminal charges, but the central role played by a group of unindicted and largely unacknowledged co-conspirators, her powerful enablers, is barely noted.
Beyond her “strong relations with the business elite” who reportedly made her “untouchable” in Atlanta (according to this New York Times story), she was a national superstar for more than a decade because her work embodied the shared educational policies of the Bush and Obama administrations. In the testing frenzy that characterized both No Child Left Behind and Race to the Top, Dr. Hall was a winner, consistently praised over many years by Secretary of Education Arne Duncan for raising test scores, hosted at the White House in 2009 as Superintendent of the Year, and appointed in 2010 by President Obama to the National Board for Education Sciences. When the Atlanta scandal broke in 2011 Secretary Duncan rushed to assure the public that it was “very isolated” and “an easy one to fix.”
That’s not true. According to a recently released study by the independent monitoring group FairTest, cheating is “widespread” and fully documented in 37 states and Washington D.C.
Valerie Strauss, a veteran education reporter and columnist, wrote Friday that …only Atlanta’s [cheating scandal] has been aggressively and thoroughly investigated. “We don’t really know” how extensive the problem is, Strauss wrote, but “what we do know is that these cheating scandals have been a result of test-obsessed school reform.”
In the District of Columbia, for example, there are unanswered questions about an anomalous pattern of wrong-to-right erasures on answer sheets during the reign of famed schools reformer Michelle Rhee, who starred in the documentary “Waiting for ‘Superman’ ” and graced the cover of Time magazine.
The deeper problem is reducing education to a single narrow metric that claims to recognize an educated person through a test score. Teaching toward a simple standardized measureand relentlessly applying state-administered (but privately developed and quite profitable) tests to determine the “outcomes” both incentivizes cheating and is a worthless proxy for learning.
I recently interviewed leaders at the University of Chicago Laboratory Schools—the school Arne Duncan attended for 12 years and the school where the Obamas, the Duncans, and the Emanuels sent their children—and asked what role test scores played in teacher evaluations there. The answer was none. I pressed the point and was told that in their view test scores have no value in helping to understand or identify good teaching. None.
http://www.washingtonpost.com/blogs/answer-sheet/wp/2013/04/03/bill-ayers-road-to-atlanta-cheating-scandal-runs-right-through-white-house/
http://m.washingtonpost.com/opinions/eugene-robinson-the-racket-with-standardized-test-scores/2013/04/01/1bbd61ea-9b01-11e2-a941-a19bce7af755_story.html
This scandal should be on the front page too.
-Rich
The tragedy of El Paso:
El Paso Rattled by Scandal of ‘Disappeared’ Students – NYTimes.com
Students identified as low-performing were transferred to charter schools, discouraged from enrolling in school or were visited at home by truant officers and told not to go to school on the test day. For some, credits were deleted from transcripts or grades were changed from passing to failing or from failing to passing so they could be reclassified as freshmen or juniors.
[…]
Eliot Shapleigh, a lawyer and former state senator who began his own investigation into testing misconduct and was credited with bringing the case to light. Mr. Shapleigh said he believed that hundreds of students were affected and that district leaders had failed to do enough to locate and help them:
“Desaparecidos is by far the worst education scandal in the country,” Mr. Shapleigh said. “In Atlanta, the students were helped on tests by teachers. The next day, the students were in class. Here, the students were disappeared right out of the classroom.”
Update:
4-1-13:
EL PASO, TX — Eight El Paso school district administrators are scheduled for firing by the school board as the district struggles with a testing and grading scandal.
An outside audit found that systematic test rigging and grade manipulation that sent one educator to prison continued long after his arrest. Former superintendent Lorenzo Garcia is serving a 3 1/2-year federal prison sentence for fraud.
LikeLike
Thanks for the posts. Very interesting.
LikeLike