We have often heard that Mark Twain said that there are “lies, damned lies, and statistics.” I checked with Wikipedia, and it turns out that this phrase has many fathers. For example, says Wikipedia:
Mark Twain popularized the saying in Chapters from My Autobiography, published in the North American Review in 1906. “Figures often beguile me,” he wrote, “particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: ‘There are three kinds of lies: lies, damned lies, and statistics.'”
But there are other claimants to the phrase, as the article notes, including one who ranked false statements as “a fib, a lie, and statistics.” A variation on this phrase is: “simple liars, damned liars, and experts.”
And then we come to the “New Orleans Miracle.” According to recent research, test scores have improved dramatically since 2005, when Hurricane Katrina wiped out the public school district and replaced it with a district which is almost all-charter. Douglas Harris, director of the Research Alliance in New Orleans reported the results in the conservative journal Education Next, which promotes alternatives to public education. Bottom line, in his account: Wiping out the district, firing all the teachers, wiping out the union contract, hiring Teach for America to replace veteran teachers, has mostly good outcomes. Education Week reported Harris’s claim of dramatic progress.
But then there is Mercedes Schneider, who reports that the state released 2015 ACT scores for every district, and the New Orleans Recovery School District ranked 70th out of 73 districts in the state. Its ACT scores are virtually unchanged over the past three years. The RSD ACT scores of 16.6 are far below the state average of 19.4.
An average ACT score of 16.6 is low. Louisiana State University requires a composite score of 22. A composite of 20 qualifies for La’s tuition waiver to a 4-year institution; a composite of 17 qualifies for tuition waiver for 2-year technical college.
And here’s the latest study by Research on Reforms in New Orleans, comparing the Orleans Parish public schools to the reformers’ Recovery School District. “A study of three ninth grade cohorts, beginning with the 2006-07 year, shows that the percentage of OPSB 9th graders who graduate within four years is almost double that of RSD 9th graders, and the RSD’s dropout rate is nearly triple that of the OPSB.”
You may decide which statistics matter most to you. But whichever you choose, be sure to read Jennifer Berkshire’s account of what the reforms in New Orleans have produced. It is important context in which to place whatever data you think is most valuable.
Paul Thomas reviews the debate about The progress of Néw Orleans and concludes:
“So we are left with two truisms about education publications and education reform: (1) If “Education” is in the publication title, you better do your homework, and (2) if education reform is touted to achieve outcomes that seem too good to be true, then they likely aren’t true.”
Diane Ravitch's blog 
Sent from my BlackBerry 10 smartphone on the Verizon Wireless 4G LTE network.
LikeLike
(3) John White may be one of the most dangerous frauds if our times. And he may well believe almost everything he says.
LikeLike
…of our times…
LikeLike
Also recommend you read Harris’ account carefully. He allows that there may be multiple factors tied to any test score improvements he found. And clearly states that it can’t be said for certain that the market-based reforms are the basis for the increased scores he reports. I think the study is far more nuanced then the headlines imply.
LikeLike
Read it, Rick. You are right in that is more nuanced. I still have questions regarding the study though.
First, Harris works for Tulane University. This does create a bias in that Tulane has been the driving force behind administering this system.
Second, Harris claims that there is no evidence that “teach-to-the-test” methods have a significant effect on these results. I guess I just don’t see how that’s possible. In a system that allowed for schools to be closed on the basis of test scores AFTER Katrina, schools had to double down hugely on preparing kids for the test. The test scores became the means of survival. So I guess I don’t understand how this was ruled out as a factor. I teach in Michigan and our math teachers are under pressure to get higher scores. We’ve seen more state test prep than ever, had a socio-economic shift in our student population and experienced virtually no turnover in our math department. But when asked to increase test prep, our scores went up noticeably. So, that rings very false to me. And our score was not under the threat of closure. Can you imagine the intensity of test prep when the very organization needs its results for survival?
Third, I think I’m just tiring of economists doing these studies. The ONLY information that Harris uses is test scores. Sure, he looks at contributing factors and cohorts but test scores are the epicenter. Today’s American culture with its fractured households, has rendered school as a place where we do far more than educate academically. And as a teacher, I can tell you, this is not by my or any other teacher’s choice. As we get bogged down in endless data and metrics, we find ourselves in a world where nothing else seems to matter. It seems that economists are highly susceptible to ignoring the non-statistical elements of what happens in a school.
When reading Berkshire’s article, it seems quite striking that many in New Orleans are highly unsatisfied with the schools. Test scores are not all that matters. But they are when certain groups are trying to prove a point.
I’m not attacking you. I think I’m just exhausted about ALL research revolving around test scores. As a teacher I do so much more than improve essay-writing ability. In fact, I’ve received far more feedback relating to my interpersonal interactions and how meaningful it has to students who I have taught and coached. A teacher’s job is so much more than spreadsheets and data analysis
LikeLike
I’m writing on the very topic you highlight. Your argument is a worthy one. Educators need speak out as to what is important in terms of outcomes of schools. Tests say something, but as you point out, it isn’t at all clear that they capture what is most important. Their lack of predictive validity about any sort of important school, job, life, etc. outcomes is telling. JJ Heckman, Nobel Laureate economist, has written persuasively about this, arguing instead in favor of what he calls ” soft skills.”
Stay at it, your voice is important.
LikeLike
Rick, first thanks for recognizing that I wasn’t on the offensive toward you. I started typing calmly and then my emotions kept rising when I think about my annoyance at data-obsessed policymakers.
One other thing. I teach juniors and seniors. I teach AP History. My students disregard the state tests as meaningless. They have made it clear that they don’t try but just gloss over the test. The online test shuts down if they answer super fast. It took a day for them to realize this. They said they would read the questions and if they weren’t sure of an answer they’d just guess rather than truly think. The test don’t prevent them from graduating and have no effect on college admissions. (Hell, I’ve see multiple students who scored 4s and 5s on multiple AP history tests receive “Not Proficient” in state testing on social studies.) So test research at the secondary level is pointless to me. (And why should they care about what happens to the school? They have one year remaining.)
Thanks for the recommendation on Heckman. I’l look Heckman up.
LikeLike
Yes, there are so many compounding problems regarding testing at the high school level. Not that primary isn’t totally insane with utterly inappropriate testing.
LikeLike
NOLA ed reform tran$formation hasn’t made much of a dent in the city’s murder rates: http://windycityteachers.blogspot.com/2015/07/triple-digit-nola-murders-despite.html
LikeLike
Did anyone hear This American Life this weekend? The whole show is worth a listen, but if you are pressed for time, listen to part 1 at 20 min – students were allowed to switch from a failing districts to a high achieving district. The reaction of the parents in the high achieving district is telling. I guess there are many parents who do care about “test scores”.
http://www.thisamericanlife.org/radio-archives/episode/562/the-problem-we-all-live-with?act=1#play
LikeLike
I listened to it in it’s entirety about an hour ago. I hate to write but I am inspired to write about that story because it brought me back to when my own elementary school was integrated, through my middle and high school years attending integrated public schools in New Orleans to educating my children in mostly integrated public schools in New Orleans over a 23 year period, only ending in 2014.
As I watch this mess that is New Orleans education reform, all I can say is that we are missing the real issues in transforming public education for the children for whom it was broken. This is being missed at every level of policy making. I want to hear the next episode before I write, but based on what I’ve heard so far, my question is…if integration is the only thing that has proven to be successful for pool Black children, why isn’t the Recovery School District in New Orleans an integrated district. There are very few White children in RSD charter schools. There is one RSD charter school that has 36% Caucasian children and that’s only because the driving force behind creating that charter school was an integrated committee of people who live in a mostly middle class integrated community. Sure they reached out to some of their poorer neighborhoods nearby but this school is only 60% free and reduced lunch where many RSD schools are above 90%. The issue is more about economic integration than it is about racial integration. I’ll leave this right here and the rest will be on my blog by the time of the 10 year Katrina Anniversary on August 29th. You can follow the blog at http://www.edutalknola.com
LikeLike
I look forward to reading about your experiences and your thoughts on your blog.
It made me really sad to hear the comments those parents were making. 😦
LikeLike
Those parent comments reminded me of the comments parents made as William Frantz school made as Ruby Bridges integrated the school. I strongly believe in integrated schools, integrated racially, economically and by ability. In a time where we are rating schools and making decisions regarding closure, we need to make sure we are dealing with apples to apples comparison. My 2nd son graduated in 2014 from a charter school that used to be a magnet school. They used to tout their “diversity” but it was in no way economically diverse. it had the feel of a private school because they had policies that kept really poor families from actually choosing that school though it was an A rated school. Our state will close failing schools. This school tests kindergarteners and kicks out kids who don’t keep a 2.0 GPA. This school will never have high numbers of low performing students, thus will never close due to academic performance. However the schools the kids kicked out or who never make it in, will have to attend schools that can’t select and keep only the smartest students. This school used to be close to 50/50 Black and white. After almost 10 years of being a charter school, it’s 29.9% Black in a cit with around 80% Black school aged children. You see when they were almost 50/50 it was not economically diverse. Now it’s lest economically diverse and less racially diverse. They have very few students with disability and via policy, they have very few low performing students, so it’s certainly not very well integrated by ability. If we believe integration is a real solution, unless we look at ability and economic integration, we are going down a slippery slope.
LikeLike
Thank you Diane, you captured everything I wanted to say but have been too busy to write this week. Listening to the episode of This American Live another commenter mentions has sent me spinning in a historical whirlwind. I’ll write about the intersectionality of that podcast and what’s happening in New Orleans on one of my too infrequent blog posts.
LikeLike
I love how those who are against these reforms resort to trickery in much the same manner that they accuse reformers of using when it supports their case.
Does anyone think it is appropriate to compare New Orleans results to state averages and then call them disappointing? Duh, look at urban area vs. statewide anywhere.
Then, comparing OPSB, which serves a disproportionately low number of students with disabilities compared to RSD (5.49% vs. 11.1%), is equally ridiculous.
I have no respect for someone who makes up their mind first and then looks for data that supports their position and ignores the rest. The preponderance of evidence in NOLA shows improvement ranging from moderate to very high.
LikeLike
Any reports of improvement must taken into consideration that students no longer in the RSD. It’s not really about comparing the two in terms of graduation rates as far as I’m concerned. I see this report as a different way to show that there are large numbers of students no longer in the cohort which skews reports of “moderate to very high” improvement. At the end of the day, no one looked at improvement when they raised the score needed to define a failing school, it was a point in time, a line drawn in the sand. We can dual in data all day. Come join our bus tour and I’ll show you how these reforms are working out in the lives of the people. In a city where after all that financial input, loss of democratic control of our schools, all that new innovations, we still have a 52% Black Male unemployment rate. Almost 10 years, we have over 15,000 “opportunity youth” between the ages of 18-24. That’s youth not in school or not working. Looking at the RoR article on 9th grade cohorts is less about comparing RSD and OPSB and more about the validity of RSD gains.
LikeLike
As the late Gerald Bracey put it in one of his “Principles of Data Interpretation”: “When comparing groups, make sure the groups are comparable.” (READING EDUCATIONAL RESEARCH: HOW TO AVOID GETTING STATISTICALLY SNOOKERED, 2006) [See comments by Karan Harper Royal above this one.]
The New Orleans Miracle? Effective antidotes to the twisted, ever-changing, and self-serving metrics of that particular example of rheephorm include the blogs of deutsch29, crazycrawfish, and louisianaeducator.
Hint: the sneer, the jeer and the smear are no substitutes for facts, reliable and trustworthy numbers & stats, and logic.
Just sayin’…
😎
LikeLike
Reblogged this on Education Talk New Orleans and commented:
There will be many stories about New Orleans coming soon as we approach the 10 year anniversary of Katrina. It’s an anniversary, but hardly anything to celebrate. We should not celebrate the perceived improvement in our school system since it was taken over by the RSD. I encourage people to look very hard at the Recovery School District since it was the reform. Ask them about each and ever child who has been educated in the RSD over the last 10 years. The state has the ability to track student ID information and they know if those specific students are doing better or not. Yet, they tend to come up with wild ways of talking about gains and improvement. Researchers are talking about their data twins, but no the very students in the reforms. It should make you all wonder why is it that the proponents of the takeover in New Orleans talk more about schools (buildings) than children. “Fewer failing schools” they say, but what happened to the students who lower performance caused them to fail and drop out. I guess as according to Dr. Charles Hatfield’s report, large numbers of them have “left the state or country.” Stay tuned, lots more to come from New Orleans.
LikeLike
I have not followed “reform” in New Orleans closely.
However, Doug Harris has written some pretty good stuff in the past. Yeah, an emphasis on test scores might be problematic, but that’s what economists tend to use. Here is what Harris said in the New Orlenas study:
“The scores cover grades 3 through 8, are averaged across subjects, and are standardized so that zero refers to the statewide mean…before the reforms, students in New Orleans performed far below the Louisiana average, at about the 30th percentile statewide…The performance of New Orleans students shot upward after the reforms. the performance gap between New Orleans and the comparison group closed and eventually reversed…”
Harris then says there might be complicating factors:
“The goal of any analysis like this is to rule out explanations for the changes in outcomes other than the reforms themselves. Our main comparisons deal with many potential problems… we consider in more depth four specific factors that could bias the estimated effects on achievement: population change, interim school effects, hurricane-related trauma and disruption, and test-based accountability distortions.”
Harris notes that “it is hard to rule out other potential test-based accountability distortions with our data.” And he adds that ” there are many challenges to estimating the effects of the New Orleans package of school reforms. ”
After assessing those factors, Harris concludes that their effects are fairly small, and says that, overall, “The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry.”
Yet, there’s an important caveat. Harris notes that ” the effects of even the most successful programs are often not replicated when tried elsewhere.” And he adds this: “There is more to the debate than we can cover here, including fundamental philosophical issues about whose objectives and values should count in making schooling decisions.”
But seriously, to take the grades and “reforms” that Doug Harris is referring to and then try to compare them to ACT scores is, well, irresponsible.
LikeLike
“After assessing those factors, Harris concludes that their effects are fairly small, and says that, overall, “The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry.”
Calling these results “fairly small” doesn’t seem accurate. Tennessee class size reduction effect size was .25 standard deviations. The black/white achievement gap is 0.7 nationally.
LikeLike
Even if those are precisely the same measures, you’re comparing city stats to state and national stats. Wider margin of error the smaller the population, and it’s easier to fudge those numbers by school policies at more local levels. Look at NYC.
LikeLike
Akademos, you probably remember that the national press was overflowing with stories about the “NYC Miracle” when Bloomberg was mayor (great P.R. staff). Now the local press (the NY Post in particular) runs story after story about “our failing public schools.” How could they have gone from reform miracle to failure in only two years?
LikeLike
John, what do you think of the ACT scores after 10 years of “reform”? 70th out of 73 districts in one of the nation’s lowest performing states. Far below what is expected for entry to college. That seems significant too, don’t you think? It reminds me of Eva Moskowitz’s high test scores, standing alongside the fact that not a single 8th grader from Success Academy charters was able to pass the test to get into a selective public high school in NYC. What should we make of that?
LikeLike
IMO, what we should make of that is that change is very hard, which we all know.
I think we should acknowledge improvement where we see it, not find a statistic (e.g. comparing against state data) in order to demonstrate a point.
Same with your comment regarding NYC data. There can be great improvement yet still be unsatisfactory results. It’s disappointing to see people cherry pick statistics to support points.
If the exact same data existed regarding an intervention that you liked, I don’t think you’d be saying the same thing.
LikeLike
Certainly the average is impacted by the students who are no longer there, right? Dr. Hatfield’s paper speaks to the students who are no longer there in the RSD. There is another Tulane report that speaks to the high number of “opportunity” youth in New Orleans. You can just measure the ones who stay and say you have success! Why aren’t we asking LDOE to show us how the students who started in the RSD have fared over time? Whether they stayed in schools or whether they left, we need to know. The state has the data of real students, yet they aren’t sharing that information with researchers who are not affiliated with the “reforms.” Even after being sued, they did not give the plaintiffs all of the data. Why withhold it? Dr. Hatfield analyzed some of what he did manage to get as it relates to 9th grade cohorts. I believe it’s worth looking at that number. Not as in comparison to OPSB, but to show that many students are no longer in the “reform.” Imagine if all of your lowest performing students left, the achievement of those who remain would look pretty good.
LikeLike
To be fair, RSD in New Orleans is way worse than Eva’s SA chambers.
LikeLike
Test scores this, test scores that, higher test scores, lower test scores, test scores, test scores, test scores, la la la la la la la la!! And the MMoOO*ing continues. All the while the students are being harmed by the insanity that are the standardized testing malpractices that are based falsehoods, chimeras, inanities, figments of folks’ imaginations.
When, when the hell will we move beyond this thoughtless, senseless, stupid, pretentious hogwash and hooey??
Are you a part of the problem of perpetrating one of the greatest frauds in the history of education??
*Mental Masturbation or Obligatory Onanism.
LikeLike
It’s been ten years since Katrina. And what we learn from New Orleans is an unchanging status quo of little or no progress in RSD–despite a large chunk of data they have accumulated in the last decade. Time to dispose a giant old fridge/10-year-old mainframe to trash before they are buried alive.
LikeLike