Horace Meister is a young untenured scholar who writes for this blog.

He writes:

Competing narratives underlie the disputes on how to best improve education for all students. On the one hand we have narratives of testing, accountability, and the free market. On the other hand we have narratives of collaboration, social capital, and public goods. Data are often cited in these debates to support one narrative or the other. But there is a dark art to the use of data, an art at which the powerful forces of corporate reform and school districts operating under their paradigm excel.

Let’s take a look at how reformer think tanks and “research” organizations manipulate data and how school districts mimic those strategies. The New York Times editorial page recently gushed over “Michael Bloomberg, who improved graduation rates and college acceptances in poor neighborhoods by shutting down schools that were essentially dropout factories and starting afresh with smaller schools, new teachers and new leadership [1].” The editorial board does not realize or acknowledge that in New York City “student outcomes have not improved compared to similar districts, which did not implement the market-based reforms [2].” The editorial board also does not realize or acknowledge that the MDRC papers, the “research” often cited as supporting the shuttering of community schools and their replacement with small schools of choice, are deeply biased and flawed [3].

Additional flaws and biases with the MDRC “research” can be added to the top 10 list in the piece cited in endnote #3. MDRC seems to have deliberately biased their sample so as to come to conclusions that support the corporate reform approach [4]. MDRC only looked at high schools– ignoring elementary and middle schools that were also subjected to closure and re-opening (and, in some cases, re-closure and re-re-opening). The data show that the new middle schools that opened under Bloomberg performed worse than the older middle schools, when controlling for student need [5]. The data also show that of “154 public elementary and middle schools that have opened since Mayor Bloomberg took office, nearly 60% had passing rates that were lower than older schools with similar poverty rates [6].”

MDRC only studies new small high schools that opened up by 2008, the very years during which the new small high schools were allowed to exclude special education students and English Language Learners. By now they could have added to their sample additional student cohorts, but they have not. Due to threats of a lawsuit since 2008 new small schools are no longer officially permitted to exclude students [7]. Does MDRC know that without this “competitive advantage” the new small school data wouldn’t look so good? When a purportedly objective “research” organization manages to exclude entire categories of schools and when including the excluded schools would lead to a more objective and less positive evaluation of a policy, we are witnessing the dark art of data manipulation.

MDRC did not consider alternative hypotheses, a basic requirement of the scientific method as taught by every science teacher. So let’s consider an alternative hypothesis for the editorial board of the New York Times. Here is the hypothesis: “Large community high schools and large high schools of choice have better student outcomes than other high schools serving similar students.” Indeed the data support this hypothesis [8]. The New York City Department of Education produces report cards that evaluate schools on their “peer percent of range.” According to this data the largest high schools in New York City, those serving over 2,000 students, outperform peers by +14.7% on weighted graduation rate (a metric that takes into account the quality of the diploma such as whether or not it is Regents-endorsed or an advanced Regents diploma) and by +20.1% on college readiness [9].

Rather than favoring certain types of schools over others and forcing schools to compete with one another, as Bloomberg did and the New York Times editorial board wants to continue, let’s have schools collaborate and work together in an equitable policy environment [10]. This approach to creating great schools is supported by the (non-manipulated) data [11].

Unfortunately, school districts operating under the corporate reform paradigm do not want to follow such an approach. Instead they manipulate data in ways that are biased towards their ideological agenda. As we just saw, large high schools in New York City do a great job on college and career readiness metrics. This must have put Bloomberg’s Department of Education in a bind. They had all the data showing that the large high schools were outperforming their peers in college and career readiness, an important part of what high schools are all about. But they couldn’t allow the new small high schools created under Bloomberg to look bad. So when including college and career readiness metrics in the school report cards they only allowed them to count as 10% of the total school grade (and not 20% or 25% or 30%– percentages that would seem more important given the importance of college and career readiness). This minimized the negative effect that these metrics would have on the grades of schools created under Bloomberg [12].

This sort of manipulation is not uncommon. Corporate reform school districts believe in privatization and charter schools. So they do not address how creaming and the sky-high attrition rates at many charter schools explains their “results [13].” They believe in accountability and evaluating schools. So they grade schools using metrics that are deeply flawed and penalize schools that serve the neediest students [14]. They believe in accountability and testing. So they pretend not to manipulate cut-scores on exams for political ends [15].

Next time you see data cited, even it is from your own school district, question it.

[1] http://www.nytimes.com/2014/11/01/opinion/when-to-shut-down-failingōschools.html

[2] https://dianeravitch.net/2013/12/20/tweed-insider-where-the-bloomberg-administration-went-wrong-on-education/

[3] https://dianeravitch.net/2014/10/23/are-small-high-schools-the-magic-bullet/

[4] The following criticisms are aimed solely at the MDRC claim that the portfolio strategy as employed by the Bloomberg administration was a success. Small schools, if implemented fairly in an equitable policy environment, may provide a level of personalization and support that is valuable for many students. Large schools can also offer personalization and support through smaller structures such as academies or advisories. But this is a topic distinct from the specific one discussed here.

[5] http://www.edwize.org/new-middle-schools-same-old-challenges

[6] http://www.nydailynews.com/new-york/bloomberg-new-schools-failed-thousands-city-students-article-1.1119406#ixzz21NV9BDG3

[7] http://www.advocatesforchildren.org/Empty%20Promises%20Report%20%206-16-09.pdf?pt=1

[8] http://schools.nyc.gov/NR/rdonlyres/7E390ED1-1689-4381-BF70-E228840E5589/0/2012_2013_HS_PR_Results_2014_01_16.xlsx

[9] The high schools with over 2,000 students run the full gamut, from community high schools that serve all local students to selective high schools where admission is based on exams to comprehensive high schools serving students who choice-in from across the city. The Bloomberg administration tried to close some of these schools. The peer percent of range metric is designed to compare each school only to other schools serving students of comparable incoming performance and demographics.

[10] https://dianeravitch.net/2014/05/02/a-triumphant-return-to-professionalism-in-new-york-city/

[11] http://www.nytimes.com/2013/02/10/opinion/sunday/the-secret-to-fixing-bad-schools.html?pagewanted=all

[12] Note that this strategy of developing metrics in such a way that they favor specific school types and policies is distinct from the outright corruption of Tony Bennett, the former Indiana education commissioner, who changed the grades of individual schools. https://will.illinois.edu/news/story/former-indiana-superintendent-feels-heat-of-grading-scandal

[13] https://dianeravitch.net/2014/08/28/beware-the-charter-attrition-game/

[14] http://withabrooklynaccent.blogspot.com/2014/01/corporate-reform-versus-child-centered.html

[15] http://withabrooklynaccent.blogspot.com/2014/03/on-misuse-of-statistics-in-testing-by.html