Let me say at the outset that I am neither for nor against small schools. Sometimes they work well, because they have small classes and extra attention, sometimes they don’t, especially when they don’t provide classes for English language learners or advanced courses or foreign languages. As always, it depends.
Recently a report by a research organization called MRDC asserted that New York City’s Gates-funded small high schools were surprisingly successful.
But an underground researcher in the NYC Department of a education says, wait a minute. Review the evidence.
How to Reform a Portfolio District
In what has become an annual propaganda exercise, MDRC (yes, their corporate name is just the initials), a “research foundation” in New York City, has self-published a non-peer reviewed paper on their website claiming that the new small high schools created under the Bloomberg administration are a success. The New York Times followed up with an editorial claiming that “the Bloomberg approach has been vindicated” and that de Blasio should continue the same educational policies.
Is there any truth to these claims? Does the data support any of this? The answer is “no.” The papers self-published by the MDRC are shoddily researched with clear biases and poor grounding in reality. It order to keep the size of this essay to a manageable length let’s limit ourselves to a Top 10 list of the paper’s flaws.
- The Gates Foundation provides the funding for these papers. The Gates Foundation also funded many of the new small high schools in New York City. What we have here is a circular process of self-congratulation. The peer-review process might be expected to uncover the biases produced by this unholy alliance. But these papers have, of course, never been peer reviewed. They are self-published by MDRC on their website and then touted in press releases and newspaper editorials.
- It is becoming standard practice for researchers to publicly post data-sets used in such studies. MDRC has refused to release the data-set. This makes it impossible for their results to be independently verified or questioned.
- The papers claim that the new high schools “are open to any student who wants to attend.” This claim invents its own reality and ignores the existing literature that has shown how schools manage their admissions and enrollment processes so as to selectively screen out more challenging students.  It also ignores the facts on how the lottery process for these new high schools actually worked. In reality the new high schools used such tools as required attendance at information sessions, applications with essays, student biographical data, and listing mandated uniforms in the high school directory to screen out more challenging students prior to the lottery process as well as post the lottery process prior to enrollment.  A review of earlier papers in the MDRC series concluded that “carrying out the lotteries using the method described in the report may have resulted in nonrandom differences between the study groups.”  MDRC has never addressed these issues and continues to self-publish these papers on their website. It seems MDRC is more interested in continued funding than actually figuring out what really works for all students.
- The new small high schools have been found to engage in questionable academic practices and the manipulation of data at a higher rate than other high schools. For example, the new small high schools represent about 25% of all New York City high schools, yet in one year they made up 60% of the schools with patterns of data so suspicious that the Department of Education did not give them a grade. This should raise some serious concerns that MDRC does not address.
- For mysterious reasons MDRC excluded 33 small new secondary schools, a potential 30% increase in the number of schools examined in their self-published papers, even though 9th graders also apply to these schools through the high school admissions process. This may lead to significant bias in the results, especially since 6-12 schools are included in the comparison group.
- The new small high schools were closed at the same rate as existing schools, raising serious doubts about claims that the new small high schools as a whole were an improvement over existing schools. This reality, of course, also biases the outcomes of the MDRC papers. Since the closed and closing new small high schools cease to accept 9th graders, their lower student outcomes would have a smaller impact on outcomes in these reports. The closed and closing new small high schools include, Manhattan Theatre Lab, Gateway School For Environmental Research and Technology, International Arts Business School, Global Enterprise High School, High School of Performance and Stagecraft (renamed Performance Conservatory), Urban Assembly Academy for History and Citizenship for Young Men, and the School for Community Research and Learning. Interestingly enough about 7% of the schools in the MDRC sample have closed, which is very close to the effect sizes MDRC claims the new small high school have produced.
- The MDRC papers only examine what they term “oversubscribed” new high schools. Only about 85% of the new small high schools meet this criteria. Meaning that 15% of new small high schools did not have enough applicants to fill their seats. Remember this is in context of students listing up to 12 high schools on their application. This means that the comparison groups are not equivalent. The outcomes of the presumably much weaker new small high schools are excluded. In order to make the comparison equivalent, the 15% of comparison schools with the weakest outcomes that the matched students attended should have been excluded as well. MDRC did not, of course, make this correction.
- Even with their biased methodology the MDRC papers have shown that the new small high schools have no significant impact on mathematics outcomes for students. Given the greater constraints in scoring on math exams, this difference suggests that any positive effects in the new small high schools are due to more relaxed grading policies rather than true increases in educational attainment. There is a lot more evidence suggesting the same thing, none of which MDRC addresses. For example, examination of credit accumulation in New York City schools has shown that while new high schools grant more credits, the credits do not correspond to a rigorous college ready curriculum.
- When analyzing outcomes of specific student populations the MDRC papers lump students into very broad categories such as English Language Learner and Special Education Status. Given the data showing, for example, that the new small high school serve fewer of the neediest special education students, such comparisons are clearly biased in favor of the new small high schools.
- MDRC does not acknowledge the special “favors” that were granted to Bloomberg’s new small schools. This includes receiving a higher percent of their Fair Student Funding formula than other schools , having more available facility space than other schools , and excluding special needs students  and English Language Learners in their early years . Any comparisons made in such an inequitable policy environment are ridiculous.
The current popularity of the portfolio district approach can be attributed to the following factors:
- a) Superintendents of urban districts and other district officials with no background in education- with zero expertise in education they have no clue how to improve teaching and learning.
- b) A reluctance on the part of districts to take ownership and responsibility for the success of their schools- this leads to the strange but increasingly familiar scenario of districts trashing the public schools they are actually supposed to be supporting and improving while praising and granting special favors to charter schools (see Newark, New Jersey and Camden, New Jersey).
- c) The short time-frame of most superintendents and other district officials in each posting. With no long-term accountability they can play the portfolio game for a couple of years- closing schools, opening schools, closing even more schools- giving off the impression of activity and hard work. Though no real progress is made, by the time this becomes obvious, they have transitioned into other positions at reform think tanks and foundations. 
Mayor de Blaiso and Chancellor Farina, please do not continue the education policies of the previous administration as the New York Times demands. Thankfully, you have already made very clear that you do not intend to, as the data show that the portfolio district approach employed by the previous administration was a failure.
Here is what you should do instead:
+ Develop rich, engaging curricula that support student learning and train teachers in implementing these curricula with fidelity while having the flexibility to customize the curricula to the needs of their students.
+Return to a geographic approach of school support and governance based on feeder patterns between elementary, middle and high schools. This will allow for articulation and alignment of supports as students progress from one grade band to the next.
+ Improve the metrics currently used to evaluate teachers and schools. The current metrics penalize schools that serve more challenging students and are open to manipulation. The initial revisions to the school Progress Reports are a good first step in what needs to be an iterative and ongoing process.
+ Focus on equity and fairness at every level of the organization. Enrollment practices must be reformed so that all students are educated by every single school. Tracking practices must be reformed so that every student receives a challenging academic program. Funding practices must be reformed so that schools are funded at levels appropriate to the students they serve.
This is how New York City will progress and truly serve every single student.
 Peer review is, of course, not the perfect solution for identifying bias in research. For example, the famous Chetty et al. study that was used to support value-added measures to evaluate teachers and played a role in the California tenure lawsuits now appears to have significantly exaggerated its claims. See Jesse Rothstein’s working paper at http://eml.berkeley.edu/~jrothst/workingpapers/rothstein_cfr_oct2014.pdf where he notes:
“Like all quasi-experiments, this one relies on an assumption that the treatment – here, teacher switching – is as good as random. I ﬁnd that it is not: Teacher switching is correlated with changes in students’ prior-year scores. Exiting teachers tend to be replaced by teachers with higher measured VA when students’ prior achievement is increasing for other reasons, and by teachers with lower measured VA when student preparedness is declining. CFR have conﬁrmed (in personal communication) that this result holds in their sample as well.
The evidence that the teacher switching “treatment” is not randomly assigned implies that CFR-I’s quasi-experimental analyses, which do not control for changes in student preparedness, cannot be interpreted causally…
It is not clear that the association between VA and long-run outcomes can be interpreted causally. The evidence of bias in VA scores means that the association between a teacher’s VA and students’ long-run outcomes may reﬂect the student sorting component of the VA score rather than the teacher’s true eﬀect. Moreover, even if this issue is set aside there is still a concern that students assigned to high-VA teachers may be advantaged in ways that are predictive of the students’ long-run outcomes, implying that the estimated “eﬀect” of being assigned to a teacher with high estimated VA is upward biased. In both CFR’s district and the North Carolina sample, teachers’ measured VA is correlated with students’ prior scores and other observables. Neither CFR-II’s observational estimates nor their quasi-experimental estimates of teachers’ long-run eﬀects control fully for students’ observed, predetermined characteristics.”
 See Jennings, Jennifer L. (2010) School Choice or Schools’ Choice? Managing in an Era of Accountability. Sociology of Education 83: 227-247 “Although district policy did not allow principals to select students based on their performance, two of the three schools in this study circumvented these rules to recruit and retain a population that would meet local accountability targets.”
 http://www.nyccej.org/wp-content/uploads/2012/01/school-closures-report.pdf. See http://blogs.edweek.org/edweek/eduwonkette/2008/06/why_has_the_education_press_mi.html and http://eduwonkette2.blogspot.com/2007/10/turnaround-at-evander-childs-nyc-small.html for other significant demographic differences between older schools and the new small high schools.
 See, for example, the cases of Joel Klein, former chancellor of the New York City Department of Education, who now works for the Murdochs and of Marc Sternberg, former deputy chancellor of the now shuttered Division of Portfolio Planning at the New York City Department of Education, who now works for the Waltons.