Archives for category: Research

A couple of days ago, New York Times’ writer David Leonardt wrote a column endorsing Question 2 in Massachusetts, which would expand charters by 12 a year for the indefinite future. He presented some studies to buttress his view that charters are “schools that work,” which he defines as “high expectations, high support.” He visited the MATCH charter school and talked to some of the researchers, who were excited about their findings. Leonardt acknowledged that not all charter schools were as effective as the ones in Boston, but nonetheless thought it was a good idea to authorize more charters in Massachusetts.

I invited Jersey Jazzman, who is a teachers and also a graduate student at Rutgers, to review the column and the evidence. Here is his response, as he is an expert on charter research.

He begins by suggesting that the comparisons in the study cited by Leonhardt are inadequate.

Because simply showing that charter school students in Boston get better test scores than similar students in the Boston Public Schools is not, I’m afraid, nearly enough evidence to support lifting the cap. In fact, the more time I spend looking at this research, the more questions I have about whether Massachusetts can reasonably expect charter expansion to improve its schools:

– Are the students who enter charter lotteries equivalent to the students who don’t? This is a critical limitation of these studies that is often ignored by those who cite them. The plain fact is that the very act of entering your child into a charter school lottery marks you as different from the rest of the population; you are taking an affirmative step the majority of public school parents are not taking in an attempt to improve your child’s education. There’s a real likelihood your family is not equivalent to a family that doesn’t enter the lottery…

It’s not always clear how to calculate the overall target population in these studies; I used the Ns that made the most sense to me.** But even if we’re not quite sure about the exact numbers, the scope of the issue is clear: the study sample is only a fraction of the total population. Which would be fine — if the sample was randomly drawn from the target population.

But clearly, that’s not the case: The sample is self-selected, because families have to choose to enter the lottery. Which means the results of the study can only be generalized to that population, because there may be characteristics of the students in the sample that are different from the entire Boston population and affect test scores….

First of all, how do we know the charters are any different than the traditional public schools regarding these school practices? Angrist, Pathak & Walters (2012) surveyed charters for their practices, which is fine… except we don’t really know how they compare to the public district schools. If we’re going to ascribe effects to these practices, we should know how they differ across our treatment and control schools.

We can say, however, that the charters have longer school days and years. This is probably a significant contributor to any effects the charters show. But is it necessary to expand charters to lengthen instructional time? Can’t Boston just do that in its district schools?

JJ points out that charters are known for their reliance on inexperienced teachers who burn out and leave within 2-3 years. Does the Boston area have enough wannabe teachers to staff a growing number of charters? Is it really a good idea to rely on policies of churn-and-burn for teachers, continually recycling inexperienced teachers to meet the demand for them?

He says that the question of cost is central to the proposal for expansion. Parents understand that more money for new charters means less money for public schools, their own community’ public schools.

JJ writes:


And, yes, there are costs. As this clever model developed by a couple of public school parents shows, districts can’t easily absorb the costs of charter expansion, which is why the state offers extra funds. Unfortunately, the state has not fully funded this program in recent years; if they can’t find the money now, how will they find even more funding in the future?

We know that charters place fiscal burdens on hosting districts, largely because they educate students who would otherwise go to private school and they replicate administrative and other costs by creating multiple systems of school governance. We know that charters are not held to the same standards of transparency and accountability as public district schools, because they are not state actors. This has created major problems in other states, incentivizing behaviors that are not in the public’s interest.

Is it not possible, given all this, that Boston’s charters are getting good results because of the cap? That limiting their expansion has increased quality and stopped the abuses that have plagued states like Michigan, Ohio, and Florida, which have let charter expansion run wild?

He concludes:

I understand Boston’s children can’t wait any longer for real improvement in their schools and their lives. But lifting the cap largely on the basis of these limited studies is not, in my opinion, smart public policy. The good people of Massachusetts have every right to question whether voting yes on Q2 is in the best interests of students both in and out of charters, and to consider the limits of the evidence presented to them as they make their decision.

And I would add, taking money away from the schools that serve the overwhelming majority of children in Massachusetts so as to open schools for a tiny minority of other students makes no sense from a public policy standpoint–or common sense. Why weaken the public schools that serve more than 90% of children in public schools to benefit the few?

As for David Leonhardt, I suggest that he visit charter schools in Minnesota, Nevada, Ohio, and many other states, where charter schools are among the lowest performing schools in the state. Anecdotes do not make good public policy, nor does one visit to a “no-excuses” charter school in Boston.

Peter Greene wants to warn readers about a new study of Rocketship charter schools that he finds not credible. Maybe it is because the Rocketship corporation always relies on the same evaluators to do their studies, and he suspects they have become too interdependent over the years. Maybe it is because they measure what is not easily quantified. Maybe it’s because the study will soon appear as part of a sales pitch. Whatever. He finds it bizarre to speak of “months of learning gained,” a metric that is of dubious merit. From his perspective, this is just so much edubizness hype.

Teacher and historian John Thompson writes here about the reflection that seems to be occurring among “reformers” as they realize that their test-and-punish reforms produce limited gains and limited outcomes. He wonders how different our federal and state policies would be had reformers strived to implement research-based reforms instead of ideas that had intuitive appeal.

He writes:


Something important is stirring in terms of education research. We’ve always gone through cycles, mostly notably in the aftermath of the Coleman Report, during debates over the so-called “culture of poverty,” and during the contemporary data-driven, market reform era, where scholars have had to think twice when analyzing where the evidence leads. This last month, however, a variety of social scientists have candidly expressed the facts that corporate reformers deride as an “excuse.”

Heather Hill’s review of the Coleman Report recalls the seminal study’s finding, “One implication stands out above all: That schools bring little influence to bear on a child’s achievement that is independent of his background and general social context.” Hill reviews the subsequent analyses of Coleman, and the findings of Tony Bryk and Stephen Raudenbush, who “show that differences among schools accounted for about one-fifth of the variability in student outcomes.” The bottom line, she reports is that “schools still pack a weaker punch than many imagine.”

Neither did the Chalkbeat editors pull any punches. Its subtitles clearly convey the message that has been condemned as heresy over the last two decades:

Meanwhile, evidence mounted for one central conclusion: schools matter – but not as much as people might think; and

The logical conclusion: You can’t fix schools without trying to fix broader social inequality, too.

Similarly, Stephen Dubner’s begins his recent Freakonomics Radio program with the words, “in our collective zeal to reform schools and close the achievement gap, we may have lost sight of where most learning really happens — at home.” Dubner concludes, “Most of us probably think too much about cognitive skills and not enough about non-cognitive. Most of us probably put way too much faith in the formal education system, when, in fact, the path to learning begins way before then, at home.” In between, we hear from economist John List, “Schools only have kids for a handful of hours per day, but who, really, will mold kids through their lives are the parents.” Also, early education expert Dana Suskind concludes, that we need preventive, not remediative programs. “About the only way” that we can “move the needle,” she says, is through science-based programs which begin the learning process at birth or before.

Even the most steadfast true believers in accountability-driven, competition-driven reform seem to finally be facing reality. The first words of a NBER paper by John List, Roland Fryer and Stephen Levitt are President Barack Obama’s 2009 statement that, “There is no program or policy that can substitute for a mother or father who will attend those parent-teacher conferences … Responsibility for our children’s education must begin at home.”

And, even the TNTP seems to be questioning its blind faith that the answers for poverty can be found inside the four walls of the classroom. Its modest pilot project taught Ariela Rozman, Timothy Daly and David Keeling that, “We have a new appreciation for the annual catastrophe that is summer learning loss—and what a headache summer is for the families we work with in general. The out-of-school opportunity gap has received increased attention in recent years … because it is becoming clearer that it is a substantial driver of long term inequality.” After a year of working with real-life families in actual schools the TNTP acknowledges:

Some people argue the post-Katrina choice-based system has led to large, sustained improvements in performance and should become a model for the rest of the country. Others say it’s still largely a low performing system and the process of creating it profoundly disrupted its workforce and community.

That brings us to the research of Douglas Harris, the Tulane University Education Research Alliance, and their recent conference on early education in New Orleans. Harris has documented major post-Katrina gains in New Orleans test scores, while acknowledging that “critics are concerned that schools under reforms are too focused on test scores.” Moreover, he notes that “disadvantaged groups always see a smaller effect than the advantaged groups early in the reforms.” Especially before 2012 or so, there were “real horror stories about how special education students and others were suspended and expelled at high rates,” and “it remains unclear whether the problems are solved.” Harris sees “signs that high school dropouts are being under-reported,” and he says that NOLA’s decentralization can “negatively impact vulnerable groups.”

I sometimes question Harris’s confidence that oversight and accountability can mitigate such problems, but I trust his judgment in regard to the initial beliefs of NOLA reformers, “The original idea was that charters would create some degree of choice and competition, allow some schools more autonomy, facilitate innovation and diversify options. “Replacing” traditional public schools was almost never part of the conversation.” On the other hand, he doesn’t deny the current threat to traditional public schools, “Yet, this is exactly what is happening in New Orleans, Detroit, and some other cities (albeit to very different effect).”

I also sense that the participants in the ERA conference saw the multiple, often contradictory, outcomes of the radical NOLA reform, and that they are mostly preoccupied with addressing its remaining weaknesses. While they may or may not be fully aware of the national campaigns to impose their charter-driven system on cities across the nation, conference attendees mostly see the NOLA model as a “done deal” in their city. They are more concerned about the need to organize, fund, and implement early education programs than in other districts’ need to beat back corporate reforms.

I can appreciate those feelings, but I may have been alone in seeing one graphic as telling the most important story about New Orleans preschool, at least in terms of the lessons it holds for the rest of the country. Pre-kindergarten is only one part of the early education system that we need, but it is illustrative of the “opportunity costs” of the contemporary school reform movement. The percentage of NOLA’s students who attended pre-k dropped from 60% in 2007 to 40% in 2011. That’s a 33% drop at a time when the city’s schools were being funded at a level beyond the imaginations of most educators. Yes, the percentage of students who attend pre-k has increased since then, but in NOLA and across the U.S., we are now facing budget crises.

It’s bad enough that reformers let pre-school slide but, worse, the money for the gold-plated corporate reforms is gone. I doubt that anyone would claim that these reforms were cost effective, and now we have to tackle the complex early education challenge at a time when all of the participating education and social service providers face enormous budgetary constraints.

And that brings us back to the question of what would have happened if we had followed a science-based path to school improvement, as opposed to the test, sort, reward, and punish experiment, known as corporate reform. Granted, Katrina took New Orleans by surprise. It’s not like the city had the time and the inclination to study education research, debate policy options, and plan and implement the best possible reform policies. Not surprisingly, when offered a test-driven, competition-driven model, as well as enormous amounts of funding, they rushed the Billionaires Boys Club’s preferred approach into place.

On the other hand, if Katrina hadn’t hit during the accountability-driven, choice-driven craze, if edu-philanthropists had been assisting a science-based campaign to provide high-quality early education and to align and coordinate socioemotional supports, think of the great good that could have come from the rebuilding of New Orleans education and social service systems. In such a case, NOLA could have turned to state of the art, evidence-based solutions, not the endless edu-politics of destruction.

Yeah, in addition to the down sides of NOLA reforms, bubble-in scores are up. Those metrics probably reflect some meaningful learning, as well are the learning of the destructive habits that are nurtured by retrograde, teach-to-the-test instruction. Is there any doubt that students and families would have chosen humane, high-quality, aligned and coordinated early education programs over the competitive culture of today’s NOLA? And, had such a nurturing, science-based system been built in New Orleans, wouldn’t educators across the nation be welcoming – not shunning – help in replicating it?

Two researchers at Teachers College, Columbia University, surveyed parents who opted their children out of state tests and confirmed what leaders of the test refusal movement have long asserted. Parents don’t opt out because they are controlled by unions. They don’t opt out because, as Arne Duncan once said, they are fearful that their child is not as smart as they thought.

“Teachers College unveiled the findings of Who Opts Out and Why?—the first national, independent survey of the “opt-out” movement—which reveals that supporters oppose the use of test scores to evaluate teachers and believe that high-stakes tests force teachers to “teach to the test” rather than employ strategies that promote deeper learning. The new survey also reports concern among supporters about the growing role of corporations and privatization of schools.

“For activists, the concerns are about more than the tests,” said Oren Pizmony-Levy, TC Assistant Professor of International and Comparative Education, who co-authored the study with Nancy Green Saraisky, Research Associate and TC alumna. “We were surprised that the survey reveals a broader concern about corporate education reform relying on standardized test-based accountability, and the increased role of ‘edu-businesses’ and corporations in schools.”


Who Opts Out and Why? also reveals that opt-out proponents oppose high-stakes, standardized testing because they believe it takes away too much instructional time.”

This is an instance where research confirms common sense.

Chalkbeat interviewed one of the authors of the study, who said:

It’s the breadth of the movement that’s noteworthy, explains Oren Pizmony-Levy, one of the report’s authors.

“It’s not just about the tests. They’re saying something bigger about the direction of education reforms in the U.S.,” Pizmony-Levy said. “It does bring together all sides of the political spectrum.”

The most common reason opt-out supporters cited for boycotting the tests was opposition to using test data to evaluate teacher performance, with 36.9 percent of respondents listing that as one of their top two reasons to support opting out (45 percent of the respondents work in education). That was followed by concerns over teaching to the test (33.8 percent), opposition to the growing role of corporations in schools (30.4 percent), fears that the tests cut into instructional time (26.5 percent), and opposition to Common Core standards (25.8 percent).

Roughly half of those surveyed self-identified as liberal, while nearly 18 percent identified as conservative.

The authors noted that there is some potential bias in the data because it depends on accurate self-reporting, and was disseminated electronically, which largely excludes those who don’t have internet access.

But Pizmony-Levy said the survey still begins to sketch out a more detailed profile of who opts out and why. (On the most recent math and English exams, 21 percent opted out across New York state, as did 2.5 percent in New York City.)

“I think what this is telling us is activists disagree with the current direction of education reforms [which include] … ideas about accountability from the business world,” he said. “They’re saying maybe there are other directions we should go.”

Wendy Lecker, civil rights attorney and columnist, says that everything the corporate reformers are doing is contrary to science and research. Kids are not helped by closing their schools and firing their teachers.

What matters most is not pressure, incentives, rewards, and sanctions: What matters most is relationships.

Connecting students to mentors and to teachers who care about them makes a huge difference in their attitude towards school.

It stands to reason that school mechanisms promoting a personal connection improve learning as well as social development. Neuroscientists have found that the brain does not recognize a sharp distinction between cognitive, social and motor functions. Consequently, research has shown that feelings of social isolation impair key cognitive abilities involved in learning.

Though they require substantial initial investments, educational policies that foster relationships save money in the long run.

Developmentally-appropriate preschool, with an emphasis on play, enables children to acquire the skills necessary to form healthy relationships. There is near universal consensus that quality preschool benefits children, increasing the chance of graduation, higher earnings, and decreasing placement in special education, involvement in the criminal justice system and the need for other social services. It also can save society as much as $16 for every dollar spent on preschool, by avoiding the costs of these later interventions.

Small class size, which fosters closer relationships between children and their teachers, has been proven to provide similar benefits, increasing graduation rates and earning potential, and decreasing the likelihood and cost to society of risky behavior. Research also shows that increasing class size has detrimental and costly long-term effects on at-risk children.

In the next round of reform, when the current era of test-and-measure, rank-and-stigmatize is thrown into the garbage heap, can we focus instead on connecting kids to adults who care about them?

I am reposting this because the earlier version lacked a link and the conclusion of the study.

The study is called “Charter Schools and Labor Market Outcomes.”

Click to access texas_charters.pdf

(Note: the source has been deleted. Google the title to find it. This seems to be the latest iteration: https://scholar.harvard.edu/files/fryer/files/charters_appendix_aej_11.26.2017.pdf)

This is an astonishing study, not just because of its findings but because of its authors. Will Dobbie and Roland Fryer are economists who have frequently studied charters, incentives and their effects on test scores. Fryer’s research institute at Harvard was started with several millions from the Broad Foundation. Fryer is a member of the Massachusetts State Board of Education.

Here is the abstract of their study of charter schools in Texas and labor market outcomes:

“We estimate the impact of charter schools on early-life labor market outcomes using administrative data from Texas. We find that, at the mean, charter schools have no impact on test scores and a negative impact on earnings. No Excuses charter schools increase test scores and four-year college enrollment, but have a small and statistically insignificant impact on earnings, while other types of charter schools decrease test scores, four-year college enrollment, and earn- ings. Moving to school-level estimates, we find that charter schools that decrease test scores also tend to decrease earnings, while charter schools that increase test scores have no discernible impact on earnings. In contrast, high school graduation effects are predictive of earnings effects throughout the distribution of school quality. The paper concludes with a speculative discussion of what might explain our set of facts.”

The paper concludes with this speculation:

“Charter schools, in particular No Excuses charter schools, are considered by many to be the most important education reform of the past quarter century. At the very least, however, this paper cautions that charter schools may not have the large effects on earnings many predicted. It is plausible this is due to the growing pains of an early charter sector that was “building the plane as they flew it.” This will be better known with the fullness of time. Much more troubling, it seems, is the possibility that what it takes to increase achievement among the poor in charter schools deprives them of other skills that are important for labor markets.”

Apparently, the obedience and conformity taught in No Excuses charter schools do not help people in jobs where initiative and independent thinking are valued.

Bill Honig was State Superintendent of Schools in California in the late 1980s. I came to know and admire him at that time. Bill Honig has spent many years dedicated to the improvement of education. He continues to work in schools, providing ideas and support.

He recently created a website to share what he has learned about education.

The site is designed to present the research and experience supporting the “build and support” approach and show why the more extreme measures of the “test and punish” approach haven’t worked. It has 16 short articles about the major issues in the debate including a piece about experience in California and is designed for educational and political policy makers and members of the media. The site provides accessible background, research, and evidence and could be a useful tool. If you read the home page and the introductory remarks you can get the flavor of the effort.

I urge you to read it. Bill is a staunch friend of public education.

A new study conducted by Jennifer Heissel, a researcher at Northwestern’s School of Education and Social Policy, concludes that students who study Algebra I online do not perform as well on tests as their peers who learned the subject in traditional classes.

The study was published in the journal Economics of Education Review.

The study exploited a 2011 district policy change in North Carolina that allowed advanced eighth-graders to take Algebra 1 online. Prior to the change, none of the middle school students took Algebra 1; instead they waited until ninth grade to take it in a regular classroom.

North Carolina has developed one of the leading virtual education systems in the country, allowing rural middle school students the chance to take high school courses that would be otherwise unavailable. The virtual Algebra 1 middle school program increased equity in access at a lower cost than a traditional classroom, and most advanced students passed the course.

“However, equity in access does not guarantee equity in outcomes,” Heissel wrote in the study. “Policymakers should carefully weigh these tradeoffs.”

What surprised Heissel most was that the effect was seen in students who normally perform above average.

“Generally, no matter what you throw at high achievers, they end up fine,” Heissel said. “That’s what concerns me: If even the advanced students can’t do well, why would we think it would work well for all?”

Christopher Lubienski reviews two recent voucher studies on behalf of the National Education Policy Center in this post. (The post summarizes the findings and contains links to Lubienski’s report.)

The two studies under review purport to show the success of vouchers. One was prepared by the pro-voucher Friedman Foundation for Educational Choice, whose role is to cheerlead for vouchers. The other comes from the Department of Education Reform at the University of Arkansas. Lubienski concludes that neither proves the success of vouchers.

William J. Mathis has drawn together the research on class size to explain why it matters.

It is costly to reduce class size, but it is very likely the most effective intervention to help students who are struggling.

It also happens to be the single reform that parents want most. When class size was put on the ballot in Florida, it was overwhelmingly approved. Despite numerous attempts by Jeb Bush and his allies to get rid of it, the caps on class size have remained intact.

Reformers disregard the research on class size because they don’t want to spend more money to do what works. They prefer changes in governance, like charter schools, vouchers, mayoral control, state takeovers–anything but reducing class size. They claim that reducing class size benefits unions because it requires more teachers. But the biggest benefit of reducing class size is to the children, who get the attention and time they need to learn.