Early in her tenure as Secretary of Education, Betsy DeVos admitted that she is not a “numbers person.” She is also not a research person. The research shows that none of her favorite reforms improve education. Bu that never deters her. When the U.S. Department of Education study of the D.C. voucher program showed that the students actually lost ground as compared to their public school peers, she didn’t care. Nonetheless, she did recently cite a study from the Urban Institute claiming that the Florida tax credit program (vouchers) produced higher enrollments in college.

William Mathis, research director of the National Education Policy Center and Vice-Chair of the Vermont Board of Education, took a closer look at the study and found that the study did not prove what she thinks it does and offers no support for vouchers because of the confounding variable of selection effects. Someone at the Department should explain to her what a “variable” is and what “selection effects” are.

Do Private Schools increase College Enrollments for Poor Children?

A Closer Look at the Urban Institute’s Florida Claims

William J. Mathis

A review of:

Chingos, Matthew M. and Kuehn, Daniel (September 2017). The Effects of Statewide Private School Choice on College Enrollment and Graduation; Evidence from the Florida Tax Credit Scholarship Program, Urban Institute. 52 pp.

The Urban Institute reports that low income students who attended a private school on a Florida tax credit scholarship (“neovouchers”), in pre-collegiate grades had higher percentage enrollments in community colleges than traditional public school students. Using language such as the “impact of” and “had substantial positive impacts,” the findings are presented as causal. This purported effect was not found by the study’s authors in four year institutions or in the awarding of degrees – just in matriculation to community colleges.

Nevertheless, for school choice advocates, this report was hailed as good news on the heels of recent negative statewide school voucher reports coming out of Louisiana, Indiana, DC and Ohio. While community colleges are non-selective, most would agree that increased community college attendance is a good thing.

That said, a closer look indicates there is less to this latest report than first meets the eye. The primary problem—selection effects—is obliquely acknowledged by the report’s authors but is far too critical to push to the background.

There are at least three important differences that likely exist between the voucher group and the non-voucher group.

• Motivation, Effort, and Seeking Out Education Options – The very act of opting to enroll in a private school signals a very significant difference between the groups. Such an action requires considerable effort on the part of parents and students in selecting, applying, and transporting the child to the private school. These private school parents demonstrate, almost by definition, a higher involvement in their child’s education. Logically, these families would also be more likely to seek out community college options.

• Finances – While the program is available only to less affluent families, private schools can charge an amount higher than the $6,000 maximum available through the neovoucher. (Currently, eligibility rules require that the student’s household income not exceed 260 percent of the federal poverty level). Parents who can arrange or pay these supplemental tuition and fees to attend a private school represent the upper economic end of this means-tested group.

• Admissions – Private schools can continue their usual admissions policies, which may exclude children with special needs or deny admission on the basis of other characteristics. We cannot know the specific differences this introduces between the treatment and comparison groups, but we can be reasonably certain that these differences exist.

The study is based on “matching” private school students with traditional public school students and then comparing the two groups. While a common technique in voucher research, troubles arise when trying to pair up each student with her doppelganger from the other camp. As the authors acknowledge, “the quality of any matching can vary” (p. 12). While the researchers did an admirable job of matching, the entire process runs the risk of leaving out very important and determinative missing variables, as described above.

The study’s regression analysis also attempts to control for differences among students. In theory, an absolutely inclusive model can “confirm” a theory, and thus the researcher can claim a causal effect. But that’s a slippery slope. Regression is simply multiple correlation – and despite many inferences in the report, that is not causation. This is particularly true in this case, where selection effects are so strong.

In summary, it is the selection effects that primarily limit the study. A reasonable interpretation of the data is simply that the difference between the groups in their enrollment rates at community college is primarily due to different characteristics of families and students. In any case, the claim of private schools causing higher community-college attendance rates—let alone high college attendance in general—is a reach too far.