Researcher Beth Zirbes, a teacher of advanced mathematics, used her skills to dissect a charter school study produced at Harvard. The study was reported by Paul Peterson in The Journal of School Choice; Peterson, like the Journal, is an outspoken advocate for charters and vouchers. The study claimed that charter schools outperform public schools, and that the charter schools in Alaska were best among all states.
The governor of Alaska cited the study as a reason to increase the number of charter schools.
Zirbes doubted that this was true and decided to do her own analysis. What she found, amazingly, was that the Harvard study ignored vital demographic factors.
She wrote:
When I first saw the results of the Harvard study concerning charter schools I was simultaneously unsurprised and skeptical. I was unsurprised as I have seen many very bright young students in my AP classes come from charter schools. I was skeptical as I suspected much of this performance could be attributed to the type of student who attends Alaska’s charter schools. As a comparative analysis of Alaska’s charter schools and neighborhood schools had not been done, I set out to do one myself.
To determine whether charter schools outperform neighborhood schools I looked at the performance of all schools on the 2018-2019 PEAKS ELA (English Language Arts) assessment from the Alaska Department of Early Childhood Education and Development’s (DEED) report card to the public, as this year was within the same time frame as the data from the Harvard study. The performance of each school is given under the “2018-2019 Performance Evaluation for Alaska’s Schools (PEAKS)” tab. I used the data on this page for every school in the state which allowed me to analyze test scores and demographic characteristics such as the proportion of the school who are economically disadvantaged, English language learners (ELL), and special education (SPED). Demographic characteristics are only given for the set of test-takers and thus all summaries and analyses are for students in grades 3-9 during the 2018-2019 school year. I also removed all correspondence schools from my dataset as these students were not included in the Harvard study, do not take NAEPs tests, and have very low participation rates on state tests. For my analysis on performance, I also restricted my dataset to include schools only in districts where charters are an option to ensure that the student populations were as similar as possible.
At first glance, it appears that charter schools are more successful than neighborhood schools. At charter schools 52.5% (1,866 out of 3,554) of students were proficient on the ELA assessment versus 40.1% (18,655 out of 46,574) of students at neighborhood schools. However, these differences could be explained by the differences in demographics of the student bodies at these schools. To rule this out as a potential issue, statisticians control for these variables in their mathematical models. We can then ask, do the charter schools outperform neighborhood schools that have similar characteristics? Or do charter schools do any better than we would expect, given their student populations? In short, the answer is no, they do not. When I fit a model which controlled for socioeconomic status alone, the type of school (charter versus neighborhood) was not significant. (For anyone who knows statistics, the p-value associated with type of school was 0.57. It wasn’t even close.) In summary, there is no evidence that charter schools outperform neighborhood schools in terms of ELA proficiency once we consider their socioeconomic make-up.
During my data exploration, I discovered that charter schools, on average, have very different student bodies than neighborhood schools. Charter schools have far fewer economically disadvantaged students, far fewer ELL students, and are comparable to neighborhood schools in terms of SPED populations. Here is a summary of how these populations differ for all Alaskan students in the relevant grades in all of Alaska’s brick and mortar schools for the 2018-2019 school year:
- Neighborhood schools were 52.2% economically disadvantaged (30,780 out of 58,929 students) compared to 31.3% in charter schools (1,219 out of 3895 students).
- Neighborhood schools were 15.5% ELL (9,150 out of 58,929 students) compared to 9.3% in charter schools (363 out of 3895 students).
- Neighborhood schools were 16.3% SPED (9,162 out of 58,929 students) compared to 13.7% in charter schools (532 out of 3895 students).
However, these summaries are highly influenced by a few outliers and obscure some large discrepancies, especially in terms of the economically disadvantaged and ELL students.
- Of the charter schools, 46.4% (13 out of 28) have economically disadvantaged rates below 20%, compared to just 3.5% of neighborhood schools (15 out of 426).
- Only 10.7% of charter schools (3 out of 28) have ELL percentages above 10%, compared to 36.9% of neighborhood schools (157 out of 426).
Even if we did a comparison of charter schools and neighborhood schools and found that charter schools did better, we still cannot conclude charter schools are causing the performance difference we observe. A comparative study like this is an example of an observational study and because it is impossible to control for all confounding factors, such as parental involvement, we can’t conclude success is caused by the school type. To definitively conclude that charter schools were causing the observed difference in success compared to neighborhood schools we would have to randomly assign some students to go to a charter school and some students to go to neighborhood schools. After some time, we would then compare the results. Obviously, this is impractical as many charter schools do not have busing, require volunteer hours, can remove students for poor attendance, and some do not even have lunch services.
Before the state uses the results of the Harvard study to change the approval process for charter schools we need to understand if charters are better and, if so, why. So far, I have not seen convincing evidence that charter schools outperform neighborhood schools when we control for various student characteristics. I have an idea for further study which I believe should be completed before any changes to policy are made. We can examine the performance of students who got admitted to charter schools via the lottery to those who applied but did not get in and attended their neighborhood schools instead. The group who was admitted is likely similar to those who applied but were not. This would be as close to a randomized experiment as one could hope to have. From this one could determine whether various factors were causing differences in performance, such as class size and teaching methodology. Additionally, we could use results of such a study to determine which factors correlate with success and apply these strategies in all our schools. As more than 90% of the students at our in-person schools are in neighborhood schools, such reforms will be more wide-reaching than simply adding a few more charter schools.
2018-2019 PEAKS ELA, Beth Zirbes, economically disadvantaged, ELL, Guest Column, Harvard Charter School Study, NAEP, SPED
Diane Ravitch's blog 
I’ve followed the charter school sector for decades now. Data-crunching can’t really account for the many ways charter schools are able to handpick their students and/or to impose hurdles that screen for more motivated, compliant students with motivated, compliant, supportive families. All charter schools can do this and if they can do it, they do it.
I’ve had discussions about this with many charter cheerleaders. They always go the same way. Charter cheerleader: We do not cherry-pick our students. Me: (Explains the numerous ways charter schools cherry-pick their students.) Charter cheerleader: What’s wrong with cherry-picking our students? Gaslighting is an overused term, but…
Expert Public School Math Teachers. Gotta love them!
Thank You Teacher Beth Zirbes. Let us continue to ADVANCE the vitality & necessity of Public Schools in our Democracy.
Analyses are still only as good as the assumptions they are built on. If you assume TEST SCORE hubris steers people away from vouchers or choice schools, WHY hasn’t the score hubris worked? If you assume people get vouchers, because of what they don’t know, has the “know-that” strategy reduced the use of vouchers? Rumor has it that ass-u-me is the mother of mistakes. If a strategy doesn’t provide meaningful change, who benefits from it?
Harvard, especially the Graduate School of Education, has been a rats’ nest of privatizers for the last 20 years or more. Everything with a Harvard provenance must be looked at askance. Beth Zirbes got the message.
To be specific about Harvard, the Program on Education Policy and Governance at the John F. Kennedy School of Government is dedicated to school privatization. Its director is Paul Peterson, who is a reputable scholar but a full-throated cheerleader for charters and vouchers. Any seminar or report or book connected to PEPG puts the best possible spin on school choice.
One of my former students did her MA at HGSE. When I asked about her experience, she said she was perpetually enraged by the demeaning and dismissive way public schools and public school teachers were seen, especially the Boston system. In one seminar, she spoke up by explaining that as a child of an immigrant Colombian family of modest means, her public schools had made the trajectory of her life possible. Love her!
Good for her. She was in the wrong program. Public schools are viewed by PEPG as the last resort for dead-end kids.
Maybe not; that credential likely gave her entrée to her job at MA DESE while it was in thrall to Charlie Baker. At least we have someone working on the inside!
Then that’s a good outcome.
while I applaud this person for their assiduous dismantling of the Harvard “study,” I worry that this and other misuses of data undermines the public trust in research in general and the higher education that generates it. No wonder people believe charlatans instead of scientists. There is much misuse of numbers among the people whom we are supposed to trust. This leads to general societal distrust of science, a modern problem.
I wouldn’t worry on that score, Roy. This is a very old scenario! Think of the age of the adage “there are 3 kinds of lies: lies, damned lies, and statistics.” It was in frequent use by the 1890’s, and its origins appear to go back at least to the 1830’s (by a French physiologist dismissing ‘stats’ that supported the use of blood-letting to cure fevers).
My impression is that the general public has long had an instinctive skepticism of statistics used to peddle any product, expenditure, or political position. The present-day popularity of debunking higher education/ academic elites– whether science/ scientists & their theories, or otherwise– is a phenomenon kicking around the US since its early days, and will not be dislodged by debates among researchers.
I definitely didn’t dismantle the study. I just ran a side by side comparison of AK’s charter schools and non charter schools. AK is particularly tricky because we have a large urban / rural divide and all of the charter schools are located in urban areas.
Hi Diane. My post couldn’t go through but wanted to let the Alaska teacher know what a great job her article was and I hope that people in education and politics really listen. Marilyn Cook
We need statistical experts to examine data that omit significant factors like socioeconomics in their conclusions. The charter lobby is known for distorting its “findings.”
The Harvard paper didn’t omit those characteristics. They just didn’t compare charters in a state to non charters in a state. It just compared charters state by state.
”Amazingly” ignored demographic factors? How about “intentionally”. There, fixed it. Oldest charter stats trick in the book, besides the “100%” grad rates.
He didn’t actually ignore those factors. He controlled for lots of them! The Harvard paper just didn’t do a comparison of charters in a state versus non charters in the state, which is what I did.
There was a high school math teacher in the district where I taught for thirty years, who analyzed the district budget covering years to determine if the district was correct when it said it didn’t have the money to pay teachers the increase in pay, we were asking for during a year we were negotiating a new contract.
That math teacher discovered the district hid the money by moving it around to different accounts so that it wouldn’t show at the end of the year as a surplus.
After that came out, with all the number crunching that math teacher did, we the teachers got the contract we wanted. It’s been a while, but I recall the school board also lost some conservative blame teachers for everything board members that year who were replaced by new board members supported by the teacher’s union in that district.
I still remember when I realized that education studies at Harvard and Stanford were based on the politics of the people paying for the studies and not on legitimate scholarship. My question: Do people like Peterson really not know the critical importance of demographics when evaluating test scores, or are they purposely steering their research in the direction desired?
Hi Linda,
Author of the article here… Peterson did control for those factors, many of them in fact. He also had access to student level data (ie. data about a student’s background) whereas I only had it for a school as a whole.
He just didn’t compare charter schools to non charter schools, which is why I did what I did. He simply produced a ranking which put AK at the top, which never happens with any educational ranking and Alaska.
I suspected that our charter schools were quite different demographically than our neighborhood schools and that is what I hoped to quantify and summarize.
Hello Diane!
This is the real Beth Zirbes who wrote the article you posted, and I just thought I should correct one thing.
You said: “What she found, amazingly, was that the Harvard study ignored vital demographic factors.” and that’s not exactly right, the difference is slightly more nuanced than that.
The Harvard study controlled for a whole host of factors that I couldn’t dream to have access to with publicly available data, including the ones I looked at.
However, the study was NOT a comparison of charter schools in one state to non charter schools in the same state. It was a ranking of charter schools only. The comparison to non charter schools simply wasn’t done, but the study was being used as if it had done so.
I’m happy to talk more on the issue!
Beth
Thanks for taking the time to respond to my question. It’s so disappointing to see great universities misusing “research” in this way.
I love math teachers! This is a great example of politicians applying research information inappropriately. It’s great that AK’s charter schools rank at the top nationally, but that doesn’t mean they’re fundamentally better than non charter schools, and we should therefore spend scarce educational funding on more charter schools. As a retired chemistry teacher I see this sort of distortion of research findings all the time and it’s disturbing. Even if the original study was near perfect in design, and without any bias, it still wouldn’t warrant the conclusion that Alaskan charter schools out perform non charter schools.
As Ms. Zirbes states:
”Additionally, we could use results of such a study to determine which factors correlate with success and apply these strategies in all our schools. As more than 90% of the students at our in-person schools are in neighborhood schools, such reforms will be more wide-reaching than simply adding a few more charter schools.”