Gary Rubinstein has a deep aversion to hypocrisy, hypes, and propaganda.
He read a widely publicized report saying “research shows” that graduates of KIPP have higher college completion rates than their peers.
But then he discovered that the research shows no significant difference between KIPP students and their peers in college completion rates.
His post debunks Richard Whitmire’s erroneous claim that KIPP students finish college at a rate three to five times greater than students who went to public schools. It is also a valuable lesson in reading and interpreting research findings or claims that “research shows.”
He begins:
The way reformers misuse data follows a very simple and predictable plan: First they get some skewed data, then pick a ‘researcher’ to interpret the skewed data. The ‘researcher’ then writes a report which gets touted in The74, EduPost, and eventually even makes it into more mainstream publications like USA Today and The Wall Street Journal. Since the report is filled with nonsense and half-truths, within a few weeks the truth comes out and the report is discredited, but not before the damage was done and the spin has made it into folklore. When this happens, the reformers will then ‘move the goalposts’ and get some more skewed data and start the process over again.
An example of this is the July 2017 report by Richard Whitmire called ‘The Alumni‘. Whitmire has written books about both KIPP and about Michelle Rhee so I think you get the idea of what his point of view is. In this poorly researched project he concludes that “Data Show Charter School Students Graduating From College at Three to Five Times National Average“.
This was probably the easiest report I ever debunked. The biggest flaw was that for most of the charter schools, they were only counting the percent of graduating seniors who persisted in college and then comparing that percent to the overall percent of all low-income students — an apples to oranges comparison. Whitmire acknowledges this in another post about the methodology in which he says that only KIPP counts students who leave the school before they graduate and that their numbers are much lower, but still at 38% which is at least triple the expected graduation rate for low income students.
A second flaw, and this one is very difficult to compensate for, is that charter school students are not a random sampling of all students since many families choose no to apply to them. So you get a biased sampling even if you do count all the students who get into the charter school and not just the ones who make it to graduate from the charter school. And even though I and others have discredited his report, it is something that still gets quoted in the main stream media.
Just recently, however, I learned of a report generated by Mathematica and funded by the John Arnold Foundation. I think that Mathematica is a very reputable company and even though reformers often hire them to produce reports, sometimes those reports reach conclusions that reformers were not expecting.
In this case, the report called “Long-Term Impacts of KIPP Middle Schools on College Enrollment and Early College Persistence” , reached a result that completely contradicts Whitmire’s claim that “Charter School Students Graduating From College at Three to Five Times National Average”.
Read on to see just how overblown is the KIPP myth about the college success of their students
Here’s the relevant summary of what they found:
Diane Ravitch's blog 
“The Alumni in Partnership with The 74” is the outlet who blasted out the claim as gospel.
The 74. Who we’re told over and over is a real news outlet and not a mouthpiece to promote the political and ideological goals of ed reformers.
I encourage all public school supporters to read the ed reform echo chamber. It’s enlightening. You’ll understand a lot more about what’s happening in and to your public schools when you read where it all comes from.
Whatever I read there is then pushed in the Ohio legislature and lands in (or on) public schools. Our lawmakers have apparently completely outsourced their jobs to 150 national charter and voucher cheerleaders. They can’t rubber stamp these schemes fast enough and they never, ever say “no”.
It’s always interesting to me that so many ed reformers work for or are affiliated with colleges and universities yet the entire “movement’ blames all college performance on the schools that come prior- K-12 schools. “College completion” is now wholly the job of K-12 schools? Wow. That’s nice for colleges.
They hold themselves harmless as far as students. Convenient, that.
What if colleges cleaned up their own side of the street instead of blaming K-12 schools for everything? Then some prestigious and powerful people and institutions would be implicated, and we can’t have that. It’s much easier to beat up on a middle school principal than the president of a university.
Chiara. You are right about blame shifting, but there are federal efforts now to hold postsecondary programs responsible for the economic outcomes from graduates of specific programs. S.800 co-sponsored by Elizabeth Warren is one of several multi-year attempts to link the performance of individual students to their academic majors or certificated programs (community college, vocational programs) to income, including IRS data, The bill is marketed as if addresses the student debt problem. It does not. This federal law will certainly discourage students from enrolling in programs of study known to have low return on investment, especially if measured at graduation and at at five year internals. Among these programs are studies in the arts, humanities, and liberal arts. Teaching is also relatively low in ROI.
This bill (S. 800) and similar efforts to put an economic value on postsecondary education are soon to be matched by reports on the per-pupil expenditures in every K-12 school that receives federal funds. ESSA requires these calculations and they are sure to lead to renewed claims that K-12 education needs to increase its productivity.
The proposed methods for getting more bang for the buck are familiar. Assign one teacher more students and add paraprofessionals to assist Increasing the class size without such aides. Cut the number of AP courses. Offer more instruction online (and so on).
The federal watchdog for this facet of ESSA is Dr. Marguerite Roza of the Edunomics Lab, Georgetown University. She and Dr. Paul Hill of the Center for Reinventing Public Education are experts on studying how to cut budgets for public education, as is McKinsey & Company. McKinsey & Company conjured the Obama/Duncan era RESPECT program for “Recognizing Educational Success, Professional Excellence And Collaborative Teaching” in 2013. RESPECT was nothing more than a pitch from McKinsey & Co. for teachers to give up all quests for job security and embrace longer hours, fewer days off, tiers of merit pay for raising test scores (minimum “a year’s worth of growth” every year) and so on.
What a great post, Laura! I am always really impressed by the depth of your knowledge and research and by your ability to put the facts forward in so succinct and clear a manner. Fascinating. And frightening!
There’s a deep and tragic irony in the fact that Deformers/Disrupters are all about the data but base all their policies on bad data and false extrapolations from data. There’s a parallel, here (one of many) to the Eugenics Movement in the early 20th-century United States, which was all about instituting social policy based on supposed data from genetics research, mostly that being conducted by the Eugenics Records Office, located in Cold Spring Harbor, New York. Based on this pseudoscientific research data from this organization, which was funded by US oligarchs, the US passed incredibly restrictive, white supremacist immigration quotas, and many states instituted mandatory sterilization policies to “protect the gene pool.” Hitler, not yet Chancellor of Germany, held up the US policies as a model for the world. It took the horror of the war against the Nazis to take the steam out of US Eugenics.
The first big lie in Deform was that US schools were failing. This was based on apples-to-oranges comparison of US students’ results on international standardized tests to results for students in other countries. The truth was that if you corrected the data for the socioeconomic level of the students taking the tests, US students were consistently near the top. The US had many students living in dire poverty, and this uncontrolled-for factor is, of course, strongly determinative. It’s an elementary failure in research design on the part of those touting data and research.
The second big lie in Deform was that if we simply held teachers and schools and kids accountable by attaching high stakes to standardized tests, achievement gaps would narrow (and eventually disappear), and the US would move to the top of the world rankings. Well, we’ve been following this policy for decades now, and neither has happened. In other words, data-based deform has failed, utterly failed, by its own preferred measures, test scores.
The third big lie in Deform is that the tests used to generate the “data” on which Deform is based are valid. They are not, especially in Reading/Language Arts. These ELA tests supposedly measure whether students have met the “standards” on the puerile, backward, vague, abstract, mostly content-free Gates/Coleman bullet list, the CC$$. But a moment’s reflection would make it clear that the tests don’t do that. They contain one or two multiple-choice questions about each very, very broad “standard.” One simply can’t ask a single MC question and determine, on that basis, whether a student is able to, say, make inferences from texts GENERALLY. It’s as though I asked you where the restrooms are located on airplanes in order to judge your suitability for piloting a 747. But somehow, all this invalidity of individual questions testing individual “standards” is supposed to add up to overall validity. It doesn’t.
I won’t go into the rest of the issues with Deformer/Disrupter “data.” Those are the big ones. Basically, data-driven deform is, like eugenics before it, numerology resulting in extremely dangerous social policy. In the case of deform, the resultant social policy has turned our schools and our curricula into trivialized test prep, robbed an entire generation of students of humane education, enriched profiteers, and created stress levels in our schools so high that teachers are quitting in droves and teen suicide and depression levels are at historic highs.
If they were smart they’d marginalize the cheerleaders.
They overpromise. Most of the blowback against charters in Ohio is based on that fact that they WILDLY oversold these schools to the public. They were so ideologically driven to privatize public schools they made promises they can’t keep. The first thing to go was the claim that they could do it cheaper. Now all they do is lobby for more funding.
And, ofc, Ohio has the great misfortune of being home to the Fordham Institute for the Payment of Big Bucks to the Officers of the Fordham Institute, dedicated to the diversion of tax dollars into private profits.
The Deformers are always looking for ways to monetize bad ideas. They are creating SEL software that they claim improves “executive functioning.” Notions such as these are simplistic, naive applications to complex problems. The bogus “information” gets sold to third party vendors.
yes: typically the reformers establish their invasions/programs on grants and short-lived funding, take their cut and then pass the burden on to the public
The goal of the “reformers” is not to create independent schools, but to establish publicly funded independent schools that are unaccountable and non-transparent.
If Bill Gates wanted independent schools that acted as labs for innovation, he would have funded them.
He wants government to pay, not him.
Great post, Bob, & a terrific companion to Rubinstein’s succinct & informative piece taking down deform stats. I’m salting both away in my compendium of background info for [in Mad Magazine parlance], Snappy Answers to Stupid Questions — i.e., my responses to ignorant comments at MSM ed articles.
Thank you, bethree!
Gary Rubenstein, I thank you profusely for your work.
Me, too!!! Thank you, Gary!!!!
Recent data from the CDE shows that KIPP Bridge in Oakland (used to be a 5-8, now it’s a K-8) lost more than half their 6th grade class from attrition between 6th and 8th grade. I don’t have local data on high schools because Oakland doesn’t have a KIPP high school. But if KIPPs attrition rates are this bad elsewhere, it absolutely plays a role in that skewed, misleading college data. As usual, KIPP and charters refuse to reveal the role that attrition and self-selection plays in their secret sauce of college and career “success”.
Robert Pondiscio wrote an entire book about Success Academy in which he made it clear that he himself and the ed reformers DO NOT CARE about attrition because deep down they believe that the kids who drop out are worthless and they deserve every abuse the charter hands out. According to Pondiscio, if a white charter CEO who just happens to be the favorite of the same billionaires who fund him pronounces a child to be unworthy, then Pondiscio absolutely knows that child IS unworthy and that child becomes absolutely invisible to him. Except if that child reappears in a public school that Pondisicio and his buddies in the ed reform industry like to bash for “low graduation rates”.
Check out not just attrition but “retention” rates and how many students are held back 1, 2 or even 3 years with the message sent that the parent better pull the student because he is never going to move up to the next grade in the charter (which is of course the child’s fault and not his specially trained charter teacher’s fault).
And the ed reform industry has convinced reporters like Eliza Shapiro at the NY Times to play along and treat all the parents whose kids have left as just as worthless or invisible.
I suspect that the fact that the majority of those dumped children have parents who are low-income African-American and Latinx is why reporters like Eliza Shapiro also embrace the charter belief that attrition is irrelevant.
I also suspect that if a significant number of college educated white parents were filing complaints about how often their struggling children were made to feel humiliated and punished and suspended that Eliza Shapiro would actually think that was worth looking into and want to know more about real attrition rates. I suspect if a group of other white parents said “oh no, the charter is nice to my kids,” that Eliza Shapiro would not tell the many white parents whose kids were being mistreated that their opinions are now worthless because some other more privileged white parents said the charter was nice to their kids.
(FYI, this happened in the Success Academy Hudson Yards Middle School, where affluent college educated parents complained that their kids were being mistreated and they were taken very seriously instead of their kids private records being released, while the many years of low-income parents doing the same were ignored by white reporters who clearly believed their 5 year old deserved it)
BASIS Charter Schools in Arizona have an exceptionally high attrition rate but the students leaving were mostly affluent white kids and the students there were mostly white and Asian. Therefore, education reporters were unwilling to push the narrative that the students at BASIS would be abject failures in public schools and their entire academic success was entirely due to BASIS. Education reporters were unwilling to push the narrative that the students who left were either dangerous or disturbed or had parents who despise good schools and wanted their child to have a substandard education and be a failure. Since the students who left were mostly white, that false narrative could not be pushed. But racism allows them to push that narrative when it comes to urban charters serving low income students who are mostly African-American and Latinx. And that is why data like attrition rates and retention rates at charters are protected as if they are top secret.
The students who enroll in BASIS know that they will be expected to pass multiple AP tests. That is a screening device, yet BASIS has high attrition rates anyway.
Yes, that is a screening device, much like Eva Moskowitz telling a group of parents “If you know you cannot commit to all that we ask of you this year, this is not the place for you” is also a screening device.
One would think that having that kind of screening device, which already gives a charter an enormous advantage over public schools that take all students, would be enough. However, it seems that the goal of some charter CEOs is not teaching kids, but establishing a charter that will undermine public schools, so just starting with that huge advantage is not enough.
BASIS has a high attrition rate despite parents understanding in advance what is required, because BASIS is not interested in teaching students who are not high achieving. Success Academy had high attrition rates despite Eva Moskowitz warning parents not to enroll unless they committed to do everything the school required. That is because her charters are not interested in teaching students who struggle — the video of the “model” teacher punishing a student for a wrong answer demonstrated that far more clearly than anything else. Naive parents believed that if they did what Success Academy required, the charter would teach their child. They did not realize that the most important requirement was not “doing all that the charter asked” but “having a child who excels in school when parents do what is asked”.
I always marvel that education reporters have so little understanding of statistics that they marvel at the number 17,000 students in a charter network and accept without question that their academic success could never be cherry picking in a city of over one million students where 47% of those 1 million students are proficient on state tests. I suspect that is why Moskowitz has shown no desire to expand into a smaller NY urban area like Albany or Buffalo, which teach a tiny fraction of the number of students in NYC and proficiency rates are significantly smaller. Surely even Eliza Shapiro would not accept without question that a charter teaching 13 students per grade — or 2% of the students in the Albany school district, could never, ever cherry pick students. Or perhaps if it was Eva Moskowitz who told her she wasn’t cherry picking those 13 students, Shapiro would report it without question, no matter how many other parents were complaining about their children’s treatment.
I know from thirty years of experience that when parents are involved, the child learns and ends up with a great GPA.
But real public schools are not allowed to use strong-arm tactics and threats like that as Malevolent Eva, the Wicked Witch of Deform Land, does.
There is one data point that the ed reformers depend the co-opted media — especially education reporters who are terrified of numbers and statistics — to cover up.
To wit — colleges and universities are obligated to publish their 4 year graduation rates and unlike charters, they have to look at every entering freshman and look at whether that very same freshman graduated 4 or 5 years later. Colleges can’t say “well these students left so we don’t have to count them” and they can’t say “we want to count all these students that came in sophomore and junior year and that way it won’t seem like we lost so many of our freshmen students.” It is much harder to cook the books and colleges and universities don’t have willing think tanks who say “we’ll do a study that hides how many students who started did not graduate because our funders want you to look really successful so the states and federal government take money away from public universities and give them to your for-profit private ones instead (and so-called not-for profit private ones, too).
I never hear private colleges like Washington and Lee planting newspaper articles and running non-stop advertising about how they are far superior to University of Virginia because their graduation rates are better. Washington and Lee does not have PR firms demanding that Virginia give at least 25% of the money designated to University of Virginia and Virginia Tech to Washington and Lee because W & L has proven via a “totally accurate” study that they are far superior to those worthless state universities wasting money with lesser graduation rates.
The charter studies are never published in peer reviewed journals. And if KIPP wants a better think tank more than willing to make them look good, may I suggest MDRC or Rebecca Unterman who came up with one of the most laughable studies desperately trying to prove that Success Academy was a charter that worked miracles for all students who walked in the door. Why, you can read the entire MDRC study without ever learning exactly how many students started in Kindergarten and how many remained whose results are in the study! Imagine a study with a control group that could have 100 students or 1000 students but the researchers felt that information was top secret and should not be included so no inconvenient questions are asked.
I suspect MDRC would do a job that was far more pleasing to KIPP than Mathematica. Maybe KIPP can hire them next.
Proof that Donald Trump is not alone in practicing the old Nazi tactic of repeating a lie until a lot of people start to believe it is the truth.
Note that KIPP utilizes a number of ways to impose hurdles in the admission process that keep out less able students from unsupportive families. Some years ago, when a happy KIPP parent in my community announced that his daughter had “tested into” KIPP San Francisco Bay Academy, I applied to that KIPP school for my own then-7th-grader to see if they would give her a test, and they did indeed contact us to schedule the test. (We didn’t follow through, for the record.) KIPP also uses pre-enrollment “counseling” that of course screens out less-able students and less-high-functioning families — and basically whatever other methods individual KIPP schools choose.
Also, Mathematica is NOT so pure — in fact, “tainted” wouldn’t be overstating it. Some years ago KIPP hired it to study its attrition, and Mathematica produced a report that conflated “attrition” with “mobility” and was used to deliberately obfuscate KIPP schools’ staggering attrition. That’s not righteous and destroyed Mathematica’s image of respectability and honesty.
“Some years ago KIPP hired it to study its attrition, and Mathematica produced a report that conflated “attrition” with “mobility” and was used to deliberately obfuscate KIPP schools’ staggering attrition.”
There is something very wrong with the people at Mathematica who would deliberately obfuscate attrition numbers to please a paid client instead of having the desire to present those numbers in the most accurate way. Children’s lives are at stake here. It is no different than a science researcher deliberately obfuscating the terrible side effects of a new drug to please a pharmaceutical company and the people who do that clearly have no concern whatsoever about who they harm when the price is right.
I don’t know how it is that education reporters have no interest in attrition rates and are delighted to embrace and report whatever numbers they are given without bothering to check them. It is similar to how education reporters always repeat the nonsense that there are millions (or even “billions” if a charter CEO says so) of students on charter wait lists without ever once asking for a bit of proof.
And they believe that they can hide their lazy reporting by saying “we only wrote that the charter CEO said there were tens of thousands of students on wait lists so why blame us” as if they believe that the role of a reporter is to simply report “he said she said” and the truth no longer matters.
Mathematica trashed its ethics (and this should trash its reputation with anyone who thinks it’s pure) with that slippery bit of deceit.
Savvy, experienced, ethical education reporters pretty much don’t fall for it, but there’s a lot of turnover on that non-prestigious beat. And it’s kind of a logic issue — the difference between attrition and mobility, and even grasping that you have to factor in the attrition before touting the supposedly fabulous college acceptance rate — and sorry to say this about my own colleagues, but journalists tend to be math-challenged and often don’t see that logic.
As I’ve said, even honest, sincere wait lists turn out to be hollow, as I learned while I was on the board of a parent-run co-op preschool. So the school can’t actually know if it has a solid wait list, plus of course the entire charter sector is built on brazen lies. This often simply doesn’t occur to reporters.
What is wrong with the Mathematica report cited in this blog entry?
It certainly is not a peer reviewed paper, only a “report” by a private company hired by another one, hence it cannot be judged as such though KIPP may pretend otherwise.
Reports by self-interested researchers paid by the entities they’re “studying” are widely given credence as though they were peer-reviewed research.
Actually, I really would like to know if anybody found flaws in the Mathematica report quoted in the blog?
Gary Rubinstein did.
I read Gary’s post and I cannot see where he states that he found a flaw in Mathematica’s methods in the report. (It’s another matter that the Mathematica report directly contradicts KIPP’s earlier claims on their great effectiveness that is supposed to be 4-5 times greater than the rest of the schools’.)
There is a book called How to lie with statistics. I think that’s the KIPP researchers’ textbook to interpret data.