John Thompson is a teacher and historian in Oklahoma who writes frequently here, at Huffington Post, and on other blogs.
Ironically, the Center for Education Policy Research (CEPR) revisionist studies, “Evaluating Newark’s Education Reforms” by Tom Kane et. al, were released as Bill Gates announced his latest, new approach to school reform. This is important because think tank papers consistently perform two basic functions. They first provide pro-reform spin for the mainstream media. Secondly, they reassure the “Billionaires Boys Club” by presenting the case that their critics are wrong. These studies typically imply that if educators and journalists had bought into the Gates’, Mark Zuckerberg’s, and other venture philanthropists’ theories, their policies (such as closing schools and expanding charters) would have worked.
Zuckerberg and the CEPR weren’t likely to be happy with the first headlines prompted by their new research on test score gains produced by the infamous $200 million Newark experiment. USA Today announced that the research found “a bit of progress.” Veteran journalist Greg Toppo also noted:
The study was funded by the Chan-Zuckerberg Initiative and conducted by a number of Harvard researchers, including Tom Kane, who said that the study’s results were independent of its funding source.
Toppo reported that the Zuckerberg-led grant “made a difference — in a limited way.” He summarized CEPR’s claims, “Newark students improved sharply in English. In math? Not so much.”
https://cepr.harvard.edu/evaluating-newark-school-reform
Toppo recalled Dale Russakoff’s “widely admired book” which found that “the effort produced ‘at least as much rancor as reform.’” He also cited Kane on the disappointing math results which the corporate-funded researcher said may look different when data from the spring 2017 tests become available,” and scores “could” rise in the future.
I stress Kane’s use of the word “could” because he has a long history of using that sort of word when spinning the modest results he documents in research studies that put Gates Foundation experiments in the best light. For instance, Kane’s study of the Gates’ value-added teacher evaluations concluded that teachers’ effectiveness “can” be estimated, although he reported little or no evidence that they would be estimated accurately enough to make those evaluations valid and reliable. After driving for the change in the laws of more that forty states, the Gates Foundation merely concluded, “It is possible to develop reliable measures” such as those that the law required, while not offering a plausible scenario for doing so.
And that leads to the one quarrel I have with Toppo’s wording. What does he mean when he says that test score growth in English improved “sharply?” And what do Kane et. al mean when reporting that those test scores improved “significantly?”
English growth scores only improved “sharply” in one year, 2015. After five years, the $200 million investment’s one success resulted in less than .08 standard deviation Newark’s test score growth in English relative to similar NJ students.
It’s beyond my expertise to explain how such a meager gain, measured by comparing such small numbers of test results from Newark on a new test, could be seen as significant according to the dictionary definition of the term, as opposed to just being statistically significant. But reading the CEPR evidence, it seems that asking questions that are relevant for real world policy decisions is beyond Kane et. al. They acknowledge one major problem with the new scores; schools that began early in teaching to the new PARCC tests would be more likely to have higher scores. On the other hand, their discussion of an even more important point, missing student scores, completely misses the point.
Kane et. al present two charts that reveal patterns that are virtually identical. As with ELA results, before 2014 the Newark math value-added scores dropped in comparison to that of similar New Jersey students. In 2015, math scores soared by nearly .1 std. But during that year, the percentage of students with missing scores increased dramatically, by almost .2 std! The next year, as the percentage of students missing scores dropped just as dramatically, math scores declined so much that all of the five year gains were wiped out.
Rather than print a similar for graph ELA, the authors merely said, “The plot for ELA was similar.”
Why didn’t Kane et. al see the need to address the most logical correlation? When the percentage of missing scores goes up, Newark test score growth goes up. When the percentage of students with missing test scores goes down, test score growth goes down.
And this leads to the implicit recommendations by Kane et. al, as well as the questions they should have asked before making them. They attribute the gains to closing schools and expanding charters. They indicate that such an approach (which, of course, is dear to the hearts of “the Billionaires Boys Club”) could institutionalize better results for students who attend high-poverty neighborhood schools. The few relevant numbers they reported argue against such a theory.
In the first place, Newark charters had previously served higher-performing students, with the non-representative KIPP and North Star Academy being the charters that expanded the most. So the first question is whether those charters could change their model so that the higher-challenge students in neighborhood schools could be retained. Newark’s free and reduced lunch student population averaged 88% over the five-year study; charters were six points lower. However, KIPP and North Star tend to serve a relatively larger percentage of low income and a smaller percentage of poor students than traditional public schools in the inner city. For instance, Newark’s North Star has served far fewer poor children (14.6% fewer free lunch) while serving relatively more students whose higher family incomes qualify for reduced lunch.
Similarly, Newark’s charters served about 60% fewer students with disabilities, but that is just part of the story. For example, North Star has a record of “serving hardly any children with disabilities and few or none with more severe disabilities.”
http://jerseyjazzman.blogspot.com/2014/04/uncommon-comes-to-camden-let.html
These charters also have a long record of benefitting from greater rates of attrition when raising test scores. North Star has a history of suspending students “at an alarming rate,” and that is a reason why “only about half (of 5th graders) ever made it to senior year.” Similarly, as Richard Kahlenberg shows:
The big difference between KIPP and regular public schools…is that whereas struggling students come and go at regular schools, at KIPP, students leave but very few new students enter. Having few new entering students is an enormous advantage not only because low-scoring transfer students are kept out but also because in later grades, KIPP students are surrounded only by successful peers….
http://educationnext.org/student-attrition-explain-kipps-success/
In other words, Kane et. al should have asked questions relevant to policy-making. For instance, how different were the tested charter students’ poverty and disability rates in comparison to their classmates who were enrolled in the first quarter? Why was it that the missing test score patterns seemed to have a far bigger effect on “within-school” outcomes? Why did they assume that Newark’s charters can be scaled up?
Did Gates-affiliated researchers overlook these obvious questions because they still are oblivious to realities within school systems? Or did they only ask the questions that they knew would produce answers that would please their bosses?
Diane Ravitch's blog 
Amongst the flaws in that analysis, it states: “The big difference between KIPP and regular public schools… is that whereas struggling students come and go at regular schools, at KIPP, students leave but very few new students enter. Having few new entering students is an enormous advantage not only because low-scoring transfer students are kept out but also because in later grades, KIPP students are surrounded only by successful peers….”
Cited in supposed support of that argument is this article
http://educationnext.org/student-attrition-explain-kipps-success/
which in reality states:
“Overall, however, the proportion of late entrants at KIPP (15 percent) is nearly identical to the overall proportion in the comparison middle schools (14 percent).”
[…]
“Even if the largest estimates of peer effects are correct, however, the improvement in peers’ prior test scores would appear to benefit KIPP students’ achievement only by about 0.07 to 0.09 standard deviations after four years at KIPP. KIPP’s cumulative impacts in middle school are three times that size, so even the largest estimates of the size of peer effects suggest that they are unlikely to explain more than one-third of the cumulative KIPP impact.
“Moreover, the best available evidence shows that KIPP produces large impacts on students in their first year at a KIPP school—before late-entering students could possibly have any effect. Consequently, the true peer effect resulting from late entrants is likely to be substantially below the back-of-the-envelope estimate of 0.07 to 0.09 standard deviations.”
[…]
“But the data available for this analysis clearly show that KIPP’s impacts cannot be explained by advantages in the prior achievement of KIPP students. Even when attrition and replacement throughout the middle school years are taken into account, the limited range of potential peer effects at KIPP schools does not explain the large cumulative impacts on student achievement identified by prior studies. One implication of these findings is that the KIPP model may include practices that could be effective outside schools of choice.”
LikeLike
EdNext is a Journal of opinion. Like my blog. It is not peer reviewed
LikeLike
EdNext: the same group at Harvard PEPG that invited DeVos.
LikeLike
To Stephen Ronan,
JOHN Thompson asked me to post his reply as WordPress is not letting him comment:
Mr. Ronan,
You and the study you cite dispute Richard Kahlenberg’s balanced use of quantitative and qualitative evidence over time. You do so with a study using reductionistic, even silly definitions, that indicate a lack of curiosity about what happens in real schools. For instance, these studies tend to reflect a misunderstanding about the differences between mobility and attrition. This study is extreme, even by corporate reform standards, because it not only shows the normative misunderstanding of low income vs poverty but it defines attrition as “All transfers out of the district or to private schools were classified as attrition.”
LikeLike
Thanks for passing that on, Diane. FWIW, I seem to have somewhat more success posting to WordPress sites using Internet Explorer rather than Chrome… Could you link to where Mr. Thompson’s analysis originally appeared? I’m a little confused by his rancorous debunking now of a source that he seemingly cited.
The assertion by Richard Kahlenberg that Mr. Thompson and I quoted was immediately followed by citation of two sources. I presumed that Mr. Thompson provided those, but perhaps viewing the original I’ll find different.
Those two sources were:
Click to access kippstudy.pdf
and
http://educationnext.org/student-attrition-explain-kipps-success/
The first of those, dated March 2011, by Miron et al was largely speculative since the researchers did not have access to student level data. It estimated “attrition based on the drop off in the number of students in the grade cohorts… There is also the general assumption that students are not held back at any point.” Recognizing properly (pg. 6) that such methods can lead to inaccurate results, near the end of their paper they posed questions to the Mathematica researchers who, unlike themselves, had access to extensive student level data:
“Below we list and briefly discuss some additional questions that we would hope that KIPP and Mathematica will consider in subsequent years of this ongoing evaluation.
Student attrition and replacement
• How many students leave during the academic year, and what portion of these depart after the autumn headcount?
• Does KIPP admit any students after the autumn head count? If so, what are KIPP’s practices concerning how it fills places?
• How many of the students leaving KIPP are replaced between school years and why does this practice vary so much by region and school?
• What are the characteristics of the small number of students that KIPP invites in after the initial entry grade?”
Such questions were in fact addressed by Mathematica researchers in that second (2014) source that Mr. Thompson seemingly cited and from which I quoted, “Does Student Attrition Explain KIPP’s Success?” by Ira Nichols-Barrer, Brian P. Gill, Philip Gleason and Christina Clark Tuttle of Mathematica. That analysis used “detailed student-level data to compare patterns of entry, attrition, and replacement in 19 KIPP middle schools and in traditional public middle schools in the districts in which the KIPP schools are located.” It concluded that patterns of student attrition and replacement “could produce only a small fraction of KIPP’s actual impact on student achievement.”
Mr. Thompson states that I dispute “Richard Kahlenberg’s balanced use of quantitative and qualitative evidence over time.” The only source I find for the Kahlenberg quote about KIPP that we’ve referenced is this brief 2011 Washington Post piece that relied on the methodology that Miron et al. conceded as seriously deficient.
http://voices.washingtonpost.com/answer-sheet/charter-schools/myths-and-realities-about-kipp.html
If there is another source where Kahlenberg’s demonstrates “balanced use of quantitative and qualitative evidence over time” to make the same point, I’d be curious to know where to find it.
Mr. Thompson derides this: “All transfers out of the district or to private schools were classified as attrition.” Perhaps he can clarify what he finds so faulty about that, and what he recommends as a proper way to define and measure attrition?
LikeLike
JOHN Thompson has been blocked by WordPress.
He sent this response:
You extensively quote the Ed Next article but that doesn’t make its simplistic methodology any better. In answer to your question about what is wrong with “All transfers out of the district or to private schools were classified as attrition,” mobility is due to a bunch of factors. Mostly, it is within districts.
In the inner city, mostly it isn’t due to a choice that parents sought. They face a multitude of circumstances that aren’t under their control. They don’t intentionally choose to transfer out of higher-performing schools into lower performing ones.
Some of KIPP’s attrition is due to those dynamics that we see every day in the inner city. But the methodologies in the studies you trust don’t allow for an estimate of how much of that happens at KIPP. They don’t allow you to estimate how many students left due to circumstances beyond their or KIPP’s control, as opposed to students and parents choosing to leave, or being chosen to leave by the nature of the KIPP system.
Neighborhood school attrition in high-challenge districts is way too high. It’s tragic. It’s not to be defended (as KIPP supporters do their attrition.) But it’s a completely different critter than the attrition at KIPP.
KIPP parents chose those schools. They’d seen them as a way out of schools that face poverty and other the challenges that are far beyond anything that KIPP would ever dream of tackling. Then families often/mostly choose to give up on that supposed ticket out.
Why? They often see that KIPP wasn’t that miracle school they’d envisioned and it was a place where very, very few kids have the personality for flourishing in. (I’d have never persisted at KIPP; would you?) So, they go back to the schools they fled. Sometimes it’s completely voluntary. Others are “exited.” Exiting can be a matter of being pushed out by the overall KIPP model.
To simplify, an intellectually honest approach to KIPP attrition would be to disaggregate between mobility, attrition, and exiting. That would take research that’s far more complex. At minimum, it would be finding the data we analyzed when discovering why Oklahoma City’s highly regarded KIPP has a three-year attrition rate of 80% for low-income kids.
https://nondoc.com/2016/06/18/kipp-okc-finances-data-belie-credibility-of-its-promises/
It would also require you to listen to students. Listen to the kids and you’ll understand why KIPP’s attrition will always keep it from being scaled up.
LikeLike
I would agree with you that it would be helpful if, together with gathering attrition data, Departments of Education were to assemble, as best as possible, explanatory materials that would help us differentiate between the variety of causes of attrition.
And it would be helpful if schools were able to provide some comparisons of academic progress between students who stay and those who depart. As you may know, CREDO cited the Brooke charter schools of Boston as demonstrating the greatest academic progress among their students of any CMO in the country. Not only do they have exceedingly low attrition rates but they have, commendably, tried to look closely at some measures of differences among those who stay and leave. See, for example, pp 32-33 here:
Click to access item3-tabA1-1.pdf
Meanwhile, statements like yours that “Neighborhood school attrition in high-challenge districts” is “a completely different critter than the attrition at KIPP” don’t seem well-substantiated.
Incidentally, you write: “Oklahoma City’s highly regarded KIPP has a three-year attrition rate of 80% for low-income kids.” and then immediately link to your article that instead states: “Four-year attrition rates for low-income and special education students are about 70 percent.”
There’s no source provided for either of the seemingly contradictory assertions.
And you compare the number of fifth and eighth grade special education students being tested in reading and apparently jump to the conclusion that the difference is due to attrition. You seem not cognizant of alternative, feasible, at least partial explanations including front-loaded grade level retention, and special education declassification. In respect to the first of those, note that the EdNext article you originally cited stated: “Part of the explanation for this trend is the comparatively high rate of students who repeat grade 5 at KIPP schools.” And in respect to the second, I would encourage you to read: “Special Education and English Language Learner Students in Boston Charter Schools: Impact and Classification”
Click to access SEII-Discussion-Paper-2016.04-Setren-1.pdf
Its finding included for schools here: “Charter schools remove special needs classifications and move special education students into more inclusive classrooms at a rate over two times higher than traditional public schools.” As Brooke states: “In order to qualify for an IEP, a student needs to both have a disability and be making inadequate progress. We strive to ensure that all our students, regardless of disability, make adequate academic progress, and this practice leads to a lower rate of IEP eligibility than district schools.”
I don’t know the degree that similar factors may be at work in OKC. But fear you may have neglected to consider any such possibilities.
You write: “I’d have never persisted at KIPP; would you?”
KIPP classes seem as if they may be too effervescent for my taste… The lulling drone of a teacher reading aloud from a textbook, as I lapsed into deep slumber, those are fond memories of public district school… Not to mention algebra and geometry teachers who kept me engaged and awake without any annoying chants. But to each his own. There’s a KIPPster nearby who seems to be persisting just fine, thanks.
LikeLike
Stephen,
JOHN can address your comments. He will soon realize that you will nitpick ad infinitude or until the cows come home, whichever is first.
I ask: don’t you ever tire of echoing DeVos’ love of the market? Were you unaffected by the referendum in Mass in which your fellow citizens voted overwhelmingly against expanding charters? Why should there be more when the public voted NO?
LikeLike
“The lulling drone of a teacher reading aloud from a textbook, as I lapsed into deep slumber, those are fond memories of public district school…”
I don’t know when it was that you attended public school, Stephen, but we’ve come a very, very long way from those days.
You might not be of this camp, but statements such as what I quoted above are often used by “education reformers” who want to cast a bad light on public education in order to further their cause. They’re intentionally derogatory and misleading. They create and then play off of people’s fears of an out of touch and antiquated system of education.
LikeLike
Stephen must be 90 years old.
LikeLike
The games reformers play is to embrace high scoring students while pushing out those who are not good test takers. Simply changing the name of a school does not change anything in real time.
While keeping restraints on traditional public schools through Common Core, NCLB and others, they create the artificail perception of failed schools thus allowing them to spin charters as success through trickery and deceit.
Take the shackles of traditional public school teachers and watch them soar.
LikeLiked by 1 person
When a perception carefully and strategically created to be nothing but artifice becomes our nation’s new “reality..”
LikeLike
Bill Gates and all of Silicon Valley’s billionaires’ inability to accept and admit being wrong coupled with their destructive will to ‘disrupt’ civil society is infuriating, the part of which most difficult to accept being the impenetrable, insular echo chamber of Wired tech enthusiasts who cheer on their every move and insist to them that they can do no wrong, thinning their skin and exacerbating the growth of their uncontrolled rampage. Also the peculiar stupidity of them, that’s maddening too.
LikeLike
I worked with techies for quite a long time and they do have an annoying habit of cheering each other on and thinking that everything techie is cool, regardless of any possible negative ramifications.
They also have the annoying habit of thinking they know more than anyone else about pretty much everything.
Other than that, they are generally very nice people.
LikeLike
I refer to Ben Tarnoff from Silicon Valley for the Guardian for the second time today:
https://www.theguardian.com/technology/2016/sep/15/silicon-valley-tech-industry-elitism-mark-zuckerberg
LikeLike
I never saw the sort of “idealism” referred to in the article. But I worked in Boston and maybe it is unique to Silicon Valley
What I saw was basically “coolism”: a desire to pursue all things “cool” (meaning all things techy)
The people I worked with were not idealists and revolutionaries but engineers and scientists who liked working on the latest technology just because it was cool.
I think the point about insularity is very apt. By and large, techies are insolated (quite purposely) by the companies they work for. They enjoy work privileges and salaries that the rest can only dream about.
LikeLike
…which makes them feel like they are “special” which I think goes a long way toward explaining the arrogance of many of these people.
LikeLike
I distinguish between techies and tech billionaires. Techies can learn and change — even write poems. Billionaires of the tech industry and their closest investors cannot learn. They cannot change. Chan-Zuckerberg, Bill and Melinda, Bezos, Powell-Jobs… they are lost in hubris.
LikeLike
The billionaires are like Greek gods — Geek gods in the case of tech billionaires.
LikeLike
“Geek Gods”
The gods of Geeks
Were Titans of tech
Like Billy Gates
And Timmy Cook
The Hubris beast
Was bound to wreck
The Pearly Gates
By hook or crook
LikeLike
“Idealism” that happens to coincide perfectly with financial interests of its billionaire proponents transacts at a very high discount.
LikeLike
You can really learn a lot about Gates, approach to education by studying his approach to Windows at his own company.
Microsoft’s business model is best described as “loose something on the public which kindof works and then debug it on the fly as the bug reports roll in”.
The pre-development specification of what the software should do was minimal, as was the pre-release testing ,( and the latter was almost certainly related to the former)
Gates tried the same thing on schools with disastrous results.
With Windows, Gates had the attitude that he did not need to get it right the first time because there would be lots of opportunities to fix it and once people had it as an operating system they would be invested so would be unlikely to switch.
Common Core is the poster child for something that was hacked together with a poor specification and quickly developed and loosed on the schools with almost no piloting (testing on small scale).
Of course, the major difference with Common Core from Gates usual pattern was that there was no provision for fixing “bugs” and in fact, it was intended that it would NOT be changed. And it was even copyrighted so states could not fix things themselves.
So, knowing Gates historical approach to software, his approach to education was not at all surprising. Not was the result.
LikeLike
Exactly. When Steve Ballmer unleashed stacked ranking on Microsoft, at least he was able to stop it before it completely destroyed the company. The same cannot be said about Common Core.
LikeLike
Oops, wrong spot.
LikeLike
I think there is also an analogy between Windows and Common Core in that Common Core is a kind of operating system for schools.
Like an operating system it acts as a platform on which programs (,eg, for curriculum) can be run.
Also as with an OS, once you have Common Core and start getting stuff to work with it (textbooks and software), you become invested in it and it becomes increasingly difficult to jettison.
I suspect that several states (,NY. For example) have chosen to keep much of the Common Core and simply rename it so they can change some things rather than junk Common Core entirely. They are already hugely invested.
Gates knew all this ahead of time and it is hard to avoid thinking that he was actually depending on such “investment” by states to keep them from leaving the Common Core fold.
LikeLike
And there’s also that federal mandate thing.
LikeLike
The federal mandate was meant to rope the states in to adopt Common Core.
Gates knew that once they adopted it and started developing/buying curriculum and training teachers for it, they would be invested and be unlikely to junk it.
It was actually an extremely clever plan, which minimized the out of pocket expenses of Gates Foundation. All they had to do is put up the money to develop the standards.
The overwhelming majority of the cost for Common Core was paid for by the Federal government and the school districts.
I consider what was done a fraud on the American people and believe that Gates and Duncan should be made to pay every last cent that it has cost schools.
LikeLike
“the federal mandate for adopting standards and testing that went along with getting Race to the Top funding”
LikeLike
After almost two solid decades, they’re still gaming the system, with big money controlling the statistics that the public sees in terms success (lots for the charters) and failures (lots for the public schools).
I used to get all up in arms, not understanding how this charade could so relentlessly move forward, but last night was a perfect example of why:
I attended a dinner party with some friends who we hadn’t seen in a while. Good friends, separated by distance and very busy lives. One of my colleagues from school was there, as well. Both of us are active in the union. We were discussing Common Core, VAM, charter schools, vouchers, functional vs grade level, etc.
My friends are very intelligent and kind human beings. Everything that we told them was completely new to them. It wasn’t so much that they had swallowed all the spin and were solid behind it as that the information that they’d absorbed was on the periphery of their lives, in terms of importance. All of our kids are either in college or already graduated. So what news they’ve seen, read, and heard is always positive in terms of education reform, EXCEPT for Betsy DeVos, which was a big reason why they were finally questioning what was going on.
Those of us who are actually in the trenches know what’s going on. And the media has vilified and silenced our voices. What’s completely transparent and of such importance to those of us in the profession of education is of passing interest to so many people.
LikeLike
“Tribalism”
“Obama good and Betsy bad”
Mantra Democratic
Situation’s really sad
Problem emblematic
LikeLike