Mathematica Policy Research has good news for KIPP. Their students make significant gains. The press release follows with a link to the report and summary of the findings.
A few questions occur to me about the replicability of the KIPP model.
First, KIPP has raised hundreds of millions of dollars from philanthropists and the U.S. Department of Education. Does that extra money translate into smaller classes and other perks? If not, what is it used for?
Second, to what extent do KIPP students benefit from peer effects, in that the comparison group is attending schools with kids with more problems and issues than those in KIPP?
Third, will KIPP ever take on the challenge of an entire small impoverished (I call it “the KIPP Challenge”)? So long as they take some but not all, the suspicion of selective attrition and exclusion will linger.
Here is the press release.
New Report Finds KIPP Middle Schools Produce Significant Achievement Gains
Contact: Jennifer de Vallance, (202) 484-4692
WASHINGTON, DC—February 27, 2013—A report released today by Mathematica Policy Research shows that Knowledge Is Power Program (KIPP) middle schools have significant and substantial positive impacts on student achievement in four core academic subjects: reading, math, science, and social studies. One of the report’s analyses confirms the positive impacts using a rigorous randomized experimental analysis that relies on the schools’ admissions lotteries to identify comparison students, thereby accounting for students’ prior achievement, as well as factors like student and parent motivation. Fact sheet.
Key findings on KIPP’s achievement gains include:
KIPP middle schools have positive and statistically significant impacts on student achievement across all years and all subject areas examined. In each of their four years of middle school, KIPP schools produced positive academic impacts on state standardized tests. Significant positive impacts are evident on average as well as for the majority of individual KIPP middle schools in the study.
The magnitude of KIPP’s achievement impacts is substantial. In each of the four subjects studied, KIPP schools produced achievement gains large enough to have a substantial impact on student outcomes:
Math: Three years after enrollment, the estimated impact of KIPP on math achievement is equivalent to moving a student from the 44th to the 58th percentile of the school district’s distribution. This represents 11 months of additional learning growth over and above what the student would have learned in three years without KIPP.
Reading: Three years after enrollment, the estimated impact in reading is equivalent to moving a student from the 46th to the 55th percentile, representing 8 months of additional learning growth over and above what the student would have learned in three years without KIPP.
Science: Three to four years after enrollment, the estimated impact in science is equivalent to moving a student from the 36th to the 49th percentile, representing 14 months of additional learning growth over and above what the student would have learned in that time without KIPP
Social Studies: Three to four years after enrollment, the estimated impact in social studies is equivalent to moving a student from the 39th to the 49th percentile, representing 11 months of additional learning growth over and above what the student would have learned in that time without KIPP.
The matched comparison design produces estimates of KIPP’s achievement impacts similar to estimates of the same impacts based on an experimental, lottery-based design. Researchers found that KIPP’s achievement gains are similar for the matched comparison design and the experimental lottery analysis.
KIPP’s gains are not the result of “teaching to the test.” For KIPP students in the lottery sample, researchers administered the TerraNova test—a nationally norm-referenced test—which students had not prepared for, and which carried no consequences for students or schools. The impacts shown in the TerraNova test were consistent with those shown in state tests.
Mathematica senior fellow and study director Philip Gleason said, “KIPP is making important strides to close achievement gaps for disadvantaged students. Findings from this large and comprehensive evaluation show that KIPP schools lead to educationally meaningful increases in student achievement, not just in basic reading and math, but in a broader set of subjects, including science and social studies.”
In addition to studying academic impacts, researchers also administered surveys to students and parents in the lottery group, to assess how KIPP affects behavior and attitudes toward school. The surveys showed that KIPP students complete up to 53 minutes more homework per night than they would have at non-KIPP schools, and that winning a KIPP lottery had a positive effect on both parents’ and students’ satisfaction with school. However, they also found that KIPP students reported no discernible increase in attitudes associated with success, and had an increased incidence of self-reported undesirable behaviors, including losing their temper, arguing with or lying to their parents, or giving their teachers a hard time.
The new report—the latest from Mathematica’s multi-year study of KIPP middle schools—is the most rigorous large-scale evaluation of KIPP charter schools to date. The report confirms and adds to the findings of the first Mathematica report on KIPP schools, released in 2010. The newly released 2013 report covers twice as many schools: 43 KIPP middle schools in 13 states and in the District of Columbia. In addition, the new report includes a broader range of student outcomes, examining not only state test results in reading and math, but also test scores in science and social studies; results on a nationally normed assessment that includes measures of higher-order thinking; and behaviors reported by students and parents.
The report also describes the population of students entering KIPP schools. Researchers found that students entering KIPP schools are similar to other students in their neighborhoods: overwhelmingly low achieving, low income, and nonwhite. Ninety-six percent are either black or Hispanic, and 83 percent are eligible for free or reduced-price school meals. Before enrolling in KIPP, typical students had lower achievement levels than both the average in the elementary school they attended and the average in the district as a whole. On the other hand, KIPP students are somewhat less likely than others in their elementary schools to have received special education services or to have limited English proficiency.
About Mathematica: Mathematica Policy Research, a nonpartisan research firm, provides a full range of research and data collection services, including program evaluation and policy research, survey design and data collection, research assessment and interpretation, and program performance/data management, to improve public well-being. Its clients include federal and state governments, foundations, and private-sector and international organizations. The employee-owned company, with offices in Princeton, N.J.; Ann Arbor, Mich.; Cambridge, Mass.; Chicago, Ill.; Oakland, Calif.; and Washington, D.C., has conducted some of the most important studies of education, disability, health care, family support, employment, nutrition, and early childhood policies and programs.
Norm referenced tests do npt really show anything in absolute terms and should not be used for the type of assessment KIPP is using them for.
@ the Chalk Face has had a couple interesting posts. The one up now is spoof that KIPP cures cancer. But the one a couple days ago was really spot on. It doesn’t matter what the Mathmatica study shows. Even if KIPP increases test scores or, hell, cures cancer, it doesn’t matter because the power and control means do not justify the ends.
Diane raises some very good points. If a school is “self-selecting” their student body from the outset—because ALL “lotteries” for charter admissions are only entered by those students from families most focused on educational achievement—and if that self-selected student body is then “counseled out” or just thrown out, thus trimming KIPP’s ranks of any students prone to score lower on standardized tests, shouldn’t all of these “superior gains” on the part of KIPP be taken with a large grain of salt?
In addition to that, would KIPP have even come close to this without the massive infusion of funds it has received from the private sector?
And, is there any reason this KIPP model couldn’t, with some minor modifications, be set up within the existing public schools system virtually anywhere? Why do private companies—either paying their “executives” absolutely obscene “salaries” and/or squeezing as much “profit” as possible for investors out of their public tax dollars—have to get involved with any of this?
Is their REAL motive just to divert as many public education dollars into their bank accounts as possible? They’re not doing anything—educationally speaking—that isn’t now done, or couldn’t be done in the future, by existing public schools.
KIPP, like every other “charter” is all about adults using “education” as a way of making gobs of money at public expense.
It was brought up before. We keep using that word, “achievement,” as though it’s synonymous with learning in a KIPP environment. What about authentic learning?
When the press release switches from the high-performing third year students over to the under-achieving lottery applicants, it skips right over the possibility that the lowest achievers have been pushed out, inflating the scores of those compliant children left in the program.
Cherry-pick your kids on the front end (with lotteries and interviews) and on the back end (by “counseling out,” i.e. expelling) those who don’t “measure up,” and your test scores will improve.
So what?
Yep. That is standard private school procedure.
I remember when I taught private school, and my students in my second year showed they were in the 78th percentile on the Stanford Achievement Tests. Did that mean I was a great teacher? No. It was the makeup of the students I had (including a couple of them who were in the 99th percentile for second grade).
Anyone have a link to the Mathmatica study?
“In each of their four years of middle school, KIPP schools produced positive academic impacts on state standardized tests.”
This might be too simple of a criticism but what if you don’t care about scores on “state standardized tests?” What if, as the National Academies of Science found, that after 10 years of standardization our children are less prepared for college and careers?
What if we asked these questions:
As a result of enrollment in KIPP:
1) Are students prepared to learn?
2) Are the students capable of critical thought?
3) What levels of compassion for other humans are exhibited?
4) Are students prepared for active and participatory citizenship?
5) What happened to the students with “EXCUSES?”
Any others?
Timothy — if one learns how to read and do division more competently, is that somehow inconsistent with having compassion and citizenship? Not quite following the chain of thought there.
Someone who “learns” is quite different from someone who “scores.” And the “chain of thought” is actually pretty easy. If we only prepare kids to do well on tests and have no indicators of compassion or citizenship why bother? The world has plenty of readers and dividers. We need more compassionate citizens.
Suffice it to say that kids who get into KIPP are learning quite a bit more about math and reading than the same sort of kids who don’t get into KIPP, and there is zero reason to think that the KIPP kids are less compassionate or civic-minded than public school kids, so it’s not clear that changing the subject here is of any use.
Anyway, if all we really care about is having kids learn compassion and citizenship, we shouldn’t bother with schools. Kids should instead spend their time at churches or other religious institutions, volunteer with food banks and other social service organizations, etc.
If, on the other hand, we want to have schools, it’s probably because we think kids need to have some basic academic knowledge and skills. So those skills are the things we’re going to prioritize in the context of school.
If we started evaluating a fire department on how many fires it put out successfully, and if the fire department’s response was to say, “But what about the fact that our firemen spent hundreds of hours last month volunteering at the soup kitchen? What about all the show-and-tell sessions we do at schools?,” we’d say that fine those are good things, but maybe the fire department isn’t doing a very good job at accomplishing its primary mission if it keeps trying to change the subject. Plus, if all we wanted was an organization to do school shows and volunteer at soup kitchens, we wouldn’t need to train firemen and have a fire department in the first place.
No. The function of PUBLIC schools is not purely academic. Acting as if modeling social behaviors and attitudes are best reserved for religious institutions and not schools is not even being remotely realistic. And the function of schools is to model volunteerism in the soup kitchens and having a “show and tell” after a student volunteers in the soup kitchen is exactly why we value comprehensive (not just achievement driven academic) public schools.
We can’t immediately dismiss KIPP nor this recent study. It’s like you said in your book, The Death and Life….”When we are too certain of our opinions, we run the risk of ignoring any evidence that conflicts with our views.” There are some good charters out there that do good work in communities of need. Let’s remember that and never forget that we all share the same goal – to educate our children.
I agree that we all share the same goal, but it is dangerous to the survival of public education when charters skim the “best” students and leave the hardest to educate children to the public schools, then trumpet that they are better. Why not collaborate? Why not admit that your organization has raised over $400 million to add to the public money it is already collecting, making it possible to do what public schools cannot do?
When I spoke to KIPP in Houston in 2010, I urged humility and also asked that they take the lead in demanding higher standards of accountability and transparency for charters.
I disagree that we “all” share the same the same goal.
I’m willing to grant that KIPP teachers might – despite the institutional character of charter schools that makes them a largely destructive force – but Eli Broad, Bill Gates, Rupert Murdoch, Penny Pritzker, the Walton family, et. al. want something very different.
I’d suggest you watch what those people do, not what they say. They couldn’t care less about the welfare of the majority of public school students, except insofar as it benefits their economic, political and social engineering interests.
I agree we can’t dismiss KIPP so easily. But I would like more honesty. (Read all of the questions already posted.) But I’d like to point out something else.
What is the retention rate of KIPP students? How many students who start in their 5th grade and up finishing at 8th grade? Google Gary Rubinstein’s blog and his miracle schools wiki. If I recall correctly, some KIPPs (but not all) have up to 40% of their students leave the school, and I’d assume these students return to their traditional middle school.
Why is this not addressed? (at least to my knowledge, KIPP doesn’t really communicate this) We can see why KIPP wouldn’t advertise this. Perhaps because it looks really, really, really bad to its funders. Perhaps they don’t want to admit it. Perhaps it’s kind of an uncomfortable truth that doesn’t fit with the idea that charters can have better results with the same students and same funding.
I’d like to see more honesty. You can see why, if the retention rate example is true (which it seems to me based on what I’ve seen), how hurtful this can be to: 1) the students who dropout of KIPP 2) the local traditional school that is labeled a failure and closed down by a mayor who thinks to himself “why can’t this school be like KIPP. Oh it’s cause they’re lazy.” 3) the local traditional school teachers who are threatened to be fired and who must educate more SPED / ELLs and don’t have a lottery to self-select out students who are motivated to do a lottery.
If KIPP shared the same goal – to educate every child…why don’t they?
Or is their goal to educate only some, like the more motivated low-income students?
http://www.edexcellence.net/commentary/education-gadfly-daily/flypaper/2013/the-charter-expulsion-flap-who-speaks-for-the-strivers.html
I think Dr. Ravitch’s second question is important, especially if the answer is yes. If KIPP students benefit from positive peer effects, dispersing those students will eliminate those positive peer effects and make those students worse off. Should society do that?
Don’t we want all of our students to be “positive” and surround themselves with “positive” peer influences? Siphoning funds away from public schools and populating charters with selected students through attrition is like trying to slice an orange with a hatchet. So much damage is done to benefit so few.
Should I take it that you are in favor of sacrificing these positive peer benefits for the relatively high performing students because of the benefits the high performing students will have on the lower performing students in a mixed class? That is certainly a reasonable position, but one that the families of the high performing students are likely to disagree with.
TeachingEconomist – No child would have to be “sacrificed” if all school were properly staffed and funded.
At what funding levels do peer effects disappear? What do those added funds need to buy in order for peers to be irrelevant in education?
Yes, according to
http://www.edexcellence.net/commentary/education-gadfly-daily/flypaper/2013/the-charter-expulsion-flap-who-speaks-for-the-strivers.html
Philadelphia has two KIPP schools. The Philadelphia Public School Notebook has an article with an interview with a study co-author who said its findings are “significant but hard to explain.” Also, the end of the article is significant given the regimented way that the KIPP schools are run. (The article does not draw this conclusion about the KIPP school environment, I am.)
“Finally, as part of the study, the researchers interviewed both those students who won the KIPP lottery and those who didn’t to try to identify any effects of KIPP on student attitudes and behaviors. They found, not surprisingly, that KIPP students do more homework. But the other finding here was surprising and the only negative note in the report.
“According to the students’ self-reports, the KIPP kids are more likely to engage in kinds of undesirable behavior that lead to conflicts with teachers or parents.” The researchers are not sure why that might be, but Gill has a couple of guesses: KIPP students possess more self-awareness, or they simply have more time to get into arguments, given the longer days and additional homework.
The study did not follow KIPP students beyond their middle school years to see how well they do in high school or college.
“We don’t have data to follow the kids long enough for that,” Gill said. “We would like to do that at some point in the future.”
Sorry, forgot the link:
http://thenotebook.org/blog/135694/kipp-success-significant-but-hard-to-explain
But wait, how could there possibly be such a thing as positive peer effects from good students on each other? Didn’t we learn from the Jeannie Oakes post on tracking that mixing is good, and that good students are not in fact better off by being put in the same classroom/building?
That is only when it is someone besides ‘us’ who makes the gains. The idea that (unless you have some dysfunctional situation) highly motivated kids don’t benefit from being with other motivated kids flies in the face of what everyone who has worked with kids knows. Only people with an agenda would think about arguing otherwise. I’ve never met ONE teacher who would have agreed that there wasn’t a positive peer effect from good students being together.
You really need to look up that Jeannie Oakes tracking post from this blog, because some people who apparently are educators (if their identity hasn’t been misappropriated) do make exactly that claim: that good students are not only NOT better off being together but that they are better off if placed with slower students.
I’ve been in high need middle and high schools since I switched careers over a decade ago. Every school that I’ve been in, the admin talked about raising grade level 2 years (some even went higher) for every year in school. When I’d point out that If anyone could raise grade level for all students 2 years for 1, every year, that they’d be making millions on the book sales, I get shit for being too negative.
Here they are praising 11 months gain over 3-4 years, much lower than I’ve been told that I’m expected to do every year. My conclusion, pretty much everything about the teacher level in schools in this country is completely screwed. Unrealistic demands, ludacris goals, taking shit from everyone.
We rail against the corporate mindset. I was treated better every single year that I was there, and never held to some ridiculous standard that had never been achieved before. We’ve been eating our own long before the reformers got involved. We need to get rid of the reformers and MANY admin who have been perpetuating lies just to beat us down.
The study does point out that not all KIPP schools are high-achieveing, something the KIPP fanboys routinely ignore. For example, KIPP Nashville’s achievement is in the bottom 5% of schools in the state of TN. Despite this poor record, it’s in no danger of take over or being sent to the Achievement School District (ASD) like a low-achieving public school would. Instead they are being rewarded for their achievement…$16 million to expand and improve facilities.
Expel the worst students and have class for hours longer each day. Have fewer special Ed and English learners enrolled. (one could have a seperate lottery for those demographics to even it out?). Have required summer classes into August, and have classes on Saturday from 9AM-1PM and your students will be magically be learning more. Oh, and more homework each day too.
It is interesting that during the 19th century children worked 6 days a week in a factory which we now think of as draconian, but drive them that hard in school and it is for their own good. I believe the stress is why the researchers have to make this statement: “Kipp students …had an increased incidence of self-reported undesirable behaviors, including losing their temper, arguing with or lying to their parents, or giving their teachers a hard time.”
Kipps own advertisements say that their teachers work 10 or 11 hours a day. (at the school) Students are in school from 7:30AM until 5 PM and then go home and do homework. It is a system that is nothing like public school which prevents it from being able to be compared to a public schools.
Their ad does say that they pay their teachers 10-15% more for their extra work.
This description makes me think “well then of course the students would get higher test scores.” So, let me ask wealthy parents, then, why don’t they send their children to schools like KIPP? Or is this type of education only allowed for other people’s children?
“Achievement” is an operational definition for standardized test scores, which are very good predictors of future standardized test scored, good predictors of family income, and not very good predictors of anything else.
Previous Mathematica studies have all been funded by the same outfits that fund Kipp — isn’t that a red flag right there? Also, previous studies have all been “working papers” not subject to peer review. Highly smelly.
Is this one any different? Inquiring minds want to know.
Yep, it’s just a “Working Paper,” not peer-reviewed. http://www.mathematica-mpr.com/publications/PDFs/education/KIPP_middle_schools_wp.pdf They have enough citations for economists (such as Hanushek) in the reference list to lead me to believe that if they do publish, it’s more likely to appear in an economics journal than an education journal.
I haven’t had time to read the entire paper yet but a cursory review and search did not find any info describing who funded the study. On the Mathematica website, the KIPP Foundation is listed as one of their “Selected Clients.”
Ask the right questions, according to Parents Across America!
http://parentsacrossamerica.org/what-we-believe-2/ask-the-right-questions/
“KIPP students reported no discernible increase in attitudes associated with success,”
KIPP students are inundated with hype about grit and graduating from college throughout their schooling, from slogans, to college pennants prominently hung in the schools, to classrooms named for the college their teacher graduated from, to being in groups named according to the year they are expected to graduate from college. I would like to see a study on the impact this kind of intense dogma students are subjected to daily for years has had on the social-emotional development of the 67% of KIPP alumni who did not go on to earn college degrees, including their “attitudes associated with success” and their self-esteem.
Great point. The single year I worked at KIPP:STAR in Harlem we looked at city-wide achievement data for graduates of the KIPP middle schools. This was 2009 and the KIPP high school had just opened up the road in Washington Heights. Data that year showed KIPP graduates attending neighborhood high schools had not performed well and many had performed poorly on state tests in their 9th and 10th grade years. There was less data about kids who had gone on to private high schools or boarding schools. The staff I’d joined was really upset by this data and felt that after all the hard work they’d done getting these middle schoolers ready to be successful in high school, the traditional public high schools had let “their” kids down. A fellow writing teacher said something like those teachers in the public high schools just don’t care like we do. This sentiment was shared many times over the course of my single year at KIPP. Many references to the rubber room. Disdain for fellow public school teachers was rampant–it’s part of the culture of KIPP.
The study is here: http://www.kipp.org/mathematica
I am puzzled by the insinuations that KIPP cherry-picks students to get these results. The study was an RFT – the “r” stands for randomized. And Mathematica reported on the attrition process in 2012: http://www.kipp.org/results/mathematica-study/mathematica-2012-working-paper
Do you have specific allegations of how the randomization process was subverted? Because if not, this is a bit of a cheap shot.
It’s also extremely odd to me that people want to dismiss the results because KIPP might have a bit of extra money or because it had longer school days. Surely, the correct way to read these results would be to say that ANY school (public or private) would likely be able to generate the same results given the same conditions. Surely it wouldn’t take much imagination to turn this into an argument in favour of greater public funding.
I did not say that KIPP cherrypicks students.
What I said is that the students in KIPP benefit by “peer effects.”
Those who lose the lottery go to schools with overcrowded classes, with kids who are disabled, more likely to not speak English, some of whom are disruptive. The kids who never entered the lottery.
Not exactly a level playing field.
What if the public schools had the same resources as KIPP?
Do we even know what resources KIPP has? I think you imply above that we don’t. And do we know whether KIPP operates with the same expenses that district schools have? Are their teachers and staff unionized? Do their teachers and staff have the same medical and pension benefits as their district school counterparts?
In response to Fierper: Julian Vasquez Heilig has studied KIPP’s resources. The organization spends considerably more than public schools. It has raised more than $300 million from foundations: http://cloakinginequity.com/2013/01/20/the-teat-be-a-little-more-honest-kipp-charter-schools/comment-page-1/
Dr. Ravitch,
Do you believe we are in a sort of zero sum situation with schools like KIPP?
It would seem that allowing KIPP schools will result in better education for those at the school but worse for those not at the KIPP school due to peer effects. Outlawing KIPP schools would result in worse education for those that would have benefited from the positive peer effect of the school, but better outcomes for those students who would have not attended KIPP.
My understanding is that they are taking students whose parents voluntarily applied for Kipp. Therefore their commitment to their child’s education is different than the commitment of the parents who wouldn’t apply and simply don’t care. It creates two different groups of people. A KIPP student has 60% more time in school. It is a given that more contact time is going to produce better results. But saying KIPP is better than public school would only be valid if they reduced the contact time to the amount public school teachers get. What the report says is that if you drive students hard for long hours they will learn more, I doubt that is a revelation, but a you say, it is data.
It looks to me that they matched students tha applied to KIPP schools and were admitted to students that applied to KIPP schools and not admitted. Because admission is random, this procedure eliminates your concern about self selection bias in the sample.
You must have a lot a faith in KIPP to conduct random lotteries. Some parents have written about KIPP’s “pre-lottery test.” For example, see comments by cgrannan here: http://www.washingtonpost.com/blogs/answer-sheet/post/do-self-selection-and-attrition-matter-in-kipp-schools/2011/06/13/AG1sQeTH_blog.html
It appears that the Mathamatica study is a “Rigorous research commissioned by KIPP – and other charter school organizations – explicitly recognize that self-selection matters, which is why they seek to compare lottery winners with lottery losers.” (a quote from the blog post you reference). This issue was of concern to 25+ years in education so I pointed out that the researchers tried to deal with the problem of self selection.
The researchers could not differentiate between school and peer effects, however. Do you think peer effects are significant? If so, do you agree with reading exchange that spending on education will eliminate peer effects?
I think it’s not a fair comparison if they rigged the “lottery,” such as by narrowing down the lottery pool in advance with a “pre-lottery test,” interviews and detailed questions about students and families on admissions forms. Since they commissioned the study, they also would have known in advance that comparing their students to the low performing students their “lottery” rejected was likely to cast them in a favorable light.
Again the study design seems to have compared two students who made it through any and all pre-lottery screening. The difference between the two was the random lottery admitted one student, did not admit the other. This is about as good as it gets in these sorts of studies.
Are you accusing KIPP of rigging the actual lottery?
Right, “it’s about as good as it gets” when you are buying research, it is not peer-reviewed, you require a “pre-lottery test” and the comparison is being made with the students you rejected.
The only public schools I know of which are not selective enrollment schools that pre-screen are charter schools, and since some parents have said that KIPP administers their own “pre-lottery test,” how random is that lottery? Even if the pre-lottery test is not typically required at all KIPP schools, it would be easy for them to pre-screen, since they can readily obtain transfer students’ grades, test scores and discipline records.
Student A and student B both passed any and all pre-lottery screening. Student B was randomly selected to attend the KIPP school, student A was not. The outcome for these two students, attempting to control for other factors that influence outcomes (like poverty) were compared in the study.
You can argue that the statistical analysis was poorly done. You can argue that the KIPP lottery was not random. You can argue that attrition is a problem (it is in most studies). You can argue, as did Dr. Ravtch, that the study fails to distinguish between the peer effects and the school effects. But you can not say the study compares students that failed any pre-lottery screening to students that did not fail the pre-lottery screening because it did not. That is my only point.
School lotteries are on the honor system. Unless someone like Pricewaterhouse had oversight, there is no evidence that an actual lottery took place. We only know that some kids were accepted and some were rejected. (And there is reason to believe they screen before accepting students.)
If the lottery system is not random, the results of the study are questionable because of selection bias. Do you have a any evidence at all that the lotteries are not random?
Where’s the evidence that unregulated charters have random lotteries and don’t juke the stats? http://www.examiner.com/article/juking-the-stats-101-how-supposed-education-reformers-deceive-with-statistics
Do you have any evidence that the lotteries are riged in favor of higher performing students or is this just an auxiliary hypothesis that can be used to reconcile the findings of the study with what you know must be true?
Lotteries are not rigged. The most dysfunctional families never enter lotteries. That explains why homeless children in New York City, for example, are dramatically underrepresented in charters. There are 60,000 homeless students, in the last survey; only 110 were in charters, as compared to about 4% of the general student population.
Good to know that the lotteries are random and the issue of pre-lottery screening is not a reason to doubt the results of the study.
TE, you really are a Johnny one-note.
The one note here is that 25+ years in education’s concern about the statistical design of the study was misplaced.
Much more important is your point about peer effects, but that has been studiously ignored.
I’ve read several things by Caroline Grannan indicating KIPP uses a “pre-lottery test.” I saw no explanation why until now. I just found something by her indicating they said the test is used to determine grade level, in comments here: http://schoolfinance101.wordpress.com/2013/03/01/the-non-reformy-lessons-of-kipp/
If students can’t automatically transfer into the grade that follows the one they just completed, it would be interesting to see how many kids KIPP holds back. Maybe the KIPP data should be disaggegated by age.
Age issues are very interesting. Here is an article concerning the month of birth and admission to Cambridge and Oxford: http://www.bbc.co.uk/news/uk-politics-21579484
Nobody is talking about the measurement system Mathematica created. The group seems only to be measured against itself.
I’m not sure about the details of the research design, but whenever I see researchers touting “estimated impacts” in these kinds of studies, I have to wonder why they are not using actual figures instead of “estimates” (as with VAM).
No one is talking about the draconian military style approach they use at KIPP schools, which is why some KIPP alums refer to it as the “Kids in Prison Program”. Having attended a very oppressive military style public middle school years ago, I do not happen to believe the end justifies the means.
If students are learning more at KIPP, I am happy for those students. Whenever students learn well we should all applaud and learn what we can. It is important however to understand the conditions which contribute to the success.
First, peer effects cannot be ignored. Even if the student is compared to a match, the KIPP student has the advantage of being in a classroom with other motivated students, who will be removed for misbehavior and who have parents with the interest and savvy needed to visit schools and enter the lottery process. The higher the mean achievement in the class, the higher the student growth of all students, especially low achievers. This is not accounted for in the study.
Second, let’s learn from KIPP and give public school classrooms more funding, students more support and public school students more learning time. Certainly these factors contribute to KIPPs success.
Third, to really explore whether KIPP is model that can succeed on a larger scale, let KIPP take over a failing district. NYSED has control of several failing suburban districts. Why not bring in KIPP? How much of a difference can KIPP make when they operate in a neighborhood school with open enrollment and when they must follow the same regulations that public schools must follow?
So would you say that high performing students in classes with large numbers of other high performing students are going to get a better education than high performing students in classes without large numbers of other high performing students?
So suddenly you believe that tracking on a schoolwide level DOES help kids because it creates better peer effects?!.
JSB of the University of Arkansas (Walton-land):
Tracking may raise test scores; segregating kids by ability may raise test scores. Pushing out kids who don’t speak English will surely raise test scores as will excluding children with disabilities of certain kinds.
But are test scores so valuable that they trump every other value?
Do you believe that higher test scores measure nothing of any value? I would also think that the students at Thomas Jefferson High School think that they get more out of attending that school than just higher test scores.
At a minimum putting high performing students together in the same class or school might also reduce the social isolation these students feel in traditional schools.
Every now and then a former KIPP student enrolls in my 6th grade class (public school). Given their ability levels in math, I was under the impression that math was an elective at a KIPP school. When you kick out poor performing students, of course your scores are going to go up. Why am I even surprised that they claim huge gains.
Some people above, including Ms. Ravitch and Ms. Burris, are claiming that the study didn’t look at peer effects.
But in a link posted above, the study’s main author begs to differ:
“it’s clear, we don’t see evidence that peer effects could explain any substantial part of KIPP’s effect.”
If either one of you actually reads the entire study, could you explain whether the study’s author was correct in saying this? Thanks.
http://www.greatschools.org/ohio/columbus/9831-KIPP-Journey-Academy-Middle/?tab=test-scores
Is this the same KIPP? I don’t get what is so great about these schools. Are there schools in certain area that are great while others stink?
With all due respect, I find a bit of the above commentary to be examples of “paralysis by analysis.”
Let’s keep it real. Gary Rubinstein [well known to many of those who view and post on this blog] visited the KIPP High School in New York City. He has posted a number of firsthand and sober observations about what he saw and heard.
Start with the following link: http://garyrubinstein.teachforus.org/2012/11/14/my-visit-to-kipp/
As Gary [under]states quite accurately in the piece linked above, “KIPP is the ‘gold standard’ of charter schools, and charter schools are the crown jewels of the education reform movement.” So what does an experienced educator like himself notice when he goes to a premium unit of the KIPP chain?
Read his commentary and the many responses for yourselves and come to your own conclusions.
I’m curious about the 37% that leave KIPP schools. The study said that was close to the percent that left public schools. But where are the kids going that leave both of those? What percent of those leaving KIPP schools go to the public schools? My guess is that there are far more leaving KIPP to go to the public schools than there are leaving the public schools to go to KIPP. If the 37% leaving KIPP are leaving because they were not performing well, and then they go to the public schools, wouldn’t that bring down the scores of the public schools?
Yes. The attrition rate is meaningless without causes for attrition identified. The concern has always been WHY students are leaving KIPP, such as whether they are being counseled out or expelled because they have low test scores, behavior problems, disabilities or are English language learners –while traditional public schools take all kids and cannot just dump them for having high needs.
Riverinthesky, a lottery is a selection device, as homeless families and the least functional families are unlikely to apply. At that point, there are peer effects. The public schools must take all. Check the demographics and attrition and spending. .